ML Model Evaluation & Debugging
Debug ML performance issues with structured evaluation workflows
What You Can Do
Diagnose root causes of model performance failures, select optimal evaluation metrics based on your use case and data characteristics, and design comprehensive evaluation strategies. You'll get systematic workflows for analyzing failure modes, statistical significance testing, threshold optimization, and cross-validation planning—turning vague model problems into actionable fixes.
Features
Systematically identify root causes like data drift, class imbalance, or feature leakage
Choose evaluation metrics aligned with your business goals and data characteristics
Create comprehensive evaluation plans with cross-validation and statistical tests
Diagnose and recommend solutions for imbalanced classification problems
Interpret confusion matrices to uncover specific failure patterns
Find optimal decision thresholds and analyze precision-recall tradeoffs
Select appropriate CV methods for your data and model type
Determine if performance improvements are statistically meaningful
Example Output
Root Cause Diagnosis
- Issue: 95% accuracy but 0.45 F1 score
- Root Cause: Severe class imbalance (99% negative class)
- Evidence: Precision 0.92, Recall 0.29 → model predicts mostly negatives
- Recommendation: Switch to stratified k-fold CV, use F1/AUC-PR metrics, apply class weighting
Evaluation Plan
- Primary metrics: AUC-PR, macro-F1, per-class precision/recall
- Cross-validation: Stratified 5-fold CV (preserves class distribution)
- Baseline: Train naive classifier to compare against
- Statistical test: McNemar's test for comparing model improvements
What's Included
- `SKILL.md`: Core evaluation & debugging skill with decision trees
- Evaluation metric selection checklist:
- Failure mode diagnosis template:
- Cross-validation strategy guide:
- Class imbalance mitigation workflows:
- Statistical significance test reference:
- Confusion matrix interpretation guide:
- Production monitoring recommendations:
Who It's For
- ML/Data Scientists debugging production models
- ML Engineers selecting evaluation metrics
- Model Researchers analyzing failure modes
- Analytics Engineers validating data pipeline models
- ML Ops professionals monitoring model performance
Best For
- Diagnosing unexpected model performance degradation
- Selecting appropriate evaluation metrics for your use case
- Analyzing failure modes in classification or regression
- Optimizing decision thresholds and precision-recall tradeoffs
- Planning comprehensive evaluation strategies for new models
You might also like
This skill provides a systematic framework for preparing training data, benchmarking model performance, identifying deployment risks, and calculating true ROI before committing fine-tuning investments. You'll validate datasets, run structured evaluations against baseline Claude models, test edge cases, and iterate toward production-ready fine-tuned versions. The framework ensures your fine-tuned models deliver meaningful improvements while maintaining safety and cost-effectiveness.
by moonwell-fi
You can design statistically valid experiments with proper power analysis, choose the right statistical tests for your data type, analyze results while controlling for multiple comparisons, and generate publication-ready reports with accurate interpretation of findings. Claude helps you avoid common statistical pitfalls and ensures your experimental claims are well-supported by evidence.
by moonwell-fi
This skill automatically documents your entire analytics infrastructure by analyzing data sources, transformations, and outputs. You'll generate production-ready data dictionaries with field definitions, lineage maps showing data flow across systems, and transformation documentation that explains logic and dependencies. Save weeks of manual documentation work while keeping your analytics stack discoverable as it evolves.
by moonwell-fi
You'll build comprehensive data quality test suites that validate transformations, detect anomalies, and document standards across your dbt and SQL pipelines. This skill generates production-ready test configurations, anomaly detection protocols, and validation rules that catch data issues before they impact analytics.
by moonwell-fi
You can create comprehensive documentation for your data platforms, metrics, and analytics infrastructure that stakeholders actually understand. The skill generates clear data dictionaries, metric definitions, pipeline diagrams, and runbooks that bridge the gap between technical teams and business users. Your documentation stays consistent with your actual infrastructure while being instantly accessible to analysts, managers, and engineers.
by moonwell-fi
You can rapidly compile, organize, and format disparate health economic evidence—from clinical trials to cost-effectiveness analyses—into structured, regulatory-compliant dossiers. This skill maps your evidence to specific payer and HTA requirements, automatically generates evidence hierarchies, and produces submission-ready dossier outlines with formatting that meets regulatory standards for NICE, EUnetHTA, and other major bodies.
by moonwell-fi
You can structure comprehensive health technology assessments (HTAs) that organize clinical evidence, economic analyses, and regulatory considerations into evidence-based coverage recommendations. This skill helps you synthesize clinical trial data, health economic models, and real-world evidence into clear, defensible payer coverage determinations. You'll generate professional HTA reports that align with major frameworks like ICER, CADTH, and NICE standards.
by moonwell-fi
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
by moonwell-fi