ML Infrastructure Failure Analysis & Optimization
Diagnose ML infrastructure failures with structured root-cause analysis
What You Can Do
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
Features
aggregates logs from training jobs, resource managers, databases, and serving infrastructure to surface patterns
detects deadlocks, timeouts, communication failures, and split-brain scenarios in distributed training setups
identifies CPU/memory/GPU exhaustion, disk I/O bottlenecks, and network saturation from metrics
traces data corruption, schema mismatches, and ingestion failures from end-to-end logs
analyzes inference latency, cache misses, hardware underutilization, and load-balancing issues
ranks potential root causes by likelihood, impact, and fix complexity
generates specific actionable steps tailored to your infrastructure type
identifies minimal steps to reproduce the failure for testing fixes
Example Output
Training Job Timeout Analysis:
ROOT CAUSE: GPU memory exhaustion after epoch 3 (95% utilization)
CONFIDENCE: High (logs + metrics align)
IMPACT: All 8 worker nodes OOM-killed after 2h 14m
REMEDIATION:
1. Reduce batch size from 256 to 128 (10 min)
2. Enable gradient checkpointing in model (30 min)
3. Increase swap on worker nodes (5 min)
Serving Latency Issue:
ROOT CAUSE: Single-threaded model loading blocking requests (p99 latency = load time)
CONFIDENCE: High (latency spikes correlate with model reloads)
REMEDIATION:
1. Implement async model loader (1 hour)
2. Pre-warm cache on deployment (30 min)
3. Scale from 2 to 4 replica pods (5 min)
EXPECTED: p99 latency < 200ms
What's Included
- SKILL.md: Complete failure diagnosis workflow with decision trees and analysis templates
- Log parsing templates: Pre-structured prompts for Kubernetes, PyTorch, TensorFlow, and cloud provider logs
- Failure diagnosis checklist: Systematic investigation guide covering resource utilization, networking, data integrity, and model configuration
- Remediation runbook: Template for documenting and prioritizing fixes
- Root-cause prioritization matrix: Decision framework to rank hypotheses by likelihood and impact
- Post-mortem template: Incident documentation format for team learning and prevention
- Infrastructure reference: Common failure patterns for Kubernetes, distributed schedulers, and cloud platforms
Who It's For
- MLOps engineers — Debugging distributed training clusters and model serving infrastructure
- ML platform engineers — Troubleshooting production incidents affecting multiple users
- Data scientists — Understanding why training jobs fail or models underperform in production
- DevOps engineers — Supporting ML workloads on Kubernetes or cloud infrastructure
- Site reliability engineers — Root-cause analysis for ML system outages
Best For
- Diagnosing unexpected training job failures or hangs — Timeouts, OOM errors, deadlocks
- Troubleshooting model serving latency or throughput issues — Slow inference, high p99 latency, uneven load distribution
- Identifying resource constraints — Understanding bottlenecks in distributed systems (CPU, GPU, memory, network, I/O)
- Post-incident root-cause analysis — Systematic investigation for incident reports and prevention planning
- Infrastructure optimization — Using failure patterns to guide configuration and scaling decisions
You might also like
You can build production-grade vector search systems that retrieve semantically similar documents in sub-millisecond time using AgentDB's optimized HNSW indexing. The skill enables you to implement RAG pipelines, semantic search engines, and intelligent knowledge bases with configurable embedding dimensions, distance metrics (cosine, Euclidean, dot product), and similarity thresholds—all with built-in quantization and caching for massive performance gains.
by moonwell-fi
You'll systematically diagnose data quality problems by developing structured root cause analysis frameworks, calculating the true business impact, and creating reproducible validation tests. This skill walks you through hypothesis-driven investigation, data lineage analysis, and remediation planning — turning data issues into documented fixes and preventive measures.
by moonwell-fi
You can systematically evaluate and improve your RAG pipelines using Claude as a design partner. You'll analyze retrieval quality, identify bottlenecks in your embedding and chunking strategies, and receive actionable recommendations to reduce hallucinations and improve context relevance. By the end, you'll have a data-driven optimization plan tailored to your specific use case and performance metrics.
by moonwell-fi
You'll build comprehensive data quality test suites that validate transformations, detect anomalies, and document standards across your dbt and SQL pipelines. This skill generates production-ready test configurations, anomaly detection protocols, and validation rules that catch data issues before they impact analytics.
by moonwell-fi
You can convert complex datasets and business metrics into polished, executive-ready reports that stakeholders trust and act on. Claude generates data-driven narratives, executive summaries, actionable insights, and visualization recommendations tailored to your audience's priorities. Your reports will tell a cohesive story that connects metrics to business outcomes, eliminating confusion and accelerating decision-making.
by moonwell-fi
You can rapidly compile, organize, and format disparate health economic evidence—from clinical trials to cost-effectiveness analyses—into structured, regulatory-compliant dossiers. This skill maps your evidence to specific payer and HTA requirements, automatically generates evidence hierarchies, and produces submission-ready dossier outlines with formatting that meets regulatory standards for NICE, EUnetHTA, and other major bodies.
by moonwell-fi
You can structure comprehensive health technology assessments (HTAs) that organize clinical evidence, economic analyses, and regulatory considerations into evidence-based coverage recommendations. This skill helps you synthesize clinical trial data, health economic models, and real-world evidence into clear, defensible payer coverage determinations. You'll generate professional HTA reports that align with major frameworks like ICER, CADTH, and NICE standards.
by moonwell-fi
This skill automatically documents your entire analytics infrastructure by analyzing data sources, transformations, and outputs. You'll generate production-ready data dictionaries with field definitions, lineage maps showing data flow across systems, and transformation documentation that explains logic and dependencies. Save weeks of manual documentation work while keeping your analytics stack discoverable as it evolves.
by moonwell-fi