ML Infrastructure Optimization & Troubleshooting
Debug ML infrastructure bottlenecks, fix distributed training failures, optimize performance
What You Can Do
You can systematically diagnose performance issues across your ML infrastructure, identify root causes of distributed training failures, and receive actionable optimization recommendations. This skill analyzes metrics, logs, and configuration data to pinpoint resource constraints, communication overhead, and configuration misalignments that impact training speed and reliability.
Features
Analyzes CPU, GPU, memory, and I/O utilization patterns to identify where your pipeline loses time
Traces collective communication failures, gradient synchronization issues, and node-to-node connectivity problems
Maps compute, data loading, and network stages to spot inefficiencies from data ingestion through model checkpointing
Reviews hyperparameters, batch sizes, gradient accumulation, and communication backends for alignment with your hardware
Recommends GPU/CPU/memory configurations based on model size, data throughput, and fault tolerance requirements
Extracts and correlates anomalies across training logs, system metrics, and error traces to surface hidden dependencies
Diagnoses clock skew, network latency, and synchronization issues across distributed clusters
Provides priority-ordered fixes with implementation guidance and expected performance gains
Example Output
Example 1: Bottleneck Analysis
- 🔴 CRITICAL: GPU utilization 32% (target 80%+)
├─ Root cause: Data loading blocking GPU pipeline
├─ Symptom: 4.2s train step (2.8s data load, 1.4s forward/backward)
└─ Fix: Enable async prefetching, increase DataLoader workers from 4 → 8
Expected improvement: 65% reduction in step time
- ⚠️ WARNING: AllGather communication overhead 18% of step time
├─ Symptom: Gradient synchronization across 8 GPUs takes 1.1s
└─ Recommendation: Switch NCCL backend to use NVLink + enable gradient compression
Example 2: Distributed Training Failure
- 🚨 Failure pattern detected across nodes:
├─ Node 3 shows hanging AllReduce at iteration 45
├─ Network trace: Packet loss 0.2% between Node 1 ↔ Node 3
├─ Root cause: Network congestion during weight synchronization
├─ Temporary fix: Increase NCCL timeout from 30s → 120s (masks symptom)
└─ Permanent fix: Route collective ops through dedicated network interface
What's Included
- SKILL.md: Full infrastructure analysis workflow with decision trees for diagnosis
- Performance profiling checklist: Metrics to collect, interpret, and correlate
- Configuration templates: DDP, FSDP, Horovod, Ray Train setup examples
- Log parsing patterns: Regular expressions and extraction rules for common ML frameworks
- Troubleshooting flowchart: Decision tree from symptom → root cause → fix
- Benchmark comparison tool: Script to baseline performance against known good configurations
Who It's For
- ML infrastructure engineers — Debug production training pipelines and scale distributed models
- Research scientists — Optimize hyperparameter sweeps and reduce iteration time on clusters
- DevOps/platform engineers — Right-size hardware allocation and improve cluster utilization
- MLOps teams — Implement monitoring and automated diagnosis for training pipeline reliability
- Data center operators — Identify network and resource constraints affecting ML workloads
Best For
- Diagnosing why distributed training hangs or crashes across multiple nodes
- Identifying data loading bottlenecks blocking GPU pipeline efficiency
- Analyzing collective communication overhead (AllGather, AllReduce) in multi-GPU setups
- Right-sizing batch size, gradient accumulation, and communication backends for your hardware
- Troubleshooting out-of-memory errors and resource allocation failures
- Profiling end-to-end pipeline performance from data ingestion through checkpoint save
You might also like
You can build production-grade vector search systems that retrieve semantically similar documents in sub-millisecond time using AgentDB's optimized HNSW indexing. The skill enables you to implement RAG pipelines, semantic search engines, and intelligent knowledge bases with configurable embedding dimensions, distance metrics (cosine, Euclidean, dot product), and similarity thresholds—all with built-in quantization and caching for massive performance gains.
by moonwell-fi
You'll systematically diagnose data quality problems by developing structured root cause analysis frameworks, calculating the true business impact, and creating reproducible validation tests. This skill walks you through hypothesis-driven investigation, data lineage analysis, and remediation planning — turning data issues into documented fixes and preventive measures.
by moonwell-fi
You'll build comprehensive data quality test suites that validate transformations, detect anomalies, and document standards across your dbt and SQL pipelines. This skill generates production-ready test configurations, anomaly detection protocols, and validation rules that catch data issues before they impact analytics.
by moonwell-fi
You can design multi-dimensional evaluation strategies tailored to your model's specific capabilities and use cases, create representative test sets that expose edge cases and failure modes, implement automated scoring mechanisms for reproducible results, and generate benchmark comparison reports that contextualize performance within industry standards. This skill transforms ad-hoc testing into systematic, evidence-based model assessment—essential for production deployment decisions and ongoing performance monitoring.
by moonwell-fi
You can convert complex datasets and business metrics into polished, executive-ready reports that stakeholders trust and act on. Claude generates data-driven narratives, executive summaries, actionable insights, and visualization recommendations tailored to your audience's priorities. Your reports will tell a cohesive story that connects metrics to business outcomes, eliminating confusion and accelerating decision-making.
by moonwell-fi
You can structure comprehensive health technology assessments (HTAs) that organize clinical evidence, economic analyses, and regulatory considerations into evidence-based coverage recommendations. This skill helps you synthesize clinical trial data, health economic models, and real-world evidence into clear, defensible payer coverage determinations. You'll generate professional HTA reports that align with major frameworks like ICER, CADTH, and NICE standards.
by moonwell-fi
This skill detects your project framework (Vite, Next.js, plain Node, or monorepo) and automatically configures agent-tail to pipe dev server and browser console logs into unified log files. You'll get a proposed configuration tailored to your setup, install agent-tail with the correct plugins, and have logs immediately available for AI agents to consume and analyze.
by moonwell-fi
This skill automatically documents your entire analytics infrastructure by analyzing data sources, transformations, and outputs. You'll generate production-ready data dictionaries with field definitions, lineage maps showing data flow across systems, and transformation documentation that explains logic and dependencies. Save weeks of manual documentation work while keeping your analytics stack discoverable as it evolves.
by moonwell-fi