Optimization Research Methodology
Design optimization solutions with rigorous experimental validation
What You Can Do
You'll develop a systematic research methodology for optimization projects that combines hypothesis-driven experimentation with statistical validation. This skill guides you through designing experiments, setting up proper baselines, implementing variants, benchmarking results, and interpreting findings with statistical rigor. By following this framework, you transform exploratory optimization work into production-ready improvements with confidence in their actual impact.
Features
structure your hypothesis, baseline metrics, variant implementations, and success criteria
control isolation, sample size calculation, statistical power analysis, and duration planning
performance measurement strategy, variance reduction techniques, and data collection procedures
significance testing, confidence interval calculation, and effect size interpretation
determines sample size needed, statistical power, minimum detectable effect, and experiment duration
understand trade-offs, failure modes, generalization limits, and rollout readiness
capture methodology, findings, recommendations, and decision rationale for stakeholders
systematically investigate why optimizations underperform and extract learnings
Example Output
Example 1: API Response Time Optimization Research Plan
✓ Hypothesis: Caching user preferences reduces API latency by ~15% ✓ Baseline: Current avg response time: 240ms (n=10k requests over 1h) ✓ Variant: With in-memory cache implementation ✓ Metrics: P50 latency, P99 latency, cache hit rate ✓ Duration: 2-hour experiment with 500+ req/sec ✓ Sample size: 720k requests (sufficient for 5% significance) ✓ Rollout decision rule: Safe if P99 < 210ms and cache hit rate > 65%
Example 2: Statistical Analysis Results
✓ Variant P50 latency: 198ms (95% CI: 195–201ms) ✓ Baseline P50 latency: 240ms (95% CI: 237–243ms) ✓ Improvement: 42ms (17.5%), p-value < 0.001 (highly significant) ✓ Effect size: Cohen's d = 0.82 (large practical effect) ✓ Cache hit rate: 68% (within acceptable range) ✓ Recommendation: Safe to roll out to 10% traffic with 24-hour monitoring
What's Included
- SKILL.md: Complete optimization research methodology framework with decision trees and verification checklists
- Research plan template: Hypothesis, baseline, variant, metrics, success criteria, and rollout decision rules
- Experimentation worksheet: Sample size calculator, statistical power analysis, and experiment duration planning
- Data analysis checklist: Significance testing, confidence interval calculation, and result interpretation steps
- Statistical reference guide: Common pitfalls, test selection criteria, and when to use paired vs. unpaired analysis
- Documentation template: Methodology justification, findings summary, and stakeholder-ready conclusions
Who It's For
- Software engineers — optimize API performance, database queries, and system throughput with data-driven validation
- ML engineers — validate model improvements, hyperparameter tuning, and feature engineering with statistical rigor
- Product managers — design and analyze A/B tests for user experience improvements with confidence
- Data scientists — structure experimentation for analytics and algorithm optimization with proper baselines
- Performance engineers — benchmark and validate infrastructure, caching, and system-level optimizations
Best For
- API latency and throughput optimization projects
- Machine learning model performance tuning and validation
- A/B testing and feature flag experimentation workflows
- Database query and caching optimization benchmarking
- User experience and UI/UX experiment design
You might also like
You can build production-grade vector search systems that retrieve semantically similar documents in sub-millisecond time using AgentDB's optimized HNSW indexing. The skill enables you to implement RAG pipelines, semantic search engines, and intelligent knowledge bases with configurable embedding dimensions, distance metrics (cosine, Euclidean, dot product), and similarity thresholds—all with built-in quantization and caching for massive performance gains.
by moonwell-fi
You'll systematically diagnose data quality problems by developing structured root cause analysis frameworks, calculating the true business impact, and creating reproducible validation tests. This skill walks you through hypothesis-driven investigation, data lineage analysis, and remediation planning — turning data issues into documented fixes and preventive measures.
by moonwell-fi
You can systematically evaluate and improve your RAG pipelines using Claude as a design partner. You'll analyze retrieval quality, identify bottlenecks in your embedding and chunking strategies, and receive actionable recommendations to reduce hallucinations and improve context relevance. By the end, you'll have a data-driven optimization plan tailored to your specific use case and performance metrics.
by moonwell-fi
You'll build comprehensive data quality test suites that validate transformations, detect anomalies, and document standards across your dbt and SQL pipelines. This skill generates production-ready test configurations, anomaly detection protocols, and validation rules that catch data issues before they impact analytics.
by moonwell-fi
You can convert complex datasets and business metrics into polished, executive-ready reports that stakeholders trust and act on. Claude generates data-driven narratives, executive summaries, actionable insights, and visualization recommendations tailored to your audience's priorities. Your reports will tell a cohesive story that connects metrics to business outcomes, eliminating confusion and accelerating decision-making.
by moonwell-fi
You can rapidly compile, organize, and format disparate health economic evidence—from clinical trials to cost-effectiveness analyses—into structured, regulatory-compliant dossiers. This skill maps your evidence to specific payer and HTA requirements, automatically generates evidence hierarchies, and produces submission-ready dossier outlines with formatting that meets regulatory standards for NICE, EUnetHTA, and other major bodies.
by moonwell-fi
You can structure comprehensive health technology assessments (HTAs) that organize clinical evidence, economic analyses, and regulatory considerations into evidence-based coverage recommendations. This skill helps you synthesize clinical trial data, health economic models, and real-world evidence into clear, defensible payer coverage determinations. You'll generate professional HTA reports that align with major frameworks like ICER, CADTH, and NICE standards.
by moonwell-fi
This skill automatically documents your entire analytics infrastructure by analyzing data sources, transformations, and outputs. You'll generate production-ready data dictionaries with field definitions, lineage maps showing data flow across systems, and transformation documentation that explains logic and dependencies. Save weeks of manual documentation work while keeping your analytics stack discoverable as it evolves.
by moonwell-fi