ETL Pipeline Design & Optimization
Design and optimize data ETL pipelines for production performance
What You Can Do
Analyze your data pipeline architecture, identify performance bottlenecks, and receive detailed optimization strategies. You'll get architectural recommendations for scalability, data quality assurance patterns, and concrete implementation guidance for improving throughput and reducing latency in your ETL workflows.
Features
Assess current ETL design and identify structural improvements for better performance
Pinpoint which stages consume the most time and resources
Design pipelines that handle growing data volumes without performance degradation
Implement validation, reconciliation, and error handling at each stage
Get concrete techniques to reduce latency and increase throughput
Evaluate tools and formats for your specific pipeline requirements
Identify resource-intensive operations and suggest cost-effective alternatives
Example Output
Pipeline Performance Analysis:
- ✅ Current state: Your daily ETL processes 2.5M records in 4 hours with 12% failure rate
Identified Bottlenecks:
-
JSON Serialization (Est. 40% of runtime)
- Root cause: Parsing 1.2M intermediate records as JSON
- Fix: Switch to Parquet format → 2.1x speedup
- Implementation: Add Parquet encoder before aggregation stage
-
Unindexed Lookups (Est. 32% of runtime)
- Root cause: Sequential scans of 500K reference table per transform
- Fix: Pre-load reference table into memory hash map
- Expected improvement: 3x faster join operations
Recommended Architecture:
Ingest → Validate → Transform → Aggregate → Parquet Sink
(fail fast) (cached refs) (vectorized) (batch I/O)
Data Quality Checklist:
- Row count reconciliation before/after each stage
- Field-level type and null validation
- Duplicate detection on primary key
- Late-arriving dimension handling
What's Included
- SKILL.md: Complete ETL design framework with analysis workflows, optimization techniques, and architectural patterns
- Pipeline Architecture Template: Structured format for documenting current and proposed pipelines
- Bottleneck Detection Checklist: Common performance anti-patterns and where to look
- Optimization Workbook: Proven techniques for throughput, latency, and cost reduction
- Data Quality & Validation Guide: Testing strategies, reconciliation queries, and alerting patterns
Who It's For
- Data engineers — Optimize production ETL pipelines and reduce processing time
- Data architects — Plan scalable data integration strategies from scratch
- Analytics engineers — Improve data warehouse load performance and reliability
- Platform engineers — Debug slow pipelines and implement resilient workflows
- Performance engineers — Analyze bottlenecks and measure optimization impact
Best For
- Diagnosing why ETL pipelines are slow or failing
- Designing new data integration architectures
- Reducing pipeline execution time and infrastructure costs
- Planning data quality validation and error handling
- Evaluating technology choices (Spark, Airflow, tools, formats)
You might also like
Analyze Solidity and other smart contract code for security vulnerabilities, gas inefficiencies, and best practice violations. Get detailed reports with risk scoring, remediation suggestions, and optimization recommendations. Whether you're auditing before deployment or reviewing third-party contracts, this skill identifies critical issues faster than manual review.
You can systematically diagnose database performance bottlenecks by sharing your schema, slow query logs, and execution plans with Claude. It identifies root causes—missing indexes, inefficient joins, lock contention—and provides prioritized recommendations with ready-to-implement SQL. Skip the manual log analysis and get tuning strategies tailored to your workload.
This skill provides a structured approach to analyzing process problems, identifying root causes, and recommending capacity optimizations. You'll get clear bottleneck identification, data-driven recommendations, and a framework to validate whether your solutions actually work. Perfect for diagnosing why workflows are slow and finding the leverage points that matter most.
Use Claude to systematically analyze your database queries, execution plans, and schema to identify performance bottlenecks. The skill generates actionable optimization recommendations with SQL rewrites, index strategies, and configuration tuning. You'll receive detailed before-and-after performance analysis to validate improvements and prioritize work by impact.
You'll design and implement mobile features with architectural rigor, cross-platform considerations, and edge-case handling built-in. This skill generates complete system designs, platform-specific implementation strategies, performance optimization approaches, and testing frameworks. The output is production-ready guidance spanning iOS and Android with security, offline resilience, and deployment strategies included.
Design and optimize injectable formulations by analyzing your active pharmaceutical ingredient (API), selecting compatible excipients, and predicting stability outcomes. You'll receive systematic workflows that guide you through API characterization, formulation architecture, and risk mitigation—enabling faster development cycles and regulatory-ready documentation.
You'll develop systematic approaches to injectable formulation design, from API selection through sterilization strategy. Claude helps you troubleshoot failed batches by analyzing root causes, recommends regulatory pathways (505(b)(2), ANDA, NDA), and provides science-backed solutions for stability, compatibility, and manufacturability challenges.
You can architect multi-node ROS control systems from scratch, including node design patterns, communication flows, and real-time constraints. You'll debug complex node interactions using publisher/subscriber analysis, service call tracing, and action server diagnostics. You can optimize motion controllers through PID tuning, trajectory planning validation, and performance profiling to achieve precise, responsive robotic behavior.