Databricks SQL & Pipeline Optimizer
Optimize Databricks SQL queries and pipeline performance
What You Can Do
You can analyze and optimize your Databricks SQL queries and data pipelines to dramatically improve performance and reduce compute costs. This skill identifies bottlenecks, suggests architectural improvements, and generates actionable optimization strategies backed by execution plan analysis and cost-benefit calculations.
Features
Parse execution plans and identify inefficient operations, missing indexes, and shuffle operations
Analyze multi-stage data pipelines for bottlenecks, skewed partitions, and resource contention
Calculate compute cost savings from specific optimizations (caching, partitioning, file format changes)
Break down complex query plans with clear explanations of each operation and its cost
Recommend cluster settings, auto-scaling policies, and SQL warehouse configurations
Design end-to-end pipeline improvements considering data volume, frequency, and dependencies
Generate before/after test plans with specific metrics to measure improvement
Verify your queries and pipelines against Databricks performance and cost best practices
Example Output
Optimized Query Example
Original query: 45 seconds, $0.82 cost
SELECT customer_id, COUNT(*)
FROM events
WHERE event_date > '2024-01-01'
GROUP BY customer_id
Optimized query: 8 seconds, $0.14 cost
SELECT customer_id, COUNT(*)
FROM events_partitioned
WHERE event_date > '2024-01-01'
GROUP BY customer_id
Improvements: Partitioning on event_date, Parquet format, eliminated unnecessary columns
Cost Reduction Plan
- Change: Enable Delta cache for fact table (5-minute retention)
- Impact: 60% faster joins, save ~$400/month
- Implementation: 10 lines of SQL configuration
- ROI: 5x within first week
What's Included
- SKILL.md: Complete skill with analysis workflows and optimization strategies
- SQL Optimization Checklist: Line-by-line query review covering joins, partitioning, caching, and aggregation patterns
- Pipeline Analysis Worksheet: Template for decomposing multi-stage pipelines and identifying opportunities
- Performance Testing Framework: Before/after testing template with metrics and cost calculation
- Databricks Configuration Guide: Recommended cluster sizes, auto-scaling, and SQL warehouse tiers by workload
- Query Plan Decoder: Decision tree for interpreting execution plans and translating findings to code
Who It's For
- Data Engineers — Optimize ETL pipelines and improve SLA compliance
- Analytics Engineers — Speed up dimensional modeling and fact table queries
- Database Administrators — Tune clusters and resource allocation
- Data Analysts — Reduce execution time for ad-hoc exploration and dashboards
- Performance Engineers — Identify and resolve workload bottlenecks across production systems
Best For
- SQL Query Optimization — Rewrite slow queries using execution plan analysis
- Pipeline Performance Tuning — Diagnose and fix multi-stage ETL bottlenecks
- Cost Reduction Analysis — Calculate ROI of optimizations (downsizing, file format, caching)
- Migration Planning — Assess and optimize legacy data warehouse workloads
- SLA Performance Improvement — Meet latency and throughput targets through targeted optimization
You might also like
This skill transforms fragmented audit observations, lab results, and monitoring records into structured compliance intelligence. You consolidate raw findings, automatically cross-reference them against applicable regulatory frameworks, prioritize corrective actions by risk and operational impact, and generate audit-ready documentation packages with systemic issue trending—reducing manual consolidation from 4-6 hours to 30-45 minutes.
by adeline117
You can design and execute professional texture profiling studies that decompose complex mouthfeel characteristics into measurable, reproducible dimensions. This framework guides you through panelist training with calibrated reference scales, standardized texture vocabulary, data collection protocols, and translation of sensory results into actionable product specifications that R&D and quality control can implement and monitor.
by adeline117
You can conduct rigorous texture evaluations that integrate descriptive sensory panels with instrumental measurements, ensuring consistency and consumer relevance. This skill guides you through descriptor standardization, panel training protocols, instrumental correlation workflows, and cross-functional communication of texture findings. You'll build validated texture specifications, troubleshoot texture defects with data-backed solutions, and benchmark competitive products against your standards.
by adeline117
You can evaluate packaging solutions across multiple critical dimensions simultaneously. Claude analyzes material compatibility with specific food products, verifies compliance with FDA, EU, and international food contact regulations, calculates environmental impact and identifies sustainable alternatives, models barrier properties against oxygen/moisture/light penetration, estimates lifecycle costs, and generates technical documentation for suppliers and regulatory submissions. This integrated approach helps you make packaging decisions that protect product quality while meeting sustainability and budget requirements.
by adeline117
You can input your current processing parameters (temperature, time, moisture, pH, and other critical control points) alongside performance metrics, and Claude will benchmark them against industry standards, identify which variables most influence your outcomes, and generate hypothesis-driven optimization recommendations. The skill helps you create reproducible protocols that reduce batch-to-batch variability while maintaining food safety compliance and regulatory requirements.
by adeline117
You can design statistically valid consumer sensory studies from protocol conception through data interpretation. This skill guides you through panel recruitment strategy, scorecard development, methodology selection (affective vs. descriptive testing), data analysis with appropriate statistical tests, and translating sensory findings into product development and marketing recommendations. You'll minimize bias, ensure valid measurement of human perception, and communicate results that drive formulation and positioning decisions.
by adeline117
You can transform raw sensory data and consumer feedback into standardized flavor architectures that drive formulation decisions. This skill systematizes descriptive analysis results, models how ingredient combinations create emergent flavor characteristics, and translates preference data into actionable formulation adjustments. You'll create evidence-based flavor briefs that align R&D and marketing stakeholder expectations while documenting flavor architecture for regulatory compliance.
by adeline117
You can submit marketing copy, label claims, or advertising language and receive a structured compliance assessment that maps claims against FDA/FTC guidance, state regulations, and case law. The skill identifies claim type (health claim, structure-function claim, comparative claim, etc.), flags regulatory risks across jurisdictions, assesses substantiation strength, and recommends revisions to reduce enforcement exposure. This accelerates claim review workflows and prevents costly post-launch violations.
by adeline117