Databricks Query Optimization & Pipeline Debugging
Optimize Databricks queries and resolve pipeline issues
What You Can Do
You can optimize Databricks SQL queries for performance, debug pipeline failures with root-cause analysis, and recommend cluster configurations. Claude analyzes execution plans, identifies bottlenecks, and provides concrete refactoring suggestions to reduce query runtime and lower infrastructure costs.
Features
Analyze execution plans to identify expensive operations, data shuffles, and scanning inefficiencies
Refactor queries with partitioning strategies, index hints, and join optimization techniques
Trace pipeline failures through logs, identify data quality issues, and recommend fixes
Right-size worker nodes, adjust shuffle partitions, and optimize memory allocation for your workload
Map data flow across jobs and notebooks to identify bottlenecks and failure propagation
Compare query performance before/after optimization and quantify resource savings
Identify fragmented data files, recommend OPTIMIZE/VACUUM operations, and analyze table statistics
Design SLAs for pipelines and configure thresholds to catch performance degradation early
Example Output
Input: Slow query execution plan with 12 shuffles
Output:
-- Optimized query with pre-filter and join reordering
SELECT customer_id, SUM(amount)
FROM sales
WHERE year = 2026 -- Filter first
JOIN customers USING(customer_id)
GROUP BY customer_id
Execution time: 45s → 8s | Shuffle operations: 12 → 2 | Estimated savings: 92% reduction
What's Included
- SKILL.md: Complete Claude skill with workflows for query optimization, pipeline debugging, and cluster tuning
- Query Analysis Checklist: Step-by-step guide for diagnosing slow queries
- Optimization Templates: Reusable SQL patterns for joins, aggregations, and window functions
- Debug Workflow: Decision tree for pipeline failure diagnosis with remediation steps
- Configuration Reference: Cluster sizing formulas and parameter tuning guide
Who It's For
- Data Engineers — Optimize ETL pipeline performance and reduce cloud spending
- Analytics Engineers — Tune SQL queries powering dashboards and reports
- Database Administrators — Manage cluster resources and enforce performance standards
- Data Scientists — Accelerate feature engineering pipelines on large datasets
- MLOps Engineers — Debug training data pipelines and optimize model training workflows
Best For
- Diagnosing slow Databricks queries and recommending optimizations
- Debugging failed Delta Lake jobs with detailed error analysis
- Right-sizing Databricks clusters and predicting cost savings
- Analyzing execution plans to identify data shuffles and scanning bottlenecks
- Optimizing SQL patterns for Databricks-specific features (Photon, Delta cache, Z-order)
You might also like
You can evaluate unfamiliar locations objectively against your specific production needs—whether that's crane access, parking capacity, noise restrictions, or permit requirements. The skill generates comprehensive scout reports that compare multiple sites using weighted evaluation matrices, flag deal-breakers early, and document infrastructure limitations so your production team makes informed decisions before committing resources to visits.
by skillslib
You'll design robust SCADA system architectures, configure industrial protocols (Modbus, Profibus, OPC-UA, DNP3), and diagnose connectivity and performance issues across distributed control networks. Get step-by-step configuration guidance, integration workflows, and troubleshooting decision trees tailored to your specific hardware and protocol stack.
by skillslib
You can establish systematic color grading workflows that accelerate project delivery, ensure visual consistency across episodes and projects, and maintain broadcast-quality standards. Claude generates reusable templates, quality control checklists, and grading decision frameworks tailored to your project's color science and deliverable requirements.
by skillslib
Generate customized legal documents and client communications automatically from your MyCase case data. This skill extracts case details, client information, and matter status to populate document templates, send email updates, and create batch processing workflows. Reduce manual data entry and standardize your document generation process while maintaining client communication consistency.
by skillslib
You can convert raw ESG metrics and narratives into professionally structured impact reports that satisfy regulatory and investor requirements. The skill automatically verifies alignment with SASB materiality categories, GRI standards, and TCFD recommendations; constructs defensible materiality matrices from stakeholder feedback; standardizes impact metrics to IRIS+ and SDG frameworks; and identifies compliance gaps between your draft and mandatory disclosure requirements.
by skillslib
Use Claude to automate your entire collection workflow, from first touch to recovery, with legally sound escalation protocols. You'll recover more money in less time, reduce bad debt write-offs by 25-40%, and build defensible collection records that withstand legal scrutiny. Claude handles customized demand letters, compliance validation, debtor profiling, and escalation strategies tailored to account age, debtor behavior, and jurisdiction.
by skillslib
You can quantify how interest rate changes will affect your bond holdings through duration and convexity analysis, calculate precise hedge ratios for Treasury futures and other instruments, and stress-test portfolios across multiple yield curve scenarios. This skill handles complex securities including callable bonds and mortgage-backed securities, enabling you to compare different portfolio structures and evaluate positioning against interest rate forecasts.
by skillslib
This skill helps you troubleshoot Wwise integration issues, diagnose audio performance bottlenecks, and generate comprehensive documentation for interactive audio specifications. You'll get step-by-step fixes for common implementation problems, performance optimization strategies, and best practices for audio systems across different game engines and platforms.
by skillslib