Grafana Alert Rule Engineering
Engineer production-grade Grafana alert rules with intelligent escalation
What You Can Do
Design sophisticated multi-threshold Grafana alert rules that intelligently route incidents to the right teams based on severity and context. Reduce false positive alerts by 40-60% through dynamic thresholding, baseline-aware triggers, and alert quality evaluation. Generate complete incident response runbooks and escalation workflows that integrate directly with your notification channels and on-call systems.
Features
Automatic severity escalation based on conditions
Route alerts to specialists based on service ownership
Pre-written remediation steps for each alert type
Quantify and reduce alert noise systematically
Slack, PagerDuty, email, and webhook routing rules
Document alert rule decisions and evolution
Test alert behavior with synthetic data
Align alerts with service level objectives
Example Output
Alert Rule: High API Latency with Escalation
AlertName: APILatencyHigh
Thresholds:
- Warning: p95_latency > 200ms (5 min)
- Critical: p99_latency > 500ms (2 min)
Escalation:
- Warning → Slack #platform-oncall
- Critical → PagerDuty (team: API Platform)
Runbook: https://wiki.company.com/runbooks/api-latency
Escalation Policy Template
Level 1 (15 min): Slack notification to team channel
Level 2 (20 min): PagerDuty escalation to primary on-call
Level 3 (30 min): Escalate to team lead
Level 4 (45 min): Escalate to manager + VP Engineering
False Positive Reduction Report
✓ Alert: HighMemoryUsage
- Baseline: 150 false positives/month
- Dynamic threshold applied (p99-based)
- Result: 12 false positives/month (92% reduction)
- ROI: 98% fewer notifications, 0 missed incidents
What's Included
- SKILL.md: Complete alert rule engineering workflow with decision trees for threshold selection, escalation design, and runbook creation
- Alert Rule Templates: Ready-to-customize rules for CPU, memory, disk, latency, error rates, and availability
- Escalation Policy Templates: Time-based and severity-based escalation workflows with team routing
- Runbook Template: Standardized format for incident response playbooks with diagnosis and remediation steps
- Alert Quality Evaluation Checklist: Criteria for scoring and ranking alerts by business impact vs. noise ratio
- False Positive Reduction Playbook: Systematic approach to identifying and eliminating noisy alerts
- Prometheus Query Optimization Guide: Best practices for writing efficient, stable alert queries
Who It's For
- SRE and DevOps engineers designing monitoring systems for production services
- Platform engineers building alerting infrastructure and runbook automation
- On-call engineers managing alert fatigue and incident response workflows
- Systems architects implementing resilience patterns and incident management processes
Best For
- Designing multi-tier alert rules with intelligent escalation policies
- Reducing false positive alert rates and on-call fatigue
- Building comprehensive incident response playbooks and runbooks
- Integrating monitoring systems with on-call and incident management platforms
- Creating organizational alert rule standards and engineering best practices
You might also like
Transform breaking news briefs into compelling visual story frameworks that guide photographers toward impactful coverage. You'll receive multiple narrative angles, detailed shot lists organized by scene and purpose, and complete assignment briefs with sourcing guidance. Each output ensures comprehensive emotional and contextual storytelling from first frame to final edit.
by manwithstyle
You'll receive data-driven recommendations for chart types, color schemes, visual hierarchies, and narrative structures tailored to your audience and message. This skill guides you through selecting encodings that accurately represent data while maximizing audience comprehension and engagement—whether you're designing publication-ready graphics or interactive dashboards.
by manwithstyle
You can rapidly evaluate public datasets to uncover newsworthy patterns and develop story angles backed by reproducible analysis. Claude helps you assess data quality, identify anomalies, generate multiple narrative angles, and document your methodology so editors and fact-checkers can verify your findings. Turn raw data into compelling stories faster than traditional research.
by manwithstyle
This skill positions Claude as your Grafana technical advisor. You can optimize PromQL queries for performance, design clear dashboards for incident response, create effective alert rules that minimize false positives, and troubleshoot visualization issues—all grounded in monitoring best practices and your specific metrics context.
Analyze Docker container logs instantly to identify root causes of failures, crashes, and resource bottlenecks. Get actionable recommendations for resource allocation, network configuration, and architectural improvements that eliminate production issues before they cascade.
by manwithstyle
Quickly architect scalable, reusable Terraform modules that follow HashiCorp best practices and organizational standards. Get automated reviews that catch common pitfalls—variable naming, provider configuration, resource dependencies—before they reach production. Standardize your infrastructure-as-code across teams with instant feedback on module quality, security posture, and cost optimization opportunities.
by manwithstyle
Manage your entire story development pipeline—from assignment briefs and source research guidance to fact-checking verification and copy editing—all within Claude's context. You'll generate assignment templates, receive real-time editorial feedback, identify verification gaps, and receive copyediting suggestions with tracked changes. The skill handles complex multi-source stories, deadline pressure, and maintains editorial standards across your publication.
by manwithstyle
Rapidly triage infrastructure incidents across all GCP services using guided diagnostic workflows, root cause decision trees, and remediation procedures. You get structured steps to isolate failures in Compute Engine, Cloud Run, Cloud SQL, networking, and storage, complete with gcloud commands, health checks, and rollback procedures.
by manwithstyle