Grafana Alert Rule Engineering
Engineer production-grade Grafana alert rules with intelligent escalation
What You Can Do
Design sophisticated multi-threshold Grafana alert rules that intelligently route incidents to the right teams based on severity and context. Reduce false positive alerts by 40-60% through dynamic thresholding, baseline-aware triggers, and alert quality evaluation. Generate complete incident response runbooks and escalation workflows that integrate directly with your notification channels and on-call systems.
Features
Automatic severity escalation based on conditions
Route alerts to specialists based on service ownership
Pre-written remediation steps for each alert type
Quantify and reduce alert noise systematically
Slack, PagerDuty, email, and webhook routing rules
Document alert rule decisions and evolution
Test alert behavior with synthetic data
Align alerts with service level objectives
Example Output
Alert Rule: High API Latency with Escalation
AlertName: APILatencyHigh
Thresholds:
- Warning: p95_latency > 200ms (5 min)
- Critical: p99_latency > 500ms (2 min)
Escalation:
- Warning → Slack #platform-oncall
- Critical → PagerDuty (team: API Platform)
Runbook: https://wiki.company.com/runbooks/api-latency
Escalation Policy Template
Level 1 (15 min): Slack notification to team channel
Level 2 (20 min): PagerDuty escalation to primary on-call
Level 3 (30 min): Escalate to team lead
Level 4 (45 min): Escalate to manager + VP Engineering
False Positive Reduction Report
✓ Alert: HighMemoryUsage
- Baseline: 150 false positives/month
- Dynamic threshold applied (p99-based)
- Result: 12 false positives/month (92% reduction)
- ROI: 98% fewer notifications, 0 missed incidents
What's Included
- SKILL.md: Complete alert rule engineering workflow with decision trees for threshold selection, escalation design, and runbook creation
- Alert Rule Templates: Ready-to-customize rules for CPU, memory, disk, latency, error rates, and availability
- Escalation Policy Templates: Time-based and severity-based escalation workflows with team routing
- Runbook Template: Standardized format for incident response playbooks with diagnosis and remediation steps
- Alert Quality Evaluation Checklist: Criteria for scoring and ranking alerts by business impact vs. noise ratio
- False Positive Reduction Playbook: Systematic approach to identifying and eliminating noisy alerts
- Prometheus Query Optimization Guide: Best practices for writing efficient, stable alert queries
Who It's For
- SRE and DevOps engineers designing monitoring systems for production services
- Platform engineers building alerting infrastructure and runbook automation
- On-call engineers managing alert fatigue and incident response workflows
- Systems architects implementing resilience patterns and incident management processes
Best For
- Designing multi-tier alert rules with intelligent escalation policies
- Reducing false positive alert rates and on-call fatigue
- Building comprehensive incident response playbooks and runbooks
- Integrating monitoring systems with on-call and incident management platforms
- Creating organizational alert rule standards and engineering best practices
You might also like
You can rapidly evaluate public datasets to uncover newsworthy patterns and develop story angles backed by reproducible analysis. Claude helps you assess data quality, identify anomalies, generate multiple narrative angles, and document your methodology so editors and fact-checkers can verify your findings. Turn raw data into compelling stories faster than traditional research.
by manwithstyle
Analyze Docker container logs instantly to identify root causes of failures, crashes, and resource bottlenecks. Get actionable recommendations for resource allocation, network configuration, and architectural improvements that eliminate production issues before they cascade.
by manwithstyle
Quickly architect scalable, reusable Terraform modules that follow HashiCorp best practices and organizational standards. Get automated reviews that catch common pitfalls—variable naming, provider configuration, resource dependencies—before they reach production. Standardize your infrastructure-as-code across teams with instant feedback on module quality, security posture, and cost optimization opportunities.
by manwithstyle
Manage your entire story development pipeline—from assignment briefs and source research guidance to fact-checking verification and copy editing—all within Claude's context. You'll generate assignment templates, receive real-time editorial feedback, identify verification gaps, and receive copyediting suggestions with tracked changes. The skill handles complex multi-source stories, deadline pressure, and maintains editorial standards across your publication.
by manwithstyle
You can transform complex datasets into compelling interactive narratives that maintain coherence while inviting exploration. This skill helps you architect data-driven stories where insights unfold naturally through guided discovery points, keeping readers engaged without limiting their agency. Readers navigate your narrative with purpose, uncovering patterns and relationships that would remain hidden in static presentations.
by manwithstyle
Create production-ready Grafana dashboards that visualize your metrics effectively and perform efficiently. Claude helps you write optimized PromQL and other database queries, configure smart alerts with custom thresholds, and design intuitive layouts that surface the metrics that matter most. You get dashboard JSON, query recommendations, and alert configurations ready to deploy.
by manwithstyle
Map complex investigations into structured evidence chains that verify source credibility, identify logical gaps, and build defensible narratives before publication. You can validate citations, assess counter-arguments, and document your research methodology in publication-ready format. This ensures your findings withstand scrutiny and legal challenges.
by manwithstyle
Transform breaking news briefs into compelling visual story frameworks that guide photographers toward impactful coverage. You'll receive multiple narrative angles, detailed shot lists organized by scene and purpose, and complete assignment briefs with sourcing guidance. Each output ensures comprehensive emotional and contextual storytelling from first frame to final edit.
by manwithstyle