Test Flakiness
Detect flaky tests by analyzing CI logs and pass rate history
What You Can Do
Analyze CI run logs and test result history to detect tests that fail non-deterministically, calculate per-test pass rates across multiple runs, and distinguish between genuinely broken tests and intermittent failures. You get actionable recommendations on whether to quarantine or fix each flaky test, plus an updated registry to prevent masking real failures.
Features
automatically detects your CI platform
Example Output
Input: CI logs from 15 recent runs
Output:
## Flaky Tests Detected (Pass Rate < 95%)
### High Priority (40-80% pass rate)
- `test_player_spawn_collision` — 65% pass rate — Likely: race condition in physics engine initialization
- **Recommendation:** Quarantine immediately, add synchronization to test setup
- **Severity:** Critical (masks real failures)
### Medium Priority (80-95% pass rate)
- `test_ui_animation_complete` — 92% pass rate — Likely: timing-dependent, platform-specific
- **Recommendation:** Increase wait timeout or add retry logic
- **Severity:** Moderate
What's Included
- `test-flakiness` SKILL.md instruction file with multi-platform CI log parsing:
- Flakiness detection algorithm with statistical thresholds:
- Quarantine registry template for regression-suite.md:
- Pass-rate calculation framework and failure pattern analyzer:
- Remediation recommendation matrix (quarantine vs. fix guidance):
Who It's For
- QA/Test Engineers — diagnose why CI is unreliable and decide quarantine vs. fix strategies
- Release Engineers — reduce false positives during release validation runs
- Software Engineers — identify which flaky tests to prioritize fixing during Polish phase
- Platform Teams — maintain test reliability and prevent test rot across CI/CD pipelines
- DevOps Engineers — monitor test suite health and flag systemic flakiness trends
Best For
- Detecting intermittent test failures across multiple CI runs without manual log review
- Quarantining known flaky tests to prevent masking real failures in CI
- Diagnosing root causes of test non-determinism (timing, race conditions, environment sensitivity)
- Building a historical registry of flaky tests and remediation status
- Polish phase regression testing when sufficient CI data has accumulated
You might also like
Query your local Screenpipe instance to retrieve screen recordings, audio transcriptions, UI element accessibility trees, keyboard/mouse input logs, and productivity analytics. You can search by keywords, filter by content type (audio, OCR, accessibility, input), set time ranges, and extract structured data about your applications, meetings, and work sessions without sending data to external servers.
by az9713
You can submit architectural design documents, floor plans, and project parameters to receive a comprehensive compliance analysis that cross-references IBC, local amendments, zoning ordinances, and ADA accessibility requirements. Claude identifies specific gaps, cites applicable code sections, explains why each gap matters, and provides concrete remediation strategies—all before formal design review submission when corrections are costly and schedule-impacting.
by az9713
You can leverage Claude to synthesize regional seismic hazard data from authoritative sources (USGS maps, published studies), interpret probabilistic versus deterministic earthquake scenarios, and analyze how site conditions modify ground motions. Claude helps you justify seismic design parameters to clients and regulators, evaluate liquefaction and earthquake-induced landslide risks, and develop defensible recommendations aligned with ASCE 7 and current guidelines—accelerating report preparation without replacing specialized geotechnical software.
by az9713
You can compare building materials across embodied carbon, water usage, recyclability, supply chain impacts, and lifecycle costs—all grounded in industry standards like EN 15804 and ISO 14040/44. This skill transforms material selection from intuition-based guessing into data-driven specification, producing analysis outputs ready for client presentations, value engineering discussions, and regulatory compliance documentation.
by az9713
You systematically evaluate your internal developer platform across dimensions like onboarding, deployment, tooling, and documentation to identify quantifiable friction points. Claude analyzes your platform's usability, creates a friction scoring matrix, and generates a prioritized improvement roadmap ranked by team impact and implementation effort. You get a data-driven strategy to reduce developer toil and accelerate delivery.
by az9713
You can audit BIM models across five critical domains: geometric accuracy and LOD consistency, information requirements and metadata, coordinate systems and project setup, element naming and classification standards, and documentation completeness. This skill generates detailed compliance reports that identify every gap and provide specific remediation guidance, enabling you to enforce consistent standards across projects, reduce rework, and ensure models are ready for handoff to downstream disciplines or coordination workflows.
by az9713
You can conduct rigorous architectural design reviews that verify compliance with building codes, accessibility standards, energy regulations, and client requirements. The skill produces evidence-based, severity-ranked findings with specific remediation recommendations and regulatory citations, enabling you to identify issues before design phase submission or construction document preparation.
by az9713
You can accelerate HVAC design workflows by inputting building parameters—envelope characteristics, occupancy data, climate conditions, and existing systems—and receiving calculated heating/cooling loads, equipment sizing recommendations, and zone-by-zone analysis. Claude organizes complex building data into actionable design parameters and specification summaries for client review, permit submission, and professional validation against ASHRAE and ACCA standards.
by az9713