Flaky Test Detection
Quick Reference
What & Why
Definition
>= 90% of flaky tests detected and fixed within 1 sprint. Flaky rate < 2%.
Business Value
Catches 70% of bugs before production deployment and reduces QA cycle time by 50% through comprehensive automated test coverage Achieving < 2% test flaky rate is a key milestone toward this goal.
Context
This capability is part of the Foundation milestone's focus on establish baseline practices (testable, releasable, monitorable). Essential for teams targeting CFR, LT improvements.
Success Criteria
Target
< 2% test flaky rate
Measurement
Test retry/flake analytics from CI
Evidence
- Flaky test dashboard
- Quarantine list
- Fix velocity metrics
In Practice
Real-World Implementation
CI tracks test pass/fail history, flags tests with <100% pass rate, teams quarantine or fix flaky tests within 1 sprint.
Concrete Example
Implementation Guide
Implementation Steps
Follow the measurement approach: Test retry/flake analytics from CI
For detailed step-by-step guidance, refer to the Testing Strategy & Quality Gates Implementation Kit.
Resources
Implementation Kit
Testing Strategy & Quality Gates KitTemplates
Browse all templatesRelated Resources
View learning pathsRelated Capabilities
Enables
What this unlocks
Complementary
Often adopted together, from the Testing Strategy & Quality Gates epic
Troubleshooting & FAQs
Common Issues
Issue: Target metric not improving
Solution: Verify measurement is accurate, check if prerequisites are fully implemented, review evidence artifacts for completeness
Issue: Team resistance to adoption
Solution: Start with pilot team, demonstrate value with metrics, provide training and support during transition
Issue: Inconsistent implementation across teams
Solution: Create shared templates and guidelines, establish regular sync meetings, use automation to enforce standards
Frequently Asked Questions
Can we implement this before completing prerequisites?
While possible, it's not recommended. Prerequisites ensure foundational practices are in place, making this capability more effective and easier to adopt.
How long does implementation typically take?
Most capabilities can be implemented within 90 days when tackled as part of the Foundation milestone. Individual timelines vary based on team size and existing practices.