Chaos Engineering Practices
Quick Reference
What & Why
Definition
>= 60% of critical services undergo monthly chaos experiments (pod failures, network latency, resource exhaustion).
Business Value
Improves system uptime from 99.5% to 99.9% and reduces blast radius of failures by 70% through circuit breakers and chaos engineering validation Achieving >= 60% services chaos tested monthly is a key milestone toward this goal.
Context
This capability is part of the Acceleration milestone's focus on scale automation, embed compliance, improve speed & reliability. Essential for teams targeting MTTR, CFR improvements.
Success Criteria
>= 60% services chaos tested monthly
Measurement
Chaos experiment execution rate
Evidence
- Chaos experiment definitions
- Experiment results
- Resilience improvements from experiments
In Practice
Real-World Implementation
Teams use Chaos Mesh/Litmus to run experiments: kill random pods, inject network latency, fill disk. Validate auto-recovery, monitor blast radius.
Concrete Example
Implementation Guide
Implementation Steps
Follow the measurement approach: Chaos experiment execution rate
For detailed step-by-step guidance, refer to the Resilient Operations & Chaos Engineering Implementation Kit.
Resources
Implementation Kit
Resilient Operations & Chaos Engineering KitTemplates
Browse all templatesRelated Resources
View learning pathsRelated Capabilities
Prerequisites
Implement these first
Enables
What this unlocks
Complementary
Often adopted together, from the Resilient Operations & Chaos Engineering epic
Troubleshooting & FAQs
Common Issues
Issue: Target metric not improving
Solution: Verify measurement is accurate, check if prerequisites are fully implemented, review evidence artifacts for completeness
Issue: Team resistance to adoption
Solution: Start with pilot team, demonstrate value with metrics, provide training and support during transition
Issue: Inconsistent implementation across teams
Solution: Create shared templates and guidelines, establish regular sync meetings, use automation to enforce standards
Frequently Asked Questions
Can we implement this before completing prerequisites?
While possible, it's not recommended. Prerequisites ensure foundational practices are in place, making this capability more effective and easier to adopt.
How long does implementation typically take?
Most capabilities can be implemented within 90 days when tackled as part of the Acceleration milestone. Individual timelines vary based on team size and existing practices.