Alerting Rules
Quick Reference
What & Why
Definition
>= 80% of services have alerting for high error rate (>= 5% 5xx), high latency (p95 >= 1s), and down status.
Business Value
Reduces mean time to detection (MTTD) from 4 hours to 15 minutes and enables proactive issue resolution for 60% of incidents before user impact Achieving >= 80% services have critical alerts is a key milestone toward this goal.
Context
This capability is part of the Foundation milestone's focus on establish baseline practices (testable, releasable, monitorable). Essential for teams targeting MTTR, CFR improvements.
Success Criteria
Target
>= 80% services have critical alerts
Measurement
Alerting rule coverage analysis
Evidence
- AlertManager/PagerDuty rules
- Alert firing history
- Alert response metrics
In Practice
Real-World Implementation
Teams define Prometheus alerting rules, send to PagerDuty, set severity (critical, warning), include runbook links.
Concrete Example
Implementation Guide
Implementation Steps
Follow the measurement approach: Alerting rule coverage analysis
For detailed step-by-step guidance, refer to the Observability & Monitoring Foundations Implementation Kit.
Resources
Implementation Kit
Observability & Monitoring Foundations KitTemplates
Browse all templatesRelated Resources
View learning pathsRelated Capabilities
Prerequisites
Implement these first
Enables
What this unlocks
Complementary
Often adopted together, from the Observability & Monitoring Foundations epic
Troubleshooting & FAQs
Common Issues
Issue: Target metric not improving
Solution: Verify measurement is accurate, check if prerequisites are fully implemented, review evidence artifacts for completeness
Issue: Team resistance to adoption
Solution: Start with pilot team, demonstrate value with metrics, provide training and support during transition
Issue: Inconsistent implementation across teams
Solution: Create shared templates and guidelines, establish regular sync meetings, use automation to enforce standards
Frequently Asked Questions
Can we implement this before completing prerequisites?
While possible, it's not recommended. Prerequisites ensure foundational practices are in place, making this capability more effective and easier to adopt.
How long does implementation typically take?
Most capabilities can be implemented within 90 days when tackled as part of the Foundation milestone. Individual timelines vary based on team size and existing practices.