How Incidents Work
Understand how MonitoringDog detects failures, opens incidents, and moves them through their lifecycle.
An incident is not opened the moment a probe fails. MonitoringDog uses a two-phase confirmation process to distinguish real outages from transient noise.
Opening an Incident
- A worker reports a probe failure → an
ISSUE_DETECTEDevent fires internally. - The failure must persist through the check's confirm period before an incident is created.
- Once confirmed, a
STARTEDevent fires and an Incident record is created with statusSTARTED.
When the confirm period is set to 0, a single failed probe opens an incident immediately.
The confirm period is time-based, not count-based — it does not count consecutive failures, it waits a fixed duration.
Incident Lifecycle
STARTED → ACKNOWLEDGED → RECOVERED → RESOLVED
↘ ISSUE_REAPPEARED| Status | Meaning |
|---|---|
STARTED | Incident opened, escalation policy is firing |
ACKNOWLEDGED | A team member has seen it and is working on it |
RECOVERED | Probes are passing and the recover period has elapsed |
RESOLVED | Incident closed |
ISSUE_REAPPEARED | The service failed again after a RECOVERED state |
Automatic Recovery
When probes start passing again after an incident, the check enters VALIDATING state. After the recover period elapses with no further failures, the incident moves to RECOVERED and resolution alerts are sent per the escalation policy's resolutionAlerts config.
If the service fails again during the recover period, the incident status moves to ISSUE_REAPPEARED and escalation resumes.