Incident Milestones & Lifecycle Phases

Incident milestones describe the current status of the incident and communicate to stakeholders the team's progress in resolving the issue.

As responders work through incidents on FireHydrant, they will typically transition the Milestone, and FireHydrant automatically logs the timestamps of these changes. This allows FireHydrant to collect data for holistic incident metrics out-of-the-box like MTT*, Impacted Infrastructure, Responder Impact, and so on.

Lifecycle Phases and Default Milestones

Milestones are grouped by lifecycle phases. We've defined four primary phases to cover the entire incident lifecycle, and each phase comes with several default milestones.

To learn more about customizing milestones, visit Custom Milestones.

Phase 1: Started

Indicates the starting stages of an incident where alerts are just coming in. This phase has the following default milestones:

1. Started - When the affected system began having problems.

2. Detected - When a monitoring system (or human) noticed that the system was having problems.

All milestones in this first phase default to the time the FireHydrant incident was opened. For an incident created from an alert, they default to the time the alert started if that is earlier. It is common practice to update the Started milestone to an earlier timestamp if it is discovered that actual impact began earlier than when the FireHydrant incident was opened.

📘

Note:

Milestones don't have to be in chronological order, with one exception: the first milestone must be earlier than all others, since it is used to calculate MTTx metrics. If you set another milestone earlier, FireHydrant moves the first milestone back to match.

In the web UI, the Started row then shows "This timestamp has been updated to be the earliest." The API's bulk milestone update endpoint (bulk_update_incident_milestones) doesn't adjust Started for you: if any milestone would end up earlier than Started, the whole request is rejected with an error such as "Acknowledged must be at or after Started", and no milestones are changed.

Phase 2: Active

Indicates that the incident is active (e.g., responders are working on fixing things). By default it will be populated with the following milestones:

3. Acknowledged - When someone responding to the incident acknowledged the situation.
4. Investigating - When the first concrete step toward triaging and identifying the problem was taken.
5. Identified - When the problem was identified and corrective actions began.
6. Mitigated - When the system is no longer exhibiting problems to users, but the team is still monitoring the situation. For example, the team may be waiting to see if signals or SLIs normalize after the corrective action.

The lifecycle phase an incident begins in depends on whether an alert is attached when the incident is created.

An incident declared without an alert, for example from the web UI, Slack, or the API, begins in the Active lifecycle phase. The timestamps for all milestones in the Started phase match the time the incident was declared, and by default the Acknowledged milestone is set to that time as well.

An incident created with an alert attached, for example one started from Signals or by an Alert Routing rule, begins in the Started lifecycle phase. It moves to the Active phase when its primary alert is acknowledged or when a user transitions it. See "Alert-driven milestone updates" below.

All milestones beyond the first one in this Active phase must be manually transitioned to.

Alert-driven milestone updates

When an incident is created with exactly one alert, that alert becomes the incident's primary alert. A primary alert can also be set later, either manually via the Command Center "Alerts" tab or through the API.

When the status of a primary alert changes, FireHydrant updates the incident's milestones to match the alert. Each update sets the milestone for every status recorded for the alert, so resolving an alert that was previously acknowledged also sets the first Active milestone:

• When the alert is acknowledged, the incident moves to the Active phase and the first Active milestone is set to the time the alert started.
• When the alert is resolved, the incident moves to the Post-incident phase and the Resolved milestone is set to the time the alert resolved.

While the primary alert is still open, an incident that was created with an alert remains in the Started phase. An incident with no primary alert changes milestones only when a user transitions it.

When you set a primary alert on an incident, the incident's Started-phase milestones are set to the time the alert started, and the first milestone of the phase matching the alert's status is set to the time of that status. This applies even if the alert's time is later than the incident's existing timestamps.

Linking an alert to an incident does not make it the primary alert; the primary alert is set separately.

If a field required at or before the target milestone is empty, the automatic update does not apply and the incident stays in its current phase. The incident timeline shows "could not automatically resolve because required fields are missing" with the field names.

Phase 3: Post-incident

Indicates that mitigation is complete and the incident itself is no longer active. By default it will be populated with the following milestones:

7. Resolved - When the system is confirmed to be working again with no relapse. This is also the time when temporary fixes to mitigate the issue are removed, and the system is now behaving as normal.
8. Retrospective Started - When the team has actively begun to review the incident.
9. Retrospective Completed - When the team has finished reviewing the incident, clarified learnings and follow-ups, and published findings.

The Resolved milestone is set when a user manually transitions the incident to it, or automatically when the incident's primary alert resolves (see "Alert-driven milestone updates"). The remaining milestones in this phase are transitioned by a user.

Phase 4: Closed

Indicates that the incident’s process has been fully completed and no further work will commence. By default it will be populated with the following milestone:

10. Closed - Indicates all tasks mid- and post-incident are completed.
    1. The default metrics (MTTD, MTTA, MTTM, and MTTR) do not use the Closed milestone.

This milestone must be manually transitioned to by a user.

Updating Milestone Times

When you transition a Milestone, the timestamp at which you performed the action is filled in. However, you can change these values at any point.

You can change the values by clicking the Milestone dropdown in an incident's Command Center:

Local vs. UTC Time

By default, milestone timestamps are displayed in your local time zone. If you prefer to work in UTC time instead, you can toggle between local and UTC time using the timezone toggle in the milestone editor. This is useful when coordinating incidents across teams in different time zones or when your incident tracking system uses UTC internally.

Click the timezone toggle at the top of the milestone editor to switch between local time and UTC. All timestamp values will convert automatically based on your selection.

Alternatively, you can go to any event in the timeline, click the ellipses, and then use that particular event's timestamp as a value for a chosen milestone. For example:

Milestone Required Fields

Organizations can enforce mandatory data entry at specific milestones throughout the incident lifecycle. This ensures that critical information is captured consistently, facilitating more effective incident analysis, reporting, and regulatory compliance.

To modify these settings, go to Settings > Incident Settings and click the Pencil icon next to the field you want to edit. Check or uncheck "Required at and after milestone," then select the milestone where the field should be required. The field will be required at the chosen milestone and any subsequent milestones.

Attempting to transition a milestone with any required fields empty will result in an error via all methods: web UI, Slack, MS Teams, and API.

Default Incident Metrics

Incident metrics are crucial for helping you understand the health and effectiveness of your services, environments, functionalities, and incident response teams.

FireHydrant offers the following metrics based on the default Milestones:

• MTTD: Mean Time to Detection 
  time of detection - time of incident start
• MTTA: Mean Time to Acknowledged
  time to acknowledgment - time of incident start
• MTTM: Mean Time to Mitigation
  time to mitigation - time of incident start
• MTTR: Mean Time to Resolution
  time to resolution - time of incident start
• Healthiness: (MTTM * incidents) / time window
  • We use time to Mitigation by default, as the Mitigated milestone generally indicates the time customers no longer see impact on an incident.
• Impact: Within a given date range, multiple incidents are added up to calculate the time a service, functionality, or environment was degraded.

To learn more about customizing these metrics, visit Custom Milestones.

Next Steps

With a basic understanding of FireHydrant's incidents, dive into the details of conducting one by visiting the following pages:

• Starting Incidents
• Adding Responders
• Custom Milestones