Creating Incidents from Alerts

Alerts are the first step in the Incident Management process: they help you assemble the right person or team to determine if a Signal from your system might be an incident. Let’s take a look at how Alerts in FireHydrant connect to the Incident Management

Fine-Tuning Alerts for Better Incidents

One of the best ways to improve your incident response is to ensure that your teams are alerted for things that are most likely to turn into incidents. FireHydrant provides a few ways for you to fine-tune your alerts.

🔀 Run a Back-Test of Signals using CEL
On the Alert Rules page, you can quickly run a backtest of events by running a CEL query in the page's filter builder. Running a backtest can give you an idea of the volume of any query, and you can easily create new Alert Rules from a query.

📊 Alerting Analytics
Use our new Alerting Analytics page to understand how incoming events are turning into Alerts, how many alerts are being acknowledged, and how many alerts are then turning into incidents. You can also see alerting metrics by multiple facets including teams, service catalog components, or rules matched.

Opening Incidents from Alerts

Alerts are one of the best leading indicators of a potential incident, and in FireHydrant, you have a few ways that you can open an incident from an Alert

• When a responder receives an Alert, one of the ways that they can respond to that Alert is to Open an Incident. It’s as easy as sending a code via SMS, responding to a voice call, or clicking a button in a Slack DM, an Email, or in the mobile app.

  ⚡

  Open Flow

  When an incident is opened from Mobile App, SMS, WhatsApp, or Voice alerts, FireHydrant skips the declare incident form and directly opens an incident. This subsequently also bypasses any required Incident Fields settings. You can later add details once the incident is opened.

  If the incident is opened from Web UI, Slack, or Email alerts, we will open the incident declaration webpage or Slack modal with pre-filled information from the alert.

Regardless of the alert's state (e.g., Open, Acknowledged, Dismissed, Resolved, etc.), you can always declare an incident from the alert.

Snoozing Alerts

When you receive an alert but aren't ready to act on it immediately, you can snooze it to temporarily pause notifications. A snoozed alert will re-notify you after the snooze period expires.

How to Snooze an Alert

In the Alert detail view or alert list, select the Snooze action. You'll be prompted to choose a snooze duration:

• Relative time — Snooze for a duration from now (e.g., 30 minutes, 1 hour, 2 hours)
• Absolute time — Snooze until a specific date and time (e.g., tomorrow at 9 AM)

The alert will move to a "Snoozing" state and won't send additional notifications during the snooze period.

Canceling a Snooze

To cancel a snooze before it expires, change the alert's status to any other action (Acknowledge, Escalate, Open Incident, etc.). Once you change the status, the alert will exit the "Snoozing" state immediately.

Re-opening or Re-escalating Acknowledged Alerts

Sometimes after acknowledging an alert, you realize you need additional help or want to escalate it further. FireHydrant allows you to re-open or re-escalate an already-acknowledged alert.

Use Cases

• Hand-off to another team: You've acknowledged the alert but realize a different team should handle it
• Need for higher-level escalation: You've started investigating but the issue is more complex than expected and requires escalation up the chain
• Correction: You acknowledged by mistake and want to re-engage the escalation policy

How to Re-open or Re-escalate

After acknowledging an alert, you can still take the following actions:

• Re-open the alert to transition it back to "Open" state and resume receiving notifications
• Escalate to send the alert to the next step in the escalation policy, even though it was already acknowledged

These actions are available in the alert detail view, Slack/MS Teams messages, and the mobile app.

Connecting alerts to an ongoing Incident

In cases where your team already has an ongoing incident for an Alert that comes into FireHydrant, responders can acknowledge the Alert and then connect it to the ongoing incident. In the web app, Slack, and the mobile app, they can take the Connect to Existing Incident action to select any incident in FireHydrant for that Alert.

You can link an alert with an existing incident regardless of the alert's current status, just like opening incidents.