Working with Alerts

Alerts help assemble your team. They are responsible for notifying your team when human intervention on an incoming event is deemed necessary (thanks to Alert Rules). Let's dive into Alerts and understand the core things that make an Alert what it is.

How Alerts get to humans

The most important thing an alert does is notify the right people about a potential incident in your system. Thanks to things like Escalation Policies and On-Call Schedules, an Alert can quickly be routed to the right person who is responsible for a team or service at any given time. You can also send Alerts directly to a team using that team's unique webhook. Let’s quickly look at the lifecycle of an Alert to see how it can make its way to a human on your team:

1. Alert Creation - An alert can be created in a few ways: (1) automatically when an event matches an Alert Rule, (2) manually when someone creates a page in Slack or the web app or (3) when an event is sent directly to a team's unique webhook. In the first two cases, the rule or the user will choose something to notify: a team, a service, an escalation policy, an on-call schedule, or an individual user. When sending an event to a team, the default escalation policy will be used.

2. Alert Notification Target - Inherent to creating an alert is specifying who or what that alert should be sent to. Let’s take a look at the ways that an alert can be routed:

3. Alert Notifications - An alert sends notifications based on a user’s Notification Preferences. For instance, a user may ask to have text messages and an email sent when an Alert is fired off.

4. Alert Actions - When the Alert reaches a human, they can respond to the alert by taking action: acknowledging the alert, escalating it, or opening an Incident.

Receiving Notifications from Alerts

Users can set up their Notification Preferences to receive a different type of Alert based on their working needs. Here’s a quick recap of the types of notifications that a user can receive:

• SMS or Text Messages - Users can add their mobile phone number to receive a text message (SMS) with information about the alert. Users can respond to text messages with codes for different alert actions. Each message will provide the correct codes for users to respond with to take the correct action.
• Voice Calls - User can also use their mobile number to receive a voice call from FireHydrant. The voice call allows them to respond verbally to prompts to take the correct action on the alert.
• Mobile App Notifications - Users who have installed the FireHydrant mobile app can also receive push notifications for any alerts. Clicking on the push notification allows them to respond to the alert easily by clicking on the correct action on the Alert page in the mobile app.
• Slack Messages - Users can also receive a Slack direct message from the FireHydrant app about Alerts. These messages will include action buttons to allow them to quickly choose the correct action when responding to the Alert.
• Email - Users can choose to receive an email every time an Alert is sent to them. The email will contain links to quickly choose the correct action for responding to the Alert.

Alert Status

Every Alert has a status that helps your team understand what others have done regarding responding to the Alert.

• Open - The default state is Open when an alert is created. This means the Alert exists, and no one has responded to it. As long as an alert is Open, it will continue attempting to notify users, moving through notification preferences and escalation policies.
• Acknowledged - When an Alert reaches a responder, the primary action someone can take is to Acknowledge it. An Acknowledged Alert will stop attempting to send any additional notifications and Escalation Policies will stop.
• Resolved - When an alert has been investigated and fixed or the upstream problem that triggered the alert is no longer a problem. If an incident is opened from this alert and linked, then resolving the incident will also resolve the alert. A Resolved alert will stop attempting to send any additional notifications and Escalation Policies will stop.
• Dismissed - When a responder has investigated an Alert and determined it was not an issue or a false alarm. Underneath the hood, FireHydrant treats Resolved and Dismissed the same (dismissed alerts will stop attempting to send any additional notifications and Escalation Policies will stop), but for organizations that don't care about differentiation, Dismissed and Resolved can effectively be treated as the same thing.

Alert actions

When responding to an Alert, responders will have a few Alert Actions they can take to respond, depending on the state of the Alert.

• When an Alert is Open
  • Acknowledge - This stops an alert from sending out additional notifications. It allows the responder to investigate the alert and determine the right next steps.
  • Escalate - This moves an alert to the next step in an escalation policy. This is useful if the responder is away from their keyboard and can’t make it back within the response SLA.
  • Open Incident - When an alert is an incident, a responder can easily open an incident right from a text message, Slack DM, or wherever they’ve received the Alert.
• When an Alert is Acknowledged
  • Open Incident - A responder can easily open an incident after acknowledging it. If an incident is opened, the Alert will automatically be attached.
  • Connect to Incident - An inbound Alert is sometimes related to an ongoing incident. After acknowledging it, responders can easily link that Alert to any existing incident.
• When an Alert is Dismissed
  • Connect to Incident - Even after an Alert has been dismissed, it may need to be connected to an existing incident.

Sending Alerts Directly to a Team

If you need to bypass Alert Rules when creating monitors, you can directly send Alerts to team by using the team's webhook URL available on the team's Alert Rules page. Any events sent to this url will create Alerts and send them to the team's default escalation policy.