Auto-Alerting Services and Teams

FireHydrant allows you to automatically page teams when a Service or Functionality is impacted in an incident. This enables engineers to focus on remediation while knowing that the right service owners are being notified about an incident.

Enable Auto-Alerting

  1. Confirm your alerting provider is configured on FireHydrant. This feature is currently supported for the following integrations:

    1. Signals (FireHydrant)
    2. Opsgenie
    3. PagerDuty
    4. Splunk On-Call (VictorOps)
  2. If using FireHydrant Signals, ensure your teams have default Escalation Policies and set them as responders to the service.

    1. Learn about configuring Escalation Policies for Teams in FireHydrant and then associate those Teams as Responders for a Service or Functionality. Skip to step 4 after this.
  3. If using an external Alerting provider, link your external services to FireHydrant.

    1. For FireHydrant to know which external services/escalation policies to page, you ormust import and/or link those external services to FireHydrant Services or Functionalities.

πŸ“˜

Note:

If using Splunk On-Call (VictorOps), you'll want to create alert routing keys in VictorOps for each Service in FireHydrant you'd like to link to an Escalation Policy.

  1. Turn on Automatic Alerting.
    1. On the page for each Service or Functionality you'd like to Auto-Alert, click Edit in the top right corner.
    2. Scroll down to the "Alerting" section and check the box for "Automatically alert if added to an incident."
Automatic alerting setting for a service/functionality

Automatic alerting setting for a service/functionality

  1. Scroll down and Save edits.

Using Auto-Alerting

After configuring the setting above, you only need to mark a service as impacted in an incident. FireHydrant will then send out a page to your alerting provider via the linked service and whichever Escalation Policy is assigned as the default for that service. The FireHydrant Incident Timeline also reflects that an alert was generated via that service.

This can happen anytime during an incident, including at declaration and mid-incident.

Example timeline event/Slack message when an alert is created via automatic service paging

Example timeline event/Slack message when an alert is created via automatic service paging

Next Steps