Runbook Conditions

FireHydrant's Runbooks offer the powerful ability to customize execution based on various conditions. This allows controlling when certain Runbooks, as well as steps within them, should attach/execute. If you configure any Custom Fields, they will also be available as conditions.

Runbook Conditions

Runbook-level conditions will determine whether a Runbook will attach to an incident. This allows you to create Runbooks tailored to specific situations, such as only SEV1 incidents, or only incidents where my-specific-service is impacted.

FireHydrant is constantly scanning your incidents for matching incident conditions. This means Runbooks can attach to any of your incidents at any point in the incident lifecycle (with some exceptions. See Condition Expiration below).

How to edit Runbook conditions

1. Click Runbooks in the navigation, then click the Runbook you want to configure.
2. In the edit page, you can configure Runbook conditions on the right panel under Execution rules.

For the full list of available conditions and triggers, visit All Available Conditions below.

Step Conditions

Individual steps within Runbooks can also be conditionally executed.

• Available rules
  • Runbook execution rules include all of the available conditions for Runbook-level conditions, in addition to several execution-specific conditions
• Scheduling
  • You can configure certain Runbook steps to repeat on intervals or specific transitions. See Repeating Steps.
  • You can also configure whether a step should immediately evaluate and execute, or if it should be triggered manually by a user. This can be done via the UI or via /fh runbooks in chat apps (see below).

🚧

Note:

All steps within a Runbook execute concurrently as soon as the Runbook attaches. If you want to ensure sequential execution of steps, use the Previous Runbook step condition and reorder the steps in the order you'd like them to execute.

How to edit step conditions

1. Follow the same steps above to navigate to the Runbook.

2. Find the specific step you'd like to add conditions to and click the writing icon on the right side.

   

3. In the modal that opens for that step, click the Triggers tab. This opens your options for applying rules that dictate when this specific step runs.

For the full list of available conditions and triggers, visit All Available Conditions below.

Repeating steps

Some Runbook steps allow for repetition and rescheduling, depending on the conditions selected. This allows, for example, things like reminding the incident channel to post updates every 30 minutes or to send an email on every incident update or note. If a step can be repeated, you will see extra checkboxes.

Repeat on Interval

By default, the checkbox for Repeat After Interval is available for any steps that can be repeated. When checked, an additional input is shown to specify the repetition time. This interval repeat behavior is mutually exclusive to repeating on transitions, covered in the next subsection.

📘

Note:

The shortest interval allowed today is 5 minutes.

Rerun on every transition

The other repetition option is for specific incident updates or transitions. This checkbox is only enabled when certain conditions are selected:

• Current milestone [changes, moves forward, moves backward]
• Current severity [changes, escalates, de-escalates]
• Incident Note [is added]

When this checkbox is enabled, the step will automatically reschedule and re-evaluate conditions after the specified change triggers it. This repetition option is mutually exclusive to interval repetitions.

Condition Expiration

Runbooks and Steps will not always execute or be on the lookout for forever. FireHydrant expires condition checks for Runbooks and Steps based on various conditions.

Runbook Expiration

FireHydrant will stop checking incidents for matching Runbook conditions when one of the following occurs:

• The incident reaches the Resolved milestone
  • After an incident is resolved, no new Runbooks will automatically attach, even if conditions match. You will need to attach them manually. "When Current milestone = Resolved," etc., conditions will still not automatically attach the Runbook.
• The Runbook has already attached
  • We do not support unattaching and reattaching the same Runbooks to the same incident currently. You may stop/restart the Runbook, but this will re-execute the same steps configured before. Other Runbooks not yet attached will still be evaluated as normal.
• The incident is older than 30 days
  • All condition checks expire after 30 days, even if an incident has not been resolved yet. So, any changes to an incident after 30 days will not trigger a new Runbook attachment.

If you want Runbooks to attach post-resolution, you can workaround this limitation by adding a Attach a Runbook step to other Runbooks that do attach pre-resolution.

Alternatively, you can still manually attach a Runbook to an incident at any time.

Step Expiration

FireHydrant will expire steps or certain aspects of steps when one of the following occurs:

• When the incident reaches the Resolved milestone**
  • FireHydrant terminates any recurring steps when an incident is resolved
  • **This does not apply to post-incident steps/conditions (e.g. When current milestone is Retrospective Completed). Those will still execute as normal post-resolution.
• When a step has been polling for longer than 30 days
  • If a step has been waiting and checking for matching conditions for longer than 30 days, the step will expire.

All Available Conditions

• Incident Slack channel - Whether the incident has an existing Slack channel
• Incident Microsoft Teams channel - Whether the incident has an existing Microsoft Teams channel
• Current milestone - If the current milestone matches the supplied conditions. Supports transition-based repetition. Milestones specifically support additional triggers, including:
  • Changes - Triggers when the milestone changes in either direction
  • Moves Forward - Triggers when the milestone moves forward (e.g., Acknowledged -> Investigating)
  • Moves Backward - Triggers when the milestone moves backward (e.g., Mitigated -> Identified)
• Current severity - If the current severity matches the supplied conditions. Supports transition-based repetition. Severities specifically support additional triggers, including:
  • Changes - Triggers when the severity changes in either direction
  • Escalates - Triggers when the severity increases (e.g., SEV2 -> SEV1)
  • De-Escalates - Triggers when the severity decreases (e.g., SEV1 -> SEV2)
• Current priority - If the current priority matches the supplied conditions
• Previous Runbook step (STEP only) - Specifies for a Runbook step only to execute if the step prior has started, errored, or completed.
• Incident tags - If the current tags match the supplied conditions
• Incident ticket - Whether the incident already has an incident ticket created
• Time since incident opened - Checks the time elapsed from when the incident was declared/started on FireHydrant
• Time since last incident note - Checks the time elapsed from when there was last an incident note or update (not general chat messages, but specifically incident updates)
• Incident assigned roles - Whether the specified role(s) has/have at least one person assigned
• Incident assigned teams - Whether the specified team(s) have been assigned to the incident
• Incident impacted infrastructure - Whether the incident has any of the specified Service Catalog components marked as impacted
• Incident impacted service tiers - Whether any of the specified Service Catalog components impacted on the incident match the specified tiers
• Incident attached Runbooks - Whether the specified Runbooks are also attached to the Incident
• When invoked (RUNBOOK only) - Sets the Runbook only to attach when specified by a human during an incident
• Incident Note is added - Specific trigger for when an incident update/note is posted. Supports transition-based repetition
• Incident type - Specific trigger for when an incident is created from a specific Incident Type
• [Custom Fields] - Specifies conditions according to your defined custom fields
• Time since milestone [...] - Checks the time elapsed from when the incident first entered the specified milestone

Resolved and Retrospective States

When any incident reaches the Resolved Milestone or beyond, any automation on Runbooks will change to a delayed model that executes every 10 minutes.

For example, if you have a step to Archive Incident Channel when the Milestone is Retrospective Completed, it may not archive immediately after you've completed your retro.

However, steps will consistently execute within 10 minutes after an incident is resolved.

Race Conditions

Several Runbook steps have prerequisites. As noted above, all steps within a Runbook execute concurrently upon attachment to an incident. Subsequently, you'll need to be aware of potential race conditions for steps that require certain things. Some examples:

• Let's say you want to Add a Bookmark to Incident Channel. You will need to, for example, ensure in the conditions for that step that the Incident channel exists. This will delay the step from executing until the channel is confirmed created, otherwise it may try to set a bookmark in a non-existent channel and fail.
• If you want to name the incident channel after the Jira ticket, then you'll want to either set the condition to Previous step has completed and put the Slack channel step just after the Jira step. Or, you can use the Incident ticket exists condition.

In general, if a Runbook step depends on some condition that another Runbook step changes, then you'll want to pay careful attention to the conditions of downstream, dependent steps to avoid race conditions.

Next Steps

• Read about Runbook Best Practices for strategies on how to organize your automation
• Browse the list of available Runbook Steps