Datadog Event Source

The Datadog Integration for Signals allows users to create events in FireHydrant from monitors in Datadog. Anytime Datadog sends an event to FireHydrant, we’ll evaluate the event payload to see if it matches a rule one of your teams set up. If a rule matches, we’ll alert the team. Learn more about Alert Rules here.

Configuring Datadog Webhook

1. In FireHydrant, navigate to the Signals Sources page (Signals > Sources). Here, you’ll find a webhook URL you will use when creating a webhook in Datadog.

2. In Datadog, visit the Integrations Page and navigate to the Webhooks Integration. If you haven’t installed it, go ahead and install it.

3. In the left-hand column of Webhooks, click the “+ New” button to add a new webhook. In the Name field, enter a name that you will later use to reference the webhook when setting up notifications for your monitor. In the URL field, paste in the webhook from FireHydrant that we covered in Step 1.
4. In the Payload field, we recommend using the following format:

{
    "summary": "$EVENT_TITLE",
    "body": "$EVENT_MSG",
    "unique_key": "$ALERT_ID",
    "level": "$ALERT_TYPE",
    "status": "$ALERT_TRANSITION",
    "links": [{"href": "$LINK", "text": "Datadog Monitor"}],
    "images": [{"src": "$SNAPSHOT", "alt": "Snapshot from $EVENT_TITLE"}],
    "tags": "$TAGS",
    "annotations": {
      "datadog/priority": "$PRIORITY"
      ...(any other annotations/data you want from Datadog here)
    }
}

(See “Fields in a Signals” for more details on how you can customize this payload in combination with Datadog’s built-in variables).

Click Save to add your new webhook.

Using Webhook as Notification Target

1. Visit any monitor where you’d like to send alerts to FireHydrant as Signals and click Edit.

2. For Monitors, scroll to step 3, “Notify your team” and type @ followed by the name of your webhook in the notification message. This should bring up a list of possible notification targets. Click on your new webhook.

   If you’re working with a synthetic monitor, you will add the webhook in Step 6, “Configure the monitor for this test.”

3. Click “Save” at the bottom of your page.

Create a test Signal from Datadog

1. For Monitors, you can send a test alert when editing the monitor. Visit the monitor you’d like to test and click Edit in the menu in the top right corner.

2. At the bottom of the page, click “Test Notifications.” In the ensuing modal, select any alert level you’d like and click “Run Test.”

3. Visit the Signal Logs in your FireHydrant app to see if a Signal was successfully created.

Field Mappings/Behaviors

The payload above directly maps to FireHydrant's Events Data Model. The following table explains the behavior once the payload hits our system:

These mappings mean that an inbound webhook from Datadog with the following content:

{
  "unique_key": "18913k2b63ifs",
  "level": "warning",
  "summary": "[P2] [Triggered] [TEST] Test Monitor",
  "body": "This is the content of the monitor.",
  "annotations": {},
  "images": [
    {
      "src": "null",
      "alt": "This should not come through"
    },
    {
      "src": "https://example.com/pic.png",
      "alt": "Snapshot from [P2] [Triggered] [TEST] Signals Test Monitor"
    }
  ],
  "links": [
    {
      "href": "https://datadoghq.com/event/event?id=238472398472582",
      "text": "Datadog Monitor"
    }
  ],
  "tags": "monitor, name:myService, role:computing-node, region:us-west2",
  "status": "Recovered",
  "received_at": "2023-11-08T21:56:54.000+00:00"
}

Will be transposed to the following FireHydrant Signal:

{
  "summary": "[P2] [Triggered] [TEST] Test Monitor",
  "body": "This is the content of the monitor.",
  "level": 1,
  "links": [
    {
      "href": "https://datadoghq.com/event/event?id=238472398472582",
      "text": "Datadog Monitor"
    }
  ],
  "images": [
    {
      "src": "https://example.com/pic.png",
      "alt": "Snapshot from [P2] [Triggered] [TEST] Signals Test Monitor"
    }
  ],
  "tags": ["monitor", "name:myService", "role:computing-node", "region:us-west2"],
  "idempotency_key": "18913k2b63ifs",
  "status": 1
}