Webhook + bridge recipeU0 · act immediatelyTemplate checks passed 2026-09-06

Datadog monitor phone calls for persistent production incidents

Wait for a production P1 to persist, confirm customer impact, and call only while the incident is active, unacknowledged, and owned by someone who can act.

Download the Datadog webhook payload Run a supervised self-call

Datadog remains the source of alert state. ClawdCall adds one contextual phone handoff, captures acknowledgment, and returns the result to the incident workflow.

Independent workflow recipe; not endorsed by or affiliated with Datadog. Automated checks verify route data, downloadable assets, fail-closed policy gates, secret scans, and the production build. This is not a live vendor-account certification. Complete one supervised call to yourself before production use; live calls can consume calling minutes.

Quick Answer

How can a Datadog monitor trigger an on-call phone call?

Trigger a Datadog webhook or Workflow Automation from a monitor only after the critical state persists. Require a production P1, confirmed customer impact, no acknowledgment, a trusted on-call owner, and a fresh alert-cycle key. ClawdCall then places one contextual phone call and returns the acknowledgment to the incident workflow without replacing Datadog On-Call or PagerDuty.

01Datadog monitor

02Persistent-impact gate

03ClawdCall phone handoff

04Acknowledgment returned

Why Phone?

Escalate persistent impact—not the first spike.

Keep transient alerts in Datadog's normal notification channels. Add a contextual phone call only after the monitor persists, customer impact is confirmed, and the on-call owner can begin investigation or request fallback.

Verified trigger

Wait for a persistent critical state

A Datadog monitor sends a Renotify transition after the critical condition persists for the configured interval.

Policy gate

Confirm impact and ownership

Require production scope, P1/critical priority, confirmed customer impact, active state, missing acknowledgment, a known on-call owner, and a fresh alert-cycle key.

Returned outcome

Return acknowledgment to the incident path

Record acknowledgment, owner, and action started against the incident or workflow; use call ID/transcript retrieval or a configured callback for the downstream step.

Before You Start

Requirements

Datadog Webhooks integration access and a monitor that supports renotification
A private bridge with a hidden shared token or OAuth authentication
A verified ClawdCall account with its bearer API key stored in the bridge
Production, customer-impact, and owner tags or an equivalent routing map
A durable idempotency store keyed by alert cycle and transition

When It Calls

All filters must pass

alert_transition equals Renotify—not the first transient trigger
alert_priority equals P1 and environment equals production
customer_impact is explicitly true
The monitor remains active and no acknowledgment exists
A fixed on-call owner is resolved from trusted routing data
alert_cycle_key + transition has not already produced a call

Spoken Preview

What the expected recipient hears

“This is a test Datadog escalation. The synthetic checkout monitor is still critical after the persistence window, and customer impact is confirmed. Please say investigating or request fallback.”

Field Map

Build the call from safe, explicit fields.

SourceDestinationTreatment

$ALERT_CYCLE_KEY + $ALERT_TRANSITIONdedupe keyPersist atomically before calling.

$ALERT_TITLE + safe service tagtasksSummarize the affected service; exclude raw logs.

$ALERT_PRIORITY + $TAGS[env]policy gateRequire P1/critical and production.

trusted on-call routing maptargetDo not accept a phone number from arbitrary monitor text.

safe dashboard/runbook labelstasksDescribe where to look; do not speak signed or private URLs.

Copyable Gate

Fail-closed Datadog policy gate

export function shouldEscalate(event, wasAcknowledged, wasProcessed) {
  const customerImpact = String(event.customer_impact).toLowerCase() === "true";
  const ownerKnown = String(event.owner_known).toLowerCase() === "true";

  return Boolean(
    event.alert_transition === "Renotify" &&
    event.alert_priority === "P1" &&
    event.environment === "production" &&
    customerImpact &&
    ownerKnown &&
    !wasAcknowledged(event.alert_cycle_key) &&
    !wasProcessed(event.alert_cycle_key, event.alert_transition)
  );
}

// Recheck the current monitor/incident state before calling.
// Build speech from safe allowlisted fields, never raw logs or signed URLs.

Implementation

Set up the recipe

01
Create a private webhook bridge and store its shared token and the ClawdCall API key in a secret manager.
02
In Datadog Webhooks, create a JSON webhook using the downloadable custom payload. Add a hidden CLAWDCALL_BRIDGE_TOKEN custom variable and send it in custom headers as {"X-Workflow-Token":"$CLAWDCALL_BRIDGE_TOKEN"}, or use a supported OAuth method.
03
Configure the monitor to renotify after the persistence window, then mention @webhook-<name> inside an is_renotify block.
04
Map environment, customer impact, service, and owner from trusted tags or incident fields; never from free-form alert text alone.
05
Run the synthetic payload through the bridge in dry-run mode and verify that Triggered, Recovered, non-production, and unconfirmed-impact variants stay quiet.
06
Explicitly approve one self-call, then replay the same cycle key to prove the bridge prevents a duplicate.

Download the Datadog webhook payload Download the synthetic Renotify event

Safe Test

Synthetic event to one self-call

Download the Datadog payload template and synthetic Renotify fixture.
Keep the bridge in dry-run mode while validating transition, priority, environment, customer impact, owner, and dedupe behavior.
Change Renotify to Triggered and confirm no call request is created.
Restore Renotify, route only to yourself, and explicitly approve one live test that may consume calling minutes.
Replay the same alert_cycle_key and confirm zero additional calls.

Start a supervised self-call

Expected result shape

status: completed
acknowledged: true
owner: on-call self-test recipient
action: investigation started
fallback required: false

Treat the live ClawdCall response and transcript as the source of truth. Do not fabricate a call ID, completion state, or human response.

Callback and Correlation

Write the call acknowledgment back to the Datadog incident path

Keep the monitor event, phone call, and downstream incident update joined by one stable alert-cycle key. The callback should update ownership and action state, while the current Datadog monitor remains the source of truth for recovery.

01
Persist alert cycle key, transition, monitor ID, and returned call ID before waiting for completion.
02
Authenticate the completion callback and recheck the monitor or incident state before applying the spoken response.
03
Write acknowledgment, owner, investigation status, and fallback requirement to the incident workflow using a strict field allowlist.
04
Deduplicate callback retries and suppress every write after the monitor has recovered or the incident has already been acknowledged.

Production Guardrails

Keep routine noise and sensitive data off the phone.

Use Renotify or an equivalent persistent-state check so a transient first alert stays in text channels.
Do not place raw logs, traces, customer records, tokens, or signed dashboard URLs in the call task.
Resolve phone routing from trusted configuration, never from a free-form Datadog message.
Suppress Recovered, Warn, No Data, test, staging, already-acknowledged, and duplicate events by default.
Stop after acknowledgment and use the existing incident escalation policy for fallback.

Troubleshooting

Fail closed, then inspect the gate.

The webhook fires on the first alert instead of after persistence.

Place the webhook mention inside an is_renotify block and configure the renotify interval and occurrences on the monitor.

Logs monitors do not populate alert_status.

Datadog requires ALERT_STATUS to be added manually to the Webhooks integration payload for Logs Monitor alerts.

Datadog retries the bridge unexpectedly.

Return a non-5xx response after safely recording rejected or duplicate events; Datadog retries 5xx responses and missed connections.

Recovered events create calls.

Fail closed unless alert_transition is exactly Renotify and the current-state check still reports active customer impact.

FAQ

Questions about this workflow

Why wait for Datadog Renotify?

It creates a persistence window. The first alert can stay in normal channels while a still-active, customer-impacting incident earns a stronger interruption.

Does this replace PagerDuty or Datadog On-Call?

No. Use it as a contextual agent-to-human handoff when synthesis, a bounded spoken response, and workflow writeback add value beyond existing paging.

Is this an official Datadog integration?

No. It is an independent webhook-and-bridge recipe based on Datadog's documented webhook variables and monitor renotification behavior.

Verification Sources

Vendor behavior and API references

Datadog WebhooksWebhook setup, payload variables, retry behavior, and authentication options.
Datadog monitor notificationsRenotification intervals and the is_renotify escalation block.
ClawdCall Agent APILive API contract for outbound calls and transcript retrieval.

Continue Building

Test, verify, and price the complete phone workflow.

First Call

Verify the full path with one call to yourself.

Once the self-call completes and the workflow returns a real outcome, replace the fixture with a narrowly filtered production event.