---
title: 'Synthetic checks as the deploy verification gate'
description: 'More deploys are started and rolled back by automation than by people watching a dashboard. A post-deploy gate needs a structured answer within minutes, and synthetic checks running against the live deployment are the natural source for it.'
date: '2026-09-15'
author: 'Drew Post'
tags: ['synthetic-monitoring', 'opentelemetry', 'observability', 'ai-infrastructure']
canonical_url: 'https://yorkermonitoring.com/blog/synthetic-checks-as-the-deploy-verification-gate'
---

![A rocket lifting off from its launch pad.](/blog/synthetic-checks-as-the-deploy-verification-gate/00-hero.jpg)

A deploy used to end with someone tailing logs for ten minutes and glancing at a dashboard. That assumed a human triggered the deploy and a human would notice if it went wrong. Now deploys get kicked off by a merge queue, a bot auto-merging a dependency bump, or an agent shipping its own change, a shift I've written about [before](/blog/synthetic-monitoring-when-agents-write-the-code). Someone still has to answer "did that deploy work", but often nobody is watching when the question comes up, and whatever is asking wants something it can parse.

Synthetic checks fit that job. Post-deploy smoke tests are as old as deploys, so novelty isn't the argument. What matters is that they produce structured results from an unattended run against the thing that's live, which is what an automated gate consumes, as long as you set them up to be read by a script and not just watched. I build synthetic monitoring for a living, so weigh the argument with that in mind. It holds up without a product name, and I'll say where naming one becomes unavoidable.

## What a deploy gate actually needs

Unit and integration tests run against the code before it ships. They tell you the change is internally correct. They don't tell you whether the deployed artifact works in the account currently serving traffic. A gate needs three things a test suite doesn't give you: it has to exercise the thing you just deployed, not a proxy for it; it has to be independent of the change, so a broken deploy can't also disable its own verification; and it has to answer in a shape a script can act on.

A real user journey hitting the live URL from outside the deploy's blast radius and returning a structured pass or fail is the job description of a synthetic check. Canary analysis on live traffic is the other obvious candidate, and it's powerful, but it needs traffic. A service that redeploys at 3am to near-zero load won't accumulate enough canary signal inside a reasonable verification window. A synthetic check generates its own traffic on demand.

## Smoke gate, and the monitor that keeps running after it

Two things get conflated here. Post-deploy smoke is a check triggered on demand right after a deploy, evaluated against a fixed assertion, and expected to answer fast enough to block or unblock a pipeline. Continuous monitoring is the same check on its normal schedule, running regardless of what the gate decided and building a longer memory of what normal looks like.

You need both. A CI job waiting on a gate has a bounded budget: someone pays for that runner and the next deploy is queued behind it, so the gate can only catch regressions severe enough to trip a fixed threshold within a few minutes. Slower regressions, the ones that surface once real traffic exercises the new code path or an hour has passed, belong to the check running on its own schedule.

## How many locations before you trust green

One green location doesn't prove the deploy is healthy everywhere, and one failing location out of several doesn't prove it's broken. Regional issues (a flaky resolver at one point of presence, a transient TCP reset) happen independently of what you just shipped, and a gate that rolls back on the first failing location will roll back for nothing more often than it catches a real regression.

![Two grids of check runs after a deploy, one row per location: a single failed run in one location should let the gate pass, while failures spreading across all three locations inside the verification window should fail it. Illustrative, not measured data.](/blog/synthetic-checks-as-the-deploy-verification-gate/02-verification-window.svg)

This is the problem Yorker's `multi_location_failure` [alert condition](/docs/reference/api#alert-conditions) exists to avoid for continuous alerting: it requires failures from `minLocations` within a `windowSeconds`, because a single-location failure and a systemic one look identical from inside that one location's result. A CI gate consuming ad hoc trigger results needs the same policy, and today it has to implement it itself: how many of the locations you triggered have to report failure, inside what window, before you call the deploy bad. There's no built-in default for a gate. Set the threshold too low and the gate is flaky; set it too high and it stops catching anything.

## Baseline, not threshold

Fixed-threshold [assertions](/docs/reference/assertions) (`response_time` under a max, `status_code` equals 200, `body_contains` a string) are fast, binary, and right for a go/no-go decision the moment after a deploy. They can't know what's normal. A `response_time` assertion set to 2000ms either tolerates a real 40% regression that lands under the ceiling, or flags a request that's slow for boring reasons (higher latency at 3am from a particular region) that have nothing to do with the deploy.

That's the case for scoring against a baseline instead of a fixed number. Yorker's `baseline_anomaly` alert condition does this continuously: it compares each of the last `consecutiveCount` successful runs against a 14-day rolling baseline bucketed by hour-of-day, day-of-week, and location, and fires when the deviation crosses a sigma threshold in the configured direction. "Slow" gets judged against what that check normally does on that day, at that hour, from that location, not a number someone picked once and forgot to revisit.

![A response-time line steps up about 40% after a deploy: it clearly leaves the check's baseline band but never gets near a fixed 2000ms assertion, so only the baseline notices. Illustrative, not measured data.](/blog/synthetic-checks-as-the-deploy-verification-gate/03-baseline-vs-threshold.svg)

Two details matter if you're building a gate around this:

- That baseline isn't a snapshot frozen at deploy time. It's the check's rolling history. It catches a deploy regression because a regression is abnormal for its slot, and Yorker does not take a "before" reading the moment you push. If you want a baseline pinned to "the state before this deploy," that's a fixed threshold you choose deliberately, not what `baseline_anomaly` gives you.
- The anomaly score is an OTLP log-event attribute on `synthetics.check.completed`, delivered to your own OTel backend once you've configured one. It is not present in the `GET /api/checks/:id/results` response a CI gate would be polling; that endpoint returns `status`, `responseTimeMs`, `httpStatusCode`, and a `timing` breakdown, not a sigma deviation. If you want the gate itself to reason about baseline deviation rather than a fixed assertion, today that query has to go against your OTel backend from inside the pipeline, not the REST API. Check this before you design a gate around a field that isn't there.

## Tying the failure back to the version that shipped

A gate that reports "checkout flow failed" is useful. A gate that reports "checkout flow failed, and the failing request shares a trace ID with three 503s in `payments-api` v2.18.0" hands someone a diagnosis before they've opened a laptop. Every HTTP, browser, and MCP check Yorker runs injects a W3C `traceparent` header into its requests; if your backend continues that context, which most OTel-instrumented services do by default, the synthetic run and the spans it triggers land in one trace. Record the deploy alongside it (`POST /api/events/deployments` takes `service`, `version`, and `commit_sha`, turns it into a `synthetics.deployment.created` event, and writes a row the dashboard lines up against check-result anomaly windows) and a failing gate check points at a trace, and the trace points at a version.

![A HyperDX trace view of one synthetic check run. The traceparent Yorker injected carries through every downstream span, so the check and the backend services it touched sit in one trace.](/blog/synthetic-monitoring-is-changing/07-clickstack-trace-with-traceparent.png)

This is the same argument I made in [the missing input to your AI-SRE tool](/blog/the-missing-input-to-your-ai-sre-tool), applied to a deploy gate instead of an incident. A status dot and a pile of screenshots suit a person triaging at their own pace. A trace ID, a check ID, and a structured status field suit whatever consumes the gate's decision, whether that's an `if` block in a workflow file or an agent deciding whether to roll back its own change.

## Wiring it into CI

There's no single command that does this end to end yet. What exists is `POST /api/checks/:id/trigger`, which forces an immediate run across a check's assigned locations, and `GET /api/checks/:id/results`, which reads results back, both documented in the [REST API reference](/docs/reference/api). The gate itself (the polling loop and the pass/fail policy) is something you write.

![The deploy gate as a pipeline: record a deploy marker, trigger the critical checks, poll their results until every location reports, then exit 0 or 1. The three REST endpoints are built into Yorker; the loop and the policy around them are DIY glue.](/blog/synthetic-checks-as-the-deploy-verification-gate/01-deploy-gate-pipeline.svg)

Here's a version of it as a step after a deploy job:

```bash
#!/usr/bin/env bash
set -euo pipefail

API="https://yorkermonitoring.com"
AUTH="Authorization: Bearer ${YORKER_API_KEY}"

# Check IDs for the journeys that matter enough to block a deploy on.
CRITICAL_CHECKS=("chk_checkout_flow" "chk_login" "chk_orders_api")

MAX_FAILED_LOCATIONS=1   # tolerate one regional blip, not more
POLL_TIMEOUT_S=180
POLL_INTERVAL_S=10

# Record the deploy so it's on the record for later correlation,
# independent of the pass/fail decision below.
curl -sf -X POST "$API/api/events/deployments" \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"service\":\"checkout-api\",\"version\":\"${GITHUB_SHA}\",
       \"environment\":\"production\",\"commit_sha\":\"${GITHUB_SHA}\",
       \"source\":\"github-actions\"}" > /dev/null

overall_exit=0

for check_id in "${CRITICAL_CHECKS[@]}"; do
  triggered_at=$(date -u +"%Y-%m-%dT%H:%M:%S.000Z")

  expected_locations=$(curl -sf -X POST "$API/api/checks/${check_id}/trigger" \
    -H "$AUTH" | jq -r '.locations')

  elapsed=0
  seen_locations=0
  failed_locations=0

  while [ "$elapsed" -lt "$POLL_TIMEOUT_S" ]; do
    sleep "$POLL_INTERVAL_S"
    elapsed=$((elapsed + POLL_INTERVAL_S))

    results=$(curl -sf "$API/api/checks/${check_id}/results?limit=20" \
      -H "$AUTH" | jq --arg since "$triggered_at" \
      '[.results[] | select(.startedAt >= $since)]')

    seen_locations=$(echo "$results" | jq '[.[].locationId] | unique | length')
    failed_locations=$(echo "$results" | \
      jq '[.[] | select(.status != "success")] | length')

    [ "$seen_locations" -ge "$expected_locations" ] && break
  done

  if [ "$seen_locations" -lt "$expected_locations" ]; then
    echo "::error::${check_id} only reported from ${seen_locations}/${expected_locations} locations in ${POLL_TIMEOUT_S}s"
    overall_exit=1
  elif [ "$failed_locations" -gt "$MAX_FAILED_LOCATIONS" ]; then
    echo "::error::${check_id} failed from ${failed_locations}/${expected_locations} locations"
    overall_exit=1
  fi
done

exit $overall_exit
```

A few choices in that script are deliberate. `expected_locations` comes straight off the trigger response's `locations` field, so the loop knows how many distinct `locationId` values to wait for instead of guessing. The failure policy is "more than `MAX_FAILED_LOCATIONS` failed" rather than "any failure fails the build", for the same reason `multi_location_failure` exists. And there's a real gap to know about: the trigger response doesn't return a `runId`, so the loop filters results by `startedAt` after the trigger call rather than by run. That's fine for one trigger per check per deploy; if you're triggering the same check back to back inside a pipeline, you'll need to track more carefully than this does.

If you'd rather not hand-roll the curl and jq, the same JSON envelope and exit-code conventions already back `yorker validate`, `diff`, and `deploy` for config changes, documented in [CI/CD Integration](/docs/guides/ci-cd). There's no equivalent command for the deploy-verification loop above: the REST API is built, and the gate on top of it is DIY. A first-class command to close that gap is on the list. It hasn't shipped.

One adjacent point: config that lives with the code it's gating tends to get maintained. I made that case in [shipping the monitor with the pull request](/blog/ship-the-monitor-with-the-pull-request), and it applies directly here. If nobody remembers configuring the check IDs a gate script references, nobody will trust it when it finally fires.

[Start free, no credit card required →](/sign-up)