Ship the monitor with the pull request

A feature ships. Three weeks later it breaks in production for two days before anyone notices, because nothing was watching it. The postmortem has an action item: "add monitoring for X." It gets assigned to whoever's on call that sprint, not the engineer who built the feature. They open the monitoring dashboard, half-remember what the flow does, click through a form, and ship a check that roughly approximates the thing it's meant to protect.

That check was created weeks after the code, by someone without full context, through a UI that leaves no trace in git. It works, roughly. Nobody reviewed it the way they'd review the code it's watching.

That doesn't have to be the default. If your monitoring config is plain text in your repo, there's no structural reason the monitor can't exist before the incident. It can go in the same pull request as the feature, written by the feature's author and approved by the feature's reviewer. Coding agents make this close to free: the session that writes the endpoint can also add the check.

Why post-incident monitoring always lags

Incentives and information both point the wrong way.

On incentives: nobody's sprint has "write a monitor for a feature that hasn't broken yet" as a task, because nothing is visibly wrong. Monitoring competes with shipping, and shipping wins until there's a fire. When the fire comes, the monitor becomes urgent for the on-call engineer putting it out, not for the person who best understands the feature.

On information: whoever wrote the invite flow knows exactly which step is fragile, which API call actually matters, and what "working" looks like on the page. The person patching coverage together after an incident is reconstructing that from logs and guesswork. Their monitor tends to check the happy path they can see, not the failure mode that caused the page.

Put those together and you get monitors that arrive late, check an approximation of the flow, and sit disconnected from the change that created the need for them. Better tooling alone doesn't fix that. The workflow has to change so that writing the monitor is as cheap and as normal as writing the test.

Put it in the definition of done

Most teams already have a line in their PR template for "did you add tests." Very few have one for "how will we know this works in production." The ask is small: one question, answered in a sentence, next to the description of what changed.

## What journey does this change?

<!-- e.g. "Adds the team invite flow: settings > invite, email sent, accept link works" -->

## How will we know it works in prod?

<!-- Existing monitor updated / new monitor added in yorker.config.yaml,
     or: no user-facing flow changed, nothing to add -->

The second question does real work. It makes "no user-facing flow changed" a stated decision instead of a silent gap, and it makes "yes, but I didn't get to it" show up in review rather than in a postmortem. Teams already expect a changed flow to ship with end-to-end test coverage; this applies the same expectation to the period after the flow goes live. I wrote more about that overlap in end-to-end tests and synthetic monitors converging.

What the agent makes free

A policy you enforce by nagging tends to decay. This one costs almost nothing to follow, because the coding agent that wrote the feature can write the check in the same session.

Say the change is a new "invite teammate" flow: a modal on the settings page, an email field, a POST /api/workspace/invites route that sends the invite. The feature diff is roughly:

+ apps/web/src/app/api/workspace/invites/route.ts    (invite handler, sends email)
+ apps/web/src/components/workspace/invite-modal.tsx  (email field, submit, confirmation toast)
~ apps/web/src/app/settings/team/page.tsx              (renders the "Invite" button)

In the same session, ask the agent to add coverage for it. yorker.config.yaml is plain YAML with a documented schema, so the agent doesn't need to guess at a proprietary format. It reads the configuration reference or the existing checks already in the file and writes something that fits:

// monitors/invite-teammate.ts
// @step: Open workspace settings
await page.goto("https://app.example.com/settings/team");
await page.waitForSelector('[data-testid="invite-button"]');

// @step: Open invite modal
await page.click('[data-testid="invite-button"]');
await page.waitForSelector('input[name="email"]');

// @step: Submit invite
await page.fill('input[name="email"]', "synthetic-probe@example.com");
await page.click('button[type="submit"]');

// @step: Confirm invite sent
await page.waitForSelector('text=Invite sent');

# yorker.config.yaml
monitors:
  - name: "Invite teammate"
    type: browser
    script: "./monitors/invite-teammate.ts"
    frequency: "10m"
    locations:
      - loc_us_east
      - loc_eu_west
    labels:
      - service:workspace
      - flow:invite
    alerts:
      - name: invite-flow-down
        conditions:
          - type: consecutive_failures
            count: 2
        channels:
          - "@ops-slack"

Each // @step: comment becomes a labelled frame in the failure screenshot filmstrip, so when this check goes red, the question is "which of these four steps stopped working," not "something about invites is broken, go read logs." The yorker.config.yaml path is one of several ways to create a check in Yorker (the others are in five ways to create a synthetic test), and it's the one that fits a coding agent's existing loop of editing a file, running validation and proposing a diff.

Run yorker validate to catch schema errors before anyone reviews it, then yorker diff to see the plan:

Yorker deploy plan for "acme-web"

  Checks:
    + CREATE  browser  "Invite teammate"  (600s, 2 locations)

  Alerts:
    + CREATE  "Invite teammate:alert:0"  (consecutive_failures)

  Summary: 2 to create, 0 to update, 0 to delete

Both files go into the same PR as the feature diff, with no dashboard step and no follow-up ticket.

Reviewing a generated monitor

An agent-written check needs the same scepticism as agent-written application code. The review is quick because the surface area is small. What I actually check:

• Does the assertion match the real UI, not a guess at it? A generated selector like text=Invite sent only works if that string is really what the confirmation shows. If the agent hasn't looked at the actual component, ask it to, or check yourself.
• Are waits tied to real state, not a fixed delay? page.waitForTimeout(3000) instead of page.waitForSelector(...) is a check that's fast to write and flaky to run. It's the same code smell as a sleep() in a test.
• Is anything hardcoded that should be a secret? Credentials go through {{secrets.NAME}} interpolation, resolved from the CI environment at deploy time, never written into the YAML.
• What does it leave behind in production? The invite check above sends a real invite every ten minutes. That's fine if synthetic-probe@example.com is a sink address and the workspace is a dedicated test tenant, and a mess if it isn't. Agents rarely think about cleanup unless you ask.
• Is the frequency and location list proportional to the flow's importance? A checkout flow probably wants more locations and a tighter interval than an internal admin page. The agent doesn't know your traffic patterns; you do.
• Does this test the flow the PR actually touched? It's easy for a generated check to cover the surface described in the prompt rather than the specific branch the code takes. Read the assertions against the diff, not against your memory of the feature.

None of this takes longer than reviewing the test file in the same PR.

Wiring the diff into CI

That review happens in the pull request, before merge, because yorker validate and yorker diff run in CI and post the result as a PR comment. Here's the trimmed shape of the workflow (the full file, including the PR-comment script and fork-PR handling, is in the CI/CD integration guide):

# .github/workflows/yorker.yml (trimmed)
on:
  push:
    paths: ["yorker.config.yaml", "monitors/**"]
  pull_request:
    paths: ["yorker.config.yaml", "monitors/**"]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g @yorker/cli
      - run: yorker validate

  diff:
    if: github.event_name == 'pull_request' &&
        github.event.pull_request.head.repo.full_name == github.repository
    needs: validate
    runs-on: ubuntu-latest
    permissions: { pull-requests: write, issues: write }
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g @yorker/cli
      - run: yorker diff --json   # parsed and posted as a PR comment

  deploy:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g @yorker/cli
      - run: yorker deploy

On the pull request, that produces a comment like:

### Yorker Diff

| Action   | Type  | Name                     |
|----------|-------|--------------------------|
| + create | check | Invite teammate         |
| + create | alert | Invite teammate:alert:0 |

2 change(s) will be applied on merge.

The comment updates in place on every push, so it stays accurate through review rounds instead of accumulating stale copies. On merge to main, the deploy job runs yorker deploy and the check goes live. The reviewer approving the feature diff sees the monitor diff on the same screen, in the same review, so verification in prod stops being a separate workstream from the change that needed verifying.

Keeping the UI from drifting back out

This quietly breaks when someone edits the check in the web UI later, say tweaking a threshold at 2am during an incident, and never brings that change back into the YAML. Yorker's CLI tracks this: the next yorker deploy compares the local config hash and the remote updatedAt timestamp, and if the remote side moved without a matching local change, it aborts with a drift report instead of silently overwriting the fix someone made under pressure.

From there you choose: yorker deploy --force if the YAML should win, yorker deploy --accept-remote if the dashboard edit should stand, or yorker pull to re-export the current remote state (including that UI edit) back into your config file so the repo catches up. The UI isn't forbidden. Drift just gets surfaced instead of silently accepted, and the repo stays the thing you trust.

Tell the agent to do this every time

The last piece is making this default behaviour, so nobody has to remember to ask. A short instruction in CLAUDE.md or AGENTS.md does it:

## Monitoring

When a change adds or materially changes a user-facing flow (a new page,
a new step in an existing flow, a changed contract on an API route the
frontend calls), check `yorker.config.yaml` for a monitor that covers it.

- If one exists and the flow changed, update its script or assertions
  to match.
- If none exists, add one: a `browser` monitor with `@step` markers for
  a multi-step flow, or an `http` monitor with `status_code` and
  `response_time` assertions for a single endpoint. See
  /docs/reference/assertions for the full list.
- Run `yorker validate` before finishing. Don't run `yorker deploy`
  locally; CI deploys on merge to main.

The agent already has the context of what it just built. Pointing it at the config file it should also touch costs one paragraph, written once, in a file it reads every session anyway.

I build Yorker, so weigh that accordingly: none of this requires our product specifically, only a monitoring config that's plain text, versioned and diffable in the same place your application code lives. What it does require is treating "how will we know this works in prod" as a real question in code review, rather than one answered three weeks later in a postmortem.

Start free, no credit card required →