---
title: 'Synthetic monitoring matters more when agents write the code'
description: 'When the model that writes your code also writes the test for it, the test stops being an independent check. Outside-in synthetic monitoring becomes more necessary as coding agents take on more of the diff, not less.'
date: '2026-07-08'
author: 'Drew Post'
tags: ['synthetic-monitoring', 'ai-infrastructure', 'observability', 'perspective']
canonical_url: 'https://yorkermonitoring.com/blog/synthetic-monitoring-when-agents-write-the-code'
---

![Hands typing on a laptop with source code on screen.](/blog/synthetic-monitoring-when-agents-write-the-code/00-hero.jpg)

I use a coding agent for most of the diffs that land in Yorker now. It writes the implementation and the unit test, and both go green in the same session, often without me reading either line by line. I don't think that's reckless. I think it's where most of us are heading, and it quietly breaks an assumption the testing pyramid was built on.

The assumption was that writing the code and deciding whether it's correct happened in different passes of attention, if not by different people. You write the function, come back later with fresher eyes, and write the test against your memory of the requirement rather than the code in front of you. That gap caught a lot of bugs. Agents close it. The same model, in the same context window, with the same understanding (or misunderstanding) of the requirement, writes both sides. If it's wrong about what "correct" means, it's wrong in the same direction in both places, and the test passes.

I'm not arguing against coding agents. I'm arguing that the thing sitting outside the code, which never read a line of it and doesn't care how it was written, now does more useful work than it used to.

![A requirement says guests can check out without an account; one agent session misreads it as login required and writes both the code and the test from that reading, so CI passes, while a synthetic check running the guest checkout journey against the deployed system fails.](/blog/synthetic-monitoring-when-agents-write-the-code/01-correlated-blind-spots.svg)

## Who checks the checker

Unit and integration tests check the code. Someone with an understanding of what the code should do encodes that understanding as assertions. When one person writes the code and another (or the same person, later, with different context) writes the test, you get two independent passes at the requirement. Both can still be wrong, but they're wrong independently, which is why disagreement between them is informative.

When a coding agent writes both in one session, that independence goes away. The test checks the code against the model's own representation of the requirement, produced by the same pass that produced the code. If the model misreads an edge case, an auth scope or a rounding rule, it typically misreads it the same way in the assertion. You get two artifacts that agree with each other and are both wrong, and CI reports that as success.

This is the old "who checks the checker" problem with a new checker. It used to be a question about test coverage and review rigour. Now it's a question about whether the thing verifying your code shares its blind spots by construction, and most of the time it does.

None of this is specific to one model or one agent harness. It's structural. Any system that produces the implementation and the verification from the same reasoning, in the same pass, will have correlated failure modes between the two. A better model doesn't fix that. A second oracle that never saw the reasoning does.

## Verifying intent is not verifying outcome

A unit test verifies intent: does this function do what I, the author, believe it should do? That's a legitimate question and you should keep asking it. It is a different question from whether the deployed system behaves that way for a real client, over a real network, right now.

Those two questions used to converge, because the gap between "what I wrote a test for" and "what's actually running" was usually small. Coding agents widen that gap. They're good at making code internally consistent with its own tests and blind to everything the tests don't model: the auth proxy in front of the service, a CDN cache-control header that a config change quietly flipped, a third-party script that now loads in a different order, a feature flag default that shipped correctly to staging and wrong to production. A test that mocks a dependency doesn't know the dependency changed, and the agent that generated the mock can't notice either, because it never talked to the real thing.

As more of the intent-verification layer gets generated by the same process as the implementation, outcome verification is left doing the job neither of them can do for itself.

## Green CI is necessary, not sufficient

I want to be precise, because this is easy to overstate. Green CI still means something. The code compiles, the logic holds under the scenarios the author (or the agent) thought to write down, and nothing regressed against your existing fixtures. That's real signal and I'm not telling anyone to stop writing tests.

What's changed is how much weight that signal can carry on its own. A green pipeline used to correlate fairly well with "a human thought hard about this and poked at the edges." Now it increasingly means "a model generated a self-consistent pair of artifacts and neither contradicted the other."

Treat green CI as confirmation that the code matches its own test suite, not that the deployed system does the right thing. Those used to be close enough to conflate. They stop being close when the volume of code changing per week goes up and the amount a human reads line by line doesn't.

![Schematic chart, not measured data: code changed per week rises as coding agents are adopted while human review capacity stays flat, and the widening area between them is change that ships without independent verification.](/blog/synthetic-monitoring-when-agents-write-the-code/03-unverified-change-gap.svg)

## Where AI-introduced regressions actually surface

The failures that slip through a same-session code-and-test pair aren't random. They cluster around the things a unit test doesn't reach:

- **Config.** An environment variable, a feature flag default or a timeout value that's correct in the diff and wrong in the environment it deploys to. Nothing in the test suite touches the deployed config.
- **Third parties.** A tag manager script, a payment SDK, an analytics beacon the agent didn't write and can't see. It can change behaviour or latency underneath a perfectly correct diff.
- **Auth.** Token scopes, session cookies, SSO redirects and CORS headers are all enforced by infrastructure the unit test mocks away, and they break silently when a proxy or gateway config shifts.
- **Caching and CDN.** Cache-control headers, edge rules, stale-while-revalidate behaviour. The code is right. What the CDN serves a real browser is a separate question the test never asked.

These are properties of the deployed system interacting with the real world, not properties of a function you can test in isolation. An outside-in check can see them and an inside-out test structurally can't. The outside-in check sends a real request to the real deployed thing on a schedule, and it has no access to the model's reasoning to inherit blind spots from.

![A matrix of unit tests, CI integration tests, code review and synthetic checks against six failure classes, showing that the layers inside the repo are mostly blind to misread requirements, config, third-party, auth and CDN failures that synthetic checks against the deployed system can see; illustrative judgement, not measured data.](/blog/synthetic-monitoring-when-agents-write-the-code/02-what-each-layer-sees.svg)

## What good synthetic coverage looks like when an agent ships the diff

I don't think the answer is "monitor everything." Aim coverage at the surfaces a same-session code-and-test pair is structurally weak on.

### Critical user journeys, end to end, in a real browser

Login, search, checkout: whichever two or three flows the business depends on. Unit tests for those flows aren't bad, but the journey only becomes real once it's assembled through the actual auth layer, CDN and third-party scripts, in the order a browser loads them.

### API contract checks that don't share a codebase with the implementation

If your OpenAPI spec is the contract, assert the live response against the spec itself, not against a test written by the same process that wrote the handler. A same-session pair won't catch schema drift, because the test was generated to match the handler, not the contract.

### The agent-facing surface itself

If you expose an MCP server, or any API other agents call, that surface deserves the same outside-in scrutiny as your human-facing one. A tool's input schema can drift, a tool can silently disappear from the catalog, or the handshake can start failing, while every unit test for the underlying function stays green. None of those tests exercise the protocol layer a calling agent depends on.

All three are checked from outside the process that wrote the code, on a schedule, against the deployed system rather than the source tree.

## What synthetics don't fix

Overselling this would be its own kind of dishonesty, so here are the limits. Synthetic monitoring doesn't replace code review or unit and integration tests. It catches a different category of failure, later, after deploy, from the outside. It won't tell you a function has an off-by-one error in a branch nobody's hit yet, and it's no substitute for a human (or a second, differently prompted agent) reading the diff before it merges. It sits under production as a floor; the work before merge still has to happen.

It's also exposed to the same underlying problem if you let an agent write your synthetic checks with no outside constraint. A check that asserts whatever the agent believes the page should say is intent verification wearing a different hat. The value only holds if pass or fail is decided by something the agent writing your application code doesn't control: a real request and response, asserted against a contract or a journey, run somewhere the agent's session can't reach in and edit the result.

## Where Yorker fits

I build Yorker, so weigh everything above accordingly. This is the gap I built it to sit in.

The config is plain YAML, and your coding agent can write and edit it like everything else in your repo. That doesn't contradict anything I've argued. Editing `yorker.config.yaml` and getting a check to pass are two different things done by two different systems. The agent can propose a monitor. It cannot fake the result. The check runs on Yorker's own runners, from real hosted locations, against your actual deployed URL, entirely outside the agent's session and context window.

A config covering the three coverage areas above looks like this:

```yaml
# yorker.config.yaml
project: "shop"

monitors:
  - name: Checkout flow
    type: browser
    script: ./monitors/checkout.ts
    frequency: 5m
    locations: [loc_us_east, loc_eu_west]

  - name: Payments API contract
    type: http
    url: https://api.example.com/v1/payments
    assertions:
      - type: openapi_conformance
        specId: spec_payments_v2

  - name: Docs MCP server
    type: mcp
    endpoint: https://mcp.example.com/mcp
    detectSchemaDrift: true
    expectedTools:
      - search_docs
      - get_page
```

`yorker diff` shows exactly what would change before anything deploys, and `yorker deploy` applies it: the same review discipline you'd want for any infrastructure change. See the [Monitoring as Code guide](/docs/guides/monitoring-as-code) for the full config shape, the [assertions reference](/docs/reference/assertions) for what `openapi_conformance` and the rest actually check, and the [MCP monitoring guide](/docs/guides/monitor-mcp-servers) for the protocol-level checks on the agent-facing surface, which I wrote about in more depth [here](/blog/mcp-server-monitoring).

Results come back through Yorker's own infrastructure, get scored against baselines, and go out as standard OTLP into whatever backend you already run: ClickStack, Grafana, Honeycomb or otherwise. That gives you a second, independent read on the system from something that never saw your code.

[Start free, no credit card required →](/sign-up)