---
title: 'Five ways to create a synthetic test in 2026, and when each one is right'
description: 'Authoring a synthetic check got cheap. Reviewing what got generated didn''t. A practical walkthrough of five creation paths, what each one costs to write versus maintain, and a decision table for picking between them.'
date: '2026-07-29'
author: 'Drew Post'
tags: ['synthetic-monitoring', 'monitoring-as-code', 'ai-sre', 'mcp']
canonical_url: 'https://yorkermonitoring.com/blog/five-ways-to-create-a-synthetic-test'
---

![Drafting tools laid over an engineering blueprint.](/blog/five-ways-to-create-a-synthetic-test/00-hero.jpg)

The way a synthetic test gets created has moved more in the last two years than in the ten before it. The first tools recorded you: click through a flow once, capture the DOM interactions, replay them on a schedule. Change one selector or reorder one step and the replay broke, with no vocabulary to explain why. The industry's answer was hand-written scripts, Selenium first and Playwright more recently, built on an engineer writing a deliberate assertion instead of trusting a captured click to keep working forever.

That held for a decade. Now hand-writing is no longer the only way to get a precise script. A model can produce a working Playwright flow from a sentence. A spec you already publish can produce a whole surface of HTTP checks in one call. A coding agent editing a feature can write the monitor for that feature in the same diff. Together with protocol-aware checks for MCP servers, that gives five distinct ways to get from "we should watch this" to a check on a schedule, each with a different cost curve.

![A timeline of how synthetic tests get written: record and replay, hand-written Selenium and code-first Playwright, then three generation paths in roughly the last two years, with session-derived tests and self-healing selectors drawn dashed because neither ships in Yorker yet.](/blog/five-ways-to-create-a-synthetic-test/01-authoring-timeline.svg)

My argument: generation didn't remove the work, it moved it. Writing a check used to be the expensive part. Now writing is close to free, and the cost sits in judgement (deciding what's worth monitoring) and review (reading what got generated closely enough to trust it in production). I build Yorker, so weigh that, but the shift holds whichever tool you use.

![A schematic plot of the five creation paths by cost to author against cost to maintain, showing that four paths are cheap to write and that plain English generation's maintenance cost ranges from low to high depending on how carefully the output was reviewed.](/blog/five-ways-to-create-a-synthetic-test/02-author-vs-maintain.svg)

## 1. Hand-written Playwright: still the most precise

Write the script yourself when the flow has business logic a generic description can't capture. "Add the item with the highest inventory count to cart, apply a coupon only if the cart total exceeds $50, and confirm the discount line appears" is a sentence a model can attempt, but the conditional branching and the exact assertion on the discount line are details you want under your own hands.

The cost is upfront: you write every line, including the parts a generated script gets right by default (waiting for the right selector, handling a cookie banner, scoping a locator so it doesn't match three elements). In return you get a script you fully understand, and the lowest-maintenance path of the five, provided you wrote it well. Nobody has to review your intent later because you were the intent.

In Yorker this is a scripted browser monitor: a Playwright script body (no `import`, no `test()` wrapper, the runner injects `page` and `context`) referenced from `yorker.config.yaml`, with `// @step:` comments marking the filmstrip boundaries. See [Create a Monitor](/docs/guides/create-monitor#browser-monitor-scripted-mode) for the exact shape.

```typescript
// @step: Add highest-inventory item to cart
const item = page.locator("[data-testid=product-card]").first();
await item.getByRole("button", { name: "Add to cart" }).click();

// @step: Apply coupon if eligible
const total = await page.locator("[data-testid=cart-total]").innerText();
if (parseFloat(total.replace("$", "")) > 50) {
  await page.getByPlaceholder("Coupon code").fill("SAVE10");
  await page.getByRole("button", { name: "Apply" }).click();
  await page.waitForSelector("[data-testid=discount-line]");
}
```

Use this path when the flow's correctness depends on a business rule rather than navigation alone.

## 2. Plain English to generated Playwright, refined iteratively

For flows that are mostly navigation ("go here, click that, confirm this appears"), a description is often enough, and Yorker's `/api/checks/generate` endpoint turns one into a Playwright script. The dashboard exposes this as **Describe in plain English**; the API is the same pipeline:

```bash
curl -X POST https://yorkermonitoring.com/api/checks/generate \
  -H "Authorization: Bearer $YORKER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Navigate to https://shop.example.com, add the first product to cart, and verify the cart shows it.",
    "targetUrl": "https://shop.example.com"
  }'
```

The response carries `"mode": "playwright"` and a `script` field with the generated steps. The part people skip is the refinement loop, and it's what makes this path usable. Pass the script back with `previousScript` and a `refinement` instruction instead of starting over:

```bash
curl -X POST https://yorkermonitoring.com/api/checks/generate \
  -H "Authorization: Bearer $YORKER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Navigate to https://shop.example.com, add the first product to cart, and verify the cart shows it.",
    "targetUrl": "https://shop.example.com",
    "previousScript": "// @step: Navigate to homepage\nawait page.goto(\"https://shop.example.com\");\n...",
    "refinement": "Also assert the cart badge count increments to 1, not just that the cart page loads."
  }'
```

That's the API equivalent of iterating in the dashboard's editor. Once you're happy with it, save it as a monitor and fire an ad hoc run with `POST /api/checks/:id/trigger` to confirm it passes before waiting on the schedule.

The cost curve is the opposite of hand-writing: authoring is close to free, and the debt hides in whatever you didn't read. A generated script that navigates correctly and asserts on the wrong thing (checking that a page loaded instead of that the cart actually updated) will pass for months and tell you nothing. Treat the output as a first draft from a competent but unfamiliar colleague: read every assertion before you trust it in production. See [Create a Monitor](/docs/guides/create-monitor#natural-language) for the full flow.

![The generate-then-review loop: a person decides and describes, the generate endpoint drafts a script, a trigger run tests it, a person reviews every assertion and refines with previousScript if needed, then yorker deploy schedules it; the two human steps are where the monitor earns trust.](/blog/five-ways-to-create-a-synthetic-test/03-generate-review-loop.svg)

## 3. Generated from a contract: one HTTP check per OpenAPI operation

If you publish an OpenAPI spec, you already have a machine-readable list of every operation your API supports, and that is enough input for one HTTP check per endpoint without writing any by hand. Yorker's spec-generation pipeline walks the spec, derives a canonical URL per operation from the `servers` block plus the path template, and attaches an `openapi_conformance` assertion that validates the live response shape against the spec on every run:

```bash
curl -X POST https://yorkermonitoring.com/api/checks/generate \
  -H "Authorization: Bearer $YORKER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "spec": { "source": "url", "specUrl": "https://api.example.com/openapi.json" },
    "locations": ["loc_us_east", "loc_eu_central"],
    "frequencySeconds": 300
  }'
```

This is idempotent on `(specId, method, pathTemplate)`: re-run it after you ship two new endpoints and it creates only those two, skipping the rest as `already_exists`. Sync the spec later and the same checks keep validating against the current contract without a re-deploy. Full detail, including the skip-reason vocabulary and the 50-operation confirmation gate, is in [API Specs](/docs/guides/api-specs).

The tradeoff is scope. You get contract coverage across an entire surface in one call. You don't get a user's actual experience: no filmstrip, no click-through, no "does the checkout button visually render." A 200 that matches the schema is not the same claim as "a person can complete this flow." Use it for the API surface you own and pair it with browser checks for the journeys that matter.

## 4. Your coding agent writes the monitor in the same PR as the feature

Mechanically this is the least novel path. `yorker.config.yaml` is plain YAML in your repo, so anything with write access, including a coding agent implementing a feature, can add a monitor block in the same commit as the code it watches. There's no separate monitoring API to learn.

```yaml
# yorker.config.yaml -- added in the same PR as the upsell feature
monitors:
  - name: Checkout Flow (post-purchase upsell)
    type: browser
    script: ./monitors/checkout-upsell.ts
    frequency: 5m
    locations:
      - loc_us_east
      - loc_eu_central
```

The review happens the way it already does for code: `yorker validate` runs in CI on every push touching the config, and `yorker diff --json` posts a plan as a PR comment so a human sees exactly what will change before merge.

```
Yorker deploy plan for "my-app"

  Checks:
    + CREATE  browser  "Checkout Flow (post-purchase upsell)"  (5m, 2 locations)

  Summary: 1 to create, 0 to update, 3 unchanged

  (dry run, no changes applied)
```

Merging to main runs `yorker deploy`, same as any other CI-gated change. See [CI/CD Integration](/docs/guides/ci-cd) for the full workflows. Any text generator can write YAML. What matters is that the monitor and the feature it watches live in the same diff, so "does this PR ship monitoring for what it changed" has an answer a reviewer can see, instead of a follow-up ticket that quietly never happens.

![yorker diff in a terminal, showing a per-field plan against remote state before anything deploys.](/blog/synthetic-monitoring-is-changing/10-cli-diff.png)

## 5. Protocol-aware checks for agent-facing surfaces

MCP servers don't fit the first four paths cleanly, because the thing worth checking is a session rather than a URL and a status code: an `initialize` handshake, a `tools/list` discovery call, and whatever a caller does with the tools the server advertises. Yorker's MCP monitor runs that whole session on a schedule and treats creation as declaring what you expect the session to look like:

```yaml
monitors:
  - name: Docs MCP Server
    type: mcp
    endpoint: https://mcp.example.com/mcp
    frequency: 5m
    locations:
      - loc_us_east
    expectedTools:
      - search_docs
      - fetch_page
    testCalls:
      - toolName: search_docs
        arguments:
          query: "pricing"
        expectedOutputContains: "Plans"
    detectSchemaDrift: true
```

There are no JSON-RPC calls to write. You declare which tools must exist, one representative call and what it should return, and whether to watch for schema drift. Drift detection hashes each tool's input schema and compares it run over run, so a renamed argument gets flagged the moment it ships instead of two days later when an agent starts failing silently. I've written about why an HTTP 200 tells you none of this in [MCP server monitoring](/blog/mcp-server-monitoring). For authoring, the upshot is that you declare the contract once and the protocol session runs itself every cycle.

## The decision table

| Path | Authoring cost | Maintenance cost | Use it when |
|---|---|---|---|
| Hand-written Playwright | Highest, you write every line | Lowest, if written well | Business-rule assertions a description can't capture |
| Plain English, refined | Near zero for a first draft | Hidden in what you didn't review | Mostly-navigation flows, fast iteration with `refinement` |
| OpenAPI spec to HTTP checks | Zero per operation | Low, spec changes flow through on sync | You own the contract and want the whole surface watched |
| Coding agent edits the YAML | Effectively free, a byproduct of the PR | Same as normal code review via `yorker diff` | The monitor should ship with the feature, not after it |
| Protocol-aware (MCP) | Low, declare endpoint and expected tools | Low, but drift alerts need triage | Any MCP server other agents depend on |

## What's coming, and what isn't here yet

Two ideas are on the horizon industry-wide. One is deriving synthetic tests from real user sessions: mining RUM or session-replay data to propose the flows worth watching instead of a human guessing. It's a good idea and several vendors are circling it. The other is self-healing selectors, where a broken locator gets patched automatically instead of failing the check. Neither ships in Yorker today.

Across all five paths, writing the mechanics of a check got cheap enough that it's no longer the bottleneck. What's still on you is deciding which flows are worth watching, and reading closely enough to trust whatever got generated on your behalf. Pick the path that matches the flow, not the path that's newest.

[Start free, no credit card required →](/sign-up)