How to monitor an endpoint that never gives the same answer twice

Drew Post··10 min read
synthetic-monitoringai-infrastructureai-sreopentelemetry

A trail of dice tumbling across a black surface, each one landing differently.

A synthetic check that asserts body_contains: "success" has been the default move for fifteen years, because most endpoints are deterministic: same input, same output, assert on a fixed string, done. Ship an LLM-backed feature (chat, summarise, search, an agent tool) and that assumption breaks. Ask the same question twice and you get two different answers, both correct. Assert on an exact string and the check flaps even when the feature works. Assert on nothing specific and the check stays green while the feature is broken.

A smarter string match won't fix this. What works is giving up on "the response equals X" and replacing it with layers, each one checking something that has to hold in any healthy response and staying quiet about the parts that are allowed to vary.

Five assertion layers stacked from cheapest to most expensive: transport, structure, invariants and golden prompts are deterministic checks, while the LLM judge on top costs a second model call and is itself probabilistic, so the bottom three run every few minutes and the top two run every thirty to sixty minutes.

Layer one: transport and latency

This layer checks two things: that you got a response, and how long it took. It doesn't care what the model said.

# yorker.config.yaml
monitors:
  - name: "chat completion API"
    type: http
    url: https://api.example.com/v1/chat
    method: POST
    headers:
      Content-Type: application/json
      Authorization: "Bearer {{secrets.CHAT_API_TOKEN}}"
    body: '{"prompt": "In one sentence, what year did the French Revolution begin?", "max_tokens": 60}'
    timeoutMs: 45000
    assertions:
      - type: status_code
        value: 200
      - type: response_time
        max: 15000

response_time asserts against total time to completion, not time to first byte (assertions reference). If your endpoint streams over Server-Sent Events or chunked transfer, that distinction matters a lot. Yorker's HTTP checks read the response through to the end before running assertions, so response_time on a streaming endpoint measures the full generation, not the moment the connection opens.

There's no dedicated assertion type for time-to-first-token today. The closest built-in signal is synthetics.ttfb.duration, a metric emitted on every HTTP check run regardless of monitor type, which for a streaming response tracks close to time-to-first-token. You can't gate a check on it in YAML yet, but you can alert on it from your OTel backend, and that's where an interesting failure mode lives: total time stays inside budget while TTFB creeps up over a week, meaning the model queue is backing up before generation even starts.

Two runs of the same streaming check with illustrative timings: on day one the first token arrives after one second and on day seven after five, yet both pass a 15-second response_time assertion because it measures total completion, so only the ttfb metric shows the queue backing up.

Layer two: is the response even shaped right

Before judging content, check the envelope. Structure is deterministic even when the words inside it aren't.

    assertions:
      - type: header_value
        header: content-type
        operator: contains
        value: application/json
      - type: body_json_path
        path: "$.choices[0].message.content"
        operator: exists
      - type: body_json_path
        path: "$.model"
        operator: exists

Not glamorous, but this is where "the provider changed their response shape and every downstream parser silently started returning undefined" gets caught in one check cycle instead of by a user reporting a blank chat bubble.

Layer three: invariants that must always hold

This layer earns its keep, because it covers the gap between "I can't predict the exact words" and "I have no idea if this is broken." A correct response to an open-ended prompt varies. A handful of things still have to be true of every correct response, whatever the wording:

  • Non-empty.
  • Not an apology or refusal sentinel, if you asked something the assistant should be able to answer.
  • No leaked system prompt or instruction text in the user-facing output.
  • Within a sane length range. An answer that's three words or twelve thousand words is a failure mode even when every individual word is fine.
  • Citations present, if your product promises citations.

None of these need a dedicated assertion type. body_matches runs a standard JavaScript regex (no flags) against the full body, and a regex can assert absence and bounds as well as presence:

      # Fails if a refusal or error sentinel appears anywhere in the body
      - type: body_matches
        pattern: "^(?![\\s\\S]*(?:I'm sorry, I can't|As an AI language model))[\\s\\S]*$"

      # Fails unless the whole body (JSON envelope included) is 20 to 4000 characters
      - type: body_matches
        pattern: '^[\s\S]{20,4000}$'

Both reach the regex engine as [\s\S], which matches across newlines where a bare . stops at the first one. The first is double-quoted because of its apostrophes, so YAML needs each backslash doubled.

A leaked-system-prompt check is the same technique pointed the other way: match on instruction fragments you know are in your own system prompt, and fail if they show up in what the user sees.

Layers four and five: golden prompts, then a judge

Layer three catches shape violations. It won't catch a model that answered confidently and wrongly. For that you need something closer to ground truth.

A golden prompt is a fixed, low-ambiguity question whose answer you can check deterministically enough: "what year did the French Revolution begin" should contain "1789" in any correct phrasing.

      - type: body_json_path
        path: "$.choices[0].message.content"
        operator: contains
        value: "1789"

It's cheap and fast, and it catches the loud failures: the model got swapped underneath you, the prompt template broke, a retrieval step stopped returning anything useful.

It won't catch subtler regressions in open-ended output, like tone drift or a summarisation feature that's technically accurate but has started rambling. For that the honest answer is an LLM judge: a second model call that scores the primary response against a rubric and fails the check below a threshold. I haven't built that as a first-class Yorker assertion type, and I don't think it belongs as one. It's a second network call with its own latency, its own cost, and its own non-determinism, judging the non-determinism of the first call. Two caveats before you reach for it:

  • It costs real money on every run. A judge call is a full model invocation. Run it at a fraction of your liveness check's frequency.
  • The judge isn't deterministic either. At layer five you've moved the non-determinism up one level and are betting the aggregate is more stable than the thing it's grading. Watch the judge's pass rate over a window, not a single run, or you'll spend your on-call rotation chasing noise.

In practice this is two monitors at two frequencies: a cheap structural-and-invariant check every one to five minutes, and a golden-prompt-plus-judge check every thirty minutes to an hour, because that one burns tokens twice per run. Yorker's monitor frequency runs from 10 seconds to 24 hours per monitor, so splitting them is ordinary config.

The model API is a dependency, not an implementation detail

Your feature is only as up as the provider behind it. When your chat endpoint goes red, the first question is whether it's your integration or their outage, and if the only thing you watch is your own API, you can't answer that without reading a page of logs.

Point a second, minimal monitor directly at the provider: the same shape of request your feature sends, with none of your product logic in between. When both monitors go red at once, the provider is down. When only yours does, the problem is in your code. It takes five minutes to set up and it's the most useful check in this list.

Your feature check exercises your endpoint, your app logic and the model provider in sequence, while a provider check sends the same request shape straight to the provider, so when both fail it's the provider's outage and when only the feature check fails it's your integration.

Also watch for a provider quietly swapping the model behind a stable endpoint name. A golden prompt sometimes catches this after the fact. Better: if your provider echoes back which model served the request, in the body or a header, assert on it directly with body_json_path or header_value instead of waiting to notice the vibe changed.

Watching the product, not just the API

Most users hit a UI, not your API, and a UI adds its own failure modes: the stream starts but never resolves, a loading spinner never clears, a "stop generating" control gets stuck.

The trap with browser-scripted checks against a streaming UI is asserting too early. Waiting for the first token to render tells you the connection opened. It doesn't tell you the response finished, and asserting against a half-formed answer is worse than asserting against nothing.

// @step: Ask the assistant a question
await page.goto("https://app.example.com/chat");
await page.fill('[data-testid="chat-input"]', "What year did the French Revolution begin?");
await page.click('[data-testid="send"]');

// @step: Wait for the stream to actually finish
// Most chat UIs show a "stop generating" control while a response is
// still streaming. Wait for it to appear, then wait for it to
// disappear, instead of racing the first token.
await page.waitForSelector('[data-testid="stop-generating"]', { state: "visible", timeout: 5000 });
await page.waitForSelector('[data-testid="stop-generating"]', { state: "hidden", timeout: 45000 });

// @step: Assert on the completed answer
const answer = (await page.textContent('[data-testid="last-message"]')) ?? "";
if (answer.trim().length === 0) {
  throw new Error("Assistant returned an empty response");
}
if (/I'm sorry, I can't|As an AI language model/i.test(answer)) {
  throw new Error(`Response looks like a refusal: ${answer.slice(0, 120)}`);
}
if (!answer.includes("1789")) {
  throw new Error("Golden prompt failed: expected 1789 in the response");
}

That's a scripted browser monitor: plain Playwright, page and context already in scope, no test runner wrapper, // @step: comments marking the filmstrip. A thrown error fails the check the same way a failed assertion does. On a failing run the filmstrip shows the actual half-rendered UI, which for a streaming interface is often the fastest way to tell "the model was slow" from "the frontend never handled the final chunk."

If the caller on the other end isn't a person in a browser but an agent calling you over MCP, the checks change shape again. Handshake failures and tools/list drift matter more than screenshots, and an MCP monitor's testCalls gives you one expectedOutputContains substring per tool call, not the full layered set above, so for anything past layers one and two you'll likely still want a companion HTTP check against the same underlying endpoint. I wrote about the MCP-specific version of this problem separately, and the MCP monitoring guide has the schema-drift mechanics.

None of this makes the endpoint deterministic, and it doesn't need to. You need a check that fails when something is actually wrong and stays quiet through the ordinary variance of a model doing its job, and layering the assertions gets you there.

Start free, no credit card required →

Back to Blog