Circuit breakers that hold work instead of dropping it

Someone uploads a scanned contract. Reading it needs OCR, and the OCR provider is refusing our key. There are three things a service can do here, and two of them are bad. It can fail the request, which hands the user an error and loses their document to solve a problem that is ours. It can accept and then silently drop, which is worse. Or it can accept, store the bytes, and hold the work until the capability comes back.

The third is what PersistMemory does, and it takes two pieces of state to do it honestly: a record of which capabilities are deliberately stopped, and a queue that can hold work for days.

A flag that only a decision can change

Two capabilities can be stopped, OCR and embedding, and each is a row carrying whether it is enabled, who changed it and why. A flag goes off only by a deliberate act: an operator turning it off, or the system tripping it after a failure that is clearly permanent, such as a rejected key or a refused model. A transient timeout is left to the ordinary retry policy, which is the thing retries are for.

Nothing turns a flag back on automatically. That is the part people argue with, so it is worth stating plainly: a breaker that resets itself hides the outage it exists to surface, and the next failure then looks like the first one rather than the eleventh. Tripping is also one way and only fires while the flag is on, so the second failure cannot overwrite the reason the first gave. The first reason explains the outage. The tenth identical timeout tells an operator nothing they did not already have.

The queue is in Postgres, not Redis

We run Redis, and this queue is not in it. Redis holds work that is about to happen. This holds work waiting on a human decision, a model coming back or a key being replaced, and that can be days. A queue that lives in memory or expires cannot carry that, and the failure mode when it cannot is that material a user handed us quietly stops existing.

CREATE TABLE IF NOT EXISTS "retry_queue" (
  "id"          text PRIMARY KEY,
  "user_id"     text NOT NULL REFERENCES "users"("id") ON DELETE CASCADE,
  "kind"        text NOT NULL,
  "status"      text DEFAULT 'waiting' NOT NULL,
  -- The Spaces and the conversation ride along in here. A capture held for
  -- two days has to replay into the place the user put it.
  "payload"     jsonb NOT NULL,
  "notice"      text,
  "attempts"    integer DEFAULT 0 NOT NULL,
  "last_error"  text,
  "claimed_at"  timestamp with time zone,
  "claimed_by"  text,
  "created_at"  timestamp with time zone DEFAULT now() NOT NULL,
  "finished_at" timestamp with time zone
);

The payload carrying the Spaces and the conversation is not incidental. A capture that waits two days and then replays has to land in the same Space and against the same conversation it was made in. If it lands loose, the user comes back to find their material in the wrong place, which is a worse outcome than the delay we were trying to absorb. The bytes go to blob storage before the row is written, so the queued row always points at something real.

Claiming, in one statement

Several workers drain the same table, so a row has to be claimed and marked in a single statement. Selecting a batch and then updating it leaves a window in which two workers both saw the same row as waiting.

UPDATE retry_queue
SET status = 'running',
    claimed_at = $1,
    claimed_by = $2,
    -- Counted at CLAIM, not at completion. A worker that dies mid-run has
    -- still tried, and a count incremented on success never notices the run
    -- that never came back.
    attempts = attempts + 1
WHERE id IN (
  SELECT id FROM retry_queue
  WHERE kind = $3
    AND attempts < $4
    AND (status = 'waiting' OR (status = 'running' AND claimed_at < $5))
  ORDER BY created_at ASC
  LIMIT $6
  FOR UPDATE SKIP LOCKED
)
RETURNING *;

SKIP LOCKED is what lets several workers drain together: a worker that finds a row locked takes the next one instead of queueing behind it. The stale lease in the predicate is how a crashed worker's row comes back without a human, and the lease is long enough that a slow but live run is not stolen out from under it.

Search says so, and so do we

With embeddings stopped, the embed step does not attempt at all. The memories are already written; this is only their index, so holding costs a delay while attempting spends money to rediscover a fact an operator has already recorded.

Search then has a problem it cannot see. The planner knows a strategy failed, not that an index is deliberately stopped, and with no semantic strategy attempted it would present a keyword-only answer as the whole answer. So the response carries diagnostics that say it, and both SDKs push you at them.

const found = await client.search.query({ query: "what did we decide about the ledger" });

if (found.diagnostics.degraded) {
  // Not "search is broken". It is looking with one eye, and `notice` is a
  // sentence written for the person, not for a log line.
  showBanner(found.diagnostics.notice ?? "Some results may be missing.");
}

The last piece is what the user is told, and it is written once. Every notice says what is wrong in their terms, that their material is safe, and that they will be emailed when it lands. Three wordings of that promise read as three different promises, so there is one wording per capability and a unique index on incident and user, because an operator toggling a flag twice while diagnosing must not send two apologies for one outage.