Seven migrations, proved correct, never applied

The first query against the new observability dashboard came back 42P01. Relation does not exist. The table had been in the schema, and in a committed migration, for weeks.

journal entries: 24  applied: 17

MISSING  0017_transcription_flag
MISSING  0018_otp_hash_scope
MISSING  0019_client_connections
MISSING  0020_inbound_mail
MISSING  0021_mcp_oauth
MISSING  0022_one_connection_per_platform
MISSING  0023_observability

Nothing was wrong with any of them. CI runs `yarn db:migrate` against a Postgres service container on every commit, so all seven had been proved to apply cleanly, repeatedly, for weeks. They were proved against an empty database. Nothing had ever compared them against the one people were using.

There is no deploy step that would. The five images each start their app directly, and Render pulls an image without running a release command. Applying migrations was a manual step nobody had written down, which means it was a manual step that existed only in whoever last remembered it.

Why observability was the worst place for this

0023 creates `metric_samples` and `request_events`. Without them, every flush from the event sink failed. The sink swallows flush failures deliberately, and the comment explaining why is correct: a metrics backend that throws into a request handler converts we cannot see into we cannot serve, which is the wrong trade every time.

So the system recorded nothing, reported nothing, and stayed up. The surface that exists to reveal a problem like this was the surface the problem was hiding inside. An empty dashboard and a quiet service look identical.

0021 was the other expensive one. It creates the four tables the MCP OAuth flow writes to, so every attempt to connect Claude or Cursor was writing into tables that were not there.

There were two causes, not one

After applying all seven, the dashboard was still empty. The route was not even mounted.

// runtime.ts
export interface Runtime {
  readonly operations: OperationsRepository;
  readonly observability?: ObservabilityRepository;  // declared
  // ...
}

// and, in the composition root, never assigned.

The sink is mounted behind `if (runtime.observability)`. An undefined field meant no request was ever recorded and no dashboard route was ever mounted, and it typechecks perfectly, because an optional port that nothing constructs is exactly as valid as one that does.

Either cause alone produces the same symptom. Had the migration been applied, the sink still would not have run. Had the repository been constructed, it would have written into tables that did not exist. Two independent failures, one indistinguishable outcome, and no error anywhere.

The check

The fix is not a migration runner. Applying DDL at boot means a rolled-back image meets a schema from the future, and it turns a bad migration into an outage rather than a failed deploy. What was missing was a comparison.

export async function migrationStatus(db, folder): Promise<MigrationStatus> {
  const journal = JSON.parse(await readFile(`${folder}/meta/_journal.json`, "utf8"));
  const applied = new Set(rowsOf(
    await db.execute(sql`SELECT hash FROM drizzle.__drizzle_migrations`)
  ).map((row) => String(row["hash"])));

  const pending = [];
  for (const entry of journal.entries) {
    const body = await readFile(`${folder}/${entry.tag}.sql`, "utf8");
    if (!applied.has(createHash("sha256").update(body).digest("hex"))) {
      pending.push(entry.tag);
    }
  }
  return { applied: applied.size, available: journal.entries.length, pending };
}

By content hash rather than by count, because counting calls a database up to date after somebody edits an already-applied file. It is reported in two places: the API says so on the way up, and the admin overview carries it in red.

Both, not one. A line in a startup log is precisely what failed to be read for the previous several weeks, and adding another line to the same log would have been a fix that reproduced the original failure.

The general shape is worth naming, because it recurs. A step that cannot fail teaches you nothing. CI proving migrations against an empty container is a step that cannot fail in the way that matters, and a green run said so every single time.