Production readiness and recovery drill

Why this matters

On Day 77 you deployed and wrote down how to back up. Today you find out whether any of it is true. You will deploy a release against a written checklist, restore a backup into a scratch database and count the rows, and deliberately break your configuration to see how your app behaves when it is wrong.

The idea that survives this hour: a backup is only trustworthy once you have successfully restored it. Everything else — the filename, the file size, the fact that the job ran last night — is a hypothesis.

Health checks

A health check is an endpoint that answers one question: is this instance able to do its job right now? Hosting platforms poll it and stop sending traffic to instances that fail.

There are two depths, and the difference matters:

• Shallow — "the process is running and can answer HTTP". Cheap, and true even when the database is unreachable.
• Deep — the process also checked its critical dependency, usually with a trivial query.

app.get("/healthz", async (req, res) => {
  try {
    await db.query("SELECT 1");
    res.json({ status: "ok" });
  } catch (error) {
    console.error({ where: "healthz", message: error.message });
    res.status(503).json({ status: "degraded", db: "unreachable" });
  }
});

SELECT 1 is the cheapest query that proves a live connection. The status code carries the answer, because that is what automated systems read:

curl -s -o /dev/null -w "%{http_code}\n" https://your-app.example.com/healthz

200

Keep health checks unauthenticated but boring: never include version details, connection strings, or environment values. It is a public endpoint.

Migration safety

A migration (Day 48) changes the database schema. In production it is the step that can destroy data, so it gets its own rules:

• Additive changes are safe. New table, new nullable column, new index. Old code keeps working beside them.
• Destructive changes are not reversible by rolling back code. DROP COLUMN and DROP TABLE delete data. Reverting to yesterday's release brings back code that expects a column which no longer holds anything.
• Split a rename into two releases. Add the new column and write to both; deploy; migrate the data; only in a later release stop writing the old one and drop it. Each release runs against the schema both before and after it.
• Run the migration against a copy first. A restored backup is exactly the copy you need — and it exercises your restore at the same time.

Backup and restore, tested

The drill, with Docker Compose from Day 76. -T disables TTY allocation so redirection works:

mkdir -p backups
docker compose exec -T db pg_dump -U postgres appdb > backups/appdb-2026-08-03.sql
ls -lh backups/

-rw-r--r--  1 you  staff   184K Aug  3 10:22 appdb-2026-08-03.sql

A file exists. That proves nothing yet. Restore it somewhere harmless and count:

docker compose exec -T db psql -U postgres -c "CREATE DATABASE restore_test;"
docker compose exec -T db psql -U postgres -d restore_test < backups/appdb-2026-08-03.sql
docker compose exec -T db psql -U postgres -d restore_test \
  -c "SELECT count(*) FROM maintenance_records;"

 count 
-------
   142
(1 row)

Compare 142 against the same count in the live database. Matching counts on your main tables is the evidence. Then drop the scratch database — DROP DATABASE restore_test; — and note how long the whole restore took. During an incident, "about four minutes" is the answer people need.

Failed configuration, on purpose

Most production outages are not exotic. They are a missing environment variable after a deploy. So find out now how yours fails: does it stop at startup with a clear message, or start happily and throw at 3 a.m. on the first user request?

Make it the first kind — fail fast:

const required = ["DATABASE_URL", "SESSION_SECRET"];
const missing = required.filter((name) => !process.env[name]);
if (missing.length > 0) {
  console.error(`Missing environment variables: ${missing.join(", ")}`);
  process.exit(1);
}

Note what is not in that list: AI_API_KEY. On Day 87 you built the AI feature to degrade gracefully when the key is absent, so a missing key must not stop the app from booting. That distinction — required to run, versus required for one optional feature — belongs in your checklist in writing.

Incident notes

An incident note is a short factual record written while it is fresh. Five lines is enough:

2026-08-03 14:05  Detected: /healthz returned 503; records screen showed error state.
2026-08-03 14:07  Cause: DATABASE_URL missing after redeploy; app started, queries failed.
2026-08-03 14:12  Action: restored the variable, redeployed, /healthz returned 200.
Impact: ~7 minutes, read and write both unavailable. No data lost.
Change: added startup config check so the app refuses to boot without it.

What happened, how you noticed, what you did, the impact, and the one change that prevents a repeat. No blame — the interesting question is why the system let it happen, not who typed it.

Walkthrough

Run the release checklist end to end. Write it as a file first: docs/release-checklist.md, one line per step, each with an observable result.

1. Clean tree, npm run check passes, tests green.
2. git log --oneline -1 — note the exact commit you are shipping.
3. Take a backup and restore it into restore_test, comparing row counts.
4. Apply migrations to restore_test first. Only then to production.
5. Deploy the release.
6. curl /healthz and confirm 200.
7. Run one real user workflow in the browser against production.
8. Check logs for errors in the first two minutes.
9. Record the rollback command for this release — the previous commit or image tag — before you need it.

Your turn

1. Write docs/release-checklist.md with every step above plus anything your host requires. Each step names its observable result.
2. Add the deep /healthz endpoint and the startup config check. Confirm both.
3. Deploy one release following the checklist literally. If a step turns out to be wrong, fix the file, not just your memory.
4. Backup drill: dump, restore into a scratch database, compare counts on two tables, drop the scratch database, record the elapsed time.
5. Failure drill: remove one required environment variable from a local run and start the app. Record what you saw and how long it took to diagnose.
6. Write one incident note for that simulated failure, in the five-line format.
7. Confirm your rollback path by checking out the previous release commit locally and starting it.

Common pitfalls

• Counting a backup file as a backup. Until it has been restored and the rows counted, it is an unverified file.
• Restoring over the live database "to test it". That is the one command that turns a drill into an incident. Always restore into a scratch database.
• Assuming rolling back code undoes a migration. It does not. Destructive schema changes need a restore, which is why they get their own step.
• Shallow health checks that lie. A /healthz that only proves the process is up will report 200 while every user-facing query fails.

Verify it yourself

Open today's reference, Docker's Get started, and find the part covering persisting data with volumes.

1. This lesson assumed your database data survives a container restart. Find the sentence that explains what happens to data written inside a container that is removed, and confirm your Compose file actually uses a volume for PostgreSQL.
2. Find what the docs say about how a container gets its configuration. Compare it to how your deployment supplies DATABASE_URL.

Add both findings to docs/release-checklist.md. If your database is not on a volume, that is the most urgent thing you will fix all week.