Week 11 deployment and operations

Why this matters

Yesterday one command started your stack on your machine. Today it runs somewhere the internet can reach, and you take on the part nobody warns beginners about: when it breaks, you are the person who has to fix it, possibly at an inconvenient hour, possibly having forgotten how any of it works. So today's real deliverable is not the URL. It is a runbook — written instructions that let someone who has never seen your project start it, read its logs, restore its data, and undo your last deploy.

Configuration that changes per environment

Your app runs in at least three places now — your machine, the test database, and production — and a few values differ in each: the database URL, the session secret, the port, the allowed CORS origin. Everything else is identical.

Those differing values are configuration, and they belong in environment variables: named values the operating system hands to your process at startup, read in Node as process.env.NAME. Not in the code, and never in the repository.

The rule is absolute for one reason: Git history is permanent and widely copied. A secret committed today is in every clone, every fork, and every backup of the repository forever. Deleting the line in a later commit does not remove it from the earlier one.

So the repository holds .env.example — every key, with dummy values and a comment on where the real one comes from — and .env stays in .gitignore. On the host you set the real values through its own configuration interface, not by uploading a file.

# .env.example
DATABASE_URL=postgresql://user:password@host:5432/maintenance
SESSION_SECRET=generate-with-openssl-rand-hex-32
CORS_ORIGIN=https://maintenance.example.com

HTTPS and domains

A domain name is a human-readable name that DNS resolves to your host's address. HTTPS is HTTP carried inside an encrypted TLS connection, proven by a certificate that ties the key to your domain.

For an app with logins, HTTPS is not optional. Over plain HTTP, every request — including the session cookie or token that is the user's identity — travels as readable text through every network in between. Anyone on the same café WiFi can copy it and become that user.

Two practical consequences. First, the Secure cookie flag from Day 53 tells the browser to send the cookie only over HTTPS, so it only does its job once HTTPS exists. Second, most container hosts terminate TLS for you: you point the domain at them, they obtain and renew a certificate automatically, and your app keeps serving plain HTTP inside the private network. Your job is to confirm it worked, not to hand-manage certificates.

Health checks

A health check is an endpoint whose only job is to answer "is this instance able to serve?" The platform calls it every few seconds and uses the answer to decide whether to send traffic to your container or restart it.

A shallow check returns 200 if the process is running. A deep check also verifies the things the app cannot work without — most importantly the database:

app.get("/healthz", async (req, res) => {
  try {
    await pool.query("SELECT 1");
    res.status(200).json({ status: "ok" });
  } catch (error) {
    res.status(503).json({ status: "degraded" });
  }
});

SELECT 1 is the cheapest possible query that still proves a connection can be obtained and used. Return 503 Service Unavailable when a dependency is down: that is the status that means "not me, not now", and it is what tells a load balancer to stop routing here. Keep the endpoint unauthenticated but boring — it must reveal nothing beyond a status word.

Migrations, backups, and rollback

Migrations (Day 48) are your schema's version history. On deploy they run before the new code serves traffic, because code expecting a column that does not exist yet fails on the first request. Treat them as forward-only: write a new migration to undo something rather than editing an applied one.

Backups are a copy of your data taken on a schedule and stored somewhere other than the machine holding the original. pg_dump produces one. Here is the part that gets people:

An untested backup is not a backup — it is a file you feel good about. Dumps fail silently: wrong database, empty result, truncated upload, a format your pg_restore version cannot read. You only find out on the day you need it. The only proof is a restore drill: take a real backup, restore it into a scratch database, and count the rows.

Rollback is returning to the previous known-good version quickly. Two habits make it possible:

• Tag images by commit, never latest. maint-api:9f2c1ab is a specific, reproducible build. latest means "whatever was pushed most recently", so you cannot say what is running or return to what was.
• Know what the database does. Deploying the previous image does not undo a migration. If your change dropped a column, rolling back the code leaves the schema without it. This is why risky schema changes are done in two deploys — add the new column and write to both first, remove the old one only after the new code is proven.

Walkthrough: deploy, verify, drill

Choose a host that runs containers — a platform such as Render, Fly.io, or Railway, or a small Linux VM where you run Compose yourself. Give it the repository, set the environment variables in the host's own configuration, and let it build the image from yesterday's Dockerfile.

Once it reports a running deployment, verify from the outside rather than trusting the dashboard:

curl -i https://maintenance.example.com/healthz

HTTP/2 200
content-type: application/json; charset=utf-8

{"status":"ok"}

Run migrations, then read the logs to see the real startup sequence:

docker compose exec api npm run migrate
docker compose logs --since 15m api

Now take a backup. -Fc is the custom compressed format, which pg_restore can restore selectively:

pg_dump "$DATABASE_URL" -Fc -f backups/maintenance-2026-08-03.dump
ls -lh backups/

And drill the restore into a scratch database — never over the source:

createdb restore_check
pg_restore -d "postgresql://localhost:5432/restore_check" backups/maintenance-2026-08-03.dump
psql restore_check -c "SELECT count(*) FROM equipment;"

 count
-------
    42
(1 row)

That 42, matching production, is the only evidence that your backup is real.

Your turn

Write docs/runbook.md. A runbook needs seven sections, and you must have run every command in it:

1. Start — the exact commands to deploy and to start locally, and how you know it worked.
2. Configuration — every environment variable, what it does, and where the real value lives. Values themselves never appear here.
3. Migrations — how to run them, and how to tell which have applied.
4. Logs — the command to read recent logs, plus one line naming what a healthy startup looks like so a stranger can spot an unhealthy one.
5. Backup — the pg_dump command, the schedule, and where dumps are stored.
6. Restore — the drill you actually performed, with the row count you observed and the date.
7. Rollback — the previous image tag, the command to redeploy it, and one sentence on what the database will and will not revert.

Then: add the /healthz endpoint with the database check, deploy it, and confirm it returns 200 from outside. Stop your database container and confirm the endpoint turns to 503 — a health check you have never seen fail is untested too. Finally, grep -ri "password\|secret\|api_key" . across your repository and confirm every hit is a variable name or an example, never a value.

Common pitfalls

• Writing the runbook from memory. Run each command as you write it. The step everyone forgets is the one that assumed you were already in the right directory.
• Deploying latest. You cannot roll back to a tag you cannot name. Tag by commit.
• Backups nobody has restored. Schedule the drill like any other task; a monthly one is enough to catch silent breakage.
• Setting a Secure cookie while testing over HTTP. The browser silently drops it and login appears broken for no visible reason.

Verify it yourself

Open today's reference, Docker's Get started guide, and find its material on running containers in production and passing configuration.

1. Find how it recommends supplying environment variables to a container, and compare it to the env_file approach from Day 76. Note in your runbook which you used and why.
2. Find what Docker documents about container restart behaviour. Which restart policy would you want in production, and what would it do the next time your app crashes at 3am? Add the answer to the Start section of your runbook.