Srinath Reddy

← Writing · Architecture

Production Is Never a Safe State

Instead of guessing which requests are bad, the application only serves routes on a declared list, and refuses to start if its code and that list disagree.

· 3 min read

Many inbound request marks arriving and stopping at a single red line; only three continue past it.

Any public web application gets requests for things that were never built: environment files, cloud credentials, repository configs. They arrive all the time, and none of them were ever going to work.

Refusing them is the easy part. The harder part was a question the codebase had never answered: which URLs is this application actually supposed to serve?

Why a blocklist doesn’t work

The first idea is to collect the bad paths and block them. The traffic itself shows why that doesn’t hold up.

Requests that arrived
GET  /api/.env
GET  /api/.aws/credentials
GET  /api/.git/config
GET  /api/client_secret.json
POST /api/exec

That’s an environment file, cloud credentials, a repository config, an OAuth client secret, and a way to run a command. The environment file alone was asked for under many different paths: /api/v2/.env, /api/dev/.env, /api/staging/.env. Other requests asked for phpinfo.php, a Laravel debugging page, and even a Next.js route pattern typed in literally, with its placeholder never filled in.

Nobody had looked at what this application runs. These are lists of guesses sent to every server that answers. A blocklist would mean judging each new guess as it shows up, indefinitely.

The real goal was narrower: a request for a path we never built should never reach anything.

Writing the list down

The application had plenty of endpoints, but no single list of them. Each one existed because someone had written a function, and the only way to know them all was to read all the code.

So I wrote the list: every route the application is allowed to answer on. Anything not on it is refused.

That changes the question asked of each request. Instead of “does this look suspicious?”, it becomes “is this path on the list?” The first is a judgement call that never ends. The second is a lookup.

Keeping the list honest

A list brings its own risk. Someone adds a route and forgets to add it to the list. Now the code and the list disagree, and nothing in code review reliably catches a line that’s missing.

So when the application starts, it compares the routes it actually has with the list. If a route isn’t on the list, the server doesn’t start.

Routes compared against the declared contract at startup Six routes registered by the application are listed down the left. Five are joined by a line to a matching entry in the declared contract on the right. The fourth has no counterpart: its line stops partway across and ends in a cross, with nothing opposite it. That mismatch is what stops the server from starting.
Five routes have a declaration. One does not — so the server refuses to start rather than serve an endpoint nobody accounted for.

It caught me. I added a route, forgot to declare it, and the server refused to start, naming the route that was missing.

A test in CI could check the same thing, but tests can be skipped when they’re inconvenient. Startup happens on every deploy, so this check can’t be forgotten.

I’d rather a deploy fail than have an endpoint go live that nobody accounted for. The failure shows up at the moment of the mistake, and it takes a few minutes to fix.

The day-to-day difference: writing a route is no longer enough to make it exist. It also has to be on the list.

What this doesn’t do

  • It doesn’t stop the requests. Anyone can still send them. They just don’t reach anything.
  • It says nothing about the code behind a declared route. Whether that code is correct is a separate problem.

What I’d take from it

When production shows you something unexpected, it’s tempting to add another alert or another rule. Often the better fix is to find the rule the system should have been following all along, write it down, and make the system enforce it. This narrows what can be reached. It doesn’t make the application safe on its own.

Read next One Shot at Understanding → The AI gets one try at understanding a question. The person can correct it as often as they like, for free. Most of the design follows from that.