Production Is Never a Safe State

A public application receives requests for things nobody built. Files of environment variables. Cloud credentials. A repository config. They arrive constantly, they have been arriving for months, and not one of them was ever going to work.

The interesting problem isn’t refusing them — that part is easy. It’s that deciding what to refuse means answering something the codebase had never written down anywhere: which URLs is this application actually supposed to serve?

The obvious answer is a blocklist

Collect the paths, block them, move on. It is the first instinct and it is the wrong one, for reasons the traffic itself makes obvious.

GET  /api/.env
GET  /api/.aws/credentials
GET  /api/.git/config
GET  /api/client_secret.json
POST /api/exec

An environment file, a set of cloud credentials, a repository config, an OAuth client secret, and a way to run a command — a fair summary of what anyone would want from a server they don’t own. But the shape matters more than the contents. Bots asked this application for its environment file 133 times, in a dozen path variations: /api/v1/.env, /api/v2/.env, /api/dev/.env, /api/staging/.env, /api/shared/config/.env. That is not somebody who has looked at the application. That is a tree of guesses being walked.

The rest confirms it. Requests for phpinfo.php. For /api/telescope/requests, a Laravel debugging tool. For /api/[[...slug]] — a Next.js catch-all, requested literally, the template placeholder never substituted, pasted straight out of somebody’s list.

Nobody checked what we run. There is no adversary here to enumerate, just a dictionary replayed against every host that answers. So a blocklist is answering the wrong question: it asks does this look suspicious?, which is a judgement you can be wrong about and have to make again every time someone has a new idea.

And the goal was never “nobody should ever request an invalid endpoint” — that isn’t achievable on a public application, and it was never the interesting problem. The goal is: an invalid endpoint should never quietly become a valid one.

The list didn’t exist

Which turns out to need something more fundamental than a rule.

We had endpoints. We did not have a single authoritative list of the endpoints the application was supposed to expose. They existed the way most API surfaces exist — as a consequence of people writing functions, discoverable only by reading all of them.

So it got written down. Not documentation: a contract. 451 route templates, each one a path the application is allowed to answer on — and nothing outside that list is.

Having it changes the question being asked at the door. Not does this URL look suspicious? but is this URL part of the application at all? The first is a judgement call, and there is no end to making it. The second is a lookup against a list we wrote ourselves.

The list decides

Enforcement is the easy half once the list exists. A request is resolved against the declared surface, and anything outside it is not served.

The application never has to decide whether a request is malicious. It doesn’t ask whether /api/.env is hostile, or whether /api/aws/credentials is a scanner or a confused client or somebody curious. It asks whether the path is declared. It isn’t, so there is nothing there to reach.

Then we protected the contract from ourselves

A static list creates a new failure mode, and it’s the one that actually worries me.

A developer adds a route and forgets to declare it. Now there is a surface that exists in the code and not in the contract. Nobody did anything anyone would notice — there’s no error, and nothing in review reliably catches an omission.

So the application checks itself. At startup it introspects its own registered routes, consolidates them, and compares the result against the declared inventory. If a route exists that the contract doesn’t account for, the server does not start.

It caught me. I added a route, didn’t declare it, and got exactly that:

Endpoint URLs and the declared endpoint inventory are not in sync.
Missing url /ui/v1/workspace/{workspace_id}/export

A test could ask the same question, and asking it in CI would have been easier to write. The difference is that a test has to be remembered, and can be skipped when it’s inconvenient; startup happens on every boot, including the one nobody is watching. Putting the check there means it runs whether or not anyone decided it should.

We would rather a deployment fail than an endpoint appear that nobody accounted for. The failure is loud, it lands at the moment the mistake was made, and it costs five minutes — which is the cheapest that mistake is ever going to be.

What actually changed is smaller than it sounds and shows up every day. Writing a route used to be enough to make it exist. Now it is half the job: the endpoint exists when the code and the contract agree about it, and until they do, nothing runs at all.

What this does not do

It does not stop the requests. Nothing here makes an unwanted request impossible to send; it makes an undeclared route impossible to serve. Two different claims, and only the second one is ours.

The inventory does two jobs and only one is written down. It began as a permission table and became an allowlist by fallthrough — a path’s absence is what refuses it. So deleting a line deletes an endpoint, and the file says that nowhere.

A declared surface says nothing about what is behind it. The contract asserts that a path is allowed to exist. Whether the handler on the other side is correct is a separate question this doesn’t touch.

The part that generalises

You don’t get to make the internet predictable. That traffic had been arriving for four months before anyone looked, and none of it was going to stop because we noticed.

What you do get to control is the boundary — and it turned out to have two sides. A request has to belong to the declared surface; the declared surface has to agree with the code. It is the same question — is this path declared? — asked once of a stranger’s request and once of our own code. The second is the half that was actually dangerous, because it had us on the inside of it.

When production shows you something unexpected, the durable response usually isn’t another alert. It’s finding the invariant hiding behind the surprise and making the system enforce it. This one narrows the surface. It doesn’t close it, and nothing does.