Hawzu

Most test management tools are, underneath, a database of test cases with reporting bolted on. They answer “what did we test?” well. They answer the question every release meeting actually asks — are we ready to ship? — badly, or not at all.

Hawzu is built the other way round. Execution is the centre of the model, and everything else exists to turn execution into a defensible shipping decision.

The newer half of the product follows from the same idea. Once the evidence is connected, the useful thing to do with it is ask — so there is now a query engine over a project’s records, an AI layer above it that turns a question into a query, and a read-only endpoint that lets an AI client in an editor read the same data under the same permissions.

The shape of it

What I built

The whole thing, end to end. There is no team.

Backend — FastAPI over MongoDB, with Redis as a cache. Three API surfaces share one middleware chain: /ui/v1 for the app, a public /api/v1 for automation, and /mcp for AI clients. Automation authenticates with scoped service accounts rather than user sessions. They behave similarly, but their lifetime belongs to the automation, not to a person’s browser session.

One small decision illustrates the philosophy: the middleware is ordered so that failures stay truthful. Authentication and authorization can both reject a request, and when they do the caller needs to see that — not a transport-level error standing in front of it. An outage you can’t name is worse than the outage.

Retrieval — Qdrant alongside MongoDB, holding four collections: the product’s own documentation, a customer’s specifications, defects and test-case coverage. Search is hybrid — dense vectors from OpenAI, sparse BM25 computed locally — and, like Redis, losing it slows the product rather than stopping it. Three named things sit on top: Oracle is the AI itself, Canon is the customer’s own specifications indexed so answers and generated test cases can cite them, and Atlas is a derived map of the product’s journeys with coverage drawn onto it.

Canon has exactly one read path, and that is the whole design. Callers never build a vector filter themselves, because a filter is a place to forget the tenant scope — and the difference between a shared documentation corpus and one customer’s unreleased specification is not a difference you get to discover later.

Frontend — a React + Vite SPA on Mantine, plus three static Astro sites (marketing, docs, blog) that deploy independently to CDN-backed buckets.

Infrastructure — one deployable API. Everything in front of it is static and ships on its own schedule, behind a single shared edge. A docs fix or a blog post goes out without redeploying the product, and nothing on that side can take it down.

Decisions worth writing down

Execution before storage

The data model treats a test execution as the primary record and the test case as the thing being executed. That inversion is why release readiness is computable at all: coverage, defects, and requirement traceability all hang off executions, so the evidence is already connected when someone asks whether to ship.

What it cost: almost every query is scoped through an execution, so the handful of genuinely test-case-centric views — “where is this case used?” — are the awkward ones. That was the right trade only because the shipping question is asked far more often than the inventory question.

A release snapshot, not a moving target

Release analytics used to shift under you: link a defect after sign-off and last month’s numbers changed.

The obvious fix — lock defects once a release completes — was wrong. Defects should keep moving; a bug found in production after sign-off is still that release’s bug, and its status genuinely changes. What must not move is the evidence the decision was made on. So the release, its executions and their test cases are held read-only by the editability guards while frozen, and the one remaining source of drift — the defect-derived analytics — is snapshotted at the moment the release completes, with the snapshot’s as_of becoming the frozen clock everything downstream reads from.

What it cost: a second read path. Every consumer of release analysis has to know whether it’s looking at live data or a snapshot, and the UI has to say which it’s showing. That’s real complexity bought to keep one property: sign-off means something only if the evidence behind it stops moving.

A deterministic engine, with a model on top

Asking a question in English is the feature. It is not the architecture.

Underneath sits a query engine that takes a versioned intermediate representation, not a sentence: it is free, deterministic, and either correct or broken. Above it, exactly one module is allowed to call a model, and its only job is turning a question into that IR. A test asserts nothing else in the package imports an AI service, and proves the engine still answers with the provider down.

The planner’s characteristic bug is naming a field that doesn’t exist, and the fix is structural rather than instructional. Its JSON schema is generated from the caller’s own catalog — which was already permission-filtered — so the model cannot emit an unknown field, or one this person may not read. Nothing in the prompt has to ask for that and no downstream code has to handle it.

What it cost: the model can still pick the wrong field and hand back a valid, confidently wrong answer. That failure can’t be designed out, so it’s designed around: every resolved clause is rendered back as an editable statement of what the question was understood to mean, and correcting one re-runs the deterministic path, which costs nothing. The model gets one shot at understanding; the person gets unlimited cheap corrections. That asymmetry, and what it costs to build for, is written up separately in One Shot at Understanding.

A third surface, not a third server

AI clients read Hawzu over MCP. That could have been a new service; it is one more router in the existing process, behind the same middleware, resolving the same bearer tokens. The thing that made it cheap is that the surface is read-only by construction — nothing reachable through it creates, edits, deletes or runs anything.

Two rules do the real work. Permissions are checked per tool rather than per route, because one path carries many operations, and a check inside each handler is a check a new handler can forget. And a caller who may not read something gets a refusal, never an empty list — returning zero rows would turn a permissions bug into a data bug, where the model reads “none” and reports it to someone as fact.

It also costs the customer no AI usage at all. The reasoning is done by the agent they already pay for; Hawzu is only the thing being read.

What it cost: a second token concept. An automation token is refused on /mcp and an MCP token is refused everywhere else, which is correct — a credential handed to an agent shouldn’t inherit the write access of the automation API — and it is one more thing to explain to anyone integrating. The same rule made an OAuth server part of the work, because the clients worth supporting don’t all accept a pasted token.

Over a limit is not out of bounds

Plans replaced an early-access tier that capped nothing, which meant the first question wasn’t pricing. It was what happens to a workspace that is over a limit rather than merely at one.

Existing work keeps working; growth is refused. A workspace that drops to the free tier with twenty members keeps all twenty, and every one of them can still read, run and edit. What gets refused is anything that makes the workspace bigger — a new member, a new project, an upload past the quota. The alternatives were both worse: read-only stops a QA team mid-release over a billing problem, and a hard cap locks people out of test cases they wrote, in an order that is arbitrary to everyone affected.

The enforcement turned out to already exist. Every seam checked its limit after the operation it was guarding, so each was a guard on growth before there was a policy saying so, and a test pins the property that nothing anywhere revokes access.

What it cost: a workspace can now sit permanently over a limit, which is a state the product has to name honestly. Telling someone at twenty of five seats that they are “at 5 of 5” is false, so over-limit had to become a first-class thing to render rather than a number clamped to its maximum.

Jobs in Mongo, not a broker

The background queue started on Redis/RQ and moved into MongoDB — the database already being operated — drained by a bounded thread pool inside the API process itself. Redis stayed, as a cache only.

The interesting part isn’t the store, it’s what Cloud Run does to the design. Every replica has the scheduler enabled, and Cloud Run autoscales, so every replica tries to drain the same queue. Rather than run a separate worker service, correctness comes from a Mongo-backed advisory lock with a lease and heartbeat: whoever claims a job runs it, a live holder keeps its lease fresh, and a holder that dies has its lock taken over once the lease goes stale. The same mechanism guards the periodic sweeps, which would otherwise run once per replica.

What it cost: the lock is now load-bearing infrastructure — a correctness bug there is a double-executed job, not a slow one. In exchange there’s no worker tier to deploy, scale or monitor separately.

A cache you can lose

Redis holds sessions, user metadata and permission lookups. When it isn’t there — at startup or halfway through a request — lookups fail fast and fall through to MongoDB rather than waiting on a connection that isn’t coming back. Slower, still correct.

The rule this encodes: if losing the cache takes the product down, it was never a cache. It was an undocumented database.

Each of these left behind something more general than itself. The versions I’d carry to the next system are written down as principles.

Where it is now

Live and bootstrapped, on three plans that differ by how much you use rather than by what the product can do. Nothing sits behind a paywall — the free tier carries five people, real limits, and the whole product including the AI.

There is no checkout yet: a bill is produced and invoiced by hand. The pricing page says so in those words, which felt better than a card form that doesn’t exist standing in for one that does.

CI/CD automation shipped: builds are triggered and imported from Jenkins, GitHub Actions and GitLab CI, with each result mapped back to its test case so nobody transcribes pass or fail by hand. The design problem that made it interesting was what to do with results that match no known test case. Discarding them loses data; force-matching them invents coverage nobody ran. They get their own collection instead, deliberately outside the one the analytics read from, so an unreconciled pile can never quietly become a number on a readiness scorecard.

The most recent work is the one described above — a query engine over a project’s records, the AI layer that sits on it, and the read-only MCP surface that lets an agent in an editor ask the same questions under the same permissions. That is also the current direction: the evidence was already connected, and what remains is making it answerable without a person having to know where it lives.