Snoowl Audit, Srinath Reddy

A lot of companies now put a chatbot in front of their customers, and very few check what it actually says. When it gets something wrong, the customer usually finds out first.

Snoowl Audit does the checking. You give it your docs and a way to reach your chatbot. It writes questions from the docs, asks the chatbot each one, and checks every answer against the docs. A wrong answer comes with the line from your docs that shows it’s wrong. Separately, it runs red-team tests: attempts to get the chatbot to ignore its instructions, leak data, or claim it did something it can’t do.

It’s aimed at the person who owns the support experience, not at an ML engineer, so everything happens in a web console and ends in a report you can hand on.

What I built

All of it, on my own.

Backend: a FastAPI service on MongoDB with a background worker for audit runs, which can take a while. The same backend serves Snoowl Answers; both products share projects and documents, so a company uploads its docs once.

Search: documents are split into passages and indexed for both meaning-based and keyword search, with the two result lists merged and then reranked. Writing good questions and judging answers both depend on finding the right passages.

Console: a React and Mantine app for uploading docs, reviewing the generated questions, grouping them into test sets, starting and scheduling runs, comparing runs, and exporting PDF reports. Scheduled runs send an alert only when something gets worse.

Billing: a prepaid balance charged per step (words read, questions written, answers judged), with no seats.

Decisions worth writing down

Prove the judge before building the product

The whole product rests on one claim: when Audit says an answer is wrong, it is. If that isn’t true, nothing else matters. So before building the console, I built a test: a fake company’s docs, a chatbot that answers correctly, and a second chatbot with mistakes deliberately planted in it.

The bar was set before running it. At least 95% of the serious findings had to be correct, and at least 90% of the planted mistakes had to be caught. Across four runs, all 37 serious findings were correct, all 72 planted mistakes were caught, and the correct chatbot got no serious findings at all.

That’s a test I designed, not real customers, and I say so. Real chatbots make messier mistakes. But it meant the rest of the product was built on something that had been measured rather than assumed.

Show the evidence, not just a score

Every wrong answer shows the passage from the company’s own docs that contradicts it. A score on its own invites an argument; the passage ends it.

The red-team results follow the same rule. There’s no single “security score”, just counts per category. Each finding records what the chatbot said and the evidence, and states what Audit can’t tell from the outside. For example, it can’t see whether the chatbot actually ran an action, only whether it claimed to.

Red-team tests are their own track

Red-team tests started out mixed in with the accuracy questions. That meant every accuracy feature (the question list, scores, comparisons, history) had to remember to filter them out, and one missed filter produced a wrong result.

They now run as a separate track with their own pages and runs. Accuracy and red-team results never mix, so there’s nothing to forget to filter.

Check the obvious things without a model

Where a red-team result can be decided by a plain check, it is. Some tests are built so that only a compromised chatbot would produce a particular marker word, so spotting one is a string match. Leaked card numbers, credentials and contact details are matched by code. A model reads the reply only when no plain check can decide, and a compromise found that way needs a second model to agree before it’s reported.

Comparing runs starts with what changed

People want to compare this month’s audit with last month’s. But between two runs the docs may have changed, the questions may have been regenerated, and the chatbot may be running a different model. Two scores side by side would hide all of that.

So the comparison page first lists what changed between the runs (docs, questions, setup, judging) and says whether the runs are comparable. Only then does it show the numbers. Answer-by-answer comparison is limited to questions whose expected answer and source passages were identical in every run.

Where it is now

In early access. Teams can request an audit at snoowl.com/audit, priced per step with no seats.

What’s in it:

Test sets, so the same questions can be asked again later.
Scheduled runs that send an alert when results get worse.
The same questions sent to several chatbots at once, compared side by side.
PDF reports with accuracy and red-team results kept apart.

Known limits: it asks single questions rather than holding a conversation, and it reaches chatbots through an API, not through a chat window on a website.

Next: proving the judging on real chatbots, not only on a test I built.