The Author Never Grades Its Own Work: A Classifier Built to Be Verified

The cheapest way to build an AI classifier is one model and a hopeful prompt. It is also the least trustworthy, because a lone model answers with identical confidence whether it is right or catastrophically wrong. We build the other way — the expensive way — because accuracy is an engineering problem, and we would rather solve it than market around it.

Here is the anatomy of a recent build, with the client details left at the door: a person types a business's name — just the name — and gets back the correct industry classification code, ranked, with the reasoning attached. Every step is designed around a single requirement: the answer has to be checkable.

What it queries

Before any model is allowed an opinion, the system gathers evidence. It fans out in parallel across public data sources — securities filings, federal spending and contract records, licensed-provider registries, open geographic and facility data — and adds a live web-reasoning layer, Perplexity's Sonar, to establish what the business actually does and whether the name belongs to more than one company. It also reads the company's own website, stripped to plain text. Each source runs isolated; a dead one returns empty rather than taking the request down. Exa's Company Search is going in next, to resolve which entity a shared name actually is.

How it reasons

Only then does a model weigh in, and it is kept on a short leash. Candidate codes are constrained to the real taxonomy — the model is handed the valid codes and may not invent one. It returns up to three ranked candidates, each with a confidence tier and a rationale tied to the specific evidence behind it. When a name genuinely maps to more than one business, it says so rather than flattening the ambiguity into one confident, wrong answer.

The two verifiers

This is the part most AI products skip, and the part we consider non-negotiable. The first verifier is deterministic: every proposed code is checked against the authoritative table, anything invalid is dropped, and the official title is stamped from the table rather than recalled from a model's memory. The second is a cross-model check, and the rule is the one we hold every build to — the author never grades its own work. One model drafts the answer (Claude Sonnet); an independent model from a different lab (OpenAI's gpt-oss-120b, running on Cloudflare Workers AI) judges whether the evidence actually supports it. If the reviewer disputes the top pick, it goes to a human, not to production.

Why this is the standard, not a preference

We did not invent this discipline; we just build it in. MIT researchers published a method this year for catching overconfident models, and the finding is blunt: a model can be confidently wrong, and the reliable tell is disagreement across different models — which is exactly what the cross-model check is for. MIT's teaching guidance says the same in plainer terms: evaluate the output with human judgment, verify it against authoritative sources, and ground it in trusted material before generating. NIST's AI Risk Management Framework treats a system as "valid and reliable" only when that is shown through evidence, with a human able to intervene where the system cannot catch its own errors. And the reviewer is a different model on purpose: models asked to grade their own output tend to prefer it.

The pattern is the product

The classification is not the interesting part. The shape is. Take a short, ambiguous input; gather evidence from wherever it already lives; ground the answer so it cannot be invented; let a second, independent model check the first one's work; keep a human on the thin or disputed calls; and write every confirmed answer back so the system compounds. That shape is reusable anywhere a messy input has to map to a governed, correct answer — licensing, credentialing, entity resolution, cleaning records against a controlled vocabulary. The public data is usually already out there. A second opinion is cheap. The verifier is the part that makes any of it safe to trust — and building it is most of the job.

The full engineering breakdown of this build lives on jessemyers.ai.