Reuse the foundation
How might different GenAI projects configure one evaluation system instead of rebuilding it?
SOTERIA
It checks AI answers, then checks the checker.
- Status
- Building
- Year
- 2026
- Role
- Design and code, solo
- Built with
- PythonFastAPIPostgreSQLRedisReact
OVERVIEW
A score from an AI judge looks precise and can still be wrong. Soteria checks AI answers for RAG, entity extraction and structured extraction, keeps the quote behind every judgement, and tests whether its own judges deserve trust. When it can't tell, it says so and hands the case to a person.
WHAT I OWNED
- Built the API, the worker, the evaluators and both web apps.
- Wrote the judge prompts, 15 in all, and the rules that grade them.
- Packaged it as one image that runs as the API or the worker.
- Deployed it to Azure for tests, and deleted each one after.
OBJECTIVES
Trust the evidence
How might a score show the uncertainty and missing evidence behind it?
Check the judges
How might we evaluate the reliability of the evaluators themselves?
HOW THEY BECAME FEATURES
WHY?
A judge’s answer can shift with prompt wording, input order, verbosity, style or model choice.
DESIGN DECISIONS
- Test prompt ensembles, perturbations, agreement and bias.
- If something can be measured deterministically, measure it deterministically. Use a model judge when interpretation is actually required.
TRADEOFF
Evaluator reliability also has to be evaluated. A score alone is not enough.
WHY?
Failures matter differently depending on risk.
DESIGN DECISIONS
- Keep the core engine stable; configure required facts and evaluation rules for each use case.
- Keep required-fact status, evidence and review decisions with the result.
TRADEOFF
Missing evidence must not quietly become a confident negative result.
WHY?
A judge can be unstable even when the answer has not changed.
DESIGN DECISIONS
- Keep drift comparisons alongside confidence intervals and judge metadata.
- Separate changes in the answers from variation in the judges.
TRADEOFF
A change in scores does not by itself prove a change in the system.
HOW IT WORKS
- Did the run execute?
- Is the evidence usable?InconclusiveHuman review
- Did the quality gate pass?
- Is the result ready for its intended use?
IT GRADES ITS OWN JUDGES
Every judge question is asked three ways, and checked again under small changes to the input. Soteria grades its judges A to D from the weakest of those checks, not the average. With one judge model the grade stops at B, because one model can't show its own bias.
EXAMPLE
Stability 0.99 on one metric and 0.835 on another. Grade B: the weaker one decides.
MISSING FACTS COUNT BY RISK
You list the facts an answer must contain and how much each one matters, from Low (1) to Critical (5). The judges check each fact and quote the answer as proof. If they disagree, the fact is marked inconclusive and goes to a person instead of counting as a miss.
EXAMPLE
Required: notice at least 60 days before term end. The answer: "giving 30 days notice". Contradicted, Critical, weight 5.
DRIFT HAS TO BE PROVEN
Soteria compares a run with an earlier one, case by case, and tests whether a drop is real. If the answers and context didn't change but the scores did, it counts that as the judge moving, not the system.
EXAMPLE
4 of 20 scores drop from 1 to 0. p = 0.125. Change seen, not proven. No alert.
ONE REPORT, THREE READERS
The same evidence opens three ways. Business gets the consequence and what to do next. Engineers get the case to investigate. Auditors get the quote and the decision record.
WHERE IT STANDS
- Status
- Building. Not in production yet.
- Metrics
- 13 on a RAG run, from BLEU and ROUGE-L to faithfulness and risk-weighted coverage.
- Runs as
- One container image: the API or the worker, with PostgreSQL and Redis.
- One audit run
- 20 cases, 2,020 judge calls, 52 min 37 s.
- Limits
- Not yet adopted by a live project. In internal vetting.