Snapshot
Create one configurable evaluation foundation that different GenAI projects can reuse instead of rebuilding evaluation independently.
A score can look precise while hiding unstable judges, missing evidence, prompt sensitivity or failures with very different risk.
A reusable evaluation system that connects task scores to evidence, risk, evaluator reliability, drift, review and publication state.
Soteria began with a broad evaluation brief, but the deeper problem became deciding when an evaluation result deserves trust.
01.1The brief
My manager gave me a broad problem: build something that could help evaluate GenAI outputs across projects, make it configurable and make it reusable. The problem was assigned; the solution architecture was mine.
I did not want every project to build its own collection of one-off scoring scripts. The idea became a shared evaluation core where task definitions, required facts, thresholds, schemas and judge configuration could change by use case without rebuilding the whole evaluator.
That is what "plug-and-play" means here: reuse the evaluation foundation, configure the contract for the project.
SoteriaPrecomputed demo · SYNTHETIC RUNOne evaluation foundation
Task definitions · required facts · thresholds · schemas · judge configuration
01.2The trust gap
The obvious version of an evaluation product is a dashboard full of scores. The problem is that a score can look precise even when the evaluator is unstable, the judge is biased, evidence is missing or a critical fact has been treated the same as something low-risk.
In a regulated or high-consequence workflow, "the evaluator said 0.91" is not enough. You need to know whether the run actually executed, whether the supporting evidence is valid, whether the task-specific quality gate passed and whether the result is ready for the action someone wants to take.
Trust the score only when the evaluation process can show its work.
SoteriaPrecomputed demo · SYNTHETIC RUNFour separate states
01.3The success condition
The success condition is not a prettier report. A real project should be able to configure Soteria instead of rebuilding evaluation, run its GenAI system through the same core and rely on the resulting evidence when making decisions.
The same evidence also has to work for different readers. A business reader wants to know what the result means for the use case. A risk or compliance reader wants the trail behind the decision. An engineer wants the scores, failure modes and cases that explain what should improve.
The design philosophy became: deterministic where possible, model judgement only where interpretation is necessary, and evidence around both.
02.1Deterministic before judge
The first boundary I wanted to make explicit was where an LLM judge should not be used. If a question can be answered deterministically, I prefer to answer it deterministically.
Schema validity, exact IDs, type checks, duplicate references, numerical checks and other objective conditions do not need a model to make them less predictable. Model judges are reserved for the parts that genuinely require interpretation, such as semantic support or required-fact assessment.
This keeps the evaluation easier to explain and reduces avoidable judge variance.
02.2Separate the states instead of one green/red answer
Soteria separates operational status, evidence status, quality decision and publication/use status. A run can execute successfully while the evidence remains incomplete. A task can have acceptable aggregate metrics while a critical required fact remains unresolved.
That separation matters because the correct response to uncertainty is not always failure. Sometimes the honest state is "inconclusive" and the right next step is human review.
SoteriaPrecomputed demo · SYNTHETIC RUNRun information
This is a saved demonstration dataset, not a live evaluation.
02.3Risk-weighted facts over average scores
A missed low-risk name and a missed critical obligation should not have equal weight. Soteria therefore supports required facts with explicit criticality and risk weighting.
For semantic fact checks, judges return both a status and an exact evidence quote. The system only treats an agreed, non-inconclusive judgement as authoritative. Missing evidence or judge disagreement can remain inconclusive and route to human review rather than being silently converted to zero.
The current risk mapping uses Critical 5, High 3, Medium 2 and Low 1. That allows the evaluator to report risk-adjusted recall and exposure rather than pretending every omission is equivalent.
SoteriaPrecomputed demo · SYNTHETIC RUNRequired answer points
10 checks from the supplied checklist. These are expected details in an answer.
What do these labels mean?
Covered: The answer includes the required detail. Partial: It includes only part of it. Missing: It omits the detail. Contradicted: It states a conflicting detail. Inconclusive: The judgments do not resolve the evidence.
Required points come from the supplied checklist, based on the reference and source context. They do not represent every possible policy requirement.
| Fact | Evidence quote | Criticality | Status | Exposure | HITL |
|---|---|---|---|---|---|
| Coverage begins on the first day of the month after 30 days of employment. | begins on the first day of the month after 30 days | HIGH | COVERED | No confirmed gap | Not required |
| Retain data where law requires it, including records within a legal hold. | except records under a legal hold | CRITICAL | INCONCLUSIVE | Unresolved | Human review |
02.4Evaluate the evaluator
The hardest part of the project was meta-evaluation. Once a model judge becomes part of the quality gate, the judge itself becomes a source of uncertainty.
Soteria tests that reliability by varying prompts, perturbing inputs, comparing judges and looking for position, self-preference, verbosity and style bias. Calibration summarizes the evidence around evaluator stability instead of treating the judge as an oracle.
A single configured judge cannot rule out model-specific bias, so the calibration certificate deliberately caps the grade at B in that situation. A recorded internal audit also produced a C/REVIEW result despite strong prompt/model agreement because perturbation variance and bias qualifications still mattered. That is exactly the behaviour I wanted: strong averages should not erase contradictory evidence.
SoteriaPrecomputed demo · SYNTHETIC RUNHow dependable is the evaluation?
Run-level calibration checks the judges. It does not approve the system's answers.
Illustrative calibration. Review limits remain documented.
Consistency across prompt variants.
A consistency measure, separate from observed agreement.
10 paired required-point decisions.
Input changes · 20 comparisons
Reorders context passages, or sentences within one passage, to test sensitivity to their order.
10 comparisons · mean score change 1.0 points · largest 1.0Adds a small typo to the question to test sensitivity to minor input errors.
10 comparisons · mean score change 2.0 points · largest 2.0Bias checks · both configured judges
Position bias
Checks whether placing an answer first or second changes its score.
Verbosity bias
Checks whether longer wording receives a higher score when the content stays comparable.
02.5One evaluator, three readers
The reporting layer is designed around the fact that the same evidence is interpreted differently by different audiences.
Business: What does this mean for the use case?
Risk / compliance: What evidence supports the decision and what remains uncertain?
Engineering: Which cases, failure modes and metrics explain what should improve?
I did not want three separate truths. The same run evidence should support all three views, with the interface changing the interpretation layer rather than changing the underlying record.
SoteriaPrecomputed demo · SYNTHETIC RUNThe implementation turns the trust model into an API-first evaluation lifecycle with task-specific contracts, evidence persistence and meta-evaluation.
03.1A configurable core for three task families
The v2 service supports three core task families: RAG, entity extraction and structured extraction. Each task has a native contract rather than being forced through one generic score schema.
RAG can combine deterministic metrics with semantic required facts. Entity extraction validates exact and partial matches, relationships and type-level behaviour. Structured extraction uses schema/STTM-style definitions to validate records, keys, field types, accepted values and priority-weighted accuracy.
The common layer is the run lifecycle, evidence model, failure vocabulary, review path and reporting - not a fake universal metric.
SoteriaPrecomputed demo · SYNTHETIC RUNRun information
This is a saved demonstration dataset, not a live evaluation.
03.2Run lifecycle and evidence persistence
A client submits a typed run with an idempotency key. Boundary validation checks the task contract and protected payload before the run is accepted. Development can execute inline, while the production configuration includes a Redis-backed worker path.
Case evaluation then combines deterministic metrics, optional judges, required facts, risk taxonomy and human-review routing. The system persists cases, judge calls, evidence, costs where pricing is known, certificate state and drift information.
The final run response keeps the four states separate: operational status, evidence status, quality decision and publication status, alongside task-specific scores and limitations.
SoteriaPrecomputed demo · SYNTHETIC RUNRun information
This is a saved demonstration dataset, not a live evaluation.
03.3Judge abstraction and model experiments
The judge layer is model-decoupled. During development I experimented with Llama 4, GLM-5, GPT-5.5, Mistral, Qwen 3.6, MiniMax and Claude Sonnet.
GPT and Claude were the most reliable overall in my testing, but that is not a universal model benchmark and the system is not tied to either one. The important design choice is that judge identity, prompt variant and evidence remain visible and configurable.
Standard evaluation can vary prompt variants against one judge, audit mode can compare multiple judges when configured, and a quick mode intentionally trades depth for speed.
03.4Perturbation, bias and calibration
The meta-evaluation layer perturbs the conditions around the judge rather than only re-running the same prompt. Examples include context reorder, answer swap, noise or typo injection and paraphrased golden answers.
Bias checks cover position, self-preference, verbosity and style. The output is not forced into "stable" or "unstable" when the evidence is incomplete; the system supports clear, detected, inconclusive and not-applicable states.
This turns evaluator reliability into evidence that can be inspected alongside the task score.
SoteriaPrecomputed demo · SYNTHETIC RUNHow dependable is the evaluation?
Run-level calibration checks the judges. It does not approve the system's answers.
Illustrative calibration. Review limits remain documented.
Consistency across prompt variants.
A consistency measure, separate from observed agreement.
10 paired required-point decisions.
Input changes · 20 comparisons
Reorders context passages, or sentences within one passage, to test sensitivity to their order.
10 comparisons · mean score change 1.0 points · largest 1.0Adds a small typo to the question to test sensitivity to minor input errors.
10 comparisons · mean score change 2.0 points · largest 2.0Bias checks · both configured judges
Position bias
Checks whether placing an answer first or second changes its score.
Verbosity bias
Checks whether longer wording receives a higher score when the content stays comparable.
03.5Drift and human review
Soteria also compares compatible runs instead of treating every score as isolated. The drift engine can compare paired metrics, surface significant changes, detect new failure modes and apply alert rules.
Human review is used where semantic evidence remains inconclusive. That is important because the goal is not to remove humans from evaluation; it is to reserve human attention for uncertainty the system cannot resolve safely.
03.6The correction arc
One of the more useful parts of building Soteria was allowing the evaluation system to expose weaknesses in its own implementation. Internal audits found issues around task contracts, perturbation pairing, required-fact semantics and operational proof.
The response was not to hide those findings behind a better-looking dashboard. The architecture and validation were tightened, replacement evidence was produced, and unresolved deployment or adapter gaps remained explicit.
That remediation history fits the core thesis: confidence should follow evidence, not the other way around.
SoteriaPrecomputed demo · SYNTHETIC RUNScope, method, and limitations
Task contracts · perturbation pairing · required-fact semantics · operational proof
Deployment or adapter gaps remain explicit.
03.7Runtime boundary
The system currently runs locally and in containerized/Docker form. Provider access can be configured through approved cloud endpoints, but I do not present templates or deployment runbooks as proof of a live cloud product.
The current state is internal vetting. There is no confirmed live project adoption yet.
The strongest result is the evaluation model itself: scores are now connected to evidence, risk, reliability, drift and review rather than standing alone.
04.1What exists today
A reusable service boundary for RAG, entity extraction and structured extraction.
Deterministic evaluation where possible, with model judges reserved for semantic interpretation.
Risk-weighted required facts with evidence quotes, exposure and inconclusive/HITL handling.
Meta-evaluation using prompt variation, perturbations, judge agreement, bias checks and calibration.
Drift comparison, evidence persistence, audience-specific reporting and publication-aware status.
Local and containerized runtime.
SoteriaPrecomputed demo · SYNTHETIC RUNFurther review is needed before a release decision.
The sample includes incomplete answers, a wrong deadline, and an unresolved legal exception. Review those cases before approving use.
Inspect the casesConfidence in the evaluation. This is separate from release readiness.
Required answer points
10 checks from the supplied checklist. These are expected details in an answer.
What do these labels mean?
Covered: The answer includes the required detail. Partial: It includes only part of it. Missing: It omits the detail. Contradicted: It states a conflicting detail. Inconclusive: The judgments do not resolve the evidence.
Required points come from the supplied checklist, based on the reference and source context. They do not represent every possible policy requirement.
How dependable is the evaluation?
Run-level calibration checks the judges. It does not approve the system's answers.
Illustrative calibration. Review limits remain documented.
Consistency across prompt variants.
A consistency measure, separate from observed agreement.
10 paired required-point decisions.
Input changes · 20 comparisons
Reorders context passages, or sentences within one passage, to test sensitivity to their order.
10 comparisons · mean score change 1.0 points · largest 1.0Adds a small typo to the question to test sensitivity to minor input errors.
10 comparisons · mean score change 2.0 points · largest 2.0Bias checks · both configured judges
Position bias
Checks whether placing an answer first or second changes its score.
Verbosity bias
Checks whether longer wording receives a higher score when the content stays comparable.
04.2What remains unproven
Soteria has not yet been integrated into a live project. I do not claim production deployment, compliance certification, measured cost savings or reduced manual review time.
The adoption milestone I care about is straightforward: a real project configures Soteria instead of rebuilding evaluation and actually relies on the resulting evidence.
04.3What I learned
The project changed how I think about evaluation. A single number is easy to generate and easy to over-trust. Provenance, evaluator reliability, deterministic checks, uncertainty and supporting evidence are what make that number useful.
I also became more comfortable with "inconclusive" as a valid engineering outcome. False precision is often more dangerous than admitting that the available evidence is not enough.
Evidence before confidence.
04.4Next
The next meaningful step is adoption, not another metric. I want a real project to configure the evaluator, run it in its workflow and use the evidence to make or review a decision.
If that happens, the operating model could be a centrally hosted service or separately configured containers for different projects or clients. The deployment shape should follow the trust and isolation requirements of the environment.