Configure evaluation judging

Updated 4 hours ago • October 18, 2026

A judge checks a conversation's answers against the requirements in your suite. An application judge uses a check registered by your app. A Studio judge uses a model configured for Studio.

Choose a judge

When launching a suite, select Studio judge or an available App judge. Studio saves that choice with the run. Missing judge configuration produces an error; it does not silently switch judges.

In Cloud, connect a supported provider in Settings > Providers. Self-hosted operators configure the Studio API's judge. Your application's workflow model remains configured in the app.

Judge criteria and evidence

Write checks against the answer and evidence your app actually produces. If a check requires a successful product lookup, emit the relevant tool result with toolExecution.emit: true. The judge cannot verify a lookup from an unsupported claim in the answer alone.

Use createEvals.defaults.evidence for default evidence selection and suite.evidence for case-suite overrides. Keep credentials and private account tokens out of suites and evidence. See the SDK evidence guide for the supported fields.

Good to know: Tool evidence and cost telemetry serve different purposes. toolExecution.emit: true supplies judge evidence; normal telemetry records the calls used for workflow cost attribution.

Read a verdict

Inspect each criterion's result and reason in the saved run. A completed workflow can fail a quality check. A judge request that fails, times out, or returns an invalid verdict is an evaluation error.

Next: Read and compare results.