Read and compare evaluation results

Updated 4 hours ago • October 18, 2026

Open Evals and select a saved run. Start by checking whether each conversation executed, then read the checks applied to its answers.

Use the interface

Open Evals → Suites to browse the applications and cases registered by your consumer. Select a suite to inspect its definition in a drawer, or open it as a full page. Choose cases, repetitions and concurrency, then start the run.

Evals → Runs lists saved runs. A run shows live progress and final case outcomes. Open a case to read evaluation reasons first, then expand conversation and debugging details. Inspect workflow opens the existing workflow drawer on top; closing it returns to the case. Page URLs, filters and selections survive reloads and browser Back/Forward.

Use Compare on a saved run and select a baseline from the same target/suite. The candidate is the run you are inspecting. The baseline is another saved run you choose as a reference; it is not an expected answer or a newly executed workflow. Comparison reads their saved grades without calling a judge or rerunning either application.

Cases are matched by case ID. For comparable cases, Studio compares the share of passed attempts, including repetitions:

For example, 1 of 3 passed in the baseline and 3 of 3 in the candidate is Improved. A provider error is Incomplete, not a behavioral regression. Open a comparison case to inspect both sides' attempts, outputs, criteria and workflow links.

Actor identity, external data snapshots and prompt versions are not automatically pinned. Tool results can change even when the recorded inputs are identical. Treat comparison as observed pass-rate differences, not proof that a prompt change caused them. Rerun the same suite with stable actor/data and review the actual evidence when testing a prompt change.

Good to know: Comparison reads saved grades without running your app or judge again. To test a fresh change, start another evaluation, then compare the same cases and judge settings.

Read results and compare

Passed/failed describe assessed quality. Errors describe execution or grading problems; cancelled and interrupted work do not count as successful quality evidence. Progress distinguishes suites finished from recorded attempt outcomes. A resolved interrupt does not establish successful continuation.

Compare equivalent cases, attempts, judge settings, prompt versions, and environments. Open case/attempt details for captured evidence instead of inferring quality from the aggregate status.

Grouped evaluation runs

The default Runs list shows one evaluation per trigger. A parent evaluation owns one or more suite executions for a single application/environment. Open a row to see every suite and its live progress, then open a suite to inspect cases and assessment/trace evidence. Older suite runs remain visible with their original links. Suites remains the definition/history view.

The Run evaluations drawer offers All suites or Selected suites. Launching from a suite page preselects and expands that suite. Expand any selected suite to choose its conversations independently. Suite checkboxes select or clear every conversation and show a mixed state for partial selections. Selections and expanded suites survive reloads and browser history. Without a project judge default, the drawer selects Studio when available, otherwise App; unavailable judges remain disabled with a reason. Repetitions (1 to 20) and concurrent attempts across the evaluation (1 to 20) apply to the selection. Each suite is limited to 100 attempts; one evaluation is limited to 1,000 attempts. Missing App judge configuration is explained beside the disabled option.

The Studio API snapshots all selected suite revisions and the judge before atomically saving the parent and suite jobs. A worker claims all suite jobs together and schedules attempts round-robin across suites, sharing the configured concurrency budget. A slot includes setup, execution, cleanup, and grading. Consumer apps must use an SDK that advertises attempt scheduling; older endpoints receive an upgrade message before a multi-suite run is queued. Custom worker stores must return all sibling suite leases together from claim; independent single-suite claims cannot enforce an evaluation-wide budget. Cancelling the parent cancels queued suites and requests cooperative cancellation of running suites. Cancelling one child suite stops its queued attempts and drains its active attempts through cleanup while other suites continue. A lost lease or uncertain transport outcome stops new dispatch without replaying attempts. Finished suites remain available; cancellation never rolls back tool side effects. A run remains Running until every suite is terminal. Execution/grading errors, behavioral failures, and cancellation stay distinct.

get returns aggregate counts and per-suite case/criterion pass-fail outcomes. --include-content additionally returns definitions, observations, reasons and execution evidence with best-effort credential redaction. A positional suite ID is still accepted by runs start, and now creates a single-suite parent. wait supports grouped evaluation IDs; use get to inspect legacy suite runs.

--wait and runs wait exit 0 for Passed, 1 for failed criteria, 2 for execution, grading, or waiting errors, and 130 for cancellation. A wait timeout does not cancel the evaluation. Without --wait, success only means the run was accepted. The CLI does not automatically retry POSTs. Use --idempotency-key to reuse an identical request after a trigger retry; reusing the key with changed selection, revisions, execution settings or metadata returns a conflict. Run creation is atomic, so failed validation never leaves a partial selection queued.

Eval run costs

Run history and each run's case table show model costs. The run summary and case inspector provide a Workflow / Judge / Total breakdown. Workflow cost automatically follows native Kortyx executions across all runtime sessions used within an attempt, including setup, cleanup, child workflows, retries and resumed turns. defineSuite and existing execution hooks need no changes. An application can update a record in session A and verify it in a fresh session B; both sessions stay independent and their recorded generations count once for the attempt. The attempt keeps its own identity, and unrelated work in a reused session is excluded. Project and environment boundaries still apply. Enable normal agent telemetry to record these calls. toolExecution.emit: true supplies tool evidence to the judge; it does not itself enable billing telemetry.

Upgrade Studio before upgrading the consumer's agent and telemetry SDKs together: progress and saved results now carry optional attempt identities and runtime execution associations. Older saved results retain session-based attribution. Attribution is automatic within the consumer process; custom executors that call another service do not propagate it across HTTP or queue boundaries. Such calls remain unavailable unless that integration carries attribution. An associated execution with no recorded generation keeps the subtotal partial or unavailable.

createEvalJudge captures provider usage separately from the generated verdict. Both app-owned and Studio-owned judges report it, including a paid call whose verdict fails validation. OpenRouter reports actual charges; BYOK uses upstream inference cost when supplied. Other model usage uses Studio's effective model rate cards. These calculated charges are marked estimated in the tooltip. No extra cost-related environment variable is required. Custom judges may report usage through EvalGradeInput.onUsage; without it their cost remains unknown.

A + after a displayed amount means a known subtotal, with more cost possible. A dash means unavailable, including absent telemetry or prices. Running or unfinished attempts stay partial; currencies are combined only when compatible. Historical workflow costs can be recovered from existing telemetry, but historical judge charges without recorded usage cannot be recovered. Costs cover recorded model calls, excluding infrastructure and external tool fees. They are not an independent reconciliation of a provider invoice. Cancellation can leave billing evidence incomplete when a remote call finishes after the executor disconnects.

Local kortyx evals run --entry … still executes without Studio and does not save a Studio record. Use the registered Studio execution endpoint to save runs and view their combined costs in Studio.

Timing and cost

Queue time precedes execution. Setup, execution, judging, and cleanup describe distinct phases; active duration can keep growing before completion. Workflow and judge model costs are separate, and total cost can remain partial when usage or pricing evidence is incomplete. Infrastructure and external tool fees are excluded. See Metrics.

Cloud administrators register endpoints and provider connections through Cloud evaluation settings. Self-hosted operators use evaluation service configuration.

Next: Configure judging or look up execution settings.