Operate evaluation workers

Updated 3 hours ago • October 18, 2026

Persistence and execution lifecycle

Apply packages/telemetry-db/drizzle/0005_eval_runs.sql through the regular migration runner before starting the updated API. Each queued record stores its suite and revision, selection, target and environment. PostgreSQL row locking allows a worker replica to claim a record once. A lease and heartbeat track active execution. Progress is saved incrementally; the final result contains public observations (including emitted execution events), expected behavior, optional references, grader version, reasons, and evidence. Studio receives scoped SSE change notifications backed by PostgreSQL LISTEN/NOTIFY, then reads the saved result. Runs, case drawers and history update on changes; completed runs also receive late workflow billing updates. Live mode is on by default for eval views and can be paused with ?live=false. The connection pauses while the tab is hidden or offline, reconciles on reconnect, and uses a 30 to 60 second refresh fallback during an outage. There is no periodic polling while connected. A browser reload does not terminate the run. Caller-visible observations may contain business data and require the same access and retention policy as ordinary Studio session data.

A consumer disconnect, timeout, or worker crash is an execution error rather than a failed behavioral grade. An expired lease is marked unknown/error and is never automatically retried: replaying application workflows can repeat side effects. Operators should inspect consumer execution before explicitly starting another run. Cancellation wins the final-storage race when its request is already saved. The worker actively closes an aborted response reader to signal the consumer; custom setup, tools and cleanup must honor the signal. A worker shutdown does not promise rollback or prove remote cleanup.

Studio supports saved-run comparison, including per-case outcomes, criterion verdicts, repeated attempts and workflow inspection through stacked drawers. Comparison describes observed differences. Its context notice identifies missing actor/data/prompt identity; it does not establish why an outcome changed.

Suites are authored by the application. Prompt selections can be tested through the current prompt evaluation controls.

Good to know: A browser reload does not end a saved evaluation. Inspect result status and timing before starting another run; the CLI result commands can inspect and cancel through a supported connection.

Concurrent execution recovery

Studio isolates each case and repetition: a refused attempt becomes an Error while siblings continue, and completed evidence remains inspectable. Compatible consumers queue accepted jobs, deduplicate admission, honor bounded Retry-After backoff and let Studio reconnect by event cursor. Cancellation is explicit and cooperative; capacity remains occupied through cleanup. Retrying, Reconnecting and Status unknown are visible in case inspection. Inspect an unknown attempt before running it again.

createEvalRouteHandler uses a memory execution store by default. For replicas and restart recovery, pass createPostgresEvalExecutionStore({connectionString,namespace}) from kortyx, sharing the namespace, service key, suites and capacity settings. Defaults allow one evaluation group, twenty attempts and a thousand queued jobs. Forward GET, POST and DELETE with all headers and query parameters. Keep the host alive while accepted execution runs. For planned retirement, use the handler's drain() to finish accepted local/queued jobs without aborting. Keep HTTP and the store open until Studio saves results; close() remains explicit abort. With one consumer, drain Studio workers first while the consumer still accepts their later cases, wait for durable Studio outcomes, then drain the consumer. Overlapping consumers need a compatible replacement sharing durable storage, service key, suites and limits. Upgrade agent/runtime together; custom stores need scoped queued and atomic claimQueued capabilities for drain. See retire evaluation workers for ordering and failure handling. Lost running leases are never replayed automatically; queued jobs can recover. External side effects still require application-owned idempotency. Upgrade SDK consumers first, then API and Studio together; older consumers retain legacy streams.

Next: Back up and upgrade Studio.