Skip to content
Studio previewSelf-host Studio locally.Run it
Kortyx
Open Kortyx on GitHub
Open navigation

Run evaluations through Studio

Updated 4 hours ago • October 18, 2026

These commands require a supported Studio project key. Cloud CLI account login is not implemented; use the Cloud UI for connected evaluations. Local terminal evaluations remain available for your application.

Eval suites and post-deployment CI

The CLI uses the same Studio API, project scopes, application targets, worker, and PostgreSQL records as the browser. It does not import your agent or obtain an end-user token. The application handles its test actor in createEvals.setup.

Configure the consumer endpoint and Studio targets first. Your connection key needs studio:read for discovery and result inspection, plus eval:run for starting and cancelling. The SDK telemetry write key cannot run evals. The application's eval service key stays on the Studio API server.

Diagnose deployment wiring

kortyx studio evals doctor --connection staging --target catalog --suite catalog-smoke kortyx studio evals doctor --connection staging --target catalog --judge app --json

Doctor checks read/execution scopes, matching targets/environment, authenticated consumer manifest/suites and configured judge compatibility. It uses discovery GET only; it starts no workflows, model calls or saved runs. Consumer discovery wrappers may perform their own authentication. It gives safe failure categories and remedies; older APIs may return only generic unavailable diagnostics.

--target, --environment, --suite and --judge narrow the checks. Connection environment defaults apply. Use --api-url and --api-key-env instead of --connection for direct access. --json prints one report with schemaVersion: 1, status and checks; exit 0 means setup checks passed, exit 1 means a failure. Each check has id, status (passed, failed or skipped), message and optional remedy. Provider credentials, test-user tool permissions and worker execution still need a real run. Follow the complete setup recipe.

Discover and run

pnpm exec kortyx studio evals suites list --connection staging --json pnpm exec kortyx studio evals suites get product-ambiguity \ --connection staging --target catalog --include-content --json pnpm exec kortyx studio evals runs start product-ambiguity \ --connection staging --target catalog --case choose-blue \ --repetitions 3 --concurrency 1 --json

Local Studio uses --connection local and its managed credentials. A remote profile references an environment variable containing a project-scoped Studio key:

pnpm exec kortyx connections add staging \ --api-url https://api.example.com \ --studio-url https://studio.example.com \ --api-key-env KORTYX_STAGING_STUDIO_KEY --environment staging

Profiles save URL locators and the variable name, never the remote key. Inject the key through your shell or CI secret store. Alternatively, pass --api-url and --api-key-env together for a direct connection using IDs; do not combine those flags with --connection or KORTYX_CONNECTION.

The connection environment filters discovery and history by default. Override it with --environment. If a suite exists on multiple targets, select --target. Repeat --case for multiple cases; omit it to run every case. The limits are 20 repetitions, concurrency 20, and 100 total attempts per queued run.

Start discovers the current suite revision and enqueues one request. It returns immediately with schemaVersion: 1, id, status: "queued", and studioUrl (null when the connection has no browser URL). Success means the request was accepted, not that its cases passed. A changed suite returns HTTP 409; no retry or second run is performed automatically. Refresh discovery and retry explicitly.

Good to know: A queued response means Studio accepted the request. Check the eventual saved result before calling it a pass; automation can enqueue work without waiting for grades.

Inspect and cancel

Runs started through the CLI default to the configured Studio judge. Add --judge app to use the consumer's code judge. The API records the selected judge and its version; missing configuration fails before enqueuing, without fallback.

pnpm exec kortyx studio evals runs list --connection staging --json pnpm exec kortyx studio evals runs get RUN_UUID --connection staging --json pnpm exec kortyx studio evals runs get \ https://studio.example.com/evals/runs/RUN_UUID --json pnpm exec kortyx studio evals runs cancel RUN_UUID --connection staging --json

History returns the latest 100 records in the project before the CLI applies an environment filter. Run detail includes counts, case/step statuses and criterion IDs/verdicts. Suite definitions, observations, reference facts and grade reasoning require --include-content. Recognizable credentials and resume tokens remain redacted on a best-effort basis, including when content is requested.

Pasted URLs must match a configured connection; redirects are not followed. Cancellation is cooperative and does not roll back application side effects. A run may finish before cancellation arrives. Inspect its final saved status.

Next: Automate evaluations after deployment.