Skip to content
Studio previewSelf-host Studio locally.Run it
Kortyx
Open Kortyx on GitHub
Open navigation

Run evaluations without Studio

Updated 4 hours ago • October 18, 2026

Run locally without Studio

Export the existing createEvals instance as evals (or the default export) from an application module such as src/evals/index.ts. Use a dedicated eval module that initializes the agent and suites without starting your HTTP server. TypeScript and JavaScript entries use the CLI's existing module loader and nearest tsconfig.json path aliases. The loader stays active for lazy workflow imports until execution finishes. Keep initialization synchronous; do asynchronous identity setup inside createEvals.setup.

Add a script to your application's package.json:

{ "scripts": { "eval": "kortyx evals run --entry ./src/evals/index.ts" } }
pnpm eval pnpm eval --suite product-ambiguity --case choose-blue --repetitions 3 pnpm exec kortyx evals list --entry ./src/evals/index.ts pnpm eval --suite product-ambiguity --concurrency 2 --json > eval-results.json

The command calls the exported instance's run() directly. Your existing setup, permission binding, custom executor, interrupt responders, cleanup and code judge all run in the application process. No Studio, consumer HTTP endpoint, target configuration or Studio key is required. Semantic criteria require a code judge; interaction and structured-output checks alone require no judge. Workflow models, tools and authentication still need their usual application configuration.

The CLI loads .env then overlays .env.local before importing the entry. In a monorepo it loads defaults from the Git or pnpm workspace root down to the app directory, with nearer files taking precedence. Existing shell variables always win. To select other files, repeat --env; later files override earlier ones:

pnpm eval --env /private/development.env --env /private/eval.env

For a one-line package script without repeated flags, remember paths in a gitignored .env.evals.json in the app or workspace root:

{ "envFiles": ["/private/development.env", "/private/eval.env"] }

Keep the profile and referenced files owner-only (chmod 600). Relative paths in the profile resolve from its directory. The nearest profile replaces default env loading; discovery stops at a Git or pnpm workspace boundary. Explicit --env flags replace both the profile and defaults, and resolve from the invocation directory. Missing configured files fail before loading the app; credentials are never printed. The CLI handles loading directly, so no launcher script or shell exports are required. Package scripts in monorepos can forward with pnpm --filter my-agent eval.

Model credentials remain application-owned, for example OPENROUTER_API_KEY for an OpenRouter app judge. Studio's judge settings are not used by this command.

Omitting --suite runs all configured suites in definition order. --case accepts space-separated case IDs and requires --suite. --repetitions and --concurrency override the SDK's run settings; omitted flags preserve them. --export NAME selects another named instance instead of evals/default.

Interactive terminals show an animated suite progress bar, one live row per active attempt, elapsed time, and the actual setup, workflow, interrupt-response, judging and cleanup phases. Every completed step and case gets its own result, followed by a colored suite verdict and counts. Long calls keep animating; concurrent attempts remain distinct. The bar measures completed attempts, not estimated model completion. Application logs/warnings are preserved above the live display. Piped output is plain text with per-case start and step results; --no-color or NO_COLOR disables color/live progress. Explicit --color forces the interactive report, including when NO_COLOR is set. --json always disables the terminal UI. --json replaces the report with one JSON object containing schemaVersion: 1, status, aggregated counts, and full SDK runs (including observations). Keep application logs off stdout when consuming JSON. Configuration/import failures use stderr.

Exit status is 0 when all selected suites pass, 1 for failed/errored runs or configuration errors, and 130 for cancellation. Ctrl+C forwards an abort signal, allows SDK cleanup and stops later suites. Cancellation remains cooperative: app code must honor its signal. Results are returned in the terminal/JSON; this local command does not save eval records to Studio. Agent telemetry, if configured by the application, continues to follow its existing settings.

Next: Connected evaluations if you need Studio to save and display the results.

Good to know: Local execution returns results in the terminal. To save and compare evaluation records in Studio, use connected evaluations and the results guide. Your configured agent telemetry continues independently.