Configure evaluation services

Updated 3 hours ago • October 18, 2026

This configuration belongs to the self-hosted API service. In managed Cloud, use evaluation connections instead. First expose your application's authenticated endpoint using the first-evaluation tutorial.

For planned deployments, choose a worker topology and retire evaluation workers before replacing services. Removing a target or seeing an empty queue does not confirm that active evaluations have finished.

Register applications on the Studio API server

Set KORTYX_EVAL_TARGETS_FILE to a secret-manager supplied JSON file:

The server fixes target URLs and credentials; the browser cannot submit an arbitrary URL or token. HTTPS is required unless allowInsecureHttp: true is explicitly set for a local target. Targets, history, details and cancellation are scoped to the Studio API key's organization and project. Execution also requires eval:run and an allowed project environment. Reading uses the existing studio:read scope. The current browser proxy supports local none and deployed basic Studio auth modes. Suite execution uses the existing server-side Studio key; it does not require a new login system. OSS bootstrap grants execution permission by default. Existing installations should rerun bootstrap with the same stored keys and pepper. A registered consumer target and a configured Studio or App judge are still required.

Where application metadata comes from

The target file supplies the application ID, display name, endpoint and environment. They are operator-defined labels, not inferred from model calls or the agent's telemetry service name. The application's manifest supplies the suite IDs, case definitions, revisions, named handlers and optional code judge.

Studio copies targetId, targetName and environment into each saved run. The application filter combines currently configured targets with labels from saved history. Removing a target stops discovery and new execution, but its historical runs and application filter entry remain. Suite discovery shows currently registered suites; saved runs retain the definition used when they started.

Use stable target IDs. Renaming a target changes its label for new runs; existing records keep the saved label. The filter groups by target ID and displays a label from the current configuration or saved history. Two environments can be registered as separate targets with distinct IDs and clear names.

Credentials have separate jobs

The service key authorizes starting evals; it does not impersonate an application user. The consumer's setup and execute callbacks bind that user through the same permission path used by ordinary application requests.

Studio judge configuration

Studio-triggered runs use the project judge default, or Studio judge when unset. Configure its model and provider key on the Studio API backend. A code judge provided to createEvals makes App judge available for run overrides and project defaults. If hosted judging is not configured, the run drawer explains what is missing and allows selecting an available App judge. It never silently changes selection.

The app executes the complete scripted scenario and resolves interrupts. With Studio judging, it returns captured evidence awaiting evaluation, then the Studio worker grades that evidence and stores the final results. App judging runs inside the app at each step. The browser never calls a model directly. The app needs no Studio grading key or model configuration for the hosted path.

Enable the backend judge on the API service:

The default server adapter uses Kortyx's OpenAI provider and its Responses API. Set KORTYX_EVAL_JUDGE_API=chat-completions for compatible Chat Completions endpoints. A custom HTTPS endpoint must support the selected API and structured JSON verdicts. Custom API hosts can instead inject any EvalJudge through createApiApp({ ..., evalJudge }), including a judge built with another Kortyx provider. Leaving the model unset disables hosted judging; app-side judging and ordinary Studio execution remain available.

For OpenRouter, use its model slug and server-side API key:

Choose a model/provider combination that supports structured outputs. The OpenRouter chat endpoint uses Kortyx's native OpenRouter provider, including its reported billing usage. A live OpenRouter account is needed to verify the configured model and provider route.

Docker Compose and CLI-generated Studio stacks pass these variables only to the API container. For a CLI-managed stack, add them to the existing owner-only <studio-home>/.env and restart with kortyx studio start --home <studio-home>. CLI startup preserves these private settings. For other deployments, use the API service's normal secret configuration. Provider credentials are never returned by judge discovery or included in run results.

The browser and CLI use the existing project key with studio:read and eval:run to enqueue runs. The app's eval service key stays in server target configuration. Project and environment access is enforced on the Studio API. Provider credentials are never sent to the app or browser.

Suite discovery advertises a code judge when one is configured and the consumer's Studio judging capability. The run drawer stores its selection in the URL. The API pins the selected judge identity in the saved request; the worker rejects a changed Studio judge before starting the app. The consumer similarly rejects a changed code judge. Missing configuration, provider failures and malformed verdicts produce errors instead of passing scores. Captured cases display Awaiting evaluation until Studio has graded them. Cancellation covers execution and grading.

The optional /v1/studio/evals/judge endpoint remains available for explicit createStudioEvalJudge calls from code. It authenticates each request using a project Studio key, checks its environment, pins identity and bounds input size, concurrency and grading deadlines. Ordinary Studio runs grade within the worker and do not call this endpoint. Saved-run regrading and editable evaluator libraries are not included in this release.

Good to know: The app endpoint credential and the judge provider key serve different requests. Review credential roles when discovery succeeds but execution or judging fails.

Container configuration

Mount the private target file on every Studio API replica and set the variable on the API service, using the container path. The browser Studio service does not need the file or application service key. For Docker Compose:

Use this as a deployment override alongside the normal Compose file. For a CLI-managed local installation, keep it outside generated state and apply it when starting/recreating services: studio start regenerates its base Compose file. Retain the existing environment file and project name, and rerun db-init to update the existing Studio key scopes before recreating the API.

The bootstrap output provides the organization and project IDs for target registration. Add an allowed project environment before using it in a target. A container cannot reach a host application at localhost; Docker Desktop typically uses host.docker.internal, or use an application on the same Docker network. Local HTTP targets need allowInsecureHttp: true; remote targets use HTTPS.

Targets are loaded when the API starts. Restart the API after changing the target file. Missing targets leave Suites empty; an unreachable endpoint shows the target unavailable. Run controls require eval:run on the existing key. Read-only keys can still inspect saved results. An endpoint's service key is a separate credential from the Studio key; neither contains the test user's token.

See Conversation Evals for suite design, typed setup/responders, Auth0 integration and grading boundaries.

Next: Run and verify an evaluation.