Run your first evaluation
Updated 19 days ago • October 18, 2026
You need a working Kortyx agent, access to its application server, and a Studio project. Use a read-only example your test account can complete. This walkthrough adds one case and an authenticated endpoint to that app.
The runnable catalog endpoint example provides a complete starting point. Your application executes the workflow and its permission checks; Studio starts the test and saves its results.
1. Define one case
Create src/evals/suites.ts in your application:
Choose a product your test account can actually read. Author the scenario from known data; do not discover random products during startup. A suite is JSON data, not authentication code. Add more cases to this array as you learn about failures.
For this criterion, enable evidence on the workflow's existing useReason call:
Without emitted tool results the judge cannot establish that the reported price came from the lookup. This flag captures evidence; normal agent telemetry is configured separately if you want workflow costs and workflow inspection links.
2. Create the eval instance beside your agent
Create src/evals/index.ts. Export the eval instance without starting a server:
This minimal instance is sufficient when your workflow needs no caller-specific
context. If your tools depend on an authenticated user, add setup and execute
before running it:
authenticateTestActor and withApplicationRequest above are your application's
existing helpers, not Kortyx exports. The first obtains a test identity; the
second binds the same permission checks, request-scoped tools and context that
ordinary requests use. Adapt their arguments to your app. Reuse that existing
logic rather than implement a second permission system for evals.
Configure the test account in the application service. It may use a username and password, a refreshable token, or another identity mechanism your app already supports. Studio neither obtains that token nor assigns the account's roles. The eval service key authorizes initiation; it does not grant domain data access. Never put user tokens or passwords in suite definitions, progress or judge inputs.
Add teardown only if setup acquires resources that need releasing. It is not necessarily a sign-out operation. See application authentication and execution for cancellation, typed callbacks and interrupt responders.
3. Mount and deploy the application endpoint
For Next.js App Router, create src/app/api/evals/route.ts:
Its URL is https://your-app.example/api/evals. Keep a single handler instance
so its per-process active-run limit is shared across requests.
For a Hono Node server, mount the same Web handler before starting the server:
For Express/Fastify or another Node server, use the framework's Web Request/ Response bridge. It must preserve the request abort signal, stream the returned body, and cancel its reader when the client disconnects. Do not buffer the full POST response or redirect it through the interactive chat route. See the runnable Node adapter for a complete application using Hono's Node server bridge.
Generate a separate service key, store it privately, and supply it to the app:
The HTTP route explicitly reads EVAL_SERVICE_KEY; the local eval module does not need it. Kortyx does not automatically
read application env variables or enable routes. If your application has its
own eval feature flag, enable it in the deployed application. Ensure the server
and reverse proxy expose the exact GET/POST path in the target configuration.
Restart/deploy the app with its service key, test-account configuration and normal workflow model/tool configuration. A healthy container does not prove that the route was registered.
4. Register the application on Studio
In Cloud, open Settings > Evaluations for the intended project and register the deployed endpoint and its service key. Follow Cloud evaluation connections.
For self-hosting, ask the operator to register the application on the API service. This requires configuration on the API host, not your application host.
Refresh suite discovery in Evals. The Catalog lookup suite should appear. Publishing workflow topology does not register evaluation suites.
5. Configure one judge
Select Studio judge for this walkthrough. Cloud users connect a supported provider through Settings > Providers. Self-hosted operators configure the judge on their API service. Then choose a model that can return the structured verdicts your evaluation requires.
The judge's model checks the result. Your agent continues using its own application model. Configure judging explains the alternatives.
Configuration belongs to different services
6. Diagnose setup before executing
Refresh discovery and confirm the deployed suite appears under the intended application. If it does not, check the exact endpoint path, reachability, and service key. A 404 usually means the route is absent or the URL is wrong; a 401/403 needs an authentication check.
A discovered suite confirms the endpoint answered discovery. A real run is still needed to test the workflow, provider credentials, tools, and test-user access. Self-hosted connections can also use the CLI setup checks.
Good to know: Discovery checks the connection; one real case checks execution. A successful conversation can still fail its judge criteria, so inspect both parts of the saved result.
7. Run and verify the actual workflow
Open Evals, select Catalog lookup, choose Studio judge, and run the case once. Wait for its saved final result. A queued record means Studio accepted the request; execution and judging happen afterward.
Open the case and read the answer, criterion verdict, and supporting evidence. Reload the page and confirm the saved result remains available. A quality failure can show that the integration works and the application needs improvement. An authentication, provider, or execution error needs resolving before you can assess answer quality.
Next: Read and compare results or run evaluations directly in your terminal.