AI assistant evals

Studio's AI assistant is not one thing you can unit-test. It is a model you do not control wrapped in scaffolding you do: a system prompt, a set of tool schemas, the translations that turn a validation error back into an instruction the model can act on. Change any of that and the only honest question is whether the assistant got better — which a test suite cannot answer and a hunch should not.

packages/studio/evals/ is the harness that answers it. Reach for it whenever you touch ai-system-prompt.ts , ai-tools.ts , or the few-shot examples.

The invariant

The model is held fixed; only the scaffolding is iterated. That is the whole design. Because the model, the tasks, and the graders stay constant between runs, a difference in pass-rate is attributable to the diff you just made. Change two things at once — the prompt and the model, say — and the run tells you nothing.

The second half of the design is that the harness drives the real production loop: the actual runAgentLoop , the actual @jxsuite/ai tool registry, the actual system prompt built by buildSystemPrompt . The only substitution is the LLM client, where the fake test client is swapped for a real OpenAI-compatible one. There is no parallel "eval mode" implementation to keep in sync, so a scaffolding change is exercised exactly as users will meet it.

Golden tasks

Each task in evals/tasks/*.json is one isolated, unambiguous specification:

{
  "id": "counter-button",
  "prompt": "Add a button that increments a counter, and show the current count above it. Wire up the reactive state.",
  "tags": ["intent", "state", "binding"],
  "intent": [
    "declares numeric state for the count",
    "a button increments the count on click",
    
  ],
  "initialDoc": {
    "tagName": "counter-widget",
    "state": { "count": { "type": "integer", "default": 0 } },
    "children": []
  }
}

initialDoc is the document the assistant starts from — every trial gets a fresh tab, so nothing leaks between runs. intent records the human success criteria: it is what you read transcripts against today, and the hook for a future LLM-as-judge grader. Keep the suite stable. Tasks are the benchmark, and a benchmark that moves with the code cannot measure it.

Running it

# Whole suite, k=3 trials per task (the default)
OPENAI_API_KEY=sk-… bun run eval

# One task, single trial — a smoke check
OPENAI_API_KEY=sk-… bun run eval --tasks counter-button --k 1

OPENAI_BASE_URL and OPENAI_MODEL (default gpt-4o ) are optional and mirror how the server proxy resolves its config. Without OPENAI_API_KEY the CLI exits 2 immediately — the harness calls a real model by design.

Each run writes a timestamped directory under evals/runs/ , which is gitignored:

report.md — the human summary: mean pass-rate, pass@k and pass^k counts, and a per-task table with Δrate against the previous run.
results.json — the same numbers, machine-readable, minus the transcripts.
transcripts/<task>-<trial>.md — one file per trial: the round and tool-call counts, both graders' verdicts, the final document, and the full message transcript.

Read the transcripts. A grader you have not watched is a grader you cannot trust, and the failure that matters is usually visible in the third assistant turn rather than in the summary row.

The two graders

Render critic — the primary signal. It mounts the produced document with the real @jxsuite/runtime under happy-dom, the same render path the Studio canvas uses, and fails on anything the runtime surfaces: a thrown error during scope-building or node rendering, or a console.error / console.warn — unresolved $ref s, missing $prototype classes, broken bindings. Errors are phrased as actionable corrections ("→ Fix: a template binding references state that doesn't exist"), so the same output could later be fed back into the live loop.

Schema grader — the baseline. The same validateDoc() the agent loop already self-corrects against. It is free and deterministic, so it is reported alongside for context.

A trial passes on the render critic , not the schema grader. A document can be perfectly schema-valid and still render nothing useful, which is precisely the gap the critic exists to close.

pass@k and pass^k

Models are non-deterministic, so a single trial per task measures luck. Each task runs k times (3 by default) and the scoreboard reports both:

pass@k — at least one of the k trials passed. Capability: can the assistant do this at all?
pass^k — all k trials passed. Reliability: can a user count on it?

passRate (passes ÷ k) sits between them and is what the mean and the Δrate column are computed from.

The merge rule

Run bun run eval and note the baseline mean pass-rate; collect the failing transcripts.
Read the failures, then change one scaffolding file — the system prompt, the tool schemas plus translateValidationError , or the few-shot examples. One file per experiment keeps runs comparable.
Re-run. Keep the change only if the mean pass-rate improves and regressed is empty. Otherwise discard it.

The scoreboard computes regressed by diffing against the most recent prior run: any task that passed pass@k before and does not now. The CLI exits 1 when that list is non-empty, so the same command can gate CI.

Warning

Never tune the tasks to make a scaffolding change look good. If a task is genuinely ambiguous, fix the task in its own commit and re-baseline — but treat that as a change to the benchmark, not a result.

The headless rubric harness

bun run eval:headless runs a second harness at packages/studio/tests/harness/ , which landed alongside the one above. It drives a catalog of prompts through the same production loop and scores them on rubric axes with evidence attached, running each test three times ( JX_AI_RUNS ) and reporting the worst run per axis so a borderline result cannot cherry-pick a lucky pass. It reads its key from JX_AI_KEY , falling back to OPENAI_API_KEY , and loads the repo-root .env regardless of the working directory. Use it for a qualitative read on assistant behavior; use bun run eval for the pass/fail number you gate on.

Out of scope

The harness deliberately does not do runtime UX sensors in the live assistant, LLM-as-judge grading, token accounting (the streaming client does not surface usage yet), or autonomous self-editing. The render critic's error format is LLM-ready on purpose, so a later phase can wire it into the live loop.

AI assistant — what the harness is measuring, from the user's side
Working in the monorepo — the testing policy the rest of the repo follows
Working with agents — external agents on a Jx project, and the checks that hold them honest