Agents on Soli

How we grade models on real Soli work. The table lives on /ai. Numbers only come from www/data/ai_evals.json.

What is compared

The first board freezes three coding CLIs on the same fixture and hidden tests: claude -p (Claude Code / sonnet), opencode run (DeepSeek opencode/deepseek-v4-flash-free), and Grok Build (grok --prompt-file, grok-4.6). It is not Cursor vs Claude Code as products, and it is not Soli vs Rails on the same numbers.

Corpus

Twelve one-capability tasks on evals/app, not this docs site.

Slug	Capability
hello-world	harness smoke
scaffold-resource	`resources` + controller
hash-where	portable hash `.where` with `gt`
column-sti	subclass + `type`
job-enqueue	`perform_later`
unless-guard	block `unless`
csrf-webhook	`skip_csrf` on the webhook only
form-permit	`permit`
validation	`validates`
attachment	`has_one_attached`
test-spec	`assert_eq`, not `assert_equal`
llm-stream	`sse` + `llm_stream`

Grading

Copy evals/app to a fresh worktree.
Apply prompt.md with a single-shot coding agent.
Copy hidden tests in. Run soli lint and soli test hidden.
Scan the diff for identifiers in expect.md (API recall).

Three runs per model. Accuracy is passes / (tasks × 3). Speed is median wall time. Tokens and cost are means. Failures and refusals count as fail.

Cost

About 5 models × 12 tasks × 3 ≈ 180 runs. At $0.50–$2 each, a board refresh is roughly $90–$360. That is why this is not on every pull request.

Reproduce

python3 scripts/evals/run.py --list
python3 scripts/evals/run.py --dry-run --models claude,opencode,grok --tasks hello-world --runs 1
python3 scripts/evals/run.py --models claude,opencode,grok --out www/data/ai_evals.json

Until a paid run is committed, models in the JSON is empty and /ai shows that empty state. Do not invent scores. Token/cost stay null when a CLI does not report them. Harness id: soli-evals/0.2.

AI Agents Linting