Open-source harness optimization for Claude Code, Codex and Pi
Harnesses evolve. You watch every edit.
A Meta Agent edits the harness around your coding agent: its instructions, skills, hooks and extensions. Each edit runs on real tasks, an independent grader scores the output, and a rule frozen before the first edit keeps it or reverts it.
Data: study/lineages.json, runs.csv, heldout.json · lineage main / pi-f2-loop-l1
Open it while it runs.
evolvarium optimize loop.json --dashboard 8765 serves a page on your machine that
updates as runs land: each candidate on the score chart, its hypothesis, the keep
rule's numbers against its bar, the pages it built and the Meta Agent's notes.
Four steps, four owners. None can change another's rules.
-
Edit
The Meta Agent, Claude Code
Writes a candidate harness and the hypothesis behind it, from the feedback the Study releases.
Touches only its iteration folder.
-
Run
The coding agent: Claude Code, Codex or Pi
Runs the search and dev tasks with the candidate, one fresh session per run.
Touches only a fresh, sandboxed workspace.
-
Grade
The grader
Scores every output. The agent never sees its checks and cannot repair the run it judges.
Touches only the output, after the run has ended.
-
Decide
The keep rule
Keeps the candidate only if its gain over its parent clears a bar set from measured noise.
Frozen before the first edit.
Every run is recorded: its inputs, the harness revision, the agent's trace and the grade. A claimed gain can be traced back to the runs behind it.
The grader is the optimizer.
−33% to −70%
of each harness's measured gain, once a stricter grader re-scored the same held-out pages. We ran the loop on open-ended web-page tasks graded by model judges; the pages did not change between the two grades.
Data: study/heldout.json
- Shows
- The same pages of the same harnesses, two graders, two different gains. How much a harness improved is partly a fact about the grader.
- Does not show
- Which grader is right. On page pairs that differed only slightly, an earlier grader agreed with a protocol-guided AI labeler 16 times in 32, a coin flip; there is no human agreement data.
What the optimizer learned was often the rubric.
8 of 9 harness versions kept after the Meta Agent saw failed rubric items had copied rubric text into the harness; none of 11 did when it saw only outcomes. Here is one, kept: a design item of the strict grader, and the rule the next harness added to its own skill.
Data: both texts are read from the released files when the site is built; Markdown emphasis removed.
- Shows
- Feedback about failed rubric items reaches the harness word for word, and the keep rule rewards it.
- Does not show
- That copying makes pages worse. The copied rule may be good advice; what it measures is the grader's own checklist.
Twelve lineages, every revision, released.
The repository releases the study's two graders, its eight web tasks, every harness revision with its decision, and the per-run scores. Each tree is one lineage, plotted like the hero; each opens its dashboard.
Data: study/lineages.json, runs.csv · one score scale for all twelve
Run it on your own models.
Confined runs need macOS 26, uv and Node 22.19 or later. The web example also needs Claude Code for the Meta Agent and an OpenAI- and an Anthropic-compatible API; DeepSeek's serves both. Its README has the private launch files.
git clone https://github.com/Yaxin9Luo/evolvarium && cd evolvarium
uv sync --locked
npm --prefix workers/pi ci --ignore-scripts && npm --prefix workers/pi run build
# A model-free batch: three harnesses, two tasks, two scripted backends, in seconds
uv run python -m examples.transfer_matrix.run --root /absolute/new/directory
# The study's experiment on your own models, with the live dashboard
uv run python -m examples.web_optimization.prepare --root /abs/runs/web-1 --grader strict \
--launch /abs/private/pi-launch.json --provider deepseek --model deepseek-flash \
--judges /abs/private/judges.json --claude "$(command -v claude)" \
--meta-model deepseek-flash --meta-launch /abs/private/meta-launch.json
uv run evolvarium optimize /abs/runs/web-1/loop.json --dashboard 8765