petri-bench — can your model do science?

Leaderboard (standardized sweep v2 — tier-fair L3, n=3 episodes per task)

Models are ordered by the L1 mean — the one tier every model has completed at n=3. Some queues were paused mid-run to protect subscription quotas, so some cells are partial (episode counts shown in-cell); overall appears only for models with all three tiers complete, and solved/avg calls cover each model's recorded episodes. Small print gives the range of the three per-pass means on complete cells. Every number traces to a committed episode artifact.

Score = correctness + method rigor (is the conclusion backed by a significant, isolating controlled experiment?) + efficiency, under a hard budget of 8 calls. All models run the identical agent loop (same brief, tools, budget, scoring). Transports: claude/gemini/glm CLIs on the frozen text protocol; codex sessions at xhigh reasoning effort; claude-fable-5 runs the same loop over native MCP tool-calling, because its safety layer deterministically flags text-serialized harness protocols — the finding, the isolation tests, and the transport are documented in the transport record. glm-5.2 joined the sweep and averaged 55.6 over its first 4 episodes before its queue was paused; it gets a row when its cells complete. Sweep v1's per-model L3 numbers were construct-invalid (the interaction tier was briefed with the single-factor instruction) — those 129 episodes are archived in-repo, and every L3 cell above comes from the tier-fair re-run.

What the evaluation found

• The reference gap survives every new model. Recomputed per model on v2-valid completed cells: the curated-value sweep leads by +25 to +33 points (paired Wilcoxon per model over 16–30 tasks, all Holm-corrected p < 0.001). The two models with all three tiers complete land at 60.7 (haiku) and 58.0 (gemini) against the reference's 90.8.
• The L1/L2 split is now extreme. gpt-5.6-sol posts the best L1 mean recorded (76.7) and near-floor L2 solving (1 of 30 episodes); claude-fable-5 ties opus on L1 mean with the best L1 solve rate on the board (24/30) and collapses on L2 (3/22). Identifying which parameter changed and reading which way it pushes the metric are different skills, and the second one is where frontier models fail.
• The tier-fair L3 re-run vindicated the v0.6 construct fix. Under the corrected factorial briefing glm-5.1's L3 jumped from a construct-invalid 53.2 to 66.7 with 15/21 solved — the best model L3 recorded — while haiku (59.5) and gemini (44.1) barely moved. The v1 instruction was masking a real capability difference, not adding noise.
• The hard-task wall fell. In v1, both L1 tasks rated hard by statistical power defeated all six models. claude-fable-5 solved swarm-101 in all three episodes and swarm-202 in two, plus two hard-rated L2 tasks; gpt-5.6-sol cracked a hard L1 and a hard L3. The power-based rating still orders difficulty — the ceiling just moved.
• Solve rate and score keep diverging — by design. fable-5 has the highest model solve rate on the board (52%) but pays heavy method taxes (chronic over-testing, probe-leaning submissions), landing its recorded-episode mean at 58.8. haiku still converts a 36% solve rate into the best complete overall through clean, fully backed process.

Failure fingerprints (per model, over its recorded v2 episodes)

p-hacked = the audit caught a submission that reads significant alone but survives only through redundant testing, failing a Holm correction across the episode's full test family. probe-only = the submission rests on matching the hidden world's output rather than an isolating controlled experiment. over-tested = redundant isolating tests (re-testing a parameter, or running more tests than candidates). The algorithmic solvers trigger zero flags; among models, process discipline is not ordered by capability tier or price — the cheapest model (haiku) still runs the cleanest process, gemini's chronic over-testing (42/90) replicates across scaffolds, and the newest, largest model on the board has the most distinctive fingerprint: claude-fable-5 measures compulsively (over-tests in half its episodes), leans on probes for its answers (5 probe-only submissions), and never once p-hacks — a cautious empiricist that pays its score away in method taxes.

How it works

Audit everything

Every number above traces to a self-contained episode artifact {task, log, score, audit, provenance} — the full call log with statistics, the score breakdown, the integrity audit, and a provenance stamp pinning the exact engine versions and commit. Re-score any episode from its log alone.

Browse all episode artifacts · download the full set (zip) · read the tech report

The tech report documents sweep v1 in depth (design, validation, and the L1/L2 episodes, which are unchanged here); this page carries the v2 leaderboard — tier-fair L3, two new frontier models, and the corrected transport notes. The report's v2 revision lands when the paused cells complete.

The instrument underneath

The five simulations are not toys with sliders — each is validated, headless and reproducibly on every release, against an established result from its literature (19/19 checks): the Vicsek order-disorder transition (swarm), Cont's stylized facts of asset returns (market), Deffuant/Hegselmann-Krause cluster scaling (social), and the Pearson Gray-Scott phase diagram (morph). The same engines power the in-browser playgrounds and the petri-labs-mcp server the bench runs through.

validation 19/19 deterministic per seed frozen task specs, CI-enforced power-based difficulty ratings self-contained episode artifacts

Run it on your model

{
  "mcpServers": {
    "petri-labs": {
      "command": "npx",
      "args": ["-y", "petri-labs-mcp"]
    }
  }
}

# one task, by hand
bench task market 101 my-run
bench experiment my-run volatility \
  '{}' '{"chartFrac":0.6}'
bench submit my-run chartFrac up
bench score my-run

# a whole model, one command
bench sweep configs/frontier-v1.json
bench report && bench taxonomy

Cite

Archived at Zenodo: concept DOI 10.5281/zenodo.20618024. Tech report: HTML · PDF.

@software{petri_labs,
  title  = {petri-labs: validated model organisms and a
            contamination-proof benchmark for AI-driven science},
  author = {Sozudogru, Baris},
  doi    = {10.5281/zenodo.20618024},
  url    = {https://petri-labs.org},
  year   = {2026}
}