petri-bench — can your model do science?

Leaderboard (standardized sweep v1, n=3 episodes per task)

Per-tier means over 30 episodes per cell; small print gives the range of the three per-pass means — an honest view of run-to-run variance. Every number traces to a committed episode artifact.

Score = correctness + method rigor (is the conclusion backed by a significant, isolating controlled experiment?) + efficiency, under a hard budget of 8 calls. Subscription-CLI models ran at their CLIs' default settings; model identities as reported by the CLIs (June 2026).

What the evaluation found

• The field is tight; the reference gap is not. Six frontier models span 4.4 points overall — differences that small are not distinguishable at n=3, and much of any single-episode ordering is variance (read n=1 agent-benchmark rankings with suspicion). The 29-point gap to the curated-value reference is significant against every model and stable in every pass.
• Tier profiles differ in shape. opus-4.8 posted the best L1 mean, gemini-3.1-pro the best L2, haiku-4.5 the best L3 — descriptive leaders (ranges overlap), but the shapes are real: haiku is flat where gemini swings 23 points between tiers.
• Solve rate and score diverge — by design. gemini has the highest solve rate (50%) and the lowest score: it finds answers while paying constant method taxes. haiku converts the lowest solve rate (37%) into a near-top score through clean, fully backed process. A correctness-only benchmark would invert these two models.
• The universal failure on the magnitude tier is direction, not magnitude. 69 of 180 L2 episodes end with the right parameter and the wrong direction (vs 11 magnitude misses). Decomposed against the logs: 61% of all direction errors never directly measured the determining contrast (most of those ran no probes at all); 39% had a significant measurement in their own log and contradicted it.
• Failures track the difficulty rating. Both tasks rated hard by statistical power defeat all six models; the easy band is solved at high rates. The power-based rating predicts where real models fail.

Failure fingerprints (per model, 90 episodes each)

p-hacked = the audit caught a submission that reads significant alone but survives only through redundant testing, failing a Holm correction across the episode's full test family. probe-only = the submission rests on matching the hidden world's output rather than an isolating controlled experiment. over-tested = redundant isolating tests (re-testing a parameter, or running more tests than candidates). The algorithmic solvers trigger zero flags across 90 episodes; among models, process discipline is not ordered by capability tier or price — the cheapest model (haiku) runs the cleanest process, and gemini's chronic over-testing (49/90) replicates its signature from the earlier own-scaffold pilot, so the trait belongs to the model, not the prompt.

How it works

Audit everything

Every number above traces to a self-contained episode artifact {task, log, score, audit, provenance} — the full call log with statistics, the score breakdown, the integrity audit, and a provenance stamp pinning the exact engine versions and commit. Re-score any episode from its log alone.

Browse all episode artifacts · download the full set (zip) · read the tech report

The instrument underneath

The five simulations are not toys with sliders — each is validated, headless and reproducibly on every release, against an established result from its literature (19/19 checks): the Vicsek order-disorder transition (swarm), Cont's stylized facts of asset returns (market), Deffuant/Hegselmann-Krause cluster scaling (social), and the Pearson Gray-Scott phase diagram (morph). The same engines power the in-browser playgrounds and the petri-labs-mcp server the bench runs through.

validation 19/19 deterministic per seed frozen task specs, CI-enforced power-based difficulty ratings self-contained episode artifacts

Run it on your model

{
  "mcpServers": {
    "petri-labs": {
      "command": "npx",
      "args": ["-y", "petri-labs-mcp"]
    }
  }
}

# one task, by hand
bench task market 101 my-run
bench experiment my-run volatility \
  '{}' '{"chartFrac":0.6}'
bench submit my-run chartFrac up
bench score my-run

# a whole model, one command
bench sweep configs/frontier-v1.json
bench report && bench taxonomy

Cite

Archived at Zenodo: concept DOI 10.5281/zenodo.20618024. Tech report: HTML · PDF.

@software{petri_labs,
  title  = {petri-labs: validated model organisms and a
            contamination-proof benchmark for AI-driven science},
  author = {Sozudogru, Baris},
  doi    = {10.5281/zenodo.20618024},
  url    = {https://petri-labs.org},
  year   = {2026}
}