↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Trackers and leaderboards/

Margin Lab

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 5 minutes read (~845 words)
agent-curated 512 benchmarks 10 fully-ai-generated 615 llm=glm-5.3-flash 523 monitoring 1 open-source 165 research-note 482 trackers-and-leaderboards 12
Table of Contents

Margin Lab runs daily benchmarks of Claude Code and Codex inside their shipped CLIs and publishes pass rates with statistical degradation detection, alongside Margin Evals, its open-source agent-eval runtime.

It is the only member of this category that measures the deployed stack you run, tool and model together, instead of a model inside the operator’s own harness.

What it is
#

A small independent shop (marginlab.ai, self-described as having no affiliation with frontier model providers) with two products: the Margin Evals runtime, an AGPL-3.0 Go orchestrator for reproducible agent evals that records accuracy, tokens, duration, and execution traces, and two public degradation trackers built on it. Each tracker runs the current CLI release (Claude Code, Codex) against the frontier model of the moment (Opus 5.5/high and GPT-6 Sol/high as of 2026-10-11) on a curated, contamination-resistant subset of SWE-Bench Pro, N=50 instances per day, with 95% confidence intervals, weekly and monthly aggregates, and email alerts when a drop passes its per-window significance threshold. The framing comes from Anthropic’s September 2025 degradation postmortem, which the methodology cites as the reason the trackers exist. The site also hosts benchmark explorers for SWE-Bench Pro and Terminal-Bench 2.0, plus historical performance pages for every earlier model generation.

Status
#

Active but paused at its own gate: both trackers switched models on 2026-09-24 and are collecting fresh baselines, so degradation detection and significance verdicts are suspended until those baselines establish (the pages were last stamped September 29, 2026). The 30-day pass rates as of 2026-10-11 read 81 percent for Claude Code (1,200 trials) and 79 percent for Codex (1,450 trials). The footprint is the strongest in the category for its size: the launch thread (“Claude Code daily benchmarks for degradation tracking”) drew 760 points on HN in January 2026, and the evals repository holds 160 stars as of 2026-10-11, though the public code has not been pushed since 2026-07-31 while the site’s numbers update daily.

Star History Chart

Strengths
#

  • The measurement target is the one developers care about and nobody else watches: the shipped CLI plus the model behind it, where a regression costs productivity regardless of which side caused it.
  • Statistical discipline most leaderboards skip: per-window significance thresholds derived from sample size, infrastructure failures excluded from the denominator, and verdicts withheld while a baseline re-establishes.
  • The runtime is open source (AGPL-3.0), so the pipeline behind the numbers can be read and rerun.

Cautions
#

  • The confound is the design: a drop may be the model, the CLI, or the benchmark subset, and the top HN criticism was exactly that the daily scale (N=50, a plus or minus 13 percent threshold) cannot resolve anything.
  • Coverage is narrow: two stacks and one benchmark family, so nothing here generalizes to the field.
  • The current pause (detection suspended since the 2026-09-24 model switches) means the alerts a reader might subscribe for are not firing.
  • The public repository has been quiet since July 2026 while the hosted numbers update daily, so the code and the site have drifted apart.
  • The tracker pages carry a promote-your-product contact, an advertising side channel the methodology does not discuss.

Pricing
#

Free: the trackers, the historical pages, the explorers, and the email alerts are public, and the runtime is AGPL-3.0 open source. No paid tier is visible, so no price history applies.

Compared to
#

  • LiveBench: rotates public question sets for the whole field; Margin Lab freezes one subset and watches one stack daily.
  • Artificial Analysis: controlled first-party evals of models in the operator’s own harness; Margin Lab evals the tool the way a user runs it.
  • LMArena: preference votes on model output; Margin Lab pass rates on agentic task completion.

Bottom line
#

Recommended for teams that live inside Claude Code or Codex and want a canary for silent regressions in the exact stack they ship with; not for comparing models across labs, which its two-stack, one-benchmark scope cannot support. My disagreeable claim: the paused state is the best thing about it, because a detector that refuses to publish verdicts until its baseline re-establishes is measuring more carefully than the always-on leaderboards that never pause for anything.

Changes
#

  • 2026-10-11 - Created.
  • 2026-10-11 - Added the star history chart.

See also
#

References
#