Reproducing these numbers

Everything on these pages is produced by code in the repository. Your absolute figures will differ from the published ones — different CPU, different kernel, different background load — and that is expected. What should reproduce is the shape: the ratio between depin and its direct baseline, the scaling curves, and the deterministic counts, which do not depend on the host at all.

Setup

git clone https://github.com/andrelopes-code/depin
cd depin
uv sync --group bench

benchmarks/ sits outside testpaths, so an ordinary uv run pytest does not collect it.

Run the suite once

uv run --group bench pytest benchmarks --benchmark-only

This prints the usual pytest-benchmark table, with one row per workload per implementation — the depin subject and its direct baseline side by side. It is the quickest way to see the overhead ratio on your own machine.

Check that the workloads still mean what they claim

uv run pytest tests/integration/test_workload_contracts.py tests/integration/test_workload_equivalence.py

These carry no timing. They assert that every workload states its claim in full, and that each depin implementation and its direct baseline observably do the same thing: same result, same objects constructed in the same order, same resources closed in the same order. A benchmark whose baseline has drifted fails here, before any number is produced.

Compare two revisions

The same measurement the pull-request gate runs. Point it at two checkouts:

git worktree add /tmp/base main

uv run --group bench python -m benchmarks.harness.pairs \
    --base-dir /tmp/base --head-dir . --repetitions 5 --out /tmp/measurements

uv run --group bench python -m benchmarks.harness.gate \
    /tmp/measurements --budgets benchmarks/budgets.toml

pairs measures both revisions in independent processes across five repetitions, alternating which side runs first. gate applies each workload's budget and exits 0 when everything passed, 1 on a regression, 2 on a malformed report, and 3 when a result was inconclusive — the case where the estimate is over budget but its interval still spans it.

Increase --repetitions to narrow the intervals. Five is the minimum at which the gate will reach a verdict at all.

Measure your own host's noise

Budgets are generated from a collection in which both sides are the same code, so every difference in it is noise:

uv run --group bench python -m benchmarks.harness.pairs \
    --base-dir . --head-dir . --repetitions 5 --out /tmp/nullrun

uv run --group bench python -m benchmarks.harness.calibrate /tmp/nullrun --summary

That prints each workload's measured dispersion and the budget it would produce here. Drop --summary to write the budget file itself.

The committed budgets are not produced this way. They come from the same command run on the pull-request runner, by the Calibrate workflow, because the gate they control runs there and a relative gate has to take its noise from the environment it runs in. What a local calibration is for is comparison: if your host's dispersion is far from the runner's, the difference is the host, not the code.

Run it with nothing else executing either way. A collection taken while other work was running measured one workload's dispersion at 20.9% where the quiet figure was 4.9% — which by the formula would have produced a budget looser than the uniform one this system replaced.

Confirm the gates actually fail

The repository keeps three deliberate regressions, one per class of protection. benchmarks/seeds/README.md explains each and how to apply it to a scratch worktree. Applying one and re-running the comparison above should fail the gate it targets — and, for the allocation and scaling seeds, should leave the latency gate green, which is why those gates exist.

Regenerate the results page

uv run --group bench python -m benchmarks.harness.report benchmarks/results/<dataset>

The published results page is exactly this output for the dataset committed alongside it, and a test asserts they match. A number on that page cannot have been typed by hand.