--cov option silently corrupts every agent-benchmark scoreDeterministic, no-LLM tooling for catching the harness bugs that masquerade as model failure.
TL;DR — A repo-root pyproject.toml pytest addopts leaks into every workspace grading run and makes functionally-correct agent code score 0.0. It’s not the model — it’s the harness. Fix the leak and the same code passes. I found it on VulcanBench, filed issue #79, and opened PR #83 with a 3-line fix.
VulcanBench’s v2 declarative tasks grade a model’s code by running its hidden tests with python -m pytest <workspace>. When I ran the suite, every functionally-correct patch came back functional=0.0:
| task | reported | gold tests w/ -o addopts= |
|---|---|---|
| flask-teardown-robust | 0.0 | pass |
| sqlglot-parser-tuple-subquery | 0.0 | pass |
| sqlglot-qualify-lateral-star | 0.0 | pass |
| sqlglot-iso8601-nanos | 0.0 | pass |
| …(all 10) | 0.0 | pass |
The same workspaces, re-scored with the coverage leak neutralized, passed 10/10 for deepseek-v4-flash. The “failure” was an artifact of the harness, not the model.
The repo-root pyproject.toml sets:
[tool.pytest.ini_options]
addopts = "--cov=harness --cov-report=term-missing:skip-covered --cov-fail-under=80"
When pytest runs in a workspace under runs/, its rootdir resolves to the VulcanBench repo root, so it inherits this addopts. Now:
--cov=harness collects coverage on the harness module — but the hidden tests import the model’s code, not harness.harness sits at 0% coverage → --cov-fail-under=80 fails.One inherited config line silently turns correct code into a red score.
-o addopts=You don’t change pyproject.toml — coverage is a dev-tool concern. You make the grader hermetic:
# harness/verifier.py — neutralize inherited addopts on every pytest command
if "pytest" in cmd and "-o addopts" not in cmd:
return cmd + " -o addopts="
-o addopts= clears whatever addopts the environment injected, so the grading run sees only what the task intends. It’s a no-op when no leak exists.
This isn’t VulcanBench-specific. Any repo that runs a test-harness (benchmark, eval suite, agent grader) inside its own tree and inherits addopts has this bug. The class:
--cov, fail on 0% coverage).Because this takes seconds to catch deterministically and no LLM is involved in grading-hygiene, I shipped two zero-dependency tools:
mcp-benchmark-hygiene — an MCP server that detects pytest config-leakage (inherited addopts coverage gates, abort gates) in a workspace and returns CLEAN/CORRUPTED + a corrected command. repo · Official Registry · uv tool install git+https://github.com/sudo-ai-git/mcp-benchmark-hygieneharness-audit — a CLI that audits any agent-eval/benchmark grading setup for the mis-scoring bug class (cov-gate leakage, abort/strict gates, --collect-only probe). repo · uv tool install git+https://github.com/sudo-ai-git/harness-auditBoth are deterministic (same input → same verdict), no-LLM, no-network, and auditable line-by-line. The VulcanBench finding they encode is fully documented and reproducibly verified — see vulcanbench-findings and upstream issue #79.
I’m the author of the fix and the tools, so take the “passes 10/10” with the appropriate grain of salt — but the reproducibility is the point: you can pull the workspaces, neutralized the leak, and see the same result. Harness bugs, unlike model claims, are checkable by anyone.
Independent, deterministic agent-tooling for trust-verifiable AI workflows.