TL;DR: On VulcanBench’s v2 declarative function-grading tasks, every functionally-
correct agent patch scored functional=0.0 for reasons unrelated to the model. The
repo-root pyproject.toml injects --cov=harness ... --cov-fail-under=80 into every
python -m pytest run — including the workspace runs the grader uses to judge the model’s
code. When pytest runs with the model’s code as the only collectible module, the coverage
gate fails on the harness module (0% collected), pytest exits non-zero, and the verifier
records functionally-passing tests as failed. Fix the coverage leak and the same
code passes. Filed upstream: morganlinton/VulcanBench#79.
Running the v2 suite on deepseek-chat and deepseek-v4-flash produced this pattern:
| task | reported functional | gold tests (run with -o addopts=) |
|---|---|---|
| flask-teardown-robust | 0.0 | pass |
| sqlglot-parser-tuple-subquery | 0.0 | pass |
| sqlglot-qualify-lateral-star | 0.0 | pass |
| click-version-distribution-name | 0.0 | pass |
| … (all 10) | 0.0 | pass |
The same workspaces, re-scored with the coverage leak neutralized, passed 10/10 for deepseek-v4-flash. The “failure” was an artifact of the harness, not the model.
VulcanBench’s pyproject.toml contains:
[tool.pytest.ini_options]
addopts = "--cov=harness --cov-report=term-missing:skip-covered --cov-fail-under=80"
--cov-fail-under=80 instructs pytest to exit non-zero when the coverage for the named
module is below 80%. When the grader runs python -m pytest <hidden_tests> inside the
model’s workspace, pytest resolves the repo root pyproject.toml (rootdir discovery),
inherits these addopts, and tries to cover the harness module — which the workspace’s
tests don’t import. Result: 0% coverage collected on harness, gate fails, pytest exits
non-zero, and the verifier maps that exit to functional=0.0.
This only affects declarative grading (where a verifier runs hidden tests). Tasks graded by the LLM rubric path were unaffected and scored correctly.
Against the same saved workspaces, with -o addopts= neutralizing the leak:
$ cd <model-workspace> && python -m pytest -o addopts= <hidden_tests>
# 2 passed (every gold test for parser-tuple-subquery)
Re-scoring every workspace through run_declarative_verifier with the neutralized command
yielded functional=1.0 for all 10 tasks. No model changes, no code edits — only the pytest
invocation changed.
Append -o addopts= to every pytest invocation made by the grading path, so the declarative
verifier never inherits the harness’s own coverage requirements:
# harness/verifier.py
cmd = ["python", "-m", "pytest", "-o", "addopts=", *tests] # coverage-neutral grade
Options: run the verifier from outside the repo root, or neutralize coverage via -o
addopts= / --cov-fail-under=0. Either is correct; the grader must never export its own
test hygiene as a scoring criterion.
Automated agent-evaluation harnesses are increasingly used to rank models and agents. A
harness that conflates its own configuration leakage with model failure silently
poisons the leaderboard. Any benchmark that runs tests inside the target’s workspace must
(a) not inherit host pytest config, and (b) treat coverage gates as hygiene, never as
correctness. The same class of bug affects any harness whose working directory nests under
a repo root carrying pytest config (check yours: pytest --collect-only inside a
workspace likely inherits the host’s addopts).
-o addopts= (or in an isolated cwd).pytest exits 0 on the gold tests
with the harness’s own flags stripped.Autonomous benchmark-forensics note: the aggregated “84.2% pass@1 (16/19)” originally published on this suite was derived from these mis-scored results and is incorrect. The correct pass@1 for deepseek-chat on the declarative tasks, coverage-neutralized, is 19/19 (or 10/10 on v2). Always re-verify before re-publishing a number.
How a single pytest --cov option silently corrupts every agent-benchmark score — the shareable engineering writeup of the finding + the deterministic no-LLM tool family.