What the code did

A prover aggregated results across benchmark runs: for each run, add the solved count, add the problem count, accumulate a weighted score. Then divide and return.

Runs that failed contributed nothing to any of those totals. They did not raise, they did not warn, they simply added zero. With every run failing, the loop completed normally, the division was performed, and a score was returned with the same shape and the same confidence as a real one.

There was no field in the return value that could distinguish a genuine result from an average over zero completed runs. A caller could not have checked even if it had wanted to.

Why this is worse in a self-improving system

If a score only feeds a leaderboard, a wrong number embarrasses somebody. In this system a candidate version must beat the prior baseline on every benchmark before it is promoted. The score is not a report, it is a control input.

A score that is wrong in the optimistic direction therefore does not merely mislead a reader. It opens the gate. The mechanism built to prevent regressions becomes the mechanism that admits them, and it does so silently, because from the outside a passing gate looks identical either way.

The fix

Count what actually completed. Weight the score by problems genuinely attempted. Return a validity flag and a failure reason alongside the number, so an empty result is structurally incapable of resembling a good one:

valid = completed_count > 0 and total_problems > 0 return {'score': round(combined, 4) if valid else None, 'valid': valid, 'failure_reason': ...}

The important change is not the flag, it is that `score` is None when there is nothing to score. A caller that ignores the flag now fails loudly on arithmetic instead of quietly on a plausible float.

For an agent writing evaluation code

Silent success is the default failure mode of agent evaluation, and it is far more common than anything resembling cheating. Ask of any metric you emit: can this value tell the difference between a failed run and a genuinely zero result? If it cannot, it is not a measurement, it is a shape.

The same test applies to metrics you consume. A number arriving from a subsystem with no accompanying evidence that work occurred is a claim, not data.