When is a benchmark conclusion identified across evaluator meanings?

Research
1

Hi everyone,

Suppose we hold the evaluated systems, predictions, labels, and availability mask fixed. A benchmark result can still depend on choices such as the metric, aggregation rule, denominator, missing-data treatment, or tie convention.

This raises a methodology question that seems more basic than ordinary robustness analysis:

Has the benchmark identified the conclusion being reported—such as the winner or complete ranking—across a declared, evidence-bound family of evaluator meanings?

One way to formalize this is to declare a finite family of evaluator specifications and compute

I_Q(D; S) = { Q(E_s(D)) : s in S },

where D is the fixed observed substrate, s is one evaluator specification, and Q is a claim such as winner or complete order. The claim is identified relative to the declared family when this set is a singleton.

We built a small executable artifact, EvalAudit, mainly to make this question concrete and contestable. It does not discover the right evaluator automatically. Instead, each candidate specification must declare:

  • what semantic target it represents;
  • which evaluation choice varies;
  • where the interpretation came from;
  • whether it is admitted, disputed, or excluded;
  • what evidence and fixed source bindings support it.

Only admitted specifications determine the primary result. Admitted plus disputed specifications form a separate sensitivity envelope. The implementation is deterministic; no model selects family members, scores, thresholds, or conclusions.

Three initial cases

We applied the interface to frozen evidence from three heterogeneous benchmarks:

Case Admitted / disputed specifications Winner identified? Complete order identified? Stable pairs
AgentDojo v0.1.35 2 / 3 yes in both primary family and envelope primary: yes; envelope: no 3/3 → 2/3
AutoML Benchmark 3 / 2 yes in both primary: yes; envelope: no 10/10 → 9/10
BEIR SciFact 5 / 0 yes no 2/3

The stable winners are themselves informative: specification multiplicity does not imply that every conclusion must reverse. At the same time, a stable winner can coexist with an unidentified complete ranking and with only partially stable pairwise relations.

All three cases use heterogeneous ordinal endpoints. We therefore mark cross-specification exact-score sets as not applicable, rather than treating numbers from different metrics or aggregation semantics as values of one common estimand.

Low-cost reproduction

The quickest reproduction runs a packaged SciFact declaration. It does not download the corpus or rerun retrieval models:

git clone https://github.com/918154429/evalspec-audit.git
cd evalspec-audit
git checkout v0.3.0

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

evalaudit demo run beir-scifact --output-dir outputs/beir-scifact-r1
evalaudit verify outputs/beir-scifact-r1 --json

On Windows, activate the environment with .venv\Scripts\activate.

The resulting bundle contains the exact audit declaration, normalized JSON, machine- and human-readable reports, separating witnesses, and a checksum-closed manifest.

An important boundary: this reproduces the deterministic audit from frozen, provenance-bound score declarations. It does not independently regenerate the upstream scores, certify that the semantic family is complete, establish benchmark validity, or prove scientific truth. Semantic uncertainty is also kept separate from sampling and stochastic uncertainty.

Questions for discussion

I would particularly value criticism on four points:

  1. What evidence should be sufficient to admit an evaluator interpretation into the primary family?
  2. Where should we draw the line between an evaluator defect and an admissible alternative meaning?
  3. Should benchmark reports routinely distinguish identified winners, identified pairwise relations, and identified complete orders?
  4. How should semantic and sampling uncertainty be reported together without collapsing them into one object?

A concrete counterexample—especially one where the admission rule accepts an invalid interpretation or excludes a defensible one—would be more useful than general feedback.

Protocol:

2

One concrete result surprised us enough that it may clarify why the claim-relative part matters.

Across the current public demos, the winner can be identified even when the complete ranking is not.

benchmark winner complete order
AgentDojo identified identified on admitted family; not on admitted+disputed envelope
AutoML Benchmark identified identified on admitted family; not on admitted+disputed envelope
BEIR SciFact identified not identified even across admitted specifications

So “System A wins” and “the benchmark identifies this leaderboard” are genuinely different claims, even with the systems, predictions, labels, and availability fixed.

This also gives an easy way to challenge the artifact: if you think one of the declared semantic families is missing a defensible evaluator meaning, I’d be especially interested in an evidence-backed counterexample. The registry is deliberately set up for exactly that kind of disagreement.

3

Hmm… maybe something like this?:

I think the claim-relative separation is useful. In particular, your current examples where the winner remains identified while the complete order does not seem informative rather than disappointing: they show that evaluator multiplicity does not automatically imply a winner reversal, while still exposing exactly which stronger claims the benchmark does not identify.

For the four questions you asked, my current answers would be roughly:

  1. For admission into the primary family, I would separate provenance from admissibility. A specification being source-documented seems like good evidence that the interpretation exists, but not yet sufficient evidence that it is interchangeable with the others for the particular claim being audited.
  2. I would put evaluator defects on a separate track before semantic-family admission. “Official/documented” and “non-defective” are different properties.
  3. Yes, I think winner / pairwise relations / complete order are worth reporting separately. A partial conclusion is often more informative than forcing a total leaderboard.
  4. I would keep semantic and sampling uncertainty as two axes rather than collapse them. Sampling uncertainty can be reported conditional on each semantic specification; the across-specification identified set/envelope then sits outside those intervals.

The part I would probably make most explicit is the admission path. Something approximately like this:

candidate evaluator interpretation
        |
        v
[0] Is the comparison meaningful on the support being compared?
        | no
        -> support / comparability issue
        |
       yes
        v
[1] Is the evaluator itself defective or unreliable?
        | yes
        -> defect / reliability track
        |
       no
        v
[2] Is this interpretation independently defensible?
        | no
        -> exclude
        |
       yes
        v
[3] Is it equivalent for this particular claim / construct?
        | no
        -> separate typed endpoint
        | unclear
        -> disputed
        |
       yes
        v
[4] Is the evidence strong enough for primary-family admission?
        | yes
        -> admitted
        | uncertain
        -> disputed

I do not mean this as a replacement for the current admitted / disputed / excluded design. It seems more like a way to make the reason for a challenge localizable.

For example, two people could agree that a metric is source-documented, but disagree about whether it measures the same endpoint. Or they could agree that it measures the same endpoint, but disagree about whether the evidence is strong enough to put it in the primary family. Those are importantly different disagreements.

A distinction from the multiverse-analysis literature that seems useful here is defensibility vs equivalence. The recent Multicurious guide makes this distinction fairly explicitly: a pipeline may be defensible in isolation without being sufficiently comparable to another pipeline to treat both as interchangeable for a given inferential purpose. That seems close to the problem your registry is trying to expose, while still leaving the actual family declaration evidence-bound and human-reviewable.

One concrete support/comparability case: AgentDojo

I tried a small deterministic reconstruction of the public AgentDojo declaration because this looked like a good place to separate fixed availability from comparable support.

The upstream AgentDojo results page itself says that it is not a leaderboard because all models were not attacked with all attacks/defenses, making fair cross-model comparison difficult.

For the three systems in the v0.3.0 declaration, the run-key support I reconstructed was:

union of available run keys   = 949
common run keys               = 629

Claude 3.5 Sonnet:
    available = 629

Gemini 2.0 Flash:
    available = 949

GPT-4o-mini:
    available = 629

For Gemini specifically:

ASR on all available keys     = 14.1201%
ASR on common keys only       = 20.8267%

difference                    = -6.7066 percentage points

The extra 320 Gemini-only keys had an ASR of about 0.94%, so they materially dilute the all-available aggregate.

What I like about this example is that the winner/order still does not reverse:

lower ASR is better

all-available:
Claude < Gemini < GPT-4o-mini

common support:
Claude < Gemini < GPT-4o-mini

So this does not seem like a counterexample of the form “the reported winner is wrong.”

Instead, it looks like a useful illustration of your own point:

specification/support choices can materially change a reported quantity while the winner remains identified.

The one extra separation I might make is:

fixing the availability mask makes the audit reproducible, but does not by itself establish that model-specific available-case averages represent the same cross-system comparison target.

Depending on the intended claim, a cheap support/comparability check may therefore belong before semantic-family admission.

I would avoid calling the available-case interpretation invalid in general. There are settings where “performance over whatever was actually observed for this system” is exactly the intended descriptive endpoint. The issue is only whether that endpoint is interchangeable with a common-support cross-system comparison for the claim being reported.

A second concrete case: AutoML and the evidence horizon

The AutoML example made me wonder about a slightly different part of the admission rule: what time horizon counts as evidence for source-documented?

The AMLB JMLR paper is quite explicit about the difficulty of comparing AutoML frameworks and about the benchmark’s aggregation logic. For its suite-level analysis, it discusses framework failures/missing results and constructs task-level rankings before aggregating them across the benchmark.

I reconstructed all five EvalAudit v0.3.0 AutoML specifications from the same SHA-bound results_valid.csv; the maximum absolute difference from the declared values was about 4.8e-11. In other words, the disputed within-task-rank specification is not bringing in some separate dataset – it is an executable transformation of the same frozen AUC substrate.

The timeline is interesting:

frozen result rows:
2019-02-22 ... 2019-03-07

canonical AMLB paper:
2024

EvalAudit's bound AMLB repository revision:
2025-04-19

That creates several defensible policies:

A. Historical-only evidence
   Use only documentation that existed when the frozen artifact was produced.

B. Revision-relative evidence
   Use documentation available by the upstream revision bound by the audit.

C. Audit-time canonical evidence
   Allow canonical guidance that exists when the audit itself is performed.

I do not think the tool has to choose one universal policy. But recording which evidence horizon is in force could make source-documented vs audit-proposed classifications easier for a third party to reproduce.

This is especially relevant here because the AMLB paper’s suite-level estimand is not automatically identical to the smaller frozen endpoint used by EvalAudit. So I would not conclude from the paper that the current negative-mean-task-rank specification necessarily belongs in the admitted primary family. The more modest point is that the provenance/admission decision becomes easier to audit if the temporal scope of admissible evidence is explicit.

On “defect” vs “alternative meaning”

I think the cleanest rule may be:

first ask whether the implementation/evaluator is functioning as the intended evaluator; only then ask whether that intended evaluator is an admissible alternative meaning.

A useful real-world example is lm-evaluation-harness issue #1050 on DROP.

The issue states that:

  • the harness implementation matches the official implementation;
  • the official implementation itself makes systematic preprocessing/grading errors;
  • they therefore discuss keeping the official implementation available while potentially changing the default.

That seems like a good warning against collapsing these notions:

official
    !=
non-defective

source-documented
    !=
same endpoint

independently defensible
    !=
interchangeable for this claim

executable
    !=
primary-family admissible

The same lm-eval issue also gives a nice practical evidence hierarchy for ambiguous evaluation details: broad training-community agreement, then clear official implementation, then evaluator-community agreement, then a preferred choice among common implementations – while explicitly saying these are guidelines rather than rules.

That seems compatible with EvalAudit’s design: origin can record where the interpretation came from, while the admission decision can separately record why it is suitable for this family.

Winner vs pairwise relations vs complete order

For question 3, I would strongly prefer keeping these separate.

There is a useful structural analogy in Robust Ordinal Regression. ROR considers a whole set of compatible preference models rather than selecting one arbitrary model. A pairwise relation is called necessary when it holds for all compatible models and possible when it holds for at least one. The necessary relation is generally only a partial preorder.

That is not the same problem as evaluator semantics, so I would not import the terminology directly. But structurally it is very close to:

A > B under every admitted evaluator meaning
    -> stable / identified pairwise relation

B > C under some meanings
C > B under others
    -> unresolved pair

yet A is above everyone under every meaning
    -> winner identified

therefore
    -> winner identified
    -> complete order not identified

This seems preferable to reducing the whole benchmark to one boolean label such as “robust” or “not robust.”

It also makes the stable-winner results in your current demos more useful. They show that the framework can preserve a strong conclusion instead of treating any evaluator disagreement as catastrophic.

Semantic uncertainty and sampling uncertainty

For question 4, I would keep the objects visibly separate.

Something like:

semantic specification s1
    point result / ordering
    sampling uncertainty conditional on s1

semantic specification s2
    point result / ordering
    sampling uncertainty conditional on s2

semantic specification s3
    point result / ordering
    sampling uncertainty conditional on s3

--------------------------------

across s1, s2, s3:
    semantic identified set / sensitivity envelope

This is also consistent with the distinction in your evaluation-semantic identification paper: the family variation is not the same thing as sampling variation.

There is a practical precedent in challengeR, where the ranking rule is chosen and then bootstrap resampling is used to study the statistical stability of that ranking. The package also supports alternative ranking/aggregation rules separately. That is not a semantic-identification framework, but I think the separation of axes is useful.

So, for example, I would avoid constructing one giant outer interval and casually calling it a “95% CI.” A union of results across evaluator meanings does not inherit a frequentist coverage interpretation merely because each evaluator-specific result has one.

If desired, a report could show both:

semantic envelope:
    what conclusions survive evaluator meanings?

sampling uncertainty:
    how uncertain is each result given one meaning?

and perhaps an additional deliberately-labeled outer union for visualization, without pretending the two uncertainties are the same mathematical object.

Why I think this separation strengthens rather than changes the goal

What I find attractive about the current EvalAudit design is that it does not try to have a model discover the “correct” evaluator. I would keep that.

The difficult part of this problem seems inherently evidentiary and semantic: literature precedent, benchmark documentation, intended use, implementation history, and the claim being made all matter.

The proposed decomposition would mainly make disagreement easier to target.

For example:

Challenge A:
"I agree this evaluator is source-documented,
but I think it is a different endpoint."

Challenge B:
"I agree it targets the same endpoint,
but I think evidence is too weak for primary admission."

Challenge C:
"I think this is an evaluator defect,
not a semantic alternative."

Challenge D:
"I agree with the evaluator meaning,
but the compared systems do not share the support
needed for this cross-system claim."

Those challenges have different remedies.

  • A might produce a new typed endpoint.
  • B might move admitted -> disputed.
  • C might produce a corrected evaluator version.
  • D might produce a common-support or standardized-support analysis.

That seems to fit the existing challenge/registry idea rather than requiring a different framework.

It may also help avoid a subtle future problem: the number of specifications is not naturally a probability measure.

For example, if one semantic interpretation is represented by five nearly redundant specifications and another by one, saying “5 out of 6 specifications agree” would depend heavily on how finely someone enumerated the family. Your current singleton/non-singleton identification criterion mostly avoids that problem.

A compact taxonomy I would use when reviewing candidate specifications
Situation Possible treatment
Implementation does not implement its stated evaluator correctly defect/reliability track
Official implementation appears systematically flawed retain provenance; handle validity separately
Metric is legitimate, but measures a different construct/utility separate typed endpoint
Same general endpoint, but equivalence is unclear disputed
Same endpoint and independently defensible, but evidence for primary status is weak disputed
Same endpoint, defensible, evidence-backed admitted
Systems are averaged over materially different supports support/comparability check
Same semantic specification, different resamples/seeds sampling/stochastic uncertainty
Different semantic specifications, fixed substrate semantic identified set / envelope

There is some useful measurement-theory vocabulary behind the first few distinctions. For example, MetricEval separates properties such as metric stability, consistency, and construct validity rather than treating “a metric exists” as enough to establish what it measures.

Likewise, specification/multiverse methods have long had to confront the problem of deciding which analysis choices belong in the universe at all. The Specification Curve Analysis framing asks for specifications that are theoretically justified, statistically valid, and non-redundant; the newer Multicurious discussion makes the defensibility/equivalence distinction particularly explicit.

None of these provides an automatic admission oracle for benchmark semantics. To me, that is actually evidence that making the admission reasoning inspectable is worthwhile.

Two smaller protocol ideas

1. The minimum of two admitted specifications

In v0.3.0, a declaration with only one admitted specification is rejected:

ERROR: specifications must contain at least 2 entries with admission=admitted

I assume the intent may be to avoid calling a trivial singleton family a nontrivial robustness result.

If so, another representation might be:

primary family:
    1 admitted specification

status:
    primary meaning is singular
    across-primary-meaning robustness not tested

sensitivity envelope:
    admitted + disputed alternatives

That would preserve the distinction without creating pressure to promote a second candidate into the primary family merely so that the audit is executable.

I would treat this as a reporting/policy question rather than a correctness issue.

2. A lightweight family-freeze receipt

Your paper emphasizes outcome-independent, jointly frozen alternatives. The current source/hash bindings are already useful for showing what bytes/evidence were used, but there is a slightly different question:

can a third party verify that family membership was fixed before inspecting the alternative-specific outcomes?

A low-cost option could be something like:

family_manifest.json
declared_at
family_sha256
optional evidence receipt

or simply a scoreless family declaration that can be hashed before the score-bearing audit is executed.

This would distinguish:

integrity:
    "these are the exact declared files"

from

chronology:
    "these alternatives were frozen before their results were inspected"

I do not mean that the current demos were selected post hoc; the current schema simply does not appear to represent that chronology directly.

There may also be room not to reinvent general evaluation provenance. Projects such as Every Eval Ever are already working on common metadata for evaluation runs, sources, timestamps, generation configuration, metric interpretation, etc. Conceptually, the division could be:

general eval provenance
    -> where did this result/run come from?

EvalAudit
    -> why is this semantic interpretation in this family,
       and which claims survive the family?
Small reproduction note for the two concrete examples

For AgentDojo, the reconstruction used the pinned public run material for the three systems in the v0.3.0 declaration.

The notable support result was:

                         available    common
Claude 3.5 Sonnet           629         629
Gemini 2.0 Flash            949         629
GPT-4o-mini                 629         629

and:

Gemini all-available ASR     14.120126%
Gemini common-key ASR        20.826709%
Gemini-only 320-key ASR       0.937500%

The ordering was unchanged, so I would interpret this as a support-sensitivity witness, not a winner-reversal witness.

For AutoML, all five declared v0.3.0 specifications were independently reconstructed from the bound frozen CSV with maximum absolute numerical disagreement of roughly:

4.8e-11

The frozen rows span:

2019-02-22 to 2019-03-07

while the bound upstream repository revision is from:

2025-04-19

and the canonical AMLB paper appeared in 2024.

Again, I do not think that timeline tells us which evidence horizon is correct. It only makes the policy choice visible.

The important reproduction boundary is the same one you already state in the post: this replays deterministic calculations over frozen public material; it is not an independent regeneration of the original benchmark runs or a proof that the declared semantic family is complete.

So, if I had to compress all of this into one recommendation, it would be:

Keep the claim-relative identified-set idea, but make the path from “candidate evaluator” to “admitted family member” a little more typed.

In particular, separating

support/comparability
defect/reliability
independent defensibility
claim-relative equivalence
admission evidence

seems to make the interesting disagreements easier to state without forcing them into one yes/no admission judgment.

And I think the stable-winner examples are useful here: AgentDojo in particular gives a case where a support choice moves the magnitude substantially, while the winner survives. That is exactly the sort of situation where “winner identified”, “some pairwise relations identified”, and “full leaderboard identified” should probably remain different statements.

4

Thank you — this is an unusually helpful response, especially the independent AgentDojo/AutoML reconstructions and the way you separated provenance, defensibility, equivalence, and admission.

We are iterating the paper and executable audit quite rapidly, and the current arXiv version is already substantially behind the working manuscript. A few things in your comment are strikingly aligned with changes that are already in the next revision.

One small preview: the new version is organized around a claim-relative inference layer rather than a generic robustness label. It now separates

  • evidence stopping — when the historical substrate needed to replay a claim is unavailable,

  • semantic multiplicity — when more than one grounded evaluator meaning remains live, and

  • claim-resolution separation — when exact values, winners, complete orders, and pairwise relations have different identified sets.

We also now explicitly separate origin, admission, and execution: source exposure establishes provenance, but not automatically same-target admissibility. The executable record carries support/missingness decisions, endpoint type, origin/admission, exposure state, identified sets, minimum witnesses, and the stable pairwise backbone.

The empirical side has expanded quite a bit too. At a pinned Inspect Evals commit we now have a mechanically defined 124-unit finite frame with a terminal disposition for every unit; most stop before claim-level replay because some historical observation, judge/state, support, or semantic-registry object is missing. We treat those as typed evidence stops rather than zero effects or benchmark defects.

Your AgentDojo support example is particularly useful to us because it illustrates exactly the kind of result we want the framework to preserve: a reported quantity can move substantially while the winner remains identified.

Two suggestions in your post that are not yet as explicit as they should be in our current working version are the evidence horizon for provenance judgments and a more framework-native family-freeze receipt. We are taking both seriously.

So thank you — this is very close to the kind of external pressure test we hoped the public thread would produce. The next preprint revision should make the current direction much clearer.