LLM evaluation: is that gap real?¶

You ran an eval. System A scored 72%, system B scored 58%. Should you switch?

A gap like that can be luck in two independent ways, and both have to be ruled out before it means anything:

the question	what goes wrong	mushin's answer
Would a re-run agree?	models sample, so the score moves run to run	the seed axis: `p_value` / `p_corrected`
Would other items agree?	you picked 40 questions; another 40 might rank them the other way	the item axis: `item_p` / `item_p_corrected`

The second is usually the larger risk and the one most harnesses never report. This notebook is self-contained: no API keys, no network, no model calls.

Download this notebook

Requirements: LLM evaluation lives in the eval extra: %pip install "mushin-py[eval]".

1. You already have scores: `compare_scores`¶

The lowest-friction entry point. If another harness (Inspect AI, lm-eval-harness, your own runner) already produced per-item scores, hand mushin the array and it does the statistics. It never touches your execution loop.

Shape is (n_runs, n_items): rows are repeats of the whole eval, columns are items.

In [ ]:

Copied!

import warnings

import numpy as np

rng = np.random.default_rng(0)

# 5 repeats x 40 questions. `strong` is genuinely better on most items.
difficulty = rng.normal(size=40)
strong = (rng.random((5, 40)) < 1 / (1 + np.exp(-(difficulty + 0.9)))).astype(float)
weak = (rng.random((5, 40)) < 1 / (1 + np.exp(-difficulty))).astype(float)

print(f"strong {strong.mean():.1%}   weak {weak.mean():.1%}   shape {strong.shape}")
import warnings

import numpy as np

rng = np.random.default_rng(0)

# 5 repeats x 40 questions. `strong` is genuinely better on most items.
difficulty = rng.normal(size=40)
strong = (rng.random((5, 40)) < 1 / (1 + np.exp(-(difficulty + 0.9)))).astype(float)
weak = (rng.random((5, 40)) < 1 / (1 + np.exp(-difficulty))).astype(float)

print(f"strong {strong.mean():.1%}   weak {weak.mean():.1%}   shape {strong.shape}")

Now the mushin part: two lines:

In [ ]:

Copied!

from mushin.llm import compare_scores

result = compare_scores({"strong": strong, "weak": weak})
result.comparisons
from mushin.llm import compare_scores

result = compare_scores({"strong": strong, "weak": weak})
result.comparisons

Every column you need to defend the claim:

column	answers
`mean_diff`	how big is the gap
`p_value` / `p_corrected`	would a re-run agree (seed axis)
`item_p` / `item_p_corrected`	would other items agree (item axis)
`item_ci_low` / `item_ci_high`	interval on the per-item difference
`effect_size`	Cohen's d
`significant`	`p_corrected < alpha`

The _corrected columns apply the multiplicity correction. With two systems there is one comparison and they equal the raw values; with three or more they matter on both axes: comparing more systems multiplies the chance of a spurious win wherever you look.

In [ ]:

Copied!

result.summary()
result.summary()

When the two axes disagree¶

This is the case worth internalising. Here the seed axis is decisive and the item axis is not: the systems differ consistently on these items, but a different item set could easily reverse it. Reporting only the seed p-value would call this proven.

In [ ]:

Copied!





# A tiny, consistent edge: same direction every run, but small next to how much
# items vary among themselves.
base = rng.normal(size=40)
a = np.tile(base, (5, 1)) + rng.normal(scale=0.01, size=(5, 40)) + 0.05
b = np.tile(base, (5, 1)) + rng.normal(scale=0.01, size=(5, 40))

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    row = compare_scores({"a": a, "b": b}).comparisons.iloc[0]

print(f"seed axis : p={row['p_value']:.2g}  -> looks decisive")
print(
    f"item axis : p={row['item_p']:.3f}, 95% CI "
    f"[{row['item_ci_low']:+.3f}, {row['item_ci_high']:+.3f}]  -> straddles 0"
)
# A tiny, consistent edge: same direction every run, but small next to how much
# items vary among themselves.
base = rng.normal(size=40)
a = np.tile(base, (5, 1)) + rng.normal(scale=0.01, size=(5, 40)) + 0.05
b = np.tile(base, (5, 1)) + rng.normal(scale=0.01, size=(5, 40))

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    row = compare_scores({"a": a, "b": b}).comparisons.iloc[0]

print(f"seed axis : p={row['p_value']:.2g}  -> looks decisive")
print(
    f"item axis : p={row['item_p']:.3f}, 95% CI "
    f"[{row['item_ci_low']:+.3f}, {row['item_ci_high']:+.3f}]  -> straddles 0"
)

2. Grouped items need `clusters=`¶

Eval items are often not independent: several questions about one passage, several prompts from one document, several attacks of one shape. Those items share whatever makes their source easy or hard, so resampling them individually treats correlated observations as independent and reports an interval that is too narrow.

Pass a per-item group label and the bootstrap resamples whole groups:

In [ ]:

Copied!





# 40 items that are really 5 groups of 8 — a strong per-group effect.
group = np.repeat(np.arange(5), 8)
group_effect = rng.normal(scale=1.0, size=5)[group]
ga = np.tile(group_effect, (5, 1)) + rng.normal(scale=0.3, size=(5, 40)) + 0.15
gb = np.tile(group_effect, (5, 1)) + rng.normal(scale=0.3, size=(5, 40))

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    naive = compare_scores({"a": ga, "b": gb}).comparisons.iloc[0]
    grouped = compare_scores({"a": ga, "b": gb}, clusters=group).comparisons.iloc[0]

for name, r in (("ignoring the grouping", naive), ("clusters=group", grouped)):
    print(
        f"{name:<22} item_p={r['item_p']:.4f}  "
        f"CI [{r['item_ci_low']:+.3f}, {r['item_ci_high']:+.3f}]"
    )
# 40 items that are really 5 groups of 8 — a strong per-group effect.
group = np.repeat(np.arange(5), 8)
group_effect = rng.normal(scale=1.0, size=5)[group]
ga = np.tile(group_effect, (5, 1)) + rng.normal(scale=0.3, size=(5, 40)) + 0.15
gb = np.tile(group_effect, (5, 1)) + rng.normal(scale=0.3, size=(5, 40))

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    naive = compare_scores({"a": ga, "b": gb}).comparisons.iloc[0]
    grouped = compare_scores({"a": ga, "b": gb}, clusters=group).comparisons.iloc[0]

for name, r in (("ignoring the grouping", naive), ("clusters=group", grouped)):
    print(
        f"{name:<22} item_p={r['item_p']:.4f}  "
        f"CI [{r['item_ci_low']:+.3f}, {r['item_ci_high']:+.3f}]"
    )

The clustered interval is wider, and that width is the honest one.

!!! warning "Few clusters stay unreliable" Clustering fixes the unit of resampling, not the sample size. What counts is the number of groups: a 500-item eval set drawn from 4 passages is a 4-sample problem. mushin warns below ~20 groups; believe it, and widen the variety of your sources rather than adding more items to the same few.

3. Let mushin run the systems: `compare_llms`¶

One level up. Instead of bringing scores, bring the systems and a metric, and mushin runs each system across seeds, scores every (system, seed) pair, and compares them.

A system is any callable system(inputs, seed) -> outputs. Here they are stand-ins with a known success rate so the notebook needs no network; swap in your own API calls and nothing else changes.

In [ ]:

Copied!





import hashlib

from mushin.llm import compare_llms

QUESTIONS = [f"question-{i}" for i in range(40)]


def _draw(*key) -> float:
    """Stable [0, 1) draw. blake2b, not hash(): hash() is randomised per process,"""
    """so the notebook would print different numbers on every run."""
    digest = hashlib.blake2b("|".join(map(str, key)).encode(), digest_size=8).digest()
    return int.from_bytes(digest, "big") / 2**64


def make_system(accuracy: float):
    def system(inputs, seed):
        return [
            "correct" if _draw(accuracy, seed, q) < accuracy else "wrong"
            for q in inputs
        ]

    return system


def exact_match(output, reference):
    return float(output == reference)


data = [{"input": q, "reference": "correct"} for q in QUESTIONS]

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    llm = compare_llms(
        {"gpt-ish": make_system(0.72), "claude-ish": make_system(0.58)},
        data=data,
        metric={"accuracy": exact_match},  # named, so the table reads `accuracy`
        seeds=range(5),
    )
llm.comparisons
import hashlib

from mushin.llm import compare_llms

QUESTIONS = [f"question-{i}" for i in range(40)]


def _draw(*key) -> float:
    """Stable [0, 1) draw. blake2b, not hash(): hash() is randomised per process,"""
    """so the notebook would print different numbers on every run."""
    digest = hashlib.blake2b("|".join(map(str, key)).encode(), digest_size=8).digest()
    return int.from_bytes(digest, "big") / 2**64


def make_system(accuracy: float):
    def system(inputs, seed):
        return [
            "correct" if _draw(accuracy, seed, q) < accuracy else "wrong"
            for q in inputs
        ]

    return system


def exact_match(output, reference):
    return float(output == reference)


data = [{"input": q, "reference": "correct"} for q in QUESTIONS]

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    llm = compare_llms(
        {"gpt-ish": make_system(0.72), "claude-ish": make_system(0.58)},
        data=data,
        metric={"accuracy": exact_match},  # named, so the table reads `accuracy`
        seeds=range(5),
    )
llm.comparisons

Name your metric (metric={'accuracy': ...} rather than a bare callable) or the result table labels the column score, which tells a reader nothing.

result.data is the same thing as a labelled xarray.Dataset, indexed by system and seed, useful when you want to plot or slice rather than read a verdict:

In [ ]:

Copied!

llm.data
llm.data

4. The sweep half: surviving a crash¶

Everything above is the statistics. The other half of mushin is the sweep, and for LLM work it earns its place, because every cell of the grid costs money and cells fail transiently (a rate limit, a timeout, a 503).

on_error="nan": a failed cell becomes NaN; the rest of the grid finishes
resume=True: a re-run reuses completed cells and retries only the holes
to_xarray(): results labelled by every parameter you swept

In [ ]:

Copied!





import tempfile

import mushin

OUTAGE = True  # a stand-in for a flaky provider


@mushin.sweep
def score_config(prompt: str, temperature: float, seed: int) -> dict:
    if OUTAGE and _draw("outage", prompt, temperature, seed) < 0.15:
        raise RuntimeError("429 rate limited (simulated)")
    base = {"terse": 0.55, "cot": 0.72, "role": 0.63}[prompt]
    hits = [
        _draw("score", prompt, temperature, seed, q) < base - 0.25 * temperature
        for q in QUESTIONS
    ]
    return {"accuracy": float(np.mean(hits))}


workdir = tempfile.mkdtemp()
grid = dict(
    prompt=mushin.multirun(["terse", "cot", "role"]),
    temperature=mushin.multirun([0.0, 0.4]),
    seed=mushin.multirun(list(range(4))),
)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    score_config.run(**grid, working_dir=workdir, on_error="nan")

ds = score_config.workflow.to_xarray()
failed = int(np.isnan(ds["accuracy"].values).sum())
print(f"{ds['accuracy'].size} cells, {failed} failed transiently")
for w in caught:
    if "run(s) failed" in str(w.message):
        print("mushin:", str(w.message)[:90], "...")
ds
import tempfile

import mushin

OUTAGE = True  # a stand-in for a flaky provider


@mushin.sweep
def score_config(prompt: str, temperature: float, seed: int) -> dict:
    if OUTAGE and _draw("outage", prompt, temperature, seed) < 0.15:
        raise RuntimeError("429 rate limited (simulated)")
    base = {"terse": 0.55, "cot": 0.72, "role": 0.63}[prompt]
    hits = [
        _draw("score", prompt, temperature, seed, q) < base - 0.25 * temperature
        for q in QUESTIONS
    ]
    return {"accuracy": float(np.mean(hits))}


workdir = tempfile.mkdtemp()
grid = dict(
    prompt=mushin.multirun(["terse", "cot", "role"]),
    temperature=mushin.multirun([0.0, 0.4]),
    seed=mushin.multirun(list(range(4))),
)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    score_config.run(**grid, working_dir=workdir, on_error="nan")

ds = score_config.workflow.to_xarray()
failed = int(np.isnan(ds["accuracy"].values).sum())
print(f"{ds['accuracy'].size} cells, {failed} failed transiently")
for w in caught:
    if "run(s) failed" in str(w.message):
        print("mushin:", str(w.message)[:90], "...")
ds

A labelled Dataset (prompt x temperature x seed) with the failed cells as NaN and everything else intact. A harness that discarded the whole grid on one 429 would have thrown away the cells you already paid for.

Now fix the cause and resume. Only the holes are retried:

In [ ]:

Copied!





OUTAGE = False  # 'we fixed it'

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    score_config.run(**grid, working_dir=workdir, resume=True, on_error="nan")

ds2 = score_config.workflow.to_xarray()
print("still missing:", int(np.isnan(ds2["accuracy"].values).sum()))
ds2["accuracy"].mean("seed").to_dataframe().unstack("temperature")
OUTAGE = False  # 'we fixed it'

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    score_config.run(**grid, working_dir=workdir, resume=True, on_error="nan")

ds2 = score_config.workflow.to_xarray()
print("still missing:", int(np.isnan(ds2["accuracy"].values).sum()))
ds2["accuracy"].mean("seed").to_dataframe().unstack("temperature")

Picking the winner is one reduction over named dimensions, and needs no bookkeeping over run directories:

In [ ]:

Copied!





mean_acc = ds2["accuracy"].mean("seed")
flat = mean_acc.stack(cell=("prompt", "temperature"))
best = flat.isel(cell=int(flat.argmax("cell")))
print(
    f"best: prompt={str(best['prompt'].values)!r} "
    f"temperature={float(best['temperature'].values)}  ({float(best):.1%})"
)
mean_acc = ds2["accuracy"].mean("seed")
flat = mean_acc.stack(cell=("prompt", "temperature"))
best = flat.isel(cell=int(flat.argmax("cell")))
print(
    f"best: prompt={str(best['prompt'].values)!r} "
    f"temperature={float(best['temperature'].values)}  ({float(best):.1%})"
)

Where to go next¶

LLM evaluation guide: the full API, including llm_judge, the output cache, and what the item bootstrap does and does not tell you
examples/inspect_ai_compare.py: feed Inspect AI eval logs into compare_scores, matching questions by sample.id
examples/prompt_injection_eval.py: indirect prompt-injection resistance, with a per-attack breakdown that finds a blind spot the aggregate score hides
examples/llm_prompt_sweep.py: the sweep above, end to end, with resume across processes

LLM evaluation: is that gap real?¶

1. You already have scores: compare_scores¶

When the two axes disagree¶

2. Grouped items need clusters=¶

3. Let mushin run the systems: compare_llms¶

4. The sweep half: surviving a crash¶

Where to go next¶

1. You already have scores: `compare_scores`¶

2. Grouped items need `clusters=`¶

3. Let mushin run the systems: `compare_llms`¶