llm

compare_llms

compare_llms(
    systems,
    data,
    metric,
    seeds=(0, 1, 2, 3, 4),
    *,
    test="welch",
    alpha=0.05,
    correction="holm",
    cache=None,
    item_bootstrap=10000,
    clusters=None,
)

compare_scores

compare_scores(
    scores,
    *,
    metric_name="score",
    clusters=None,
    test="welch",
    alpha=0.05,
    correction="holm",
    item_bootstrap=10000,
)

Compare systems from already-computed per-item scores.

Use this when another harness (inspect-ai, lm-eval-harness, your own runner) already produced per-item results and you only want the analysis: you get the same :class:BenchmarkResult, corrections and item-level bootstrap as :func:compare_llms without handing mushin your execution loop.

scores maps a system name to either

• a 1-D sequence of per-item scores: a single run. There is no seed dimension, so the seed-based columns are masked (a single run says nothing about decoding noise) while the item-level columns are fully populated; or
• a 2-D (n_seeds, n_items) array, repeated runs, giving both the seed-based test and the item-level bootstrap.

Every system must cover the same items in the same order: item i of one system is paired with item i of another. Call once per metric.

clusters is passed to :func:~mushin.benchmark._stats.paired_item_bootstrap. Supply it when items are grouped (several questions per passage), or the interval will be too narrow.

llm_judge

llm_judge(
    judge,
    rubric,
    *,
    parse=parse_score,
    template=default_template,
    seed=0,
)

Return a pointwise metric (output, reference) -> float that asks judge (a user-supplied, provider-agnostic judge(prompt, seed) -> reply) to score each output against rubric, parsing the reply to a float.