benchmark

compare

The compare facade: evaluate methods on a task battery and report significance.

compare

compare(
    methods,
    data,
    task="classification",
    *,
    num_classes=None,
    predict_fn=None,
    metrics=None,
    prob_metrics=None,
    test="wilcoxon",
    alpha=0.05,
    correction="holm",
    ignore_index=None,
    device=None,
)

Compare methods on a standard battery and report significance.

Parameters:

Name	Type	Description	Default
methods	dict[str, Sequence[Module]]	{method_name: [model_per_seed, ...]}; every model is evaluated on data, giving one score per (method, seed).	required
data	Iterable	A re-iterable batch source (e.g. a DataLoader); it is streamed once per model.	required
task	str or Task	A registered task name ("classification", "segmentation", "detection", "regression", "retrieval", "image_quality", "audio", or a custom one — see list_tasks()) or a Task object.	'classification'
num_classes	int or None	Required (when metrics is not provided) only for tasks whose battery needs it — "classification" and "segmentation"; ignored for the others.	None
predict_fn	callable or None	(model, x) -> (preds, probs); defaults to the task's.	None
metrics	dict[str, Metric] or None	Replaces the task's battery entirely. A custom battery gets no implicit probability routing — name the probability-consuming entries in prob_metrics.	None
prob_metrics	frozenset[str] or None	Battery entries fed probabilities instead of hard predictions. None means the task's own set when the task's battery is used, and the empty set when a custom metrics= battery is given. Every name must be a key of the battery in use.	None
test	str	Statistical test for pairwise method comparison; one of mushin.benchmark.available_tests() (default paired Wilcoxon).	'wilcoxon'
alpha	float	Significance level for the (multiple-comparison-corrected) comparisons.	0.05
correction	str	Multiple-comparison correction applied per metric across method pairs: "holm" (default), "bonferroni", "fdr_bh" (Benjamini-Hochberg FDR), or "none" for raw p-values.	'holm'
ignore_index	int or None	Label to exclude from segmentation metrics (e.g. a void/boundary class).	None
device	device or None	Evaluation device; defaults to the device of each model's parameters.	None

BenchmarkResult dataclass

Holds benchmark results.

Attributes:

Name	Type	Description
data	Dataset	Dims (method, seed), one data variable per metric.
comparisons	DataFrame	Pairwise significance results (see _stats.compare_methods).
alpha	float	Significance level used.

summary

summary(reference=None)

Publication-ready table: per method/metric mean and CI, with a "*" marker when the method differs significantly from reference (default: the first method in data).