# mushin

> Boilerplate-free, reproducible machine-learning experiment sweeps: a framework-agnostic sweep engine built on hydra-zen, with first-class PyTorch Lightning integration. Decorate an experiment, sweep its parameters, and get the results back as a labeled `xarray.Dataset`.

Facts to get right when writing code that uses mushin:

- Install with `pip install mushin-py`, import with `import mushin`. The PyPI name is not `mushin`.
- The core install is the sweep -> dataset workflow. `compare`, the metric batteries, LLM evaluation (`mushin.llm`) and `Study` need the `eval` extra: `pip install "mushin-py[eval]"`. Other extras: `viz`, `netcdf`, `detection`, `image`, `audio`, `mcp`.
- Supported Python: 3.11 to 3.14.
- The public API is the top-level `mushin` namespace plus the documented `mushin.benchmark` and `mushin.llm` symbols. Modules whose names start with an underscore are internal.
- A task returns a `dict` of metrics; each key becomes a data variable. Each `mushin.multirun([...])` argument becomes a dimension of the dataset. Use `mushin.hydra_list([...])` to pass a list as one value instead of sweeping it.
- `pip install mushin-py` also installs torch (>= 2.4) and pytorch-lightning, even for a task that never imports them. Intel macOS is not supported: PyTorch publishes no x86_64 macOS wheels after torch 2.2.
- The sweep layer is framework-agnostic (scikit-learn, XGBoost, plain NumPy). `HydraDDP`, `MetricsCallback`, auto-tuning, `compare` and the batteries assume PyTorch / Lightning.
- To set up, run and report an experiment, read the [agent guide](https://raw.githubusercontent.com/martinez-hub/mushin/main/src/mushin/agent_guide.md) first. On mushin 0.16 or later, `python -m mushin guide` prints the copy for the installed version; the web copy applies from mushin 0.10 on.

```python
import mushin


@mushin.sweep
def experiment(lr, seed):
    accuracy = 0.9 - abs(lr - 0.1) + 0.001 * seed  # train and evaluate here
    return dict(accuracy=accuracy)


ds = experiment.run(
    lr=mushin.multirun([0.01, 0.1, 1.0]), seed=mushin.multirun([0, 1, 2])
)
ds["accuracy"].mean("seed")  # dims (lr, seed); average over seeds
experiment.workflow.to_dataframe()  # the same results as a tidy pandas table
```

## Docs

- [Home](https://martinez-hub.github.io/mushin/): what mushin is and a tour of its features
- [Install](https://martinez-hub.github.io/mushin/install/): extras table and the platform, torch and NumPy support matrix
- [Quickstart](https://martinez-hub.github.io/mushin/quickstart/): run a sweep and get a labeled `xarray.Dataset` back
- [For AI agents](https://martinez-hub.github.io/mushin/for-agents/): the usage guide for coding agents, and a block to paste into a project's `AGENTS.md` or `CLAUDE.md`
- [Tutorial](https://martinez-hub.github.io/mushin/tutorial/): the full workflow end to end, from sweep to statistical comparison
- [Core concepts](https://martinez-hub.github.io/mushin/concepts/): workflows, the (method x seed) dataset, significance testing, the task registry

## Guides

- [Workflows & sweeps](https://martinez-hub.github.io/mushin/guides/workflows/): `@mushin.sweep`, `MultiRunMetricsWorkflow`, launchers, using mushin alongside W&B or TensorBoard
- [Auto-tuning batch & LR](https://martinez-hub.github.io/mushin/guides/auto-tuning/): `tune_batch_size` and `tune_learning_rate`, pinned for reproducibility
- [Comparing methods](https://martinez-hub.github.io/mushin/guides/compare/): `benchmark.compare` across seeds with significance tests
- [Built-in batteries](https://martinez-hub.github.io/mushin/guides/batteries/): classification, segmentation, detection, regression, retrieval, image-quality and audio metrics
- [LLM evaluation](https://martinez-hub.github.io/mushin/guides/llm/): `compare_llms`, `compare_scores` and `llm_judge`, with Holm-corrected p-values and an output cache
- [Studies](https://martinez-hub.github.io/mushin/guides/study/): `Study` runs the training sweep and feeds it into `compare`
- [Resilient & resumable sweeps](https://martinez-hub.github.io/mushin/guides/resilience/): `on_error="nan"`, `resume=True` across hard kills and preemption, per-run provenance
- [From exploration to a paper](https://martinez-hub.github.io/mushin/guides/exploration-to-paper/): moving from a quick exploratory grid to a paper-ready sweep
- [Packing small jobs onto GPUs](https://martinez-hub.github.io/mushin/guides/packing/): `pin_gpu_round_robin` to co-locate small sweep jobs
- [Multi-node training](https://martinez-hub.github.io/mushin/guides/multinode/): `HydraDDP` on SLURM or Elastic clusters, including preemption and resume
- [Sharded training](https://martinez-hub.github.io/mushin/guides/sharding/): `HydraFSDP` for models that do not fit on one GPU
- [Segmentation](https://martinez-hub.github.io/mushin/guides/segmentation/): comparing segmentation models with the segmentation battery
- [Custom metrics & predict_fn](https://martinez-hub.github.io/mushin/guides/custom/): custom tasks, metrics and prediction functions, `register_task`
- [Understanding the statistics](https://martinez-hub.github.io/mushin/guides/statistics/): test selection, seed counts and the Holm correction
- [Analyzing from Claude Code](https://martinez-hub.github.io/mushin/guides/mcp/): the read-only `mushin-mcp` server for inspecting completed runs from an MCP client

## API reference

- [analysis](https://martinez-hub.github.io/mushin/reference/analysis/): `show`, `best`, `diff` and CSV export over a sweep directory
- [benchmark](https://martinez-hub.github.io/mushin/reference/benchmark/): `compare` and `BenchmarkResult`
- [llm](https://martinez-hub.github.io/mushin/reference/llm/): `compare_llms`, `compare_scores`, `llm_judge`
- [study](https://martinez-hub.github.io/mushin/reference/study/): `Study`
- [workflows](https://martinez-hub.github.io/mushin/reference/workflows/): `sweep`, `multirun`, `hydra_list`, `MultiRunMetricsWorkflow`
- [lightning](https://martinez-hub.github.io/mushin/reference/lightning/): `HydraDDP` and `MetricsCallback`
- [tuning](https://martinez-hub.github.io/mushin/reference/tuning/): `tune_batch_size`, `tune_learning_rate`
- [utils](https://martinez-hub.github.io/mushin/reference/utils/): `load_experiment`, `load_from_checkpoint`, `original_cwd`

## Examples

- [Examples](https://martinez-hub.github.io/mushin/examples/): index of the runnable, CI-tested scripts in `examples/`
- [Sweeps → datasets](https://martinez-hub.github.io/mushin/notebooks/01_sweep_to_dataset/): notebook
- [Compare & batteries](https://martinez-hub.github.io/mushin/notebooks/02_compare_and_batteries/): notebook
- [Studies](https://martinez-hub.github.io/mushin/notebooks/03_study/): notebook
- [Resilient sweeps](https://martinez-hub.github.io/mushin/notebooks/04_resilience/): notebook
- [LLM evaluation](https://martinez-hub.github.io/mushin/notebooks/05_llm_eval/): notebook
- [scikit-learn (framework-agnostic)](https://martinez-hub.github.io/mushin/notebooks/06_sklearn_framework_agnostic/): notebook, a task that uses no torch
- [Analyzing your results](https://martinez-hub.github.io/mushin/notebooks/07_analyzing_results/): notebook
- [LLM examples end to end](https://martinez-hub.github.io/mushin/notebooks/08_llm_examples/): notebook

## Optional

- [Contributing](https://martinez-hub.github.io/mushin/contributing/): dev setup and the local gate
- [Changelog](https://martinez-hub.github.io/mushin/changelog/): release notes, including migration notes for breaking changes
- [AGENTS.md](https://github.com/martinez-hub/mushin/blob/main/AGENTS.md): guidance for coding agents changing mushin itself