Changelog

All notable changes to this project are documented here. The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.15.1 - 2026-10-15

No changes to mushin's code since 0.15.0. This release fixes the source tree's uv configuration, which uv 0.12.14 and later could not re-resolve (#243).

Misc

• 242, #243, #245, #247

0.15.0 - 2026-10-14

Changed

• Sweeps now require hydra-zen 0.17.0 or newer, which builds Hydra sweepers through the plugin registry. mushin no longer needs its workaround for hydra-core refusing the hydra._internal sweeper target, and that code is gone. hydra-core must now be 1.3.7 or newer: on 1.3.6, hydra-zen 0.17.0 fails to instantiate any config that carries a callable, and 1.3.7 also fixes the hydra-core security advisory GHSA-mwj6-rfh8-7qf4. (#235)

Fixed

• The statistics guide said 5 seeds make a Wilcoxon test viable; the floor is 6 (its best-case two-sided p-value is 2 ** (1 - n_seeds)), so the documented remedy for the underpowered-test warning produced the same warning. The guide no longer recommends the default Wilcoxon test for small seed counts either, and the LLM guide's seed-count advice no longer states a fixed threshold as if power were independent of effect size. (#228)

Removed

• Python 3.10 is no longer supported: it reached end-of-life on 2026-10-01, and mushin now requires Python >= 3.11. Python 3.15 is not supported yet; the install guide explains what it is waiting on. (#236)
• Intel macOS (x86_64) is no longer supported: PyTorch publishes no x86_64 macOS wheels after torch 2.2, which held mushin there to torch 2.2.x and NumPy 1.x. mushin 0.14.x, which supports Intel Macs on Python 3.10 and 3.11, is the last series that runs there (python3.11 -m pip install "mushin-py<0.15"). Apple Silicon Macs remain fully supported and tested. (#239)

Misc

• 221, #229, #240

0.14.2 - 2026-09-23

Added

• The workflows guide now documents a known upstream incompatibility: on hydra-core >= 1.3.6, a config built with both zen_partial=True and zen_wrappers= raises InstantiationException: Callable targets cannot return partial subclasses. hydra-core 1.3.6 hardened instantiate so a callable target may no longer return a functools.partial subclass, which is what hydra-zen produces when those two options are combined — either alone is fine. The hardening carries forward, so 1.3.7 rejects it too. mushin itself is unaffected (it uses zen(), a different feature, and never zen_wrappers), but the error names neither Hydra nor hydra-zen, so it is easy to mistake for a bug in your own task function. The note gives the fix: change the config — drop one of the two options — rather than the dependency. (Pinning cannot help here: the rejection is deliberate security hardening that still applies on 1.3.7, and every version an exclusion falls back to carries a high-severity advisory the newer release fixes.) (#205)

Fixed

• Sweeps work again on hydra-core 1.3.7. 1.3.7 refuses to instantiate the hydra._internal sweeper target that hydra-zen's launch names directly, so every multirun raised InstantiationException on a fresh install. mushin now marks that one target trusted around its own launch call — the same exemption Hydra applies to its own sweeper — rather than capping hydra-core, which would have withheld 1.3.7's security fix (GHSA-mwj6-rfh8-7qf4, high, hydra-core >= 1.3.4, < 1.3.7) and put a hard upper bound on a library. The fix belongs upstream in hydra-zen; it can be deleted once launch routes the sweeper through Plugins. (#211)
• Corrected the workflows guide's advice on the zen_partial + zen_wrappers incompatibility. It previously suggested pinning hydra-core != 1.3.6, which does not work: the rejection is deliberate security hardening that still applies on 1.3.7, and every version the exclusion would fall back to carries a high-severity advisory the newer release fixes. The guide now says to change the config instead, and the section is retitled for hydra-core >= 1.3.6 rather than 1.3.6 alone. (#214)

Misc

• 198, #200, #201, #203, #204

0.14.1 - 2026-08-24

Added

• New notebook LLM examples end to end walks the LLM-evaluation layer in one place: compare_scores when another harness already produced your per-item scores, why the seed axis and the item axis can disagree, clusters= for grouped items, compare_llms when you want mushin to run the systems, and the sweep half surviving a simulated rate-limit outage with on_error="nan" and resume=True. Self-contained — no API keys, no network, and deterministic, so the numbers are the same on every machine.

Changed

• Each of the three LLM examples now opens with a "The mushin part, in full" block in its module docstring, showing the actual API calls before the scaffolding that surrounds them. examples/inspect_ai_compare.py is 395 lines of which two call mushin — the Inspect adapter and the print formatting are the bulk — so a reader opening the file to learn the library could reasonably conclude they never saw any mushin code. report() in that example also drops from 117 lines to 57 by extracting its two verdict formatters, leaving the compare_scores call and the columns it returns visible rather than buried. No behaviour changes: all three demos produce byte-identical output.

Fixed

• examples/inspect_ai_compare.py now sorts question ids in their own order rather than as strings. Inspect numbers samples 1, 2, 3... by default, and sorting those lexicographically returned [1, 10, 11, ... 2, 20]. Comparisons were correct either way — every model got the same order, and building clusters= from the returned id list (as documented) gave identical results before and after — but the surprising order was a trap for anyone lining clusters= up against their own ascending item order instead. Ids with no total order (mixed types) still fall back to string ordering. Separately, a sample that errored during a real run is now reported as unscored: Inspect writes scores={} for those, not None, so the check missed them and they surfaced as a confusing "pass scorer=" error instead. The adapter is now tested against real inspect_ai EvalLog objects rather than only duck-typed stand-ins, which is what surfaced the scores={} behaviour.

0.14.0 - 2026-08-23

Added

• compare_llms now reports a paired item-level bootstrap (item_diff, item_ci_low, item_ci_high, item_p) beside the seed-based test, so you can see whether a difference would survive a different sample of eval items — usually the larger uncertainty. Disable with item_bootstrap=0. (#183)
• LLM evaluation gains clusters= for grouped eval items (several questions per passage): resampling whole groups instead of individual items, because ignoring the grouping covers a nominal 95% interval only about half the time. NaN group labels are rejected rather than silently dropped, and the small-sample warning counts groups rather than items. New compare_scores runs the same analysis on per-item scores you already have from another harness, so you can use mushin's statistics without adopting its execution loop. (#185)
• New examples/inspect_ai_compare.py answers the question Inspect AI leaves open — your eval says one model scored 60% and another 56.7%, but is that gap real? It checks whether the difference survives a re-run (Inspect epochs) and whether it would survive a different set of questions (an item bootstrap), and reports a plain verdict. Run --demo to see it on two scenarios with known ground truth. Questions are matched across logs by sample.id rather than by position, since matching positionally would compare unrelated questions. (#186)
• New examples/prompt_injection_eval.py measures resistance to indirect prompt injection — the attack arriving inside retrieved content rather than from the user — using both halves of mushin: compare_llms for whether a resistance gap survives a re-run and a different attack set, and @mushin.sweep to break the result down by attack shape. The demo shows why the breakdown matters: the system with the better aggregate score is quietly weaker against one attack family. Detection uses an inert canary; --demo needs no model or keys. (#187)
• New examples/llm_prompt_sweep.py shows the sweep half of mushin on an LLM tuning problem: prompt template x temperature x seed, where every cell costs money and cells fail transiently. on_error="nan" keeps the completed cells when a call is rate-limited, resume=True retries only the holes, and the labelled dataset makes picking the best configuration one reduction over named dimensions. Run --demo to see it without keys or network. (#188)
• compare_llms and compare_scores now report item_p_corrected beside item_p: the item-level bootstrap p-value under the same multiplicity correction, and over the same family (the comparisons within one metric), that p_value gets as p_corrected. Comparing three systems multiplies the chance of a spurious item-level "win" exactly as it does on the seed axis, so a corrected seed p-value reported next to a raw item p-value understated one of the two risks. With a single comparison the two columns are equal.

Misc

• 182, #189

0.13.0 - 2026-08-19

Added

• Python 3.14 is now supported and tested on Linux, Windows, and Apple Silicon. (#179)

Misc

• 172, #174, #175, #176, #177, #178

0.12.0 - 2026-08-13

Changed

• Migrated the MCP server to the MCP Python SDK v2 (the v2 SDK removes the mcp.server.fastmcp module the server was built on, so an uncapped 1.x install would break at import time): the mcp extra now requires mcp >= 2, < 3 (the 1.x line is in maintenance mode upstream). The server's tools, transport (stdio), and mushin-mcp --root CLI are unchanged; if you pin mcp 1.x elsewhere in your environment, stay on the previous mushin release until you can upgrade. (#162, #163)
• Moved off Hydra's deprecated 1.1 compatibility mode ahead of its removal in Hydra 1.4: run()'s version_base now defaults to "1.3", and the hydra-core floor rises from 1.2 to 1.3. Behavior is unchanged — run() has always pinned hydra.job.chdir=True explicitly, so each job still executes in its own working directory — and hydra-core 1.3.5's new Hydra14MigrationWarning no longer fires on every launch. HydraDDP / HydraFSDP now locate the job's saved config via Hydra's runtime output directory instead of the process CWD, so they also work when launched with version_base >= "1.2" semantics (no per-job chdir) outside of run(). (#167)

Fixed

• Corrected HydraDDP's docstring, which still said the saved config.yaml must live in the process's current working directory: since the version_base migration it is located via the job's Hydra output directory, so DDP re-launching no longer depends on the CWD. (#169)

0.11.1 - 2026-07-27

Changed

• The PyPI summary, docs landing page, and docs-site metadata now match the README: "reproducible machine-learning experiment sweeps" (the identity line previously said "workflows" on those three surfaces). (#152)

Misc

• 154

0.11.0 - 2026-07-23

Added

• A missing Hydra launcher plugin now raises with the exact pip install package name (e.g. hydra-submitit-launcher) instead of Hydra's bare "Could not find" listing. (#149)
• run() accepts launcher_config= — launcher fields (e.g. from submitit_slurm_config()) are applied as hydra.launcher.* overrides internally, so cluster sweeps no longer need hand-rolled override strings. (#149)

Fixed

• max_total_seconds is disabled (with a warning) for cells running as one rank of an external multi-rank launch (submitit DDP/FSDP) — per-rank deadlines could diverge and hang the ranks at NCCL rendezvous; bound those jobs with the scheduler's time limit instead. (#149)
• HydraDDP's docstring example now runs on supported Lightning (devices=2, not the removed gpus=2); the multinode example uses the standard workflow API; the multinode guide gained a "Preemption & resume" section covering SLURM requeue and recovery. (#149)
• Provenance now records the Apple Silicon MPS accelerator (device: "mps (<chip>)") when CUDA is absent — M-series runs are no longer hardware-blind; the install guide documents Apple Silicon support explicitly. (#150)

0.10.1 - 2026-07-22

Changed

• Repositioned the project description everywhere (README, PyPI, docs site + metadata, citation, paper): mushin is a framework-agnostic sweep engine built on hydra-zen, with first-class PyTorch Lightning integration — the sweep layer never imports Lightning (Lightning ships in the base install for the deep-learning path). (#146)

Fixed

• A second run() on the same workflow instance no longer reuses the previous sweep's cached overrides, which silently NaN-filled the new grid. (#145)
• ±Inf metric values survive the sidecar round-trip with their sign instead of collapsing to NaN. (#145)
• load_from_dir now recovers each cell's config-group choice, so a finished group sweep reloads with real values instead of an all-NaN grid. (#145)
• diff no longer reports metric deltas from failed/skipped cells, and a failing cell removes the metrics sidecar left by a prior successful attempt. (#145)
• show/best/diff/export.table now scope to the latest sweep's manifest, excluding stale cell dirs left by an earlier sweep in a reused working_dir (the mid-sweep live view is preserved). (#145)
• Study training-sweep resume now fingerprints the methods mapping: renaming, reordering, or editing a method (including a functools.partial's bound arguments) re-runs cells instead of silently reusing checkpoints trained by old code. One-time cost: the first resume of a sweep created before this change re-runs all cells (its cells lack the fingerprint). (#145)
• Fail-soft, sampled, and budget-skipped sweeps can now be reloaded from disk: failed/skipped cells NaN-fill instead of load_metrics raising FileNotFoundError, and a stale sidecar in a failed cell's dir is never loaded. (#145)
• compare_methods(allow_incomplete=True) masks constant (zero-variance) methods over the completed seeds and keeps ±Inf cells in the pairing instead of silently excluding them; summary(reference=...) rejects unknown method names. (#147)
• max_total_seconds runs on a monotonic clock — a wall-clock step (NTP/DST) mid-sweep can no longer stretch or prematurely expire the budget. (#147)
• show/export.table: a metric sharing a swept param's name renders as <name> (metric) instead of overwriting the param column, and a typo'd metrics= entry raises instead of silently dropping the column. (#147)
• The grid accounting (dry-run preview, confirm_above gate, sample= selection) no longer silently excludes user params whose names merely start with "hydra" (e.g. hydraulic=). (#147)
• dry_run reports the sampled cell count when sample= is set (the cell-count gate still applies to the full grid, since every cell is launched regardless of sampling); a fresh run reusing a working_dir no longer inherits the prior sweep's notes/tags, and tags=[] on resume clears them. (#147)
• MCP get_failures/get_provenance raise for a nonexistent path instead of reporting an empty-but-valid result; MetricsCallback writes standalone trainer.validate() metrics to validate_metrics.pt; seed_everything_per_rank falls back to LOCAL_RANK; provenance records dirty: null when git status cannot be verified. (#147)
• working_dir paths containing = or , (e.g. dirs named lr=0.1) now work; duplicate sweep values via raw overrides= are rejected instead of silently collapsing to one cell. (#147)

Removed

• Removed the dead pre-Lightning-1.6 HydraDDP code path (~130 lines) — the dependency floor is pytorch-lightning >= 2.4. (#147)

Misc

• Reframed the workflows guide's hyperparameter-search section: mushin does grid and random search (often the whole job for a small discrete space, with the labeled dataset + stats as a bonus); it does not do adaptive/Bayesian search over large or continuous spaces, which is where a dedicated optimizer like Optuna is complementary. Replaces the earlier "mushin is not a hyperparameter optimizer" overstatement.
• The README shows a "latest release" badge linking to the newest GitHub release and its notes, alongside the existing PyPI version badge.

0.10.0 - 2026-07-22

Added

• A pre-launch cell-count gate guards against an accidentally huge grid. run(..., confirm_above=N) refuses to launch a sweep with more than N cells, raising a ValueError that reports the count and how to proceed (preview with dry_run=True, or raise the ceiling). The MUSHIN_MAX_CELLS environment variable supplies a default ceiling when a call doesn't set one — so a shared cluster or CI profile can cap every sweep — and an explicit confirm_above= wins over it. A malformed MUSHIN_MAX_CELLS is ignored, and dry_run=True bypasses the gate so an over-limit sweep can still be previewed.
• mushin.best(root, metric) returns the completed cell that optimizes a metric across a sweep — mode="max" (default) or mode="min". Like mushin.show it reads the per-cell sidecars directly (offline, no Hydra/xarray). The returned BestResult carries the winning combo (swept params), the optimized value, the full metrics dict, the cell status, and its job dir (for locating checkpoints/artifacts). Failed/running/skipped cells and non-finite metric values are ignored; an unknown metric raises a ValueError that lists the available scalar metrics.
• mushin.diff(a, b) compares two sweep directories. It aligns cells by their swept-parameter combination and, for each shared cell, reports the delta b - a of every metric that is a finite scalar in both — printed as a Δmetric table. Cells present in only one sweep are listed (only_in_a / only_in_b), and the two runs' environment provenance is diffed field-by-field (git/packages/python/accelerator), excluding volatile fields like the per-cell timestamp. Like show/best it reads the per-cell sidecars directly (offline, no Hydra/xarray); the returned DiffResult exposes .rows, .only_in_a, .only_in_b, .provenance, and .table.
• mushin.export.table(root) exports a sweep as CSV — one row per cell with its swept parameters, status, and metrics. It reads the per-cell sidecars directly (offline, no Hydra/xarray) and writes full-precision values so pandas parses numbers numerically. Returns the CSV string, or writes to path= and returns the Path; metrics= restricts the metric columns. A durable, spreadsheet-friendly substrate for researchers who prefer pandas to xarray.
• mushin.show(root) prints (and returns) a status/metrics table for a sweep directory. It reads each cell's status and metrics sidecars directly — pure JSON, so it needs neither Hydra nor xarray and works mid-sweep — making it handy for watching a live sweep or eyeballing a finished one before the full to_xarray load. Each row carries the cell's swept parameters, its status (completed/running/failed/skipped/…), and its metric values; metrics= restricts the metric columns and sort= orders the rows. The returned ShowResult exposes .rows (one dict per cell) and .table.
• run() now validates up front that every required task (and pre_task) parameter is satisfied — by the base config, an override, or the raw overrides=[...] list — and raises a clear ValueError naming the missing parameter(s) before launching. Previously a genuinely missing parameter surfaced as an opaque per-job Hydra ConfigAttributeError after the sweep had already started, once per cell. A parameter supplied only via an override is still valid (the common decorator case, where the base config is empty), so this adds no false positives.
• run(..., cache_dir=...) adds a content-addressed cache of completed cells that is shared across working_dirs. A cell whose resolved config AND task source match a previously-computed cell — keyed on the same fingerprints as resume — reuses that result instead of recomputing, so a cell computed in one sweep is free in another (a coarse grid refined into a finer one, a re-run in a fresh directory). Newly-computed cells are stored in the cache automatically. It complements resume (which reuses only within a single working_dir); a changed non-swept config value or an edited task body is a cache miss. Cache writes are best-effort and never fail the cell.
• run(..., dry_run=True) (and @mushin.sweep .run(dry_run=True)) previews a sweep instead of launching it: it prints the cell count and each swept axis with its values — so a range typo shows up as an unexpectedly wide axis before any compute is spent — and returns a summary dict (num_cells, axes, fixed, working_dir) with no jobs run.
• run(..., max_total_seconds=T) adds a graceful wall-clock budget. Once the budget is exhausted the remaining grid cells are skipped — recorded 'skipped' in the manifest and per-cell status, NaN in the dataset, and surfaced in self.skipped (and a mushin_skipped dataset attr) — instead of the sweep running to the end. The clock starts at the first computed cell, so at least one cell always runs and resume cache hits don't consume the budget; a cell already running is never interrupted. Because a skipped cell is not completed, a later resume=True with more time finishes exactly the skipped cells. The budget is measured per launcher process, so it's best paired with the default sequential launcher.
• run(..., notes="...", tags=[...]) annotates a sweep with free-form lineage. The note and tags are recorded in the sweep manifest, exposed on the workflow as wf.notes / wf.tags (and preserved when a sweep is reloaded with load_from_dir), and carried on the dataset as the mushin_notes / mushin_tags attrs — so "why did I run this?" travels with the results. tags must be a list of strings and notes a string. A resume that does not re-pass notes/tags preserves the original run's lineage rather than wiping it.
• run(..., sample=K) runs only a random K-cell subset of the grid — the rest are skipped (NaN, and listed in self.skipped) — for fast exploration of a large grid without paying for every cell. The subset is chosen deterministically from the Hydra job indices (seeded by sample_seed, default 0), so it is reproducible and identical across a resume; resuming without sample fills in the remaining cells. sample >= n_cells runs everything. Because selection is by job index it is launcher- and axis-type-agnostic. Note: the full grid is still composed by Hydra (only the sampled cells run), and a sampled sweep is intentionally incomplete (is_complete is False).

Fixed

• A batch of defensive-handling fixes: corrupt-but-valid-JSON cell-status sidecars now degrade (resume treats a null/"3" attempt or a wrong-shape record as absent) instead of crashing; load_from_dir raises a real FileNotFoundError for a missing config (not a -O-stripped assert); the resume contextvar no longer leaks when a status write fails; RobustnessCurve.plot no longer leaks a figure when plotting raises; a malformed tuning pin reaches its friendly "delete it or retune" guard; mushin.lightning/benchmark/mcp/testing are reachable as lazy submodules after import mushin; compare/compare_llms reject an unknown correction= before any evaluation runs; the MCP get_failures/get_provenance tools tolerate wrong-type sidecar JSON; and a value_check message typo ("None orof") is fixed.
• A typo'd sweep-axis name is now caught before launch instead of silently producing a wrong dataset. Passing run(lrate=multirun(...)) when the task parameter is lr previously ran to completion and added a phantom lrate dimension of constant values; it now raises a ValueError that names the likely intended parameter. Only near-misses of a real target are rejected — deliberate overrides beyond the task's own parameters (values consumed by pre_task, config groups, interpolations) and tasks declaring **kwargs are unaffected.
• Fixed trainer.validate() under HydraDDP/HydraFSDP: the child ranks were told to run fit (train) while rank 0 validated — a collective mismatch that hangs or crashes. The launcher now maps Lightning's _validate_impl to a pl_validating flag and _pl_main dispatches trainer.validate accordingly, alongside the existing test/predict paths.
• Fixed a cluster of silent-wrong-results bugs in the override → grid pipeline: a fixed (non-multirun) string containing , or = is no longer re-split into an accidental sweep and "true"/"1" stay strings; a duplicated sweep-axis value is rejected instead of silently collapsing two cells into one; combo_key is now injective (delimiter-containing values can't collide with a different combination); a dotted sweep (model.width) on a config field that already exists no longer crashes; and a batching sweeper's multi-batch job list is flattened instead of failing after every job ran.

• Fixed five defects found by an adversarial review of the exploration features:

• cache_dir is now resolved to an absolute path (expanduser().resolve()) before launching. A relative or ~-prefixed cache_dir was previously dereferenced inside each per-cell job dir (Hydra chdirs into it), so cross-cell/cross-directory reuse silently never hit.

• resume=True with sample= is now rejected with a clear error. Combining them overwrote already-completed cells (skipped by job index) to skipped/NaN, silently discarding prior results; resume WITHOUT sample fills the rest.
• The cell count (used by sample=, confirm_above=/MUSHIN_MAX_CELLS, and dry_run) now counts sweep axes supplied via the raw overrides=[...] list, not only param=multirun(...) kwargs — so those no longer undercount the grid.
• compare_methods(allow_incomplete=True) now computes each comparison over only the seeds completed for both methods (dropping NaN cells), fulfilling its documented promise instead of feeding NaN into the tests and returning all-NaN statistics.
• mushin.show(sort=...) places NaN metric values last deterministically (they no longer break the sort's total order) and raises a clear error when sort names a column that isn't present.
• Fixed metric-value coercion into the JSON sidecar: a task returning a nested-dict metric containing numpy/torch values (or a datetime/Path) no longer crashes the sidecar write after the task succeeded (which left the cell stuck and aborted the sweep); non-finite metrics (NaN/±Inf) now serialize as valid JSON (null) instead of the non-strict NaN/Infinity literals that broke strict parsers, and read back as NaN so metric columns stay float dtype on resume; a non-scalar battery metric (numpy array/list, not just a torch tensor) now gives the crafted "must return scalar" error.
• Provenance no longer leaks secrets: mushin_provenance.json records the config with resolve=False (so a ${oc.env:SECRET} interpolation is kept as an unresolved reference instead of baking the resolved secret into the file), and values under secret-named keys (api_key, token, password, ...) or shaped like provider tokens (sk-..., hf_...) are redacted. This also fixes the MCP get_provenance(include_config=True) tool, which re-served the on-disk record.
• Reloading a sweep offline with load_from_dir(dir, "mushin_metrics.json") no longer crashes. The default metric_load_fn now sniffs the file — it reads the JSON metrics sidecar (written by any task that returns a dict, including @mushin.sweep/decorator sweeps) with json and falls back to torch.load only for torch pickle/.pt files. Previously it was hard-wired to torch.load, which raised UnpicklingError on the JSON sidecar unless you overrode the loader.
• Resume now guards against stale-code reuse: editing a task body and re-running with resume=True (same config, same swept params) re-runs the affected cells — with a clear warning — instead of silently returning the previous run’s cached metrics. A per-cell hash of the task source is stored alongside the config fingerprint; a completed cell is reused only when both still match. Legacy sweeps (no recorded code hash) resume as before.
• Resuming a sweep after changing the grid shape (adding or removing an axis) no longer silently reuses the wrong cells: previously, with legacy (pre-fingerprint) sidecars, adding an axis projected every new cell onto the old parameters, reused a stale cell across every new axis value, dropped the new dimension entirely, and reported the sweep complete. The resume now keys reuse on the current swept parameters, so a shape change re-runs the full grid, with a one-time warning.
• Resuming into a working_dir whose numeric job dirs get reused for different cells no longer leaves stale metrics behind: on a cache hit the reused cell now writes its metrics and status into the current job dir, so the dir stays consistent with the config Hydra wrote there. Previously a reused cell (e.g. a=3) kept the leftover metrics of whatever cell last used that dir (e.g. a=1), so the manifest and an offline load_from_dir mis-keyed the value.
• compare_methods now refuses a sweep that has skipped/never-run cells, not just failed ones. Previously it keyed only on mushin_failures, so a sample= subset or a max_total_seconds-limited sweep — both of which leave NaN cells — would compute statistics silently over a partial grid. It now also keys on the mushin_skipped completeness signal and raises IncompleteSweepError. A new allow_incomplete=True argument bypasses the guard (with a warning) to compute stats over only the completed cells, for exploratory analysis of a partial sweep. As before, the guard keys on the completeness signals (attrs), never on raw NaN values, so a metric that is legitimately NaN for other reasons does not trigger it.

Misc

• Documentation fixes: load_from_checkpoint docstring corrected (its three params default to None, not "state_dict"/"model"; no load_module param; returns the passed nn.Module, not a LightningModule); run()/wrapper docstrings reference hydra_zen.zen (not the nonexistent mushin.zen); capture_env/resume/load_metrics/workflow_overrides docstrings match actual behavior; the Python support matrix reads 3.10–3.13 (3.14 is not yet a required CI leg); study.md/concepts.md list all seven task batteries; and the correction= option is documented in the compare/LLM/index guides.
• Documented the sweep-analysis API. mushin.show, mushin.best, mushin.diff, and mushin.export.table now have an API reference page (reference/analysis), and the exploration-to-paper guide covers max_total_seconds, notes=, and tags=. The workflows guide also gains a "Using mushin alongside a hyperparameter search" section documenting the Optuna/Ax/Nevergrad → mushin two-phase pattern (searcher finds configs; mushin runs the reproducible final grid), with no new dependency.
• Fixed several tests that did not test what they claimed: the LLM "clear winner flagged significant" test now uses seed-varying systems and asserts the comparison is actually flagged significant (its old deterministic systems made the comparison masked, so it asserted nothing about significance); the Holm "is monotone" test now pins the exact step-down values and the monotonicity property; test_on_error_raise asserts the specific RuntimeError("boom") rather than any exception; a conditional-pass tuning test now visibly pytest.skips instead of silently returning; a workflow test no longer scatters multirun/ into the repo root; and a config-group test cleans up its ConfigStore entry.
• New guide, "From exploration to a paper-ready sweep", documenting the convention for moving from fast throwaway exploration (sample=, cache_dir=, dry_run, show/best/diff) to a clean, complete, reproducible paper run — including why the paper grid must be run fresh (code-era mixing, winner's curse) rather than grown from the exploration directory.
• Repo hygiene & example fixes: the crashing MNIST examples (compare_classifiers, study_mnist) now name torchvision in their install hint; batteries.py skips (not fails, exit 0) batteries whose optional extra is missing and drops the false "Run one"; two tracked stray mushin_seed*.json files are removed and gitignored; the omegaconf/typing-extensions dependency floors are raised to versions that actually resolve; mushin-mcp on a core install gives an actionable "install [mcp]" message instead of a bare ModuleNotFoundError; and make spell now covers examples like CI.
• The publish workflow pins pypa/gh-action-pypi-publish to its release tag (a bare commit SHA cannot resolve the Docker-based action's container image, which broke the v0.9.0 publish).

0.9.0 - 2026-07-20

Added

• Provenance records the accelerator identity (CUDA/cuDNN/device name), seed_everything_per_rank persists the effective seed to a per-rank JSON in the run dir, and a resume no longer overwrites the environment snapshot recorded for earlier cells.
• Study and evaluate_checkpoints now accept predict_fn/metrics/prob_metrics/correction and forward them to compare (custom-output models and custom batteries work through Study), and checkpoints load lazily one at a time instead of holding every method x seed model in RAM.
• Sweep axes got first-class support for nested (dotted) params and Hydra config groups (the dataset coordinate is the chosen option name), range(...) overrides expand to their values, and unsupported continuous/adaptive sweeps (interval(...)) raise a clear error instead of crashing or yielding an all-NaN grid.
• The MCP server gained get_failures (failed cells + tracebacks from the manifest) and get_provenance (git/package/accelerator records) tools, and now summarizes long coordinate axes and metric curves instead of flooding the agent's context with raw arrays.
• The wheel now ships a PEP 561 py.typed marker, so mypy/pyright/IDEs see mushin's type annotations.
• MultiRunMetricsWorkflow.to_dataframe() (and experiment.workflow.to_dataframe()) returns the sweep as a tidy pandas DataFrame in one call; README and quickstart show the pandas path alongside the dataset.
• compare/compare_methods/compare_llms accept correction= -- holm (default), bonferroni, fdr_bh (Benjamini-Hochberg), or none; the statistics guide documents the per-metric family scope.

Changed

• import mushin is ~5x faster (~0.13s from ~0.61s): torch now loads on first use, not at import time.

Fixed

• HydraDDP/HydraFSDP now track their child rank processes like Lightning's base launcher -- children are reaped if rank 0 dies (no more orphaned GPU processes), signal forwarding works, and per-rank thread pools are capped; checkpoint/experiment loading errors are real exceptions with actionable messages instead of bare asserts.
• HydraDDP/HydraFSDP rank startup is staggered by a deterministic, tunable delay (MUSHIN_DDP_LAUNCH_DELAY, default 1s) instead of a hard-coded random 1-5s per rank, and the re-launch command builds a Windows-safe hydra.run.dir override.
• Paired tests now report the paired effect size (Cohen's d_z) instead of the pooled-variance d; mushin_failures is stored as a JSON attr so failed-sweep datasets survive a netCDF round-trip; a custom metrics= battery no longer inherits the task's probability routing silently (pass prob_metrics= explicitly, unknown names now raise).
• Resilient sweeps got tougher: a corrupt (mid-write-killed) sweep manifest no longer aborts resume, atomic sidecar writes use unique temp names so concurrent writers cannot clobber each other, and on_error="nan" now writes the full traceback to mushin_error.txt in the failed cell's directory.
• Resume now verifies a config fingerprint before reusing a completed cell -- changing a non-swept value and resuming re-runs the affected cells (with a warning) instead of silently mixing two configurations into one dataset; resuming also no longer ships the full manifest to every worker.
• The MCP server now surfaces workflow-sweep metrics from mushin_metrics.json (previously only MetricsCallback .pt files were read), and a single corrupt per-run config.yaml no longer makes the whole experiment unqueryable.
• parse_score no longer misreads an incidental leading integer ("0 issues found...") as a 0.0 score -- a bare integer with trailing text now raises; the LLM guide documents cache-invalidation rules and when paired tests are valid.

Misc

• CI now tests macOS (Apple Silicon + the Intel dependency branch), executes notebooks with the netCDF engine they recommend, and the publish workflow smoke-tests the built wheel and SHA-pins the OIDC publish action; the Python 3.14 classifier is withheld until its CI leg is required to pass.
• Docs: eval-extra callouts everywhere they were missing (examples index, notebooks 02/03/05, custom/segmentation guides, example docstrings), a "Versioning & scope" statement in the README, a tracker (W&B/TensorBoard) how-to in the workflows guide, multirun/hydra_list reference entries, and the decorator quickstart now points at the decorator example. MultiRunMetricsWorkflow.run is fully documented and plot no longer leaks figures.
• Notebook 07 demonstrates xarray group-by (split-apply-combine), notebook 05 finally shows its plot on the docs site, and a netCDF round-trip test pins string-coordinate datasets.

0.8.0 - 2026-07-20

Changed

• The evaluation layer (compare, the metric batteries, LLM evaluation, and Study) now requires the optional eval extra: pip install mushin-py[eval]. This keeps the core sweep→dataset install lean — torchmetrics and scipy are no longer base dependencies. Accessing these features without the extra raises a clear install hint. The detection/image/audio battery extras now imply eval.

0.7.0 - 2026-07-18

Added

• Multi-node DDP support: submitit_slurm_config (derives tasks_per_node from gpus_per_node) and seed_everything_per_rank helpers, a fail-fast check that the launched world size matches num_nodes x devices, MetricsCallback now writes only on global rank 0, and _teardown clears only mushin-set env vars (leaving scheduler-owned vars alone under SLURM/torchrun). See the new multi-node guide.
• HydraFSDP: a Fully-Sharded Data Parallel strategy that works under Hydra --multirun. Like HydraDDP, it reattaches ranks via the job's saved config.yaml instead of re-executing with sys.argv (which a sweep would run as the wrong job), so FSDP sharded training composes with Hydra sweeps. Exported from mushin; see the new "Sharded training under Hydra" guide.
• pin_gpu_round_robin(num_gpus): an opt-in helper to pack several small sweep jobs onto each GPU. Called at the top of a Hydra task, it sets CUDA_VISIBLE_DEVICES to job_index % num_gpus so jobs round-robin across devices; run num_gpus * jobs_per_gpu jobs concurrently (via your launcher's n_jobs) to co-locate them. New "Packing small jobs onto GPUs" guide covers the joblib recipe, Ray fractional-GPU, and MPS/MIG.

Fixed

• HydraDDP/HydraFSDP docs now show the working launcher-provided-ranks pattern; an imperative @mushin.sweep task with strategy=HydraDDP(), devices=N on the local launcher silently trains on one GPU. (#95)
• A bare list/tuple passed as a sweep argument (the common slip of forgetting multirun) now raises an actionable error — lr=mushin.multirun([...]) to sweep, or mushin.hydra_list([...]) to pass the list as a single value — instead of a generic type-list TypeError.

Misc

• Added a runnable examples/parallel_sweep.py showing how to submit a sweep out-of-process (run(..., launcher="joblib")), wired into the workflows guide and examples index — the missing runnable counterpart to 0.6.0's out-of-process launcher support.
• Added an internal cluster-gated validation runbook (docs/superpowers/cluster-validation-runbook.md): a self-contained, per-PR checklist (Phase 1 single-node multi-GPU, Phase 2 multi-node/SLURM) that anyone with HPC access can run to validate HydraDDP, GPU-packing (#59), HydraFSDP (#58), multi-node DDP (#50), submitit dispatch (#86), and resume hard-kill/preemption durability (#83).
• Docs: README and the workflows guide now lead with the @mushin.sweep decorator; mushin.sweep, out-of-process launchers, and hard-kill-durable resume are covered in the README feature list + API reference; and a new "Analyzing your results" example notebook shows the common xarray.Dataset recipes (reduce over seeds, pick the best config, slice, tabulate, plot, save/reload).
• Docs: the workflows guide now explains the two axes of sweep parallelism — a Hydra launcher= (distributes the sweep across cells/nodes) vs. HydraDDP (multi-GPU training within one cell) — and how they compose (e.g. submitit + HydraDDP).

0.6.0 - 2026-07-15

Added

• New @mushin.sweep decorator: turn a plain task-style function into a runnable sweep with no subclassing — experiment.run(lr=multirun([...]), seed=multirun([...])) returns the labeled xarray.Dataset directly. Drop to experiment.workflow (the last-run MultiRunMetricsWorkflow) or experiment.workflow_cls for power features; mushin_resume and all run() resilience options carry through. The MultiRunMetricsWorkflow subclass form is unchanged.
• Resumable sweeps now survive a hard process kill (OOM, SLURM preemption): each cell records its completion durably from inside its own job, so resume=True never recomputes finished cells. A task may also declare a mushin_resume parameter to receive a ResumeContext (the cell's directory, is_resume, and the last checkpoint) and resume its own training mid-run.
• Sweeps can now use out-of-process Hydra launchers (e.g. hydra-joblib-launcher, submitit): per-cell dispatch is stdlib-picklable, so run(..., launcher="joblib") parallelizes across worker processes. Previously any process-backed launcher failed with a PicklingError. Resilience, resume, and provenance semantics are unchanged in-process and preserved out-of-process.

Changed

• Provenance capture no longer spawns git subprocesses per sweep cell. The sweep-constant part of the record (git state + package versions — three git subprocesses via _git()) is now captured once per run() and reused for every cell, instead of being recomputed per cell. An N-cell sweep now spawns 3 git subprocesses instead of 3N (e.g. ~30s less git overhead on a 1000-cell sweep). Each cell's mushin_provenance.json is byte-for-byte equivalent (only timestamp/config vary per cell, as before).
• import mushin is now ~65% faster (~1.7s → ~0.6s on a cold import): the Lightning integration (HydraDDP, MetricsCallback) and its heavy pytorch_lightning dependency (which also transitively pulled in matplotlib and scipy) now load lazily on first attribute access instead of at import time. The sweep → xarray core no longer pays for Lightning it never uses. All public names resolve unchanged; _tuning/Study still work (they import pytorch_lightning only inside functions).

Fixed

• Sweeps no longer emit Hydra's "Future Hydra versions will no longer change working directory at job runtime" deprecation warning: MultiRunMetricsWorkflow.run now sets hydra.job.chdir=True explicitly (the behavior the workflow already relies on).

Misc

• Added six runnable EQUINE-style example notebooks (sweeps, compare + batteries, Study, resilient sweeps, LLM evaluation, scikit-learn) under an "Example notebooks" section of the docs, executed in CI via nbmake so they stay current.
• CI: a batteries-clean-install job builds the wheel, installs it (non-editable) with the detection/image/audio extras into a fresh env, and runs all 7 battery examples against it — catching packaging / optional-extra issues the editable test job would miss.
• Documentation: a Built-in batteries guide covering all 7 benchmark batteries (classification, segmentation, detection, regression, retrieval, image_quality, audio) with real-model recipes and CI-tested runnable toys.
• Documentation: add a top-level Examples page indexing all runnable example scripts (sweep_to_dataset, sklearn_sweep, compare_classifiers, study_mnist, segmentation_demo, compare_llms_demo, batteries), each with a one-line description and a link, plus a pointer to the batteries guide.
• Documentation: add sweep resilience + provenance (0.5.0) to the README "What it provides" list.
• Documentation: the Built-in batteries guide now shows real captured outputs under every battery, plus a flagship notebook-style walkthrough (comparing two classifiers) with the actual summary table, significant p-values, and interpretation.

0.5.0 - 2026-07-14

Added

• Sweep resilience and provenance: run(on_error="nan") records failed grid cells as NaN and keeps going (default stays "raise"); run(working_dir=..., resume=True) re-runs only the failed/missing cells and fills them in place; compare/Study refuse statistics on an incomplete sweep (IncompleteSweepError) until you resume; every run writes per-job provenance (mushin_provenance.json: git SHA, versions, config) with an opt-in capture_env=True full dependency snapshot. Study accepts on_error/resume/capture_env for training sweeps.

0.4.1 - 2026-07-13

Added

• Added a runnable scikit-learn sweep example (examples/sklearn_sweep.py) demonstrating that the framework-agnostic MultiRunMetricsWorkflow wraps non-Lightning models (here LogisticRegression) and still returns a labeled xarray.Dataset. scikit-learn is a dev-only dependency for the example and its test; the package itself has no scikit-learn dependency.

Misc

• Documentation: add a "Frameworks: Lightning-first, sweep layer agnostic" section to Core concepts, clarifying that the sweep->dataset workflow wraps any framework (scikit-learn, XGBoost, ...) while compare/auto-tuning/HydraDDP are PyTorch/Lightning-specific.
• Documentation: add a Changelog page to the docs site that embeds CHANGELOG.md (self-maintaining, no duplication).
• Documentation: add an API reference page for the auto-tuning helpers (tune_batch_size, tune_learning_rate), which had a usage guide but no auto-generated API docs.
• Documentation: announce the 0.4.0 release with a "What's new" highlights callout on the docs home page.
• Documentation: refresh the README for 0.4.0 — add auto-tuning and the task API/batteries to the feature list, note the framework-agnostic sweep layer (scikit-learn example), and fix the stale viz extra description that referenced the now-deprecated RobustnessCurve.

0.4.0 - 2026-07-13

Added

• Four new built-in task batteries — regression, image_quality, audio, and retrieval — plus a per-Task update_fn hook for metrics whose update step is not (preds, target) (used by retrieval). LPIPS and STOI sit behind the optional [image] and [audio] extras. Each battery is exported from mushin.
• Overhauled the documentation: runnable, tested example scripts (MNIST) that the guides embed verbatim, deeper guides with annotated output, and new Tutorial, Core concepts, Custom metrics/predict_fn, and Statistics pages.
• Public task API: Task dataclass plus register_task, get_task, and list_tasks make evaluation tasks first-class and reusable. compare(...) and Study(...) now accept either a Task object or a registered task name, and the built-in batteries (classification_battery, segmentation_battery, detection_battery) are exported from mushin.
• compare(task="detection") — compare trained object detectors across seeds over the full torchmetrics.detection bounding-box family (mean-average-precision plus the IoU/GIoU/CIoU/DIoU variants), reporting every scalar metric with Holm-corrected significance. Needs the optional mushin-py[detection] extra.
• mushin.llm.compare_llms — compare LLM systems (callables or hydra-zen configs) across reproducible stochastic seeds with a metric (a plain scorer, a torchmetrics text metric, or the new llm_judge helper), reporting Holm-corrected statistical significance. Includes an on-disk output cache. Provider-agnostic: you bring the systems, data, and judge model.
• tune_batch_size / tune_learning_rate: opt-in, reproducibility-preserving auto-tuning. Lightning's batch/LR finder runs once, the result is pinned to a sidecar YAML, and later runs reuse it. tune_batch_size pins a hardware- independent effective batch, choosing the largest device batch that both fits and divides the per-device target exactly, so the effective batch is identical on any GPU count with no drift.

Changed

• Modernized the codebase to Python 3.10+ idioms now that 3.9 is no longer supported: ruff target-version is py310, and the pyupgrade auto-fixes (Optional[X]/Union[X, Y] -> X | None/X | Y) plus explicit zip(..., strict=True) have been applied. No behavior change.
• import mushin is now lightweight: the benchmark and llm subsystems load on first use instead of at import time, so a bare import no longer pulls the battery/eval machinery. Every existing top-level name still resolves. The default Hydra config/job name is now mushin_workflow (was rai_workflow), and the new mushin.original_cwd() helper anchors relative paths in task() against the launch directory rather than Hydra's per-job output directory.

Fixed

• Fixed the docs example scripts surfaced in review: the Study example now trains and evaluates on separate MNIST splits and resolves its checkpoint directory to an absolute path (so it works under Hydra's per-job chdir); the custom-metrics guide no longer implies Study accepts metrics/predict_fn (those are compare-only).
• Hardening from a repo-wide adversarial audit: load_experiment now loads DDP/nested-layout configs (was silently None) and labels each job's own working_dir; the benchmark compare() path masks zero within-group-variance comparisons (no more false-positive significance — now consistent with compare_llms, both via a single compare_methods); MetricsCallback keeps every metric series aligned to the epoch axis (NaN-padding missing metrics, reserving the epoch key); Study labels the seed coordinate with the real seed values and relocates checkpoints across filesystems (shutil.move); multirun overrides are built with Hydra's choice(...) syntax so comma-bearing and single-element values are no longer mis-split; and several legibility/robustness fixes (to_dataset empty-method error, _to_device namedtuple support, '='-bearing override parsing, load_from_dir cache reset, dropped the never-working dict-override path).

Removed

• Dropped support for Python 3.9 (end-of-life October 2025); mushin now requires Python >= 3.10. This refreshes the dependency lockfile to patched versions of pillow, urllib3, aiohttp, filelock, requests, pytest, and pytorch-lightning, clearing the Dependabot security alerts anchored on the old Python-3.9 dependency branch. The scipy (>= 1.13) and matplotlib (>= 3.9) floors are raised to their first NumPy-2-compatible releases, and the mcp extra no longer needs a Python version gate.

Deprecated

• BaseWorkflow and RobustnessCurve are deprecated at the top level and will be removed in a future release. Import them from mushin.workflows instead; accessing them as mushin.BaseWorkflow / mushin.RobustnessCurve now emits a DeprecationWarning.

Misc

• Added a @claude mention bot (GitHub Actions): mention @claude in any issue or PR comment to have Claude answer questions or make changes. PR reviews stay with the Codex connector.

0.3.0 - 2026-06-24

Added

• Optional read-only MCP server (mushin-mcp, pip install "mushin-py[mcp]", Python >= 3.10) that lets Claude Code and other MCP clients list experiments, summarize swept parameters and metrics, read configs, and inspect saved datasets — with no training or sweep launching. (#32)
• Added a documentation website (MkDocs Material) with how-to guides and an auto-generated API reference, deployed to GitHub Pages.
• The test suite now runs on Windows in CI (windows-latest) across Python 3.9-3.14, alongside Linux.

Changed

• Raised minimum dependency floors to a tested minimum and added a min-versions CI job that runs the suite against the lowest declared versions, so the floors stay honest. The old floors did not actually work: several were incompatible with NumPy 2 (the non-Intel floor), so torch (>= 2.4), pandas (>= 2.2.2), xarray (>= 2024.6), and netCDF4 (>= 1.7.0) were raised to their first NumPy-2-compatible releases. pytorch-lightning was raised to >= 2.4 (the pre-2.0 1.5 API does not run this package) and hydra-zen to >= 0.10 (for multirun). Intel-macOS keeps torch 2.2.x / NumPy 1.x.

0.2.1 - 2026-06-23

Added

• Adopted towncrier news fragments with a CI gate — each PR now adds a changes/ fragment, assembled into the changelog at release.

Fixed

• Point PyPI project links at the mushin repo (Homepage/Repository/Issues/Changelog) instead of only the upstream toolbox.
• Synced uv.lock's recorded project version to 0.2.0.

0.2.0 - 2026-06-23

Added

• Study — orchestrate a multi-seed training sweep (via Hydra/Lightning) and route the trained models into compare in one call; Study.from_checkpoints(...) for eval-only comparison of existing checkpoints.
• Segmentation support in compare (task="segmentation"): mean IoU, Dice, pixel accuracy, and macro precision/recall, with ignore_index for void labels; plumbed through Study.
• Streaming evaluation — metrics update per batch (O(C²) memory for the segmentation battery); one unified eval loop for all tasks.
• A task registry so new task types self-describe their battery and predict step.
• compare gains a prob_metrics override and rejects one-shot data iterators.

Changed

• Package author/maintainer set to Josue Martinez-Martinez (previously the original MIT-LL authors; their attribution remains in LICENSE and the README).
• README: documented compare and Study; clarified install (pip install mushin-py → import mushin); absolute logo URLs so images render on PyPI; transparent dark-mode logo.

Fixed

• Significance: compare warns when the chosen test cannot reach alpha at the given seed count (e.g. Wilcoxon with ≤5 seeds).
• cohens_d returns signed infinity (not 0.0) when within-group variance is zero but the means differ.
• Several review-driven fixes: Hydra-scalar method names in Study sweeps, NaN-safe Holm correction, ragged-results validation, and an empty-checkpoints guard.

0.1.0 - 2026-06-22

First release of mushin as a standalone package — a fork of the rai_toolbox.mushin workflow layer from MIT Lincoln Laboratory's (unmaintained) responsible-ai-toolbox.

Added

• Standalone packaging: pyproject.toml, MIT license (original MIT-LL copyright retained), README, and a vendored value_check so the package no longer depends on rai_toolbox.
• uv-based development workflow with a committed uv.lock.
• Ruff (lint + format) and codespell, with pre-commit hooks.
• Makefile developer shortcuts (make check, make test-py, ...).
• GitHub Actions CI across Python 3.9–3.14.
• Community health files: CONTRIBUTING, CODE_OF_CONDUCT, SECURITY, issue forms, PR template, Dependabot, CODEOWNERS.

Changed

• Modernized type hints (PEP 585 generics, collections.abc imports).
• Declared support for Python 3.9–3.14; NumPy >= 2 except on Intel-macOS, which is capped at NumPy 1.x / torch 2.2.x (no newer wheels on that platform).

Fixed

• Compatibility with PyTorch 2.6+, which changed torch.load's default weights_only to True: pass weights_only=False when loading trusted, self-produced metric and checkpoint files.
• test_overrides_roundtrip: exclude Hydra-reserved tokens (null/none/ nan/inf) from the generated-string strategy.
• Updated deprecated xarray.Dataset.dims to .sizes in tests.