Habits & experiments

The best way to pick up good engineering is to build it into your workflow from day one and let it run automatically.

• Reproducible environments — keep dependencies in the project (with uv), so uv sync makes the same code run on your laptop, on Colab, or on a cloud GPU.
• Repo hygiene — small, descriptive commits; a real README; a .gitignore that never includes data or model weights; one experiment = one config.
• Automate quality checks — a linter/formatter like Ruff plus a spell-checker, wired into a pre-commit hook so they run on every commit.
• Keep a lab log — every run gets a dated entry: what you ran, the settings, the numbers, what you concluded. It's how a final write-up gets written without a last-minute scramble.
• Use AI tools as drafters, not oracles — they're fast and useful, but always read and verify what they produce. And never paste anything sensitive or private into a cloud model.
• Getting unstuck — try it yourself, then search or ask your AI tools, then ask a mentor. A good question after a genuine attempt beats spinning for hours.
• Learn how research actually works — Bill Freeman's (MIT) chapter How to Do Research, in the free Foundations of Computer Vision, is a short, practical read on picking problems, making steady progress, and working with an advisor.

Start a project so it's reproducible from day one:

uv init my-project && cd my-project
uv add torch torchvision lightning scikit-learn
uv run python train.py     # runs in the project env — no activate needed

Wire up automatic quality checks that run on every commit:

uv add --dev ruff codespell pre-commit
uv run pre-commit install          # installs the git hook
uv run ruff check . && uv run ruff format .

…with a .pre-commit-config.yaml in your repo root:

# then run: uv run pre-commit autoupdate  (pins the latest versions)
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.6.9
    hooks:
      - id: ruff
      - id: ruff-format
  - repo: https://github.com/codespell-project/codespell
    rev: v2.3.0
    hooks:
      - id: codespell

A starter .gitignore — never commit data or model weights:

__pycache__/
.venv/
data/
outputs/
*.ckpt
*.pth
.env
.DS_Store

Once you're running more than a couple of experiments, these tools keep your work reproducible and your results organized. Don't front-load them on day one — reach for them when scattered scripts and logs start to hurt.

uv add hydra-zen "mushin-py[eval]" xarray   # [eval] adds compare, metric batteries & LLM eval

• hydra-zen — configure and launch experiments from Python instead of hand-written YAML. Each run saves its full configuration alongside its results, so experiments are reproducible, and parameter sweeps are one line. It has a dedicated PyTorch Lightning guide, so it layers onto your stack. Docs →
• xarray — labeled, multi-dimensional arrays: think NumPy with named dimensions and coordinates. Ideal for holding results across seeds and settings as one tidy dataset instead of scattered logs. Docs →
• mushin — a boilerplate-free sweep engine built on hydra-zen (a library I maintain). Decorate an experiment function with @mushin.sweep, sweep over parameters, and get the results back as a labeled xarray dataset — no subclassing. Your task just returns a dict, so it works with any model (scikit-learn, XGBoost, PyTorch), with first-class Lightning integration. With the [eval] extra, benchmark.compare and Study run a metric battery across seeds and report mean ± confidence intervals with statistical significance — "is this difference real, or just noise?" — and the same significance testing extends to LLM and agent evaluations. Docs →

hydra-zen — capture a config you can override and reproduce later:

from hydra_zen import builds, instantiate

OptimCfg = builds(dict, lr=1e-3, weight_decay=0.0)
cfg = OptimCfg(lr=3e-4)     # override one field
instantiate(cfg)           # -> {'lr': 0.0003, 'weight_decay': 0.0}

xarray — hold results across seeds and settings in one labeled array:

import numpy as np, xarray as xr

acc = xr.DataArray(
    np.random.rand(3, 2),
    dims=("seed", "lr"),
    coords={"seed": [0, 1, 2], "lr": [1e-3, 1e-2]},
    name="accuracy",
)
acc.mean("seed")     # average across seeds, per learning rate

mushin end-to-end — decorate a function that trains a Lightning model, sweep it across learning rates and seeds, get the results back as a labeled xarray dataset, and plot straight from it:

import lightning as L
import matplotlib.pyplot as plt
import mushin

@mushin.sweep
def lr_sweep(lr, seed):
    L.seed_everything(seed)
    model = MyLitModule(lr=lr)                   # your LightningModule
    trainer = L.Trainer(max_epochs=5, accelerator="auto",
                        logger=False, enable_checkpointing=False)
    trainer.fit(model, datamodule=dm)            # dm: your LightningDataModule
    return dict(accuracy=trainer.callback_metrics["val_acc"].item())

ds = lr_sweep.run(                               # 9 runs -> labeled xarray.Dataset
    lr=mushin.multirun([1e-3, 1e-2, 1e-1]),
    seed=mushin.multirun([0, 1, 2]),
)                                               # dims (lr, seed)
mean = ds["accuracy"].mean("seed")              # average over seeds
std  = ds["accuracy"].std("seed")               # spread across seeds

# plot straight from the xarray: val accuracy vs. learning rate, mean ± spread
plt.errorbar(mean["lr"], mean, yerr=std, marker="o", capsize=4)
plt.xscale("log"); plt.xlabel("learning rate"); plt.ylabel("val accuracy")
plt.tight_layout(); plt.savefig("acc_vs_lr.png")

Need the full toolkit — failure handling, provenance, custom analysis? Drop to lr_sweep.workflow, or subclass MultiRunMetricsWorkflow directly.