ESC-50, from Scratch

ResNet-18 on log-mel spectrograms. No pretraining, no transfer, no ensemble cheats. Trained on the standard ESC-50 5-fold split, single-file Python, runs end-to-end on one GPU in under an hour.

Final number: 83.0% mean accuracy! across the 5 official folds (83.75 / 82.5 / 82.5 / 85.75 / 80.5). For reference, PANNs CNN14 trained from scratch reports 83.3% on the same split, so this is essentially at parity with the standard published baseline, which is what I was aiming for.

I'd been meaning to do this for a while. Most of the public ESC-50 repos either use ImageNet-initialized backbones and don't say so loudly enough, or use SpecAugment with iid_masks=False (the torchaudio default - same mask coordinates across the whole batch, which kills most of the regularization), or both. So the "from scratch ResNet baseline" you find on GitHub is often a couple points worse than it should be.

Layout

main.py              everything (dataset, frontend, model, train loop, 5-fold driver)
scripts/resample.py  one-shot prep: 44.1k wavs -> 32k mono f32, cached as one .pt
scripts/infer.py     load chkpt(s), classify a wav, optional TTA + cross-fold ensemble
scripts/confmat.py   per-fold + aggregated 50x50 confusion matrices from chkpts
chkpt/fold#.pt       trained models, each holds ema + raw weights + mu/sigma + cfg
logs/                per-fold .log files plus a summary.json
test/                two random barking-dog wavs I grabbed for sanity checks
tmp.md               my planning doc from before I wrote any code, mostly notes-to-self

Running it

python -m venv .venv
source .venv/bin/activate.fish  # or .venv/bin/activate for bash
pip install numpy torch torchaudio torchvision torchcodec # also maybe matplotlib if you want

git clone https://github.com/karolpiczak/ESC-50 data/ESC-50-master
python scripts/resample.py      # writes data/esc50_32k.pt (~610 MB)

python main.py                  # full 5-fold sweep, defaults
python main.py --folds 1        # one fold for a smoke test
python main.py --epochs 300 --batch 128 --lr-max 1.5e-3

A fold took me 5 minutes or so on single RTX 5070Ti. If you have multiple GPUs the easiest thing is one fold per GPU via CUDA_VISIBLE_DEVICES; DDP within a fold buys you nothing at this dataset size.

Inference on your own wavs

python scripts/infer.py --chkpt chkpt/fold1.pt --wav some.wav

# softmax-average across all 5 fold checkpoints, with 10-crop TTA
python scripts/infer.py --chkpt chkpt/fold*.pt --wav some.wav --tta 10 --topk 5

One thing to be careful about: averaging the 5 fold checkpoints is fine for inference on out-of-distribution audio you record yourself, but you can't do it on held-out ESC-50 clips and report the number. Every fold's checkpoint has trained on 4 of the 5 splits, so the ensemble has seen everything. The 83.0% above is per-fold checkpoints evaluated on their own held-out fold only.

Confusion matrices

python scripts/confmat.py --chkpt chkpt/fold*.pt
# writes logs/confmat/{fold[1-5],aggregate}.{csv,png} + top_confusions.txt

Each fold checkpoint is scored on its own held-out fold only (no leakage), and the aggregate is the elementwise sum where every clip contributes exactly once. Re-evaluating the saved checkpoints gives 82.80% aggregate (per-fold 83.50 / 82.00 / 82.25 / 85.75 / 80.50, mean 82.80), within rounding of the 83.0 headline; the small drift on folds 1-3 is the EMA val acc at the saved-best epoch vs. a clean re-eval pass.

An...odd obervation: my original hypothesis in Things I didn't do was wrong. The dominant confusion isn't crow/rooster (doesn't even crack top 20) - it's cat --> crying_baby at 27.5%, and the model never makes the reverse mistake. The other big ones are mostly the wind/helicopter/airplane/vacuum_cleaner blob (broadband noise that's hard to separate without longer context) and the click family (mouse_click ↔ can_opening ↔ keyboard_typing).

Top 10 aggregate off-diagonal pairs (rate = errors / 40 clips per class):

true --> pred	rate	count
cat --> crying_baby	0.275	11/40
helicopter --> airplane	0.200	8/40
helicopter --> wind	0.125	5/40
washing_machine --> vacuum_cleaner	0.125	5/40
mouse_click --> can_opening	0.125	5/40
mouse_click --> keyboard_typing	0.125	5/40
water_drops --> mouse_click	0.125	5/40
wind --> helicopter	0.100	4/40
wind --> crickets	0.100	4/40
water_drops --> clock_tick	0.100	4/40

Full ranked list in logs/confmat/top_confusions.txt, per-fold matrices in logs/confmat/.

How To

The non-default choices that matter:

• 32 kHz, 5 s clips, mel(n_mels=128, n_fft=1024, hop=320), AmplitudeToDB(top_db=None)

• per-fold dataset-level mean/std, computed on training waves only (not per-instance - per-instance loses absolute-loudness cues and composes badly with mixup)

• waveform-domain mixup with the energy-preserving denominator /sqrt(λ² + (1-λ)²), Beta(0.2, 0.2). Mixing in waveform space rather than mel space because two sounds playing simultaneously is x1 + x2 in time, not in mel (the mel transform is non-linear)

• SpecAugment with iid_masks=True (the important bit), 2x freq mask param 48, 2x time mask param 96 with p=0.2

• random gain +/- 6 dB, random circular roll up to 1 second

• ResNet-18 with 1-channel stem, dropout 0.2 before fc, last-BN-of-each-residual-block zero-initialized (the ResNet-D trick)

• AdamW lr 1e-3 --> 1e-6 with 5-epoch linear warmup into cosine, 200 epochs, wd 1e-4

• label smoothing 0.1, grad clip at 5.0

• EMA (decay 0.999) for evaluation. Critically, the EMA model gets a refresh_bn pass before each eval. Without it the EMA val accuracy will look 1+ pp lower than raw and you'll think EMA is broken

• fp16 training with autocast, but the mel frontend is wrapped in autocast(enabled=False). fp16 squaring of small mel magnitudes underflows and you get NaNs from the subsequent log10

Skip any of these and you'll generally lose 1-2 pp. Skip several and you end up at 78%, which is the "I tried" number you see in a lot of writeups.

Things I didn't do

• BC-learning's SPL-aware mixup (Tokozume et al. 2018). The plain energy-preserving mix gets you to ~83; BC-learning would probably push it toward 85. Maybe later.

• PCEN. The papers on this are pretty clear that the wins are concentrated in noisy / far-field / bioacoustic data, not clean studio recordings. ESC-50 is the latter.

• AudioSet pretraining. Different project.

Notes on the code

The semicolons everywhere in main.py are a personal habit, just ignore them. The argparse blocks in various files the only chunks an LLM wrote (Claude Sonnet noted in a comment in the file) because I didn't want to type out 14 add_argument calls by hand and the alternatives (a config dict, macros, a YAML loader) were all worse for a one-file project.