I silently break training codes or configs so I made pybench [P]

It is like pytest but for statistical tests: it ensures no regression of your metrics at a statistical level.

It manages tedious things such that seeds, past benchmark results, ...

Simple CLI working like pytest but with benchmarks/ directory instead of tests/:

pybench # 1st time: samples seeds, saves a baseline, marks NEW pybench # later: reruns on the same seeds, marks PASS / FAIL pybench update # re-baseline after an intended change pybench show # print current baseline stats (--history for per commit) 

Please give me your feedback,

Github: https://github.com/AnthonyBeeblebrox/pybench

Docs: https://pybench.readthedocs.io/en/latest/

EDIT: this is for statistical regressions in metrics, not a replacement for unit test