The qlora 2e-4 default is wrong under 10k samples and nobody talks about it [D]

Every qlora tutorial on earth says start at 2e-4. Unsloth docs, hf examples, the paper itself. and for small datasets i now think that numbers is a trap.

Where does 2e-4 come from? alpaca. 52k samples. cool, except most of us are fine tuning on 5-10k samples we scraped and labeled ourselves, not 52k. at that size the model overfits inside epoch one and then youre just watching training loss go down all pretty while eval lost sits there doing nothing. or climbs.

I burned close to three weeks on this. recleaned the data set twice. rewrote the prompt template twice. spent on entire sunday hand relabeling rows while my flatmate watched football next to me (started with 8k rows, ended around 7200 after cutting garbage, i think, didnt log it properly). eval did not move. you you know what’s worse than a bad eval? seven identical bad evals in a row.

Then i changed one number. 2e-4 down to 1e-4, epochs 3 to 5. eval jumped more than everything else combined. i sat there refreshing wandb thinking it was a fluke. three more runs, same story.

And the annoying part, unsloth literally calls 2e-4 “a starting point” in their own docs. but every shared notebook has it hardcoded, zero comment. so people copy paste, get garbage, blame their data, blame their rank, lose a week. ask me how i know lol.

My rule now. above 30k 2e-4 is probably fine. under 10k, start at 1e-4 or lower and add epochs. in between, actually tune it, its one number, takes an afternoon.

If there’s real research defending flat 2e-4 on small data i want to read it. and if you all quietly figured this out in 2024 and never posted about it, im mad at every one of you individually.