•1 min read•from Towards Data Science
Stop Calling the First Significant Day a Win
Checking an A/B test until it crosses p < 0.05 can turn a nominal 5 percent false-positive rate into almost 28 percent. I use a seeded simulation to show how large the damage gets and compare the fixes that keep early stopping honest.
The post Stop Calling the First Significant Day a Win appeared first on Towards Data Science.
Want to read more?
Check out the full article on the original site
Tagged with
#A/B testing
#p-value
#statistical significance
#false positive
#early stopping
#simulation
#data science
#statistical analysis
#hypothesis testing
#nominal rate
#error rate
#seeded simulation
#stopping rule
#confidence level
#p < 0.05
#experiment design
#statistical power
#data analysis
#optimization
#machine learning