Small a/b test puzzle that broke my brain

I recently build a platform, that aims to help everyone to practice data science cases, to get hands on experience. I've been working as a DS for years. I mainly use databricks or Hex notebook, with AI assistant. So this platform can let you practice with the same tool. This is one of the best case I've built, and I want to share it with all of you---

Imagine you're testing two homepage banners. banner A vs banner B. Two weeks of traffic, lots of data. Banner A wins by a comfortable margin - cool, ship A, done.
Then for some reason you decide to split it by device before pushing the button.
Desktop: B wins
Mobile: B wins

So, banner B is better for desktop users. And banner B is better for mobile users. But added up, banner A wins overall? How the answer is the test wasn't fair. For whatever reason (caching, ad targeting, just bad luck), banner A got shown to a lot more desktop traffic than banner B did. And desktop users convert way better than mobile on almost every site. So it turns out A wasn't a better banner, it was a banner that got tested on an easier audience. Fix the traffic mix and B is the right call.

This thing has a name (Simpson's Paradox if you wanna google it) but you don't need the name to spot it. you just need to remember to slice your data before you trust the headline. If you are interested, you can practice the same case at https://www.litmetrics.ai/