How do you incorporate hats into your look?

I don’t wear hats, not because I don’t like them, but because they’ve always felt like a challenge for me style wise. However, I have been told many times that even a simple ball cap looks great on me and I tend to agree when I see the pictures.

That being said, hats are hard! Do you just have to have a large collection of colors and shapes/sizes to make it work? Or do you just buy a few and plan the outfits around the hat? I want to generally avoid looking like I’m going to the Kentucky derby on a casual day out. The one ball cap I do have is bright red, is that something you would pair with another bold bright color so it doesn’t seem out of place? Or let the hats hold their own?