Do you build outfits around your bag or treat it as an accessory?

I realized recently that I tend to build outfits around whichever bag I am carrying that day. If I grab a structured bag I end up dressing a little more polished, and if it is a slouchy tote the whole outfit goes more relaxed.

Curious how other people approach it. Do you pick your bag last or does it sometimes drive the outfit?