Why production systems keep making “correct” decisions that are no longer right [D]

I’ve been looking at a recurring failure pattern across AI systems in production. Not model failure, or data quality or infrastructure.

Something else. Where system continues to operate exactly as designed, models run, outputs look valid, pipelines execute and governance signs off

But the underlying assumptions have shifted. So you end up with decisions that are technically correct, but contextually wrong. Most organisations respond by tightening controls, reducing overrides or increasing monitoring.

Which just reinforces the same behaviour. I’ve tried to map this as what I’m calling the “Formalisation Trap”, where meaning gets locked into structure and continues to be enforced even after it stops reflecting reality.

Has anybody else seen similar patterns in production systems?