High validation accuracy can conceal production risk: Using SHAP to expose and block proxy bias at runtime [P]

To demonstrate a problem that accuracy-only model validation often misses, here is a breakdown of a synthetic hiring-screening pipeline where a model cheats the metric, and how to physically intercept the failure at runtime.

A logistic regression model was trained using three features: technical assessment score, years of experience, and a synthetic postcode indicator.

The model achieved 94.2% validation accuracy. Without feature-level analysis, it could easily pass an accuracy-only deployment gate.

But the training labels were deliberately poisoned. They were constructed directly from the postcode field, simulating a historical selection process in which location determined who progressed.

Using shap.LinearExplainer to inspect a proposed inference, the feature attribution array exposes the shortcut:

``text
technical_score: -0.0007
years_experience: -0.0680
postcode: 3.5031
`

The model was not measuring candidate suitability. It was accurately reproducing a biased historical decision rule.

An explanation makes the problem visible, but it does not enforce what happens next.

To solve this, the estimator is wrapped with an L2 semantic execution boundary (ramen-mlflow-guard). For the governed request, the application passes both the feature payload and the calculated SHAP evidence to the wrapper (RamenGovernedModel).

Before delegating to the inner model’s predict() method, the wrapper evaluates that evidence against a configured proxy-bias policy.

The policy denies the request. The wrapper raises a GovernanceDeniedException`, and the underlying estimator does not execute the governed prediction.

The application receives remediation steering, relevant statutory anchors (e.g., EU AI Act), and a locally verified Ed25519 receipt bound to the configured policy.

The model looked reliable when judged by accuracy alone. Runtime governance prevented the proxy-driven decision from producing a governed prediction.