Model reliability program for a machine learning platform
Production models drifted quietly, and the team had no consistent way to know when a model had stopped behaving.
- Industry
- Insurance and financial services
- Outcome
- [N production models] brought under monitoring across [M business lines]
Organization or industry
A national insurance and financial services company running machine learning models in production across several business lines, including underwriting support and fraud detection.
Initial problem
The company had shipped a number of models, but reliability was uneven. Training and serving code had diverged in places, so a model could score differently in production than it had in testing. Data feeding the models changed upstream without warning, and drift was usually noticed only after a downstream team complained. There was no shared definition of what healthy looked like for a model, and no consistent alerting, so incidents were caught late and diagnosed slowly.
Technical environment
- [N production models] serving predictions across [M business lines].
- An Azure-based platform with models trained in [training environment] and served through [serving layer].
- Feature data drawn from several internal systems, with limited validation between source and model.
- Existing monitoring focused on infrastructure health rather than model behavior.
Abisam team involvement
Abisam provided a focused team that worked with the company's data science and platform groups. We concentrated on the reliability layer around the models rather than the models themselves, and we set the program up so the internal teams could own and extend it.
Approach
We began by making the current behavior visible. For each model we defined what to measure, including input distributions, prediction distributions, and, where labels were available, live performance against outcomes. We closed the gap between training and serving by moving shared feature logic into a single path used by both, so a model scored the same in both places. We added data validation at the boundary, so malformed or out-of-range inputs were caught before they reached a model. On top of that we built drift and performance monitoring with thresholds agreed with each business line, and alerting that pointed to the likely cause rather than just reporting that something was wrong. We wrote runbooks so an on-call engineer could act without needing the original author.
Systems or processes delivered
- A shared feature path used by both training and serving, removing a class of silent scoring differences.
- Input validation at the data boundary, with clear handling for records that fail.
- Drift and performance monitoring per model, with thresholds owned by each business line.
- Alerting tied to runbooks, so incidents route to the right people with a starting point for diagnosis.
- A retraining and review cadence the internal teams could follow without Abisam in the loop.
What the organization was able to do afterward
The company moved from finding out about model problems through complaints to catching them through monitoring. Scoring differences between test and production stopped being a recurring surprise, incidents were diagnosed in a fraction of the time, and each business line had a clear, shared picture of whether its models were healthy. The internal teams carried the program forward and onboarded new models to it themselves.
Have a system that has to hold up?
Tell us what is not working and we will tell you where to start.