ML Monitors

What are ML Monitors?

ML monitors represent tools used to monitor ML model performance metrics and identify model performance issues. ML monitoring constitutes a subset of AI observability.  It involves measuring model performance and setting up alerts for drifts and model quality issues. In contrast, AI observability denotes a bigger picture with testing, validation, explainability, and preparedness for unpredictable failure modes.

‍

Types of ML Monitors

Data quality monitors

“The ML prediction is as good as the data used to train the model.”

• Missing data monitor: These monitors generate alerts for missing value violations exceeding user-defined thresholds. Monitoring missing data in production data streams helps you serve high-quality data to models similar to the training datasets. 
• New value monitor: A data quality monitoring tool that helps check production datasets for new feature values than the set of feature values used to train the model. 
• Data range monitor: These monitors assess range violations for numeric columns such as customer age. If the valid range is from 1-99 and the input value is a typo with an additional zero at the end for the set of rows, it affects performance with garbage input.
• Data type mismatch monitor: Data type mismatch happens when the data stream returns values that are not valid to the category. It can be due to conditions such as data source becoming unreliable, skewed data processing code, undesirable typecasting, or downstream schema change, etc.

Drift monitors

• Data drift monitors: These monitors track data distribution and skew in production and training datasets to aid data scientists in model retraining. 
• Concept drift monitors: These monitors track production datasets for target changes and newly introduced categories. 

Model activity monitors

These monitors track the number of predictions the model serves over time and help recognize deviations, model overload possibility, and potential future trends.

Performance monitors

These monitors help track ML performance metrics such as precision, recall, sensitivity, specificity, F1 score, FNR, and FPR.

‍

Why is ML Monitoring Essential?

• To identify performance issues with ML models and supporting data pipelines 
• To take the right course of action for triaging and troubleshooting ML models  
• To ensure ML model predictions are well-explicable and transparent
• To enhance ML prediction system with seamless performance and governance

‍

Leveraging ML Monitors

ML model monitoring is not difficult with advanced monitoring solutions like Censius AI Observability Platform. It automates tracking the performance of production ML models and the entire ML pipeline.

Censius Observability Platform facilitates customized monitors to drive deeper insights on your ML pipelines. At present, the platform enables setting and customizing ML monitoring solutions for:

• Data Quality Monitors
• Drift Monitors
• Model activity monitors
• Performance monitors

The platform automates ML model performance tracking and frees up your team’s bandwidth for more productive tasks.

‍

Further Reading

A Comprehensive Guide on How to Monitor Your Models in Production

Monitoring Machine Learning Models in Production