Survey on Model Monitoring in NLP

Model Monitoring is an operational stage which comes after model deployment in machine learning lifecycle. It includes monitoring ML models for changes like model performance degradation, resultant data drift, and concept drift, and ensuring that the developed model is maintaining an optimum level of performance. ML teams can use model performance metrics like accuracy, precision, recall to monitor the real time or live performance of production models. However, these metrics need ‘ground truth’ or labels for these real-time predictions. Although labels are always available in a training data set, they might not always be available in production for a given use case. In the absence of inputs or to complement the visibility of the performance metrics, monitoring live changes in production features and prediction distributions can be used as an indicator and troubleshooting tool for issues in performance. Proper model monitoring can help improve model performance, increase transparency, and ensure that models are deployed ethically and responsibly. We provide an overview of the importance of model monitoring in NLP and highlight some of the key techniques and tools used for this purpose. The main aim of this research will be to understand the capabilities of novel drift algorithms and how they can be used to automate the ML-OPS pipeline used in the industries. Monitoring NLP Models can save a lot of time and effort of data scientists. It will also help organizations in keeping their large language models to be more consumer friendly and ethical