Self-Learning Log-based Failure Prediction using LSTM for Proactive System Reliability

Distributed systems generate an immense amount of log files that contain many valuable signals regarding system health, runtime behavior, and possible failures. Many of today’s operational monitoring solutions are designed to react to failure after the fact, meaning that the alerting process occurs only after some service degradation has already occurred. This paper details a self-learning log-based framework that transitions reliability management from a purely reactive focus on detection to a proactive focus on prediction. The proposed framework uses structured log parsing with horizon-based sequence labeling and a Long Short- Term Memory (LSTM) classifier to create a system that can be periodically retrained to improve its accuracy over time as the underlying systems evolve. Raw log messages are first abstracted into stable templates, assigned integer template IDs, and fixed-length overlapping windows of template streams are then extracted and marked (labeled) based on whether or not a failure occurs within a defined prediction horizon. The labeled sequences are then used to train the LSTM, which is used to predict the likelihood of an imminent failure. As a result of continually changing software, workloads, and infrastructure, the framework uses incremental retraining as part of its overall scheduling strategy and will only replace the current model with a candidate model when the replacement candidate demonstrates superior validation performance. On the public LogPai HDFS benchmark, the framework achieved a classification accuracy of 0.93; precision of 0.91; recall of 0.89; and an F1-score of 0.90; and a ROC-AUC value of 0.95.