Abstract

The growing complexity and sophistication of cyber-attacks necessitate the development of robust intrusion detection systems for effective network security. It is also essential for businesses to identify and deal with these threats in real time to ensure operational continuity, secure sensitive data, prevent financial loss and safeguard their reputation in an increasingly competitive and fast-paced business environment. In this paper, we develop a Kafka-based Intrusion Detection System that offers real-time anomaly detection by utilizing Apache Kafka and machine learning models to detect and classify connection attempts as anomalous or not. This system employs Apache Kafka as a central messaging platform, which has efficient data streaming capabilities, fault tolerance, and scalability. It also enables the system to transition from traditional batch processing to real time intrusion detection. We apply the anomaly-based approach to develop an IDS engine that leverages historic network data, attack patterns, and a trained machine learning model to identify peculiar inputs. Random Forest Classifier model is employed for threat detection. The proposed system displays a proficient ability to recognize suspicious patterns while also generating Slack notifications to the concerned channel and log the event to the terminal.