A Web-based Indian Sign Language to Kannada Language Translation System Integrating with NLP and Real-Time Gesture Recognition

It is always difficult to bridge the communication gap between the hearing and the hearing-impaired community, especially in linguistically diverse places like India. The heard of hearing community's primary form of communication, Indian Sign Language (ISL), is difficult to integrate with regional spoken languages like Kannada, which makes it difficult to use in social situations, public services, and education. In this paper, an end-to-end ISL translation and recognition system that combines web technologies and Natural Language Processing (NLP) is presented. User-uploaded videos are processed by a Flask-based RESTful API, which uses MediaPipe and OpenCV for real-time gesture detection and classification. This system incorporates a Kannada-to-ISL pipeline that uses spaCy for dependency parsing and part-of-speech tagging, converting intricate Kannada sentences into glosses that are compatible with ISL in order to facilitate multilingual translation. TensorFlow.js and MediaPipe enable real-time client-side recognition for responsive, offline-capable interactions, while animated Three.js avatars are used to depict ISL motions. Secure MongoDB data management and scalable user authentication are made possible by the backend, which was created using Spring Boot and hosted on localhost services. The platform's potential to promote inclusive communication, education, and social engagement for Kannada-speaking deaf and hard-of-hearing people is demonstrated by the experimental findings, which indicate high accuracy and low latency processing.