Indian Sign Language (ISL) Recognition and Translation System

Communication barriers between the hearing-impaired community and general society remain a persistent challenge for social inclusion. Existing sign language recognition systems predominantly target foreign sign languages and support only unidirectional translation. This paper presents a real-time, bidirectional Indian Sign Language (ISL) recognition and translation system that overcomes both limitations. The proposed system employs MediaPipe Holistic for simultaneous hand, face, and pose landmark detection, producing a 1,563-dimensional feature vector per frame. A stacked three-layer Long Short- Term Memory (LSTM) network performs temporal gesture classification over rolling 30-frame windows. A custom ISL dataset comprising 30 gesture classes with 30 sequences each (30 frames per sequence, totalling 900 sequences and 27,000 frames) was collected from signers under varied lighting and background conditions for training and evaluation. The sign-to-voice pipeline converts recognized gestures to speech via gTTS, while the voice-to-sign pipeline uses the Web Speech API for transcription and a GIF-indexed ISL repository for sign display. The system is deployed through a FastAPI back-end with WebSocket streaming and a browsernative HTML5/CSS3/JavaScript front-end requiring no installation. As demonstrated by system output — including the recognition of ‘thankyou’ at a confidence of 0.98 and bidirectional translation of phrases such as ‘good morning’ — the system functions reliably in real-world conditions. The proposed framework is modular, extensible, and establishes a foundation for inclusive, real-time ISL communication.