From Motion to Meaning: Decoding Sign Language Gestures using Deep Learning

Automating the recognition of Indian Sign Language (ISL) is a challenging task that involves various methods for processing movement over time and spatial shapes. This study presents a dual-pathway analysis comparing deep learning architectures in two areas: static fingerspelling (36 classes) and dynamic word recognition (61 classes). The research looks at the balance between feature stability and architectural efficiency for the static pathway. We thoroughly assessed a MobileNetV2 baseline, a High-Capacity Vision Transformer (ViT-Huge), and a lightweight Custom Sequential CNN. Despite the ViT-Huge achieving 100% accuracy and showing great resilience under stress with a Robustness Score of 90.94%, its high number of parameters (706.48 M) resulted in a slow inference speed of 0.16 FPS on standard CPUs. In contrast, the Custom CNN emerged as the best choice for edge deployment, achieving 8.20 FPS and 95.44% accuracy with a small size of 0.69 M parameters. The study also looks at the spatiotemporal complexity of 61 isolated sign words for the dynamic pathway. We used a Bidirectional Long Short-Term Memory (BiLSTM) network to capture subtle co-articulation patterns through MediaPipe Holistic for real-time feature extraction (258 keypoints). This dynamic model achieved a weighted F1-score of 0.889 and a test accuracy of 89.2% on an independent test set, setting a strong benchmark. This study concludes that lightweight Convolutional and Recurrent architectures remain practical for latency-sensitive, real-time communication systems. Large-scale Transformers can provide the necessary robustness for “in-the-wild” static recognition.