Multimodal AI for Bilingual Virtual Assistants: A Low- Latency Framework Integrating Whisper, Phi-3 and Edge TTS

Speech AI systems for multilingual environments must balance real-time performance with linguistic inclusivity, especially in code-mixed contexts. This paper presents an edge-optimized framework that addresses this trade-off by combining Whisper (speech-totext), Phi-3-mini (NLP), and Edge TTS (text-to-speech). Our unified pipeline achieves 33.1ms latency—3.6 times faster than modular baselines—on consumer-grade GPUs (8GB VRAM), with 89.3% English and 59.1% Hindi transcription accuracy. For code-mixed inputs, phonetic ambiguities (38% errors) and script-transition failures reduce token accuracy to 34.3%. Key innovations include GPU-CPU hybrid execution (84% cost savings over cloud APIs), model pruning, and a Gradio interface for real-world validation (85% user preference for sub-1s latency). While accuracy deficiencies still exist, this work reframes them as actionable insights for furthering low-resource ASR, providing a replicable template for community-driven tokenization improvements and edge deployment techniques. The results demonstrate that inclusive speech technology does not have to forfeit real-time performance, even in resourceconstrained environments.