EmoGita: A Multimodal Emotion-Aware Retrieval- Augmented Framework for Personalized Bhagavad Gita Guidance

Emotional distress often shows up in subtle ways through the interplay of voice tone and language, making it difficult for systems that rely on just one type of input to interpret accurately. This paper introduces EmoGita, a multimodal framework that combines speech and text analysis to detect a user’s emotional state and provide personalized philosophical guidance from the Bhagavad Gita using a retrieval-augmented generation (RAG) approach. Speechbased emotions are captured with a Wav2Vec2 model, while text-based emotions are inferred using a BiLSTM classifier on transcripts generated by Whisper. The results from both channels are merged using a weighted probabilistic method to reduce ambiguity and enhance emotional accuracy. The combined emotional representation, together with the user’s context, is projected into a shared semantic space to query a vector database of Bhagavad Gita verses using sentence-level embeddings. A large language model then generates explanations that explicitly connect the retrieved verses to the user’s emotional state. Compared with systems that rely solely on text or speech, this multimodal fusion approach delivers guidance that is more emotionally aligned and contextually relevant. EmoGita demonstrates how affective computing and emotion-conditioned retrieval can be integrated into a single, human-centered AI system for offering sensitive and personalized guidance.