Abstract

In today's digital age, video interviews are prevalent, enabling remote interactions and necessitating user sentiment analysis for insights in recruitment, market research, and psychology. This paper outlines an advanced approach using NLP and facial emotion recognition for enhanced sentiment analysis in video interviews. It employs multimodal emotion detection models: audio-based emotion prediction via CNN for tone recognition, CNN-based facial emotion recognition (with optional media pipe face landmarks), and an LSTM network for speaker emotion prediction from video clip audio and image sequences. Hyperparameter tuning ensures optimal accuracy across datasets. OpenCV detects faces, extracting expressions for emotion analysis, while speech-to-text analysis via the Google Cloud Speech-to-Text API aids in accurate transcription. Text sentiment analysis with Scikit-learn's Naive Bayes classifier further enhances understanding. This integrated method captures verbal and non-verbal cues, facilitating nuanced sentiment analysis for informed decision-making in diverse domains.