Comparative Evaluation of Embedding Models in Retrieval-Augmented Medical Chatbots

This paper aims to conduct a comparative evaluation of MiniLM, BERT, and BioBERT embeddings in the context of Retrieval-Augmented Generation (RAG). The experiments were conducted using PubMedQA and MedMCQA as the benchmark and FAISS as the retrieval system. The experiments showed that BioBERT emerged as a winner in terms of accuracy (0.67) and F1 score (0.576), which is a testament to the effectiveness of domain-specific pretraining in medical domains. Although MiniLM was slightly less accurate, it had significantly lower latency (8.97 ms), which puts it at the forefront as the most efficient model in conversational AI. The retrieval latency was low for all three models, and generation latency was constant at 0.9 s, which confirms the fact that retrieval efficiency is impacted only by the choice of embeddings and not generation synthesis. This paper has shown that there is a tradeoff between accuracy and efficiency in medical AI. The experiments are to be extended to multiple vector databases and multi-evidence retrieval to optimize user trust in medical AI.