A Neural Universal Dependencies-based Framework for Analyzing POS Divergence in Marathi Automated Short Answer Evaluation

Part of Speech (POS) tagging is important in numerous Natural Language Processing (NLP) applications, including Automated Short Answer Evaluation (ASAE). Most traditional approaches assume that semantically identical concepts receive identical POS tags despite variations in context. However, this principle does not apply to morphologically diverse languages such as Marathi, where the same concept could exist in different grammatical forms. In this paper, POS divergence in ASAE systems is investigated via a neural UD approach based on the Stanza NLP library. POS divergence is described as a situation where identically spelled or semantically similar words receive different POS tags due to morphological variations. For the research, a manually curated short answer dataset with one-word model responses and students' replies in Marathi language is utilized. According to the results obtained via experimental testing, POS divergence occurs in most cases of Marathi ASAE. More precisely, around 23% of lemma-based matching showed POS divergence, whereas a significantly higher rate was noted among semantically accurate responses. Upon closer examination of manually processed data, it becomes clear that most differences were linguistically justified, being the result of the grammar change in the context, rather than incorrect tagging. Consequently, strict dependency on POS matching for evaluation purposes leads to false negative scores, which means that the system rejects the response, even if it is correct. Therefore, Marathi ASAE requires alternative methods of response evaluation that include not only grammatical but also semantic analysis.