Data Integration and Transformation using Large Language Models

This research paper presents a novel approach for automating data integration and transformation of CSV data using a large language model (LLM) and Python. The approach involves converting CSV data to JSON objects, and then accepting natural language queries from users to specify the transformation they require. The LLM generates Python code that manipulates the JSON objects based on the user query, accounting for variations in column names and data formats across CSV files. The approach is verified and evaluated through precision, recall, and F1 score metrics and user studies, demonstrating its potential to significantly reduce time and effort in data management and analysis. Overall, our approach showcases the potential of combining LLMs with Python to streamline data integration and transformation tasks