Linear Regression Excel in Predicting Government R and D Expenditure: ML Comparison and District Economic Clustering

This paper evaluates the performance of linear and ensemble regression associated with centroid This paper evaluates the performance of linear and ensemble regression associated with centroid and density-based cluster analysis, as ways of estimating and profiling Indian sectoral RandD expenditure for the period 2018–2021. The study used a dataset of publicly available government-related / funding dataset documented by economic activity, and with RandD expenditure attributable to the relevant activity. A linear regression of sectoral RandD expenditures predicted observations of 2020–21 with an R² ≈ 0.9989, and MAE ≈ 265, using sectoral RandD expenditure features of dimensions 2018–19 and 2019–20. The Random Forest regression used features and predicted an R² ≈ 0.9542 with RMSE ≈ 3201.7, and all features had relatively uniform feature importances against the two features with a lag. The K-Means and hierarchical clustering both showed meaningful partitioning of the activities into three groups with the identical partitioning, and their respective measures (Silhouette ≈ 0.8299, Davies– Bouldin ≈ 0.0517, Calinski–Harabasz ≈ 262.10). The DBSCAN cluster analysis default parameters terminated on a model of a single 'core cluster' with outliers (Silhouette = -1, DB = ∞) and produced a core cluster that was 'compact', with a density contrast that was not pointbased. The findings concluded, firstly, that the linear predictive relationship summing yearover- year disaggregated sectoral RandD series appeared to capture the overall trend of growth, and secondly, that centroid-based clustering produced meaningful, consistent cluster with interpretability also, and opportunities for policy considerations to assist RandD, and a basis for benchmarking services/labs/departments from similar clusters.