Policy-Induced Distributional Shift in RLHF: A Real- Time Monitoring and Penalization Framework for PPO Training

Reinforcement learning from human feedback (RLHF) remains vulnerable to reward hacking, where policies exploit imperfections in the proxy reward model rather than im-proving genuine response quality. Existing mitigations—such as KL regularization, re-ward shaping, ensembling, and behavior-supported constraints—typically lack a direct step-wise signal of emerging overoptimization. We investigate whether policy-induced dis-tributional shift in the reward model’s latent representation space can serve as such a sig-nal. We propose DSP-PPO (Distributional Shift Penalized PPO), which computes a PCA-Mahalanobis distance from the reward model’s in-distribution latent support and subtracts a graded penalty from the PPO reward. The method requires no architectural changes, no reward ensembles, and only a one-time statistics pass. On Anthropic HH-RLHF using GPT-2 and an Open-Assistant DeBERTa-v3-large reward model, the shift signal corre-lates significantly with reward-model scores (r = 0.3036, p = 1.25×10??) and temporally precedes proxy-reward inflation by 15 PPO steps (r = 0.1825, p = 0.0341), offering a pro-spective early-warning indicator absent in KL regularization alone. At ? = 0.001, DSP-PPO matches KL-only proxy reward over the final 20 steps while providing a continuous latent-drift trace at only 0.22% per-step overhead. DSPPPO offers a practical real-time monitoring framework for standard RLHF pipelines and motivates future causal valida-tion and independent alignment evaluation.