ProxyScale: Low-Cost Mixed-Precision Policy Transfer for Memory-Efficient LLM Inference

Deploying large language models on consumer hardware remains difficult because both the model weights and the Key-Value (KV) cache must fit in GPU memory. This paper presents ProxyScale, a method that profiles layer sensitivity on a small proxy model running on a single consumer GPU and then transfers the resulting mixed-precision pattern to a larger target model, removing the need for any retraining. The sensitivity profile is built from activation-norm signals: layers whose hidden states carry higher magnitude receive more bits, while redundant layers are compressed aggressively. Across three model families—GLM-Edge, Qwen2.5, and Qwen3.5—and four benchmarks (MMLU, ARC-Easy, Needle-in-a-Haystack, and WikiText-2 perplexity with 95% Wilson confidence intervals), ProxyScale reaches average bitwidths between 5.4 and 5.7 bits, which corresponds to roughly three times less memory than full FP16 storage. On the dense Qwen2.5 family, the method outperforms uniform 4-bit quantization on MMLU by 3.4 percentage points and keeps long-context retrieval intact. On the heavily distilled GLM-Edge family, performance matches the uniform baseline. On Qwen3.5, which uses a Mixture-of-Experts architecture with dynamic token routing, accuracy drops below the uniform baseline, exposing a clear architectural limitation of static sensitivity maps. These results position ProxyScale as a practical, low-cost compression tool whose usefulness depends on how closely the proxy and target share a conventional dense structure.