Late Breaking Results: Structured Expert Routing for Efficient MoE Inference under GPU-CPU Orchestration

Abstract

Hybrid GPU-CPU deployment of large Mixture-of-Experts (MoE) models is often latency-bound by CPU-side expert execution and cross-device orchestration overhead. To overcome this limitation, we propose a structured expert routing strategy, implemented via asymmetric expert skipping, which reduces expert activation in latency-critical layers while applying lightweight magnitude compensation and norm calibration to preserve model accuracy. Experiments on quantized Qwen3 (30B and 235B) models show that our method improves end-to-end generation throughput by up to 60% while achieving 3.6% higher accuracy than uniform expert reduction under the same compute budget.

Pu Zhao

Professor