[2603.28768] CRAFT: Cost-aware Expert Replica Allocation with Fine-Grained Layerwise Estimations

This page presents CRAFT, a framework for efficient expert replication in Mixture-of-Experts (MoE) architectures used in large language models. The paper addresses the problem that existing replication schemes often over-replicate experts, consuming substantial GPU memory and degrading throughput. CRAFT maximizes load balance under memory constraints through fine-grained, per-layer replication based on estimated benefits. It seamlessly integrates into existing serving frameworks without requiring additional training or model modifications. Evaluations demonstrate that CRAFT achieves 1.14× average (up to 1.2×) throughput improvements over conventional replication techniques in large-scale deployments.