X-MoE-4D: Breaking the Scaling Barrier for Trillion Parameter MoEs on Hierarchical HPC Systems

1University of Illinois Urbana-Champaign 2Oak Ridge National Laboratory

News

2026-11-17: X-MoE-4D is presented at SC 2026 in Chicago (Tuesday, November 17, 2:15 PM CST, Room W178ab, session "Scaling Complex AI Training Workloads").

Abstract

We present X-MoE-4D, a distributed training system for scaling expert-specialized Mixture-of-Experts models on hierarchical HPC platforms. We observe that existing expert-parallel approaches suffer from severe scalability limits due to cross-node all-to-all communication. To address this, X-MoE-4D adopts a pipeline-global, expert-local execution model that confines expert computation within nodes while scaling across nodes via pipeline parallelism. X-MoE-4D further introduces a Minimax Pipeline Planner that jointly optimizes pipeline partitioning and activation checkpointing to minimize bottleneck stage execution time under memory constraints. Together, these techniques eliminate cross-node expert routing and mitigate MoE Pipeline Skew.

We evaluate X-MoE-4D on the Frontier supercomputer, demonstrating the first efficient 1-trillion-parameter expert-specialized MoE training on an AMD-based HPC platform. X-MoE-4D achieves 2.24× higher aggregate throughput than X-MoE at 1-trillion-parameter scale (2.4× per GPU throughput) while using fewer GPUs (960 vs. 1,024), and sustains a 3.24× speedup when strong-scaling the 173B model to 2,304 GPUs.

Background

Mixture-of-Experts (MoE) scales large language models by activating only a subset of experts per token, so the capacity of the model grows without a proportional increase in computation. Existing MoE training systems primarily rely on expert parallelism (EP), which distributes experts across devices and dispatches tokens via all-to-all communication.

Modern MoE designs, such as Kimi K3, DeepSeek-V4, Qwen3-Next, and Llama 4, have shifted toward fine-grained experts with large top-k routing, which increase the number of experts significantly while reducing per-expert size to improve expert specialization. On hierarchical HPC platforms, this shift creates three challenges for training systems:

Cross-Node All-to-All Communication

Under EP, tokens are dynamically routed to remote experts and gathered back across devices. As the number of experts and devices increases, cross-node traffic grows rapidly and becomes constrained by the hierarchical bandwidth of HPC interconnects such as Dragonfly. When EP grows from 16 to 512 GPUs, MFU falls from 26.4% to 4.3% (Figure 1).

Per-layer execution time and MFU as expert parallelism grows from 16 to 512 GPUs

Figure 1. Impact of scaling expert parallelism (EP). As EP increases, dispatch and combine all-to-all communication becomes a dominant component of MoE execution time, reducing MFU and increasing runtime variability.

Routing-Induced Activation Memory

Fine-grained experts with large top-k routing substantially increase routing-related activation memory, which scales with both the number of experts and the top-k routing degree. This memory can dominate the per-layer footprint and makes memory and computation costs highly uneven across layers.

MoE Pipeline Skew (MPS)

Keeping experts within a node and scaling across nodes with pipeline parallelism removes the cross-node all-to-all, but pipeline execution prolongs activation lifetimes in early stages. Combined with the amplified activation memory and imbalanced compute costs of fine-grained experts, this disproportionately inflates the cost of the early pipeline stages and makes them the bottleneck for end-to-end training.

X-MoE-4D Design

X-MoE-4D overview

Figure 2. Overview of X-MoE-4D. EP-based scaling (top-left) suffers from cross-node all-to-all bottlenecks. X-MoE-4D adopts pipeline-global, expert-local execution (top-right) to eliminate cross-node routing, but introduces MoE Pipeline Skew (MPS) due to stage imbalance. ELM-PP (left) mitigates MPS by balancing pipeline stages via joint partitioning and checkpointing. Efficient expert-local execution (right) improves utilization by reducing kernel fragmentation.

Pipeline-Global, Expert-Local Execution

Expert parallelism is kept within each node, so all expert routing runs over the high-bandwidth intra-node interconnect, while cross-node scaling uses pipeline parallelism. This aligns MoE execution with the hardware topology and eliminates cross-node expert routing. The aggregate memory of a node is sufficient to host a full MoE layer even at trillion scale (512 GB across 8 GCDs on Frontier).

Expert-Localized Minimax Pipeline Parallelism (ELM-PP)

A skew-aware pipeline planner that jointly optimizes stage partitioning and activation checkpoint placement to minimize the execution time of the slowest stage under per-GPU memory constraints. It combines selective checkpointing with uneven layer partitioning, guided by a memory model and a runtime model, and solves the partitioning by dynamic programming on top of the 1F1B schedule.

Efficient Expert-Local Execution

Confining experts within each node co-locates tens of fine-grained experts (e.g., 64) on each GPU, which fragments expert computation into many small kernels. X-MoE-4D uses GroupGEMM with two backends, Primus-Turbo's Composable Kernel (CK) for AMD GPUs and a Triton implementation for cross-platform support; both aggregate per-expert work into larger persistent kernels.

Evaluation Results

We evaluate X-MoE-4D on the Frontier supercomputer (AMD MI250X GPUs connected by a Dragonfly network) with up to 2,304 GPUs, on expert-specialized MoE models from 10B to 1T parameters, against DeepSpeed-MoE, DeepSpeed-TED, Tutel, TorchTitan, and X-MoE.

MFU and aggregate throughput from 63B to 1T parameters

Figure 3. Training performance across model scales (63B–1T) on Frontier. (a) Model FLOPs Utilization (MFU). (b) Aggregate throughput (PFLOPS). X-MoE-4D achieves the highest MFU and throughput across all scales.

Scale: The first efficient 1-trillion-parameter expert-specialized MoE training on an AMD-based HPC platform, at 24.8% MFU on 960 GPUs.
Throughput: 2.24× higher aggregate throughput than X-MoE at 1T (2.4× per GPU) with fewer GPUs (960 vs. 1,024); 35.4% MFU at 537B, a 3.03× improvement over X-MoE.
Scalability: On the 173B model, a stable 36% MFU under weak scaling to 1,152 GPUs, and up to 3.24× speedup over X-MoE under strong scaling to 2,304 GPUs (Figure 4).
Weak and strong scaling of X-MoE-4D and X-MoE on the 173B model

Figure 4. Scalability of X-MoE-4D vs. X-MoE on the 173B model. (a) Weak scaling. (b) Strong scaling. X-MoE-4D maintains high efficiency and achieves faster iteration time. The performance gap widens with scale.

Memory: At 1T, per-GPU model-state memory falls from 41.1 GB with X-MoE (1,024 GPUs, EP256-TP4) to 22.8 GB with X-MoE-4D (960 GPUs, PP60-EP8), a 1.8× reduction. Pipeline stages shard the non-MoE layers instead of replicating them, so attention-state memory falls from 11.4 GB to 0.9 GB per device. The headroom lets X-MoE-4D train the 1T model on as few as 480 GPUs (Figure 5).
Per-GPU model-state memory of X-MoE and X-MoE-4D at 1T scale

Figure 5. Per-GPU model state memory at 1T scale. Comparison of the highest throughput X-MoE and X-MoE-4D configurations. X-MoE-4D reduces per-device model-state memory from 41.1 GB to 22.8 GB.

Stability and convergence: Across five independent 1T training runs, the coefficient of variation of MFU is 0.4%, against 11.8% for X-MoE, because all-to-all communication stays within a node (Figure 6). Trained for 2,000 iterations on the 10B model, both systems give nearly identical loss curves (3.583 for X-MoE and 3.543 for X-MoE-4D at iteration 2,000), so the partitioning and recomputation decisions of ELM-PP do not alter convergence.
MFU of X-MoE and X-MoE-4D over five 1T training runs

Figure 6. Coefficient of variation (std/mean) in runs. X-MoE has high variability (CV 11.8%). X-MoE-4D remains stable (CV = 0.4%).

Integration: X-MoE-4D is implemented on top of PyTorch and DeepSpeed, extending X-MoE with 1F1B pipeline parallelism and the skew-aware planner. The code is in the X-MoE repository.

BibTeX

@inproceedings{wang2026xmoe4d,
title={X-MoE-4D: Breaking the Scaling Barrier for Trillion Parameter MoEs on Hierarchical HPC Systems},
author={Zixian Wang and Liheng Jing and Emily Herron and Sajal Dash and Feiyi Wang and Minjia Zhang},
booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '26)},
year={2026},
address={Chicago, IL, USA},
publisher={IEEE},
}