Others & Etc. / ICML Workshop 2026 (AdaptFM)

The Fastest Expert Layout Depends on How Many Tokens Arrive

Experiments on one eight-GPU server show why an expert layout suited to generating a few tokens can lose when the model processes a long prompt.

Layout and Fusion Trade-offs for Mixture-of-Experts Inference under Single-Node Tensor Parallelism

June Yong Yang, Inhyuk Cho, Taehyeon Kim, Yu Jin Kim, Moontae Lee

Overview

A mixture-of-experts model sends each token to a few specialized feed-forward networks. On a server where all eight GPUs share the computation, the software must also decide how to arrange those tokens in memory. This paper times several arrangements and finds a crossover between smaller and larger token batches.

Key Findings

Eight tokens and sixteen thousand tokens pose different scheduling problems

A mixture-of-experts layer sends each token to several specialized feed-forward networks. A long prompt may supply many tokens in a batch. A generation step can supply only a few. The arithmetic follows the same broad pattern in either case. Before and between the matrix multiplications, the implementation still has to get the activations to the expert kernels in a layout they can read efficiently. This workshop paper times that choice on one server.

Move the data first, or change its order while doing useful work

The figure follows token and expert layouts through the up projection, reduction, and down projection.
The figure follows token and expert layouts through the up projection, reduction, and down projection.

The baseline is expert-major. It groups the routed tokens by expert before the first projection so the expert kernels receive contiguous data. That arrangement helps the calculation and requires a separate pass over the activations. A token-major implementation keeps the input in token order and avoids the up-front rearrangement. A third layout starts with token-major input and writes the intermediate results in expert order during the up projection. The next projection can then consume expert-major data using a persistent down kernel.

At small token counts, a separate rearrangement can dominate a short computation. The hybrid layout has more work to set up than the pure token-major path, and it loses in the smallest measured cases. With more tokens, organizing the intermediate results for the down projection can pay for itself.

A stage-by-stage clock explains the crossover

The paper's Table 1 measures BF16 on a single node with eight H200 GPUs connected by NVLink. It uses tensor parallelism across all eight, a hidden size of 4,096, a per-rank intermediate size of 256, 128 experts, and eight selected experts per token. Assignments are randomized. The values are means of 50 timed iterations after ten warmups.

TokensExpert-major baselineToken-major up/downHybrid with persistent down
80.484 ms0.249 ms0.335 ms
1280.535 ms0.352 ms0.408 ms
4,0961.406 ms0.810 ms0.785 ms
16,3844.319 ms2.486 ms2.190 ms

At 16,384 tokens, the baseline spends 2.117 ms on the pre-permutation alone. The hybrid’s advantage in that row shows what happens when a layout change removes a separate memory pass and gives the next stage an efficient input. The two nonbaseline paths cross somewhere between the measured small and large settings. The table does not locate a universal cutoff.

The kernel still has to meet the rest of the server

These totals include the local combination of expert outputs and leave out the final all-reduce, the cross-GPU operation that combines partial results. Table 2 times the reduction stages separately. At tensor-parallel degree eight the fused path took 0.049 ms with 128 tokens, compared with 0.077 ms for sequential combine and NCCL all-reduce. With 16,384 tokens, the same comparison flips to 2.101 ms and 0.858 ms respectively. That second crossover shows why layout and communication require separate choices.

A deployed service also has to deal with attention, request scheduling and uneven expert loads. Real token routing can concentrate work differently from random assignments. The paper’s result gives a concrete reason to choose a layout by the shape of the work, and the measured times describe the particular system and module the authors tested.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why does the faster layout change?

A: For small batches, avoiding rearrangement pays off. With many tokens, a persistent down-projection kernel can benefit from expert-major data written directly by the earlier projection.

Q: Should another server use the same cutoff?

A: The exact cutoff can depend on the GPU, matrix shapes, routing, and batch construction. The measured token counts give test points for a calibration sweep.

Q: Will a faster kernel make the whole request equally faster?

A: The published crossover compares feed-forward stages on one eight-H200 server with random token routing. It excludes the final all-reduce from those totals. An end-to-end request also pays for communication, switching layouts, and the rest of inference. A replay of real batches can measure which of those costs changes the crossover.

Future Ideas

Profile both layouts in shadow mode on a mixed prompt and decoding workload. Fit a small dispatch rule from batch size and expert imbalance, and compare it with both fixed policies. Check output equivalence and report time to first token, time per generated token, and tail latency under sustained load.

Count switching cost and record expert imbalance alongside the timing of the full request path.

Sources