Overview
A mixture-of-experts model sends each token to a few specialized feed-forward networks. On a server where all eight GPUs share the computation, the software must also decide how to arrange those tokens in memory. This paper times several arrangements and finds a crossover between smaller and larger token batches.
Key Findings
Eight tokens and sixteen thousand tokens pose different scheduling problems
A mixture-of-experts layer sends each token to several specialized feed-forward networks. A long prompt may supply many tokens in a batch. A generation step can supply only a few. The arithmetic follows the same broad pattern in either case. Before and between the matrix multiplications, the implementation still has to get the activations to the expert kernels in a layout they can read efficiently. This workshop paper times that choice on one server.
Move the data first, or change its order while doing useful work
The baseline is expert-major. It groups the routed tokens by expert before the first projection so the expert kernels receive contiguous data. That arrangement helps the calculation and requires a separate pass over the activations. A token-major implementation keeps the input in token order and avoids the up-front rearrangement. A third layout starts with token-major input and writes the intermediate results in expert order during the up projection. The next projection can then consume expert-major data using a persistent down kernel.
At small token counts, a separate rearrangement can dominate a short computation. The hybrid layout has more work to set up than the pure token-major path, and it loses in the smallest measured cases. With more tokens, organizing the intermediate results for the down projection can pay for itself.
A stage-by-stage clock explains the crossover
The paper's Table 1 measures BF16 on a single node with eight H200 GPUs connected by NVLink. It uses tensor parallelism across all eight, a hidden size of 4,096, a per-rank intermediate size of 256, 128 experts, and eight selected experts per token. Assignments are randomized. The values are means of 50 timed iterations after ten warmups.
| Tokens | Expert-major baseline | Token-major up/down | Hybrid with persistent down |
|---|---|---|---|
| 8 | 0.484 ms | 0.249 ms | 0.335 ms |
| 128 | 0.535 ms | 0.352 ms | 0.408 ms |
| 4,096 | 1.406 ms | 0.810 ms | 0.785 ms |
| 16,384 | 4.319 ms | 2.486 ms | 2.190 ms |
At 16,384 tokens, the baseline spends 2.117 ms on the pre-permutation alone. The hybrid’s advantage in that row shows what happens when a layout change removes a separate memory pass and gives the next stage an efficient input. The two nonbaseline paths cross somewhere between the measured small and large settings. The table does not locate a universal cutoff.
The kernel still has to meet the rest of the server
These totals include the local combination of expert outputs and leave out the final all-reduce, the cross-GPU operation that combines partial results. Table 2 times the reduction stages separately. At tensor-parallel degree eight the fused path took 0.049 ms with 128 tokens, compared with 0.077 ms for sequential combine and NCCL all-reduce. With 16,384 tokens, the same comparison flips to 2.101 ms and 0.858 ms respectively. That second crossover shows why layout and communication require separate choices.
A deployed service also has to deal with attention, request scheduling and uneven expert loads. Real token routing can concentrate work differently from random assignments. The paper’s result gives a concrete reason to choose a layout by the shape of the work, and the measured times describe the particular system and module the authors tested.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Why does the faster layout change?
A: For small batches, avoiding rearrangement pays off. With many tokens, a persistent down-projection kernel can benefit from expert-major data written directly by the earlier projection.
Q: Should another server use the same cutoff?
A: The exact cutoff can depend on the GPU, matrix shapes, routing, and batch construction. The measured token counts give test points for a calibration sweep.
Q: Will a faster kernel make the whole request equally faster?
A: The published crossover compares feed-forward stages on one eight-H200 server with random token routing. It excludes the final all-reduce from those totals. An end-to-end request also pays for communication, switching layouts, and the rest of inference. A replay of real batches can measure which of those costs changes the crossover.
Future Ideas
Profile both layouts in shadow mode on a mixed prompt and decoding workload. Fit a small dispatch rule from batch size and expert imbalance, and compare it with both fixed policies. Check output equivalence and report time to first token, time per generated token, and tail latency under sustained load.
Count switching cost and record expert imbalance alongside the timing of the full request path.