Huawei Ascend Adapts Kimi K3.1 Pro for LoRA Fine-Tuning on a 128-Card Atlas 950 SuperPoD

Huawei's Ascend computing team says it has adapted Moonshot AI's Kimi K3.1 Pro for LoRA fine-tuning and validated training on a 128-card Atlas 950 SuperPoD, according to Guandian and a Chinese technical write-up published on November 6 and 7. The work concerns training, a separate step from the day-zero inference support Ascend announced after Moonshot released the model's weights on November 4.

Kimi K3.1 Pro is an open-weight, natively multimodal mixture-of-experts model with about 2.8 trillion total parameters. The reports describe 896 routed experts with 16 activated per token, a hybrid attention design in which roughly three quarters of attention blocks use KDA linear attention alongside gated sparse attention, and a context window of up to one million tokens.

That scale creates memory pressure: the BF16 weights alone occupy roughly 5.6TB, before gradients, optimizer states and long-sequence activations. Ascend says it combined FSDP2 sharding of parameters, gradients and optimizer states with expert parallelism, distributing experts across accelerators. Validation used an FSDP2 plus EP64 configuration on the 128-card system.

On the operator side, the team says it built large fused forward and backward kernels for KDA attention and used fused AscendC operators for sparse attention, claiming more than twice the performance of the earlier small-operator approach. New residual structures were brought up through the Triton Ascend framework and the BiSheng compiler, and MoE routing steps such as token permutation and grouped matrix multiplication were fused and overlapped with communication.

A selective caching mechanism called Swap Core keeps the outputs of chosen matrix-multiplication operators in a tiered store spanning HBM and DDR memory. Ascend says this raised average training performance by 15% in the current test configuration; it did not publish absolute throughput, and the figure has not been independently verified. Chunked loss computation along the sequence and vocabulary dimensions was used to contain memory for the large vocabulary.

The team also reported loss curves from the 128-card runs. Training from random initialization started near 13.5 and fell to about 1.7 after more than 400 steps, while LoRA fine-tuning from the open weights started near 1.85 and converged to roughly 0.27 after more than 500 steps. Ascend says an agent-driven development workflow allowed the model to be brought up within days.

SEE ALSO: Huawei Ascend Adds Day-Zero Support for Kimi K3.1 Pro Across Atlas A3 and Ascend 950 SuperPoDs