Research on multi-heterogeneous hybrid training system for AI computing power scenarios

LI Panpan; NIU Hongweihua; ZHAO Wanlong; MA Huawei; WANG Yanhui; JIANG Wei; ZHANG Wenxin; LU Yiming; ZHAO Feng

doi:10.11959/j.issn.1000-0801.2025164

您当前的位置：

首页 >

文章列表页 >

Research on multi-heterogeneous hybrid training system for AI computing power scenarios

Topic | 更新时间：2025-08-07

- Research on multi-heterogeneous hybrid training system for AI computing power scenarios
- Telecommunications Science Vol. 41, Issue 7, Pages: 133-144(2025)
- 作者机构：
  
  1.中移（苏州）软件技术有限公司，江苏苏州 215123
  2.中国移动通信集团设计院有限公司，北京 100080
  3.中国移动通信集团设计院有限公司安徽分公司，安徽合肥 230041
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2025164
  CLC： TP393
- Received：21 March 2025，
  
  Revised：2025-07-02，
  
  Accepted：12 June 2025，
  
  Published：20 July 2025
- 稿件说明：
移动端阅览
李攀攀,牛红韦华,赵万龙等.面向AI算力场景的多元异构混合训练系统研究[J].电信科学,2025,41(07):133-144.

LI Panpan,NIU Hongweihua,ZHAO Wanlong,et al.Research on multi-heterogeneous hybrid training system for AI computing power scenarios[J].Telecommunications Science,2025,41(07):133-144.
李攀攀,牛红韦华,赵万龙等.面向AI算力场景的多元异构混合训练系统研究[J].电信科学,2025,41(07):133-144. DOI： 10.11959/j.issn.1000-0801.2025164.

LI Panpan,NIU Hongweihua,ZHAO Wanlong,et al.Research on multi-heterogeneous hybrid training system for AI computing power scenarios[J].Telecommunications Science,2025,41(07):133-144. DOI： 10.11959/j.issn.1000-0801.2025164.

摘要

大语言模型训练是人工智能（artificial intelligence，AI）发展的核心场景，在算力多元化和异构化趋势下，跨生态异构算力协同能力将成为十万卡级训练的关键支撑。基于此背景，设计了一套异构AI算力混合训练系统，该系统能够主动检测、适配异构AI芯片，实现异构算力间的集合通信。基于该原型系统，在一个由3种异构算力组成的RoCEv2网络互通集群实现了多种异构算力组合的混训。在异构流水线并行（pipeline parallelism，PP）混训场景下，英伟达与壁仞的最优混训效率达到99.77%，英伟达、天数智芯、壁仞的最优混训效率可达99.03%。在异构数据并行（data parallelism，DP）混训场景下，英伟达与壁仞的最优混训效率达到92.88%。

Abstract

Large language model training is a pivotal scenario in AI development. Under the trend of diversified and heterogeneous computing power

the cross-ecosystem heterogeneous computing power collaboration capability will become the key support for training at the hundred-thousand-card-scale. Based on this background

a heterogeneous AI computing power mixed training system was designed

which could automatically detect and adapt to heterogeneous AI chips

enabling collective communication among heterogeneous computing powers. Based on the prototype system

heterogeneous training was implemented using three types of AI chips in a RoCEv2-interoperable cluster. In the heterogeneous pipeline parallelism (PP) training scenario

peak training efficiency reached 99.77% using NVIDIA and Biren GPU

and 99.03% using NVIDIA

Iluvatar

and Biren GPU. For heterogeneous data parallelism (DP) training

the optimal mixed training efficiency between NVIDIA and Biren GPU reached 92.88%.

关键词

Keywords

references

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J ] . Advances in Neural Information Processing Systems , 2017 : 5998 - 6008 .

KAPLAN J , MCCANDLISH S , HENIGHAN T , et al . Scaling laws for neural language models [J ] . arXiv preprint , 2020 : 2001 .08361.

WEI J , TAY Y , BOMMASANI R , et al . Emergent abilities of large language models [J ] . arXiv preprint , 2022 : 2206 .07682.

GRATTAFIORI A , DUBEY A , JAUHRI A , et al . The Llama 3 herd of models [J ] . arXiv preprint , 2024 : 2407 .21783,.

BARKER B . Message passing interface (MPI) [C ] // Workshop: high performance computing on stampede . Houston : Cornell University Publisher , 2015 : 262 .

HARLAP A , NARAYANAN D , PHANISHAYEE A , et al . PipeDream: fast and efficient pipeline parallel DNN training [J ] . arXiv preprint , 2018 : 1806 .03377.

JIANG Y M , ZHU Y B , LAN C , et al . A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters [C ] // Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2020 : 471 - 487 .

PARK J H , YUN G , YI C M , et al . HetPipe: enabling large DNN training on (whimpy) heterogeneous GPU clusters through integration of pipelined model parallelism and data parallelism [J ] . arXiv preprint , 2020 : 2005 .14038.

JIA X Y , JIANG L , WANG A , et al . Whale: efficient giant model training over heterogeneous GPUs [J ] . arXiv preprint , 2020 : 2011 .09208.

XU S , HUANG Z X , ZENG Y , et al . HETHUB: a distributed training system with heterogeneous cluster for large-scale models [J ] . arXiv preprint , 2024 : 2405 .16256.

WU Q , WANG W H , FAN P Y , et al . Cooperative edge caching based on elastic federated and multi-agent deep reinforcement learning in next-generation networks [J ] . IEEE Transactions on Network and Service Management , 2024 , 21 ( 4 ): 4179 - 4196 .

Views

741

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on customer escalated complaints prediction based on LLM-RFE-XGBoost approach

Automated extraction and evaluation of business process entities and relations via pre-trained models

Research and application of intelligent operation and maintenance human-computer interaction system based on large language models for communication networks

Hyperscale intelligent computing cluster networks for domestic computing power: critical challenges, technical pathways, and future trends

Large language model for cloud-network configuration audit and its application in IP network

Related Author

Jiang Jun

Xu Yu

Liu Fang

Wang Qian

Qiao Yaqian

Li Shan

Wan Xiang

Liu Zesan

Related Institution

China Mobile Online Service Co., Ltd., AI Capability Center

China Mobile Online Service Co., Ltd., Shandong Branch

Electric Power Science Research Institute, State Grid Liaoning Electric Power Co., Ltd.

State Grid Information and Communication Industry Group Corporation

Aostar Information Technologies Co., Ltd.

⁰