OPENAI . Introducing ChatGPT [EB ] . 2023 .

OPENAI . Introducing OpenAI o1 [EB ] . 2024 .

郭亮 , 王少鹏 , 权伟 , 等 . 面向大模型的智算网络发展研究 [J ] . 电信科学 , 2024 , 40 ( 6 ): 137 - 145 .

GUO L , WANG S P , QUAN W , et al . Research on the development of intelligent computing network for large models [J ] . Telecommunications Science , 2024 , 40 ( 6 ): 137 - 145 .

WEINGRAM A , LI Y K , QI H , et al . xCCL: a survey of industry-led collective communication libraries for deep learning [J ] . Journal of Computer Science and Technology , 2023 , 38 ( 1 ): 166 - 195 .

KIM J , KIM H . Router microarchitecture and scalability of ring topology in on-chip networks [C ] // Proceedings of the 2009 2nd International Workshop on Network on Chip Architectures . Piscataway : IEEE Press , 2009 : 5 - 10 .

BOUKNIGHT W J , DENENBERG S A , MCINTYRE D E , et al . The illiac IV system [J ] . Proceedings of the IEEE , 1972 , 60 ( 4 ): 369 - 388 .

AL-FARES M , LOUKISSAS A , VAHDAT A . A scalable, commodity data center network architecture [J ] . ACM SIGCOMM Computer Communication Review , 2008 , 38 ( 4 ): 63 - 74 .

AL-DUBAI A Y , OULD-KHAOUA M . A new scalable broadcast algorithm for multiport meshes with minimum communication steps [J ] . Microprocessors and Microsystems , 2003 , 27 ( 3 ): 101 - 113 .

THAKUR R , RABENSEIFNER R , GROPP W . Optimization of collective communication operations in MPICH [J ] . International Journal of High Performance Computing Applications , 2005 , 19 ( 1 ): 49 - 66 .

Pjesivac-Grbovic J . Towards automatic and adaptive optimizations of MPI collective operations [J ] . International Journal of High Performance Computing Applications , 2007 , 15 ( 3 ): 45 - 60 .

HUSE L P . Collective communication on dedicated clusters of workstations [M ] // Recent Advances in Parallel Virtual Machine and Message Passing Interface . Berlin, Heidelberg : Springer Berlin Heidelberg , 1999 : 469 - 476 .

BRUCK J , HO C T , KIPNIS S , et al . Efficient algorithms for all-to-all communications in multi-port message-passing systems [C ] // Proceedings of the Sixth Annual ACM Symposium on Parallel Algorithms and Architectures . New York : ACM Press , 1994 : 298 - 309 .

BROWN T , MANN B , RYDER N , et al . Language models are few-shot learners [J ] . Advances in neural information processing systems , 2020 , 33 : 1877 - 1901

HUANG Y , CHENG Y , BAPNA A , et al . Gpipe: Efficient training of giant neural networks using pipeline parallelism [J ] . Advances in Neural Information Processing Systems , 2019 , 32 .

WANG G , QIN H , JACOBS S A , et al . ZeRO++: Extremely efficient collective communication for large model training [C ] // The Twelfth International Conference on Learning Representations(ICLR) , Vienna, Austria : ICLR , 2024 .

RAJBHANDARI S , RASLEY J , RUWASE O , et al . ZeRO: memory optimizations toward training trillion parameter models [C ] // Proceedings of the SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE Press , 2020 : 1 - 16 .

LIAN X R , HUANG Y J , LI Y C , et al . Asynchronous parallel stochastic gradient for nonconvex optimization [C ] // Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2 . New York : ACM Press , 2015 : 2737 - 2745 .

GRAHAM R L , SHIPMAN G M , BARRETT B W , et al . Open MPI: a high-performance, heterogeneous MPI [C ] // Proceedings of the 2006 IEEE International Conference on Cluster Computing . Piscataway : IEEE Press , 2006 : 1 - 9 .