Research and application of big data platform technology based on multi-centre collaborative computing

RUAN Yilong; XU Xueling; FA Hu; DONG Silun; JIANG Lei; YANG Lei; YAN Yuanyuan

doi:10.11959/j.issn.1000-0801.2024152

您当前的位置：

首页 >

文章列表页 >

Research and application of big data platform technology based on multi-centre collaborative computing

Engineering and Application | 更新时间：2024-08-05

- Research and application of big data platform technology based on multi-centre collaborative computing
- Telecommunications Science Vol. 40, Issue 5, Pages: 141-151(2024)
- 作者机构：
  
  中电信人工智能科技（北京）有限公司，北京 100010
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2024152
  CLC： TP391
- Received：18 January 2024，
  
  Revised：15 May 2024，
  
  Published：20 May 2024
- 稿件说明：
移动端阅览
阮宜龙,徐雪灵,法虎等.基于多中心集群协同计算的大数据平台技术研究与应用[J].电信科学,2024,40(05):141-151.

RUAN Yilong,XU Xueling,FA Hu,et al.Research and application of big data platform technology based on multi-centre collaborative computing[J].Telecommunications Science,2024,40(05):141-151.
阮宜龙,徐雪灵,法虎等.基于多中心集群协同计算的大数据平台技术研究与应用[J].电信科学,2024,40(05):141-151. DOI： 10.11959/j.issn.1000-0801.2024152.

RUAN Yilong,XU Xueling,FA Hu,et al.Research and application of big data platform technology based on multi-centre collaborative computing[J].Telecommunications Science,2024,40(05):141-151. DOI： 10.11959/j.issn.1000-0801.2024152.

摘要

中国电信面向横跨多个地域、拥有众多集群的大型政企机构，推出可以高效协同各类资源的广域大数据架构体系——云边智算大数据平台。该平台从集群维度对数据分区进行逻辑抽象，将独立分散的数据集整合为一个“虚拟数据集”，实现了一对多的数据集映射管理。同时，该平台的计算负载数据集具有泛化特征，能够灵活应对不同场景下的数据处理需求。另外，该平台以关系表达式为中间表示，支持多种计算引擎和调度系统，能够在复杂的大型数据处理高容错场景中高效地完成批处理任务负载。目前，云边智算大数据平台已在多种应用场景中落地，平台在5G Core能力调度子系统（5GC）多中心大数据作业开发、运营方面提效17%，且已实现8省前置大数据集群共计42 PB存储、84 TB内存、24 984 VCore计算资源的协同调度，日均完成80 308次前置-核心两级任务调度。

Abstract

China Telecom has launched a high-efficient and collaborative wide-area big data architecture system

the cloud edge computing big data platform

for large-scale governmental and enterprise organizations spanning multiple geographies and clusters. The platform logically abstracts data partitions through the cluster dimension

integrates multiple independent datasets into a "virtual dataset"

and achieves many-to-one dataset mapping management. At the same time

the computing load dataset of the platform has generalized characteristics

which can flexibly cope with the data processing requirements in different scenarios. In addition

the platform also supports a variety of computing engines and scheduling systems using relational expressions as intermediate representations to achieve batch tasks for large-scale

complex data processing in highly fault-tolerant scenarios. At present

the cloud edge computing big data platform has been applied in a variety of application scenes. The platform has improved efficiency by 17% in 5G Core capacity scheduling subsystem (5GC) multi-centre big data job development and operation

and has achieved the collaborative scheduling of a total of 42 PB of storage

84 TB of memory

and 24 984 VCore computing resources

with a daily average of 80 308 times of task scheduling between the front cluster and the core cluster.

关键词

Keywords

references

百度智能云 . 深入理解数据存储架构设计:从集中式到分布式存储 [EB ] . ( 2024-02-18 ).

Baidu Intelligent Cloud . Deeper understanding of data storage architecture design: from centralized to distributed storage [EB ] . ( 2024-02-18 ).

YI X , LIU F , LIU J , et al . Building a network highway for big data: architecture and challenges [J ] . IEEE Network , 2014 , 28 ( 4 ): 5 - 13 .

BRISBANE S . Decentralising Big Data Processing [D ] . Sydney : The University of New South Wales , 2016 .

JOHN F , GOPINATH S , SHERLY E . A decentralised framework for efficient storage and processing of big data using HDFS and IPFS [J ] . International Journal of Humanitarian Technology , 2020 , 1 ( 2 ): 131 - 143 .

朱姝 . 边缘云网下的新业务探索与应用 [J ] . 电信科学 , 2023 , 39 ( Z1 ): 55 - 61 .

ZHU S . Exploration and application of new business under edge cloud network [J ] . Telecommunications Science , 2023 , 39 ( Z1 ): 55 - 61 .

IDC中国 . 2021 H1大数据平台市场份额报告 [EB ] . ( 2021-12-27 ).

IDC China . 2021 H1 Big data platform market share report [EB ] . ( 2021-12-27 ).

IDC中国 . IDC:把握机遇,穿越周期——2022年中国大数据平台市场报告发布 [EB ] . ( 2023-07-25 ).

IDC China . IDC: seize the opportunity, traverse the cycle- China big data platform market report 2022 released [EB ] . ( 2023-07-25 ).

LI P , GUO S , YU S , et al . Cross-cloud MapReduce for big data [J ] . IEEE Transactions on Cloud Computing , 2020 , 8 ( 2 ): 375 - 386 .

FUSCO G , AVERSANO L . An approach for semantic integration of heterogeneous data sources [J ] . PeerJ Computer Science , 2020 , 6 : e254 .

YANG J , LEE T Y , LEE W T , et al . A design and application of municipal service platform based on cloud-edge collaboration for smart cities [J ] . Sensors , 2022 , 22 ( 22 ): 8784 .

SAMWEL B , CIESLEWICZ J , HANDY B , et al . F1 query: declarative querying at scale [J ] . Proceedings of the VLDB Endowment , 2018 , 11 ( 12 ): 1835 - 1848 .

LIU X , YU Y , YU M , et al . An edge-cloud collaborative computing system for real-time internet-of-things applications [C ] // Proceedings of International Conference on Computer Science and Education . Singapore : Springer Nature Singapore , 2022 : 652 - 663 .

PAN A , RAPOSO J , ÁLVAREZ M , et al . The Denodo data integration platform [C ] // Proceedings of the 28th International Conference on Very Large Data Bases . New York : ACM Press , 2002 : 986 - 989 .

HECHLER E , WEIHRAUCH M , WU Y . Data fabric architecture patterns [M ] // Data Fabric and Data Mesh Approaches with AI: A Guide to AI-based Data Cataloging, Governance, Integration, Orchestration, and Consumption . Berkeley, CA : Apress , 2023 : 231 - 255 .

LYU S , DAI X , MA Z , et al . A heterogeneous cloud-edge collaborative computing architecture with affinity-based workflow scheduling and resource allocation for internet-of-things applications [J ] . Mobile Networks and Applications , 2023 : 1 - 17 .

ISLAM A , DEBNATH A , GHOSE M , et al . A survey on task offloading in multi-access edge computing [J ] . Journal of Systems Architecture , 2021 , 118 : 102225 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Adaptive distributed cloud edge collaborative load control strategy for load management

An industrial internet business management platform that supports the cloud-native

KubeTelecom: a multi-cloud multi-container cluster management and operation engine for 5G network slicing

Application of 5G in railway smart stations

Analysis and development suggestions of operators in the field of industrial internet identification analysis system

Related Author

ZHANG Xirun

YUE Liang

DU Lishi

YU Long

JIN Li

LI Siwei

Gang AN

Guixiang ZOU

Related Institution

Key Laboratory of Smart Grid of Ministry of Education, Tianjin University

Beijing FibrLINK Communications Co., Ltd.

Research Institute of China United Network Communications Co., Ltd.

Unicom Digital Technology Co., Ltd.

Gansu Branch of China United Network Communications Co., Ltd.

⁰