浏览全部资源
扫码关注微信
基于多中心集群协同计算的大数据平台技术研究与应用
Research and application of big data platform technology based on multi-centre collaborative computing
- 电信科学 2024年40卷第5期 页码：141-151
- 作者机构：
中电信人工智能科技（北京）有限公司，北京 100010
- 作者简介：
[ "阮宜龙（1977- ），男，中电信人工智能科技（北京）有限公司大数据研发中心总经理、高级工程师，主要研究方向为大数据平台架构规划与建设运营。" ]
[ "徐雪灵（1997- ），女，中电信人工智能科技（北京）有限公司工程师，主要研究方向为大数据PaaS平台研究、存算分离、分布式缓存。" ]
[ "法虎（1998- ），男，中电信人工智能科技（北京）有限公司工程师，主要研究方向为多中心协同计算、基因大数据压缩算法。" ]
[ "董丝纶（1997- ），男，中电信人工智能科技（北京）有限公司工程师，主要研究方向为通用查询优化器。" ]
[ "姜磊（1989- ），男，中电信人工智能科技（北京）有限公司高级工程师，主要研究方向为数据虚拟化。" ]
[ "杨磊（1986- ），男，中电信人工智能科技（北京）有限公司高级工程师、PaaS研发总监，主要研究方向为多模态融合PaaS平台和多中心跨域协同技术。" ]
[ "燕媛媛（1990- ），女，中电信人工智能科技（北京）有限公司高级工程师，主要研究方向为大数据PaaS平台规划研究与管理。" ]
- 基金信息：
DOI：10.11959/j.issn.1000-0801.2024152
中图分类号： TP391收稿日期：2024-01-18，
修回日期：2024-05-15，
纸质出版日期：2024-05-20
- 稿件说明：
移动端阅览
阮宜龙,徐雪灵,法虎等.基于多中心集群协同计算的大数据平台技术研究与应用[J].电信科学,2024,40(05):141-151.
RUAN Yilong,XU Xueling,FA Hu,et al.Research and application of big data platform technology based on multi-centre collaborative computing[J].Telecommunications Science,2024,40(05):141-151.
阮宜龙,徐雪灵,法虎等.基于多中心集群协同计算的大数据平台技术研究与应用[J].电信科学,2024,40(05):141-151. DOI： 10.11959/j.issn.1000-0801.2024152.
RUAN Yilong,XU Xueling,FA Hu,et al.Research and application of big data platform technology based on multi-centre collaborative computing[J].Telecommunications Science,2024,40(05):141-151. DOI： 10.11959/j.issn.1000-0801.2024152.
摘要
中国电信面向横跨多个地域、拥有众多集群的大型政企机构，推出可以高效协同各类资源的广域大数据架构体系——云边智算大数据平台。该平台从集群维度对数据分区进行逻辑抽象，将独立分散的数据集整合为一个“虚拟数据集”，实现了一对多的数据集映射管理。同时，该平台的计算负载数据集具有泛化特征，能够灵活应对不同场景下的数据处理需求。另外，该平台以关系表达式为中间表示，支持多种计算引擎和调度系统，能够在复杂的大型数据处理高容错场景中高效地完成批处理任务负载。目前，云边智算大数据平台已在多种应用场景中落地，平台在5G Core能力调度子系统（5GC）多中心大数据作业开发、运营方面提效17%，且已实现8省前置大数据集群共计42 PB存储、84 TB内存、24 984 VCore计算资源的协同调度，日均完成80 308次前置-核心两级任务调度。
Abstract
China Telecom has launched a high-efficient and collaborative wide-area big data architecture system
the cloud edge computing big data platform
for large-scale governmental and enterprise organizations spanning multiple geographies and clusters. The platform logically abstracts data partitions through the cluster dimension
integrates multiple independent datasets into a "virtual dataset"
and achieves many-to-one dataset mapping management. At the same time
the computing load dataset of the platform has generalized characteristics
which can flexibly cope with the data processing requirements in different scenarios. In addition
the platform also supports a variety of computing engines and scheduling systems using relational expressions as intermediate representations to achieve batch tasks for large-scale
complex data processing in highly fault-tolerant scenarios. At present
the cloud edge computing big data platform has been applied in a variety of application scenes. The platform has improved efficiency by 17% in 5G Core capacity scheduling subsystem (5GC) multi-centre big data job development and operation
and has achieved the collaborative scheduling of a total of 42 PB of storage
84 TB of memory
and 24 984 VCore computing resources
with a daily average of 80 308 times of task scheduling between the front cluster and the core cluster.
关键词
Keywords
references
百度智能云 . 深入理解数据存储架构设计:从集中式到分布式存储 [EB ] . ( 2024-02-18 ).
Baidu Intelligent Cloud . Deeper understanding of data storage architecture design: from centralized to distributed storage [EB ] . ( 2024-02-18 ).
YI X , LIU F , LIU J , et al . Building a network highway for big data: architecture and challenges [J ] . IEEE Network , 2014 , 28 ( 4 ): 5 - 13 .
BRISBANE S . Decentralising Big Data Processing [D ] . Sydney : The University of New South Wales , 2016 .
JOHN F , GOPINATH S , SHERLY E . A decentralised framework for efficient storage and processing of big data using HDFS and IPFS [J ] . International Journal of Humanitarian Technology , 2020 , 1 ( 2 ): 131 - 143 .
朱姝 . 边缘云网下的新业务探索与应用 [J ] . 电信科学 , 2023 , 39 ( Z1 ): 55 - 61 .
ZHU S . Exploration and application of new business under edge cloud network [J ] . Telecommunications Science , 2023 , 39 ( Z1 ): 55 - 61 .
IDC中国 . 2021 H1大数据平台市场份额报告 [EB ] . ( 2021-12-27 ).
IDC China . 2021 H1 Big data platform market share report [EB ] . ( 2021-12-27 ).
IDC中国 . IDC:把握机遇,穿越周期——2022年中国大数据平台市场报告发布 [EB ] . ( 2023-07-25 ).
IDC China . IDC: seize the opportunity, traverse the cycle- China big data platform market report 2022 released [EB ] . ( 2023-07-25 ).
LI P , GUO S , YU S , et al . Cross-cloud MapReduce for big data [J ] . IEEE Transactions on Cloud Computing , 2020 , 8 ( 2 ): 375 - 386 .
FUSCO G , AVERSANO L . An approach for semantic integration of heterogeneous data sources [J ] . PeerJ Computer Science , 2020 , 6 : e254 .
YANG J , LEE T Y , LEE W T , et al . A design and application of municipal service platform based on cloud-edge collaboration for smart cities [J ] . Sensors , 2022 , 22 ( 22 ): 8784 .
SAMWEL B , CIESLEWICZ J , HANDY B , et al . F1 query: declarative querying at scale [J ] . Proceedings of the VLDB Endowment , 2018 , 11 ( 12 ): 1835 - 1848 .
LIU X , YU Y , YU M , et al . An edge-cloud collaborative computing system for real-time internet-of-things applications [C ] // Proceedings of International Conference on Computer Science and Education . Singapore : Springer Nature Singapore , 2022 : 652 - 663 .
PAN A , RAPOSO J , ÁLVAREZ M , et al . The Denodo data integration platform [C ] // Proceedings of the 28th International Conference on Very Large Data Bases . New York : ACM Press , 2002 : 986 - 989 .
HECHLER E , WEIHRAUCH M , WU Y . Data fabric architecture patterns [M ] // Data Fabric and Data Mesh Approaches with AI: A Guide to AI-based Data Cataloging, Governance, Integration, Orchestration, and Consumption . Berkeley, CA : Apress , 2023 : 231 - 255 .
LYU S , DAI X , MA Z , et al . A heterogeneous cloud-edge collaborative computing architecture with affinity-based workflow scheduling and resource allocation for internet-of-things applications [J ] . Mobile Networks and Applications , 2023 : 1 - 17 .
ISLAM A , DEBNATH A , GHOSE M , et al . A survey on task offloading in multi-access edge computing [J ] . Journal of Systems Architecture , 2021 , 118 : 102225 .
0
浏览量
9
下载量
0
CSCD
- 网络PDF下载
- WORD下载
- XML下载
- Meta-XML下载
- BibTeX下载
- Endnote下载
- RefMan 下载
- NoteExpress下载
- NoteFirst下载
关联资源
相关文章
相关作者
相关机构
- 技术支持由北京北大方正电子有限公司提供 京ICP备09064830号-19
京公网安备11010802024621 - 本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
- Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies
- 总访问量：70096 今日访问量：151