Practice and application of distributed data quality management system in power enterprise

Yuanning LI; Sen LIU; Shijun ZHANG; Feng CHEN; Zhiying WANG

doi:10.11959/j.issn.1000-0801.2016104

您当前的位置：

首页 >

文章列表页 >

Practice and application of distributed data quality management system in power enterprise

Electric power information column | 更新时间：2024-06-05

- Practice and application of distributed data quality management system in power enterprise
- Telecommunication Science Vol. 32, Issue 4, Pages: 169-174(2016)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2016104
  CLC： TP391
- Published Online：2016-04，
  
  Published：20 April 2016
- 稿件说明：
移动端阅览
Yuanning LI, Sen LIU, Shijun ZHANG, et al. Practice and application of distributed data quality management system in power enterprise[J]. Telecommunication science, 2016, 32(4): 169-174.
DOI：

Yuanning LI, Sen LIU, Shijun ZHANG, et al. Practice and application of distributed data quality management system in power enterprise[J]. Telecommunication science, 2016, 32(4): 169-174. DOI： 10.11959/j.issn.1000-0801.2016104.

摘要

随着企业信息化水平和企业精细化管理要求的不断提高，企业对数据管理的需求也随之增强，如何提高企业数据质量更是需要重点解决的问题。针对电力企业数据质量管理面临的挑战，创新提出了分布式数据质量管理解决方案。针对集中式数据质量系统的性能瓶颈，在研究数据质量系统特点并借鉴国内外对大数据的解决方案后，提出了基于Hadoop分布式处理框架的解决方案。利用Hadoop集群，可以把缺陷数据从Oracle中抽离，分散存储在集群里多台服务器上，以有效提高磁盘I/O性能和数据分析性能。

Abstract

As the improvement of the enterprise’s informationalization level and the increasing management requirement of enterprise refinement，the demand of data management of enterprise is becoming greater and greater，how to improve the data quality of the enterprise is the key problem needed to be solved. Aiming at the challenges of data quality management that the power enterprise faces，some solutions for distributed data quality management were proposed. After researching the system features of data quality，some foreign and domestic cases of big data were analyzed as reference，and a solution based on Hadoop distributed processing framework was given to solve the performance bottleneck of centralized data quality system. Hadoop clustering could dissociate defect data from Oracle and the data would be stored separately on multiple servers of the clustering，which could improve the I/O performance and data analysis performance of the magnetic disk effectively.

关键词

Keywords

references

田秀霞，周耀军 . 基于Hadoop架构的分布式计算和存储技术及其应用 [J ] . 上海电力学院学报， 2011 ， 27 （ 1 ）： 70 - 75 .

TIAN X X ， ZHOU Y J . The technology and application of distributed computing and storage based on Hadoop architecture [J ] . Journal of Shanghai University of Electric Power ， 2011 ， 27 （ 1 ）： 70 - 75 .

BIRMAN K P ， GANESH L ， RENESSE R . Running smart grid control software on cloud computing architectures [C ] // Workshop on Computational Needs for the Next Generation Electric Grid，April 19-20，2011，Cornell University，Ithaca.[S.l.：s.n.] ， 2011 ： 1 - 28 .

刘鹏 . 云计算 [M ] . 北京：电子工业出版社， 2010 .

Liu P . Cloud computing [M ] . Beijing ： Publishing House of Electronics Industry ， 2010 .

REESE G . Cloud application architectures：building applications and infrastructure in the cloud [M ] . New York ： OˊReilly Media ， 2009 .

辛军，陈康，郑纬民 . 虚拟化集群管理技术研究 [J ] . 计算机科学与探索， 2010 （ 4 ）： 325 - 327 .

XIN J ， CHEN K ， ZHENG W M . Studies on virtualization of cluster resource management technology [J ] . Journal of Frontiers of Computer Science and Technology ， 2010 （ 4 ）： 325 - 327 .

HDFS scalability with multiple NameSpaces [EB/OL ] . [2015-09-20 ] . http://issues.apache.org/jira/browse/HDFS-1052 http://issues.apache.org/jira/browse/HDFS-1052 .

WHITE T . Hadoop：the definitive gide [M ] . New York ： OˊReilly Media ， 2009 .

Hadoop apache project [EB/OL ] . [2015-09-20 ] . http://hadoop.apache.org. http://hadoop.apache.org. .

GHEMAWAT S ， GOBIOFF H ， LEUNG S T . The Google file system [C ] // SOSP，October 19-22，2003，Bolton Landing，New York，USA ， New York ： ACM Press ， 2003 .

陈远，罗琳 . 信息系统中的数据质量问题研究 [J ] . 中国图书馆学报， 2004 （ 1 ）： 48 - 50 .

CHEN Y ， LUO L . Research on data quality in information system [J ] . Journal of Library Science in China ， 2004 （ 1 ）： 48 - 50 .

胡金林，梅士员 . 基于元数据扩展的空间数据质量管理方法 [J ] . 现代测绘， 2004 ， 27 （ 3 ）： 21 - 24 .

HU J L ， MEI S Y . The extended metadata method of spatial data quality management [J ] . Modern Surveying and Mapping ， 2004 ， 27 （ 3 ）： 21 - 24 .

Views

1102

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Design of Telecom Operators' Network Data Sharing Platform Based on Hadoop+MPP Architecture

Big data classification method of non relational distributed submission information under differentiated requirements

Research on cloud native based distributed IoT operating system architecture

Construction of DDoS attacks malicious behavior knowledge base construction

Development trend and application analysis of blockchain with 5G

Related Author

Zhenyu Chen

Xinghui Yi

Huang Xin

Weiyu CHEN

Fei ZHANG

Jianfeng HE

Huaizhen SU

Lu HAN

Related Institution

Chongqing Branch of China Mobile Group Design Institute Co., Ltd.

China Mobile Group Guizhou Co., Ltd.

State Grid Lanzhou Siji Feitian Cloud Date Science Technology Co., Ltd.

State Grid Gansu Electric Power Company Dingxi Power Supply Company

State Grid Gansu Electric Power Company

⁰