Hadoop bottleneck detection algorithm based on information gain

Zaole TAN; Zhifeng HAO; Ruichu CAI; Xiaojun XIAO; Yu LU

doi:10.11959/j.issn.1000-0801.2016203

您当前的位置：

首页 >

文章列表页 >

Hadoop bottleneck detection algorithm based on information gain

Viewpoint aggregation | 更新时间：2024-06-05

- Hadoop bottleneck detection algorithm based on information gain
- Telecommunications Science Vol. 32, Issue 7, Pages: 115-120(2016)
- 作者机构：
  
  1. 广东工业大学计算机学院，广东广州510006
  2. 广州优亿信息科技有限公司，广东广州510630
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2016203
  CLC： TP391
- Published Online：2016-07，
  
  Published：15 July 2016
- 稿件说明：
移动端阅览
Zaole TAN, Zhifeng HAO, Ruichu CAI, et al. Hadoop bottleneck detection algorithm based on information gain[J]. Telecommunications science, 2016, 32(7): 115-120.
DOI：

Zaole TAN, Zhifeng HAO, Ruichu CAI, et al. Hadoop bottleneck detection algorithm based on information gain[J]. Telecommunications science, 2016, 32(7): 115-120. DOI： 10.11959/j.issn.1000-0801.2016203.

摘要

当今，Hadoop已经成为了大数据存储和大数据挖掘的主要平台。虽然Hadoop平台通过分布式的机器集群来实现高性能的并行计算，但由于其由廉价主机组成，故当集群负载增大时，便不可避免地在某机器上出现瓶颈。针对此问题，提出一种基于信息增益的瓶颈检测算法，该算法通过计算各个资源的信息增益来检测集群的瓶颈资源。实验证明了该瓶颈检测算法具有可行性。

Abstract

Hadoop has become a major platform for big data storage and large data mining nowadays.Although Hadoop platform achieves high performance parallel computing through a distributed cluster of machines

the bottlenecks will inevitably appear on a machine when cluster load increases

because the cluster is composed of inexpensive host.Aiming at this problem

a bottleneck detection algorithms based on information gain was proposed.The algorithm detected cluster's bottlenecks resource by computing the information gain of each resource.The experiments show that the bottleneck detection algorithm is feasible.

关键词

Keywords

references

中国互联网络信息中心第37次中国互联网络发展状况统计报告 [EB/OL ] . [ 2016 - 02 - 10 ] . http://www.cnnic.cn/hlwfzyj/hlwxzbg/hlwtjbg/201601/t20160122_53271.htm http://www.cnnic.cn/hlwfzyj/hlwxzbg/hlwtjbg/201601/t20160122_53271.htm .

China Internet Network Information Center The 37th China internet network development state statistic report [EB/OL ] . [ 2016 - 02 - 10 ] . http://www.cnnic.cn/hlwfzyj/hlwxzbg/hlwtjbg/201601/t20160122_53271.htm http://www.cnnic.cn/hlwfzyj/hlwxzbg/hlwtjbg/201601/t20160122_53271.htm .

张呈 . Hadoop集群下海量小文件优化处理 [D ] ．武汉：武汉理工大学， 2014 .

ZHANG C . Mass small files to optimize processing under the Hadoop cluster [D ] . Wuhan: Wuhan University of Technology , 2014 .

唐霞 . Hadoop调度器优化及其在舆情分析中的应用 [D ] ．北京：北京化工大学， 2015 .

TANG X . Hadoop scheduler optimization and its application in public opinion analysis [D ] . Beijing:Beijing University of Chemical Industry , 2015 .

曾婉琳，陈兴蜀，罗永刚 . Hadoop节点资源参数优化策略 [J ] . 计算机工程， 2016 ( 1 ): 1 - 6 .

ZENG W L , CHEN X S , LUO Y G . Hadoop node resource parameter optimization strategy [J ] . Computer Engineering , 2016 ( 1 ): 1 - 6 .

董新华，李瑞轩，周湾湾，等 . Hadoop系统性能优化与功能增强综述 [J ] . 计算机研究与发展， 2013 , 50 ( z2 ): 1 - 15 .

DONG X H , LI R X , ZHOU W W , et al . Hadoop system performance optimization and function enhanced review [J ] . Journal of Computer Research and Development , 2013 , 50 ( z2 ): 1 - 15 .

李怿铭 . 基于MapReduce性能优化的研究 [D ] ．上海：上海师范大学， 2015 .

LI Y M . Based on graphs performance optimization research [D ] . Shanghai: Shanghai Normal University , 2015 .

DA V , PRADHAN P , DAN R . Provisioning servers in the application tier for E-commerce systems [J ] . ACM Transactions on Internet Technology , 2007 , 7 ( 1 ): 57 - 66 .

朱显杰 . 大规模复杂系统瓶颈检测和性能预测方法的研究 [D ] ．杭州：浙江大学， 2010 .

ZHU X J . Large-scale complicated system bottleneck detection and performance prediction method research [D ] . Hangzhou:Zhejiang University , 2010 .

GHEMAWAT S , GOBIOFF H , LEUNG S T . The Google file system [J ] . ACM Sigops Operating Systems Review , 2003 , 37 ( 5 ): 29 - 43 .

DEAN J , GHEMAWAT S . MapReduce: simplified data processing on large clusters [J ] . Communications of the ACM , 2008 , 51 ( 1 ): 107 - 113 .

Apache Software Foundation Eagle: secure Hadoop in real time [EB/OL ] . [ 2016 - 02 - 11 ] . http://eagle.apache.org/ http://eagle.apache.org/ .

王小巍，蒋玉明 . 决策树ID3算法的分析与改进 [J ] . 计算机工程与设计， 2011 , 32 ( 9 ): 3069 - 3072 .

WANG X W , JIANG Y M . Analysis and improvement of the decision tree ID3 algorithm [J ] . Computer Engineering and Design , 2011 , 32 ( 9 ): 3069 - 3072 .

Views

645

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Applications of a big data platform based on PaaS

Research and implementation on acquisition scheme of telecom big data based on Hadoop

Practice and thinking on the transition of telecom operator analysis support system in big data era

An associated perception import method for big data

Discussion on Integration of Data Warehouse and Big Data

Related Author

Haiyong XU

Yan HUANG

Baoyou WANG

Jing QIAN

Shijin YUAN

Nanhai LIU

Lei LEI

Rui WANG

Related Institution

School of Software Engineering,Tongji University

Shanghai Branch of China United Network Communication Co.,Ltd.

Nanjing University of Posts and Telecommunications

Jiyuan Branch of China Telecom Co.，Ltd.

DingLi Communications Co.,Ltd.

⁰