Comparison of Open-Source Distributed Computing Framework for Big Data

Ai Fang; Xiong Xu; Bing Liang; Yuzhong Zhang; Yiping Yang

doi:10.11959/j.issn.1000-0801.2015174

您当前的位置：

首页 >

文章列表页 >

Comparison of Open-Source Distributed Computing Framework for Big Data

Wide operating technology | 更新时间：2024-06-05

- Comparison of Open-Source Distributed Computing Framework for Big Data
- Telecommunications Science Vol. 31, Issue 7, Pages: 152-157(2015)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2015174
  CLC：
- Published Online：2015-07，
  
  Published：20 July 2015
- 稿件说明：
移动端阅览
Ai Fang, Xiong Xu, Bing Liang, et al. Comparison of Open-Source Distributed Computing Framework for Big Data[J]. Telecommunications science, 2015, 31(7): 152-157.
DOI：

Ai Fang, Xiong Xu, Bing Liang, et al. Comparison of Open-Source Distributed Computing Framework for Big Data[J]. Telecommunications science, 2015, 31(7): 152-157. DOI： 10.11959/j.issn.1000-0801.2015174.

摘要

结合电信增值业务领域中对大数据处理的实际需求，对现有主流的分布式大数据处理架构（Hive、Impala、Spark）的核心进行分析与实测，比较它们在大数据处理过程中的优劣及适用的场景，从而为大数据分析所面临的架构适用性选型提供参考。

Abstract

A comparison of three open source distributed computing frameworks for big data （Hive，Impala and Spark）was conducted.Tests were run to evaluate the performance aiming at real business demands.The cost of implementation to meet business requirements was also discussed.

关键词

Keywords

references

Hadoop . http://hadoop.apache.org/ http://hadoop.apache.org/ ， 2014

耿益锋，陈冠诚 . Impala：新一代开源大数据分析引擎 . 程序员， 2013 （ 8 ）

Geng Y F ， Chen G C . Impala：next-gen open-source big data analytic engine . Programmer ， 2013 （ 8 ）

Spark . http://www.cnblogs.com/jerrylead/archive/2013/04/27/Spark.html http://www.cnblogs.com/jerrylead/archive/2013/04/27/Spark.html ， 2013

Hive与Impala比较 . http://www.linuxeden.com/html/news/20130714/141332.html http://www.linuxeden.com/html/news/20130714/141332.html ， 2013

Comparison between Impala and Hive . http://www.linuxeden.com/html/news/20130714/141332.html http://www.linuxeden.com/html/news/20130714/141332.html ， 2013

Dremel . http://www.yankay.com/google-dremel-rationale/ http://www.yankay.com/google-dremel-rationale/ ， 2012

Spark API . http://spark.apache.org/docs/latest/api/python/index.html http://spark.apache.org/docs/latest/api/python/index.html ， 2014

Resilient distributed datasets：a fault-tolerant abstraction for in-memory cluster computing . http://www.cs.berkeley.edu/-matei/papers/2012/nsdi_spark.pdf http://www.cs.berkeley.edu/-matei/papers/2012/nsdi_spark.pdf ， 2012

PageRank . http://en.wikipedia.org/wiki/PageRank http://en.wikipedia.org/wiki/PageRank ， 2014

微博 3.5 亿用户关系数据（样例） . http://www.datatang.com/data/46248 http://www.datatang.com/data/46248 ， 2014

Relationship data of 350 million weibo users（samples） . http://www.datatang.com/data/46248 http://www.datatang.com/data/46248 ， 2014

Views

1228

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Summary of Large-Scale Grapb Partitioning Algoritbms

Design of Mobile Internet Big Data User Behavior Analysis Engine Based on Cloud Computing

Security access permission authentication simulation of multi-source network big data based on blockchain

Research on theoretical ultimate value of 5G flow residence ratio based on big data

A network traffic classification method based on random forest and improved convolutional neural network

Related Author

Jinfeng Xu

Yihong Dong

Shiyi Wang

Xianmang He

Huahui Chen

Caixia Tao

Xiaojun Xie

Kang Chen

Related Institution

Research Institute of China United Network Communications Co., Ltd.

China United Network Communications Group Co., Ltd.

Zhejiang Branch of China United Network Communications Co., Ltd.

Willfar Information Technology Co., Ltd.

China Electronic Power Equipment and Technology Co., Ltd.

⁰