A Large Scale NetFlow Analysis System Based on Spark

Shengyong Ding; Shiwu Min; Yongbing Fan

doi:10.3969/j.issn.1000-0801.2014.10.009

您当前的位置：

首页 >

文章列表页 >

A Large Scale NetFlow Analysis System Based on Spark

Topic: big data technology and Application | 更新时间：2024-06-05

- A Large Scale NetFlow Analysis System Based on Spark
- Telecommunications Science Vol. 30, Issue 10, Pages: 48-51(2014)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.3969/j.issn.1000-0801.2014.10.009
  CLC：
- Published Online：2014-10，
  
  Published：20 October 2014
- 稿件说明：
移动端阅览
Shengyong Ding, Shiwu Min, Yongbing Fan. A Large Scale NetFlow Analysis System Based on Spark[J]. Telecommunications science, 2014, 30(10): 48-51.
DOI：

Shengyong Ding, Shiwu Min, Yongbing Fan. A Large Scale NetFlow Analysis System Based on Spark[J]. Telecommunications science, 2014, 30(10): 48-51. DOI： 10.3969/j.issn.1000-0801.2014.10.009.

摘要

目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统，面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题。大数据技术的快速发展尤其是内存式计算平台如Spark的出现为集中处理大规模NetFlow数据提供了可能，本文提出了基于Spark的NetFlow分析系统，验证了核心算法（如流量应用构成统计）在Spark平台的性能。实验表明，基于Spark的NetFlow分析系统具有很高的性能和很强的扩展能力，较之Hadoop MapReduce有显著的性能提升。

Abstract

The existing systems usually adopt private distributed architectures

which face scalability

openness

cost and latency problems. The development of big data technology such as Spark offers new opportunity for large scale NetFlow processing systems. A new analysis system based on Spark platform was proposed and the effectiveness of the method was verified. The experimental results show its superior performance.

关键词

Keywords

references

White T . Hadoop: the Definitive Guide . O'Reilly Media Inc , 2012

Zaharia M , Chowdhury M , Das T , et al . Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing . Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation , San Jose, CA, USA , 2012

Rossi D , Silvio V . Fine-grained traffic classification with NetFlow data . Proceedings of the 6th International Wireless Communications and Mobile Computing Conference , Shenzhen, China , 2010

Views

1405

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A network traffic classification method based on random forest and improved convolutional neural network

Delay-sensitive traffic intellisense scheduling based on optimal decision tree

Implementation of a big data anonymization system based on Spark

An improved CLTree algorithm

Comparison of Open-Source Distributed Computing Framework for Big Data

Related Author

Xiaoya GAN

Yaguan QIAN

Bensheng YUN

Xuerong WANG

Zhengzhi TANG

Yinchuan LI

Meiyu QI

Jianbo ZHU

Related Institution

Research Institute of China Telecom Co., Ltd.

Beijing Huawei Digital Technologies Co., Ltd.

Beijing Venus Information Security Technology Incorporated Company

Beijing University of Posts and Telecommunications

⁰