Research and design of distributed high-performance network reptiles based on cloud platform

Enming SHI; Xiaojun XIAO; Yu LU

doi:10.11959/j.issn.1000-0801.2017234

您当前的位置：

首页 >

文章列表页 >

Research and design of distributed high-performance network reptiles based on cloud platform

Operation technology wide Angle | 更新时间：2024-06-05

- Research and design of distributed high-performance network reptiles based on cloud platform
- Telecommunications Science Vol. 33, Issue 8, Pages: 180-186(2017)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2017234
  CLC： TP393
- Published Online：2017-08，
  
  Published：15 August 2017
- 稿件说明：
移动端阅览
Enming SHI, Xiaojun XIAO, Yu LU. Research and design of distributed high-performance network reptiles based on cloud platform[J]. Telecommunications science, 2017, 33(8): 180-186.
DOI：

Enming SHI, Xiaojun XIAO, Yu LU. Research and design of distributed high-performance network reptiles based on cloud platform[J]. Telecommunications science, 2017, 33(8): 180-186. DOI： 10.11959/j.issn.1000-0801.2017234.

摘要

随着大数据时代的到来，数据成为最宝贵的资源，而网络爬虫技术作为外部数据采集的重要手段，已然成为数据分析的标配。介绍了一种高性能、灵活和便捷的基于云平台的爬虫架构设计和实现。从爬虫的整体架构、分布式设计以及各模块的设计等角度进行了详细的阐述。爬虫各模块用 Docker 封装，Kubernetes做集群的资源调度和管理，在性能优化上采用了MD5去重树算法、DNS优化和异步I/O等多种策略组合的形式。实验表明，对比未优化的方案，爬虫在性能上具有较明显的优势。

Abstract

With the arrival of large data age

data has become the most valuable resource.And web crawler technology as an important means of external data collection

has become a standard tool for data analysis.A high-performance

convenient cloud-based crawler architecture design was introduced.The overall structure of the reptile to the distributed design and the design of the sub-module was described in detail.Each module of the crawler was encapsulated in Docker

and Kubernetes was used as the resource scheduling and management of the cluster.In the performance of optimization

the MD5 reset tree algorithm

DNS optimization and asynchronous I/O were adopted.Experimental results show that the performance of crawler has obvious advantages compared with the UN optimized scheme.

关键词

Keywords

references

NEGUS C . Docker containers from start to enterprise (includes content update program):build and deploy with Kubernetes,Flannel,Cockpit and Atomic [J ] . Vaccine , 2016 ( 19 ): S87 - S95 .

严华云 , 关佶红 . Bloom Filter 研究进展 [J ] . 电信科学 , 2010 , 26 ( 2 ): 31 - 36 .

YAN H Y , GUAN J H . Survey of Bloom Filter [J ] . Telecommunications Science , 2010 , 26 ( 2 ): 31 - 36 .

吴桦 , 龚俭 , 杨望 . 一种基于双重Counter Bloom Filter的长流识别算法 [J ] . 软件学报 , 2010 , 21 ( 5 ): 1115 - 1126 .

WU H , GONG J , YANG W . Algorithm based on double Counter Bloom Filter for large flows identification [J ] . Journal of Software , 2010 , 21 ( 5 ): 1115 - 1126 .

张进 , 邬江兴 , 刘勤让 . 4种计数型Bloom Filter的性能分析与比较 [J ] . 软件学报 , 2010 , 21 ( 5 ): 1098 - 1114 .

ZHANG J , WU J X , LIU Q R . Performance evaluation and comparison of four Counting Bloom filter schemes [J ] . Journal of Software , 2010 , 21 ( 5 ): 1098 - 1114 .

严磊 , 丁宾 , 姚志敏 , 等 . 基于 MD5 去重树的网络爬虫的设计与优化 [J ] . 计算机应用与软件 , 2015 ( 2 ): 325 - 329 ,333.

YAN L , DING B , YAO Z M , et al . Design and optimisation of md5 duplicate elimination tree-based network crawler [J ] . Computer Application and Software , 2015 ( 2 ): 325 - 329 ,333.

BRIN S , PAGE J . The anatomy of a large-scale hypertextual web search engine [J ] . Computer Networks and Isdn Systems , 1998 , 98 ( 30 ): 107 - 117 .

朱泽德 , 李淼 , 张健 , 等 . 基于文本密度模型的Web正文抽取 [J ] . 模式识别与人工智能 , 2013 ( 7 ): 667 - 672 .

ZHU Z D , LI M , ZHANG J , et al . Web content extraction based on text density model [J ] . Pattern Recognition and Artificial Intelligence , 2013 ( 7 ): 667 - 672 .

Views

1192

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Virtualized deployment of DPI based on Docker

Research on compatibility of high-performance network stack

Design and analysis of telecom service platform reconstruction based on open source software

PaaS cloud platform based on container technology

A dynamic detection method based on Web crawler and page code behavior for XSS vulnerability

Related Author

Zhao XUE

Jie PAN

Huiyou JIANG

Junfeng LI

Dan LI

Gang LU

Zhe WANG

Honghui OU

Related Institution

Guangzhou Research Institute of China Telecom Co., Ltd.

Guangdong Branch of China Telecom Co., Ltd.

⁰