大数据分析全英文 (10).pdf

《大数据分析全英文 (10).pdf》由会员分享，可在线阅读，更多相关《大数据分析全英文 (10).pdf（12页珍藏版）》请在得力文库 - 分享文档赚钱的网站上搜索。

1、Chap2-3 External Data Acquisition2External data and its acquisition1.Network big data and web crawlers2.Web crawler crawling strategy3Network Big DataBig data generated by the interaction and integration of the ternary world of human,machine,and material in cyberspace and available on the InternetNe

2、twork big data characteristics:Network big data Multi-source heterogeneousTimelinessSocialityInteractivitySuddenHigh noise4Web CrawlerA web crawler is a program or web robot that automatically browses the Internet and get data.5Web crawler crawling process234Seed URLDownloaded web library1.Identify

3、all the needed links and add them to the URL to be crawled queue.URL crawledRead URL,DNS resolution,web page downloadPut the downloaded URL into the crawled URL listExtract the new URL into the URL queue to be crawled6Different crawling strategiesCrawling strategies:1.Depth first2.Breadth first3.Par

4、tial PageRank Strategy4.OPIC(online Page Importance Computation)7Crawling Strategy 1 Depth first and 2 Breadth firstDepth firstBreadth first83 Partial PageRankQuantity and quality:Links to webpage E are far more than links to webpage C,but webpage C is far more important than webpage E.This is becau

5、se page C is linked by page B,and page B is of high importance9OPIC(online Page Importance Computation)1.The OPIC strategy assigns the same gold coins to each web page.Whenever a page P is downloaded,the gold coins owned by P are equally distributed to the linked pages contained in the web page.2.Th

6、e links in the queue to be crawled are sorted by gold coins3.OPIC calculation speed is faster than partial PageRank strategy10Distributed web crawlerDistributed Architecture model1 the master needs to be updated when expanding,the master node is under heavy pressure and easily becomes a bottleneck,a

7、nd the number of slave nodes is limited.2 All nodes need to update all other nodes when communication is expanded.No master,no single machine hot spots,and the number of slave nodes is limited.3 The master communicates with the master,and the number of slave nodes is small.1 2 3 11Summary of externa

8、l data acquisition 1.Web crawler crawling process2.Web crawler crawling strategyDepth first,Breadth first,Page Rank,OPIC(online Page Importance Computation)3.Distributed web crawler structure Haiying CheInstitute of Data Science and knowledge EngineeringSchool of Computer ScienceBeijing Institute of TechnologyThank you！