python爬蟲(chóng)實(shí)例詳解
本篇博文主要講解Python爬蟲(chóng)實(shí)例,重點(diǎn)包括爬蟲(chóng)技術(shù)架構(gòu),組成爬蟲(chóng)的關(guān)鍵模塊:URL管理器、HTML下載器和HTML解析器。
爬蟲(chóng)簡(jiǎn)單架構(gòu)
程序入口函數(shù)(爬蟲(chóng)調(diào)度段)
#coding:utf8 import time, datetime from maya_Spider import url_manager, html_downloader, html_parser, html_outputer class Spider_Main(object): #初始化操作 def __init__(self): #設(shè)置url管理器 self.urls = url_manager.UrlManager() #設(shè)置HTML下載器 self.downloader = html_downloader.HtmlDownloader() #設(shè)置HTML解析器 self.parser = html_parser.HtmlParser() #設(shè)置HTML輸出器 self.outputer = html_outputer.HtmlOutputer() #爬蟲(chóng)調(diào)度程序 def craw(self, root_url): count = 1 self.urls.add_new_url(root_url) while self.urls.has_new_url(): try: new_url = self.urls.get_new_url() print('craw %d : %s' % (count, new_url)) html_content = self.downloader.download(new_url) new_urls, new_data = self.parser.parse(new_url, html_content) self.urls.add_new_urls(new_urls) self.outputer.collect_data(new_data) if count == 10: break count = count + 1 except: print('craw failed') self.outputer.output_html() if __name__ == '__main__': #設(shè)置爬蟲(chóng)入口 root_url = 'http://baike.baidu.com/view/21087.htm' #開(kāi)始時(shí)間 print('開(kāi)始計(jì)時(shí)..............') start_time = datetime.datetime.now() obj_spider = Spider_Main() obj_spider.craw(root_url) #結(jié)束時(shí)間 end_time = datetime.datetime.now() print('總用時(shí):%ds'% (end_time - start_time).seconds)
URL管理器
class UrlManager(object): def __init__(self): self.new_urls = set() self.old_urls = set() def add_new_url(self, url): if url is None: return if url not in self.new_urls and url not in self.old_urls: self.new_urls.add(url) def add_new_urls(self, urls): if urls is None or len(urls) == 0: return for url in urls: self.add_new_url(url) def has_new_url(self): return len(self.new_urls) != 0 def get_new_url(self): new_url = self.new_urls.pop() self.old_urls.add(new_url) return new_url
網(wǎng)頁(yè)下載器
import urllib import urllib.request class HtmlDownloader(object): def download(self, url): if url is None: return None #偽裝成瀏覽器訪(fǎng)問(wèn),直接訪(fǎng)問(wèn)的話(huà)csdn會(huì)拒絕 user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)' headers = {'User-Agent':user_agent} #構(gòu)造請(qǐng)求 req = urllib.request.Request(url,headers=headers) #訪(fǎng)問(wèn)頁(yè)面 response = urllib.request.urlopen(req) #python3中urllib.read返回的是bytes對(duì)象,不是string,得把它轉(zhuǎn)換成string對(duì)象,用bytes.decode方法 return response.read().decode()
網(wǎng)頁(yè)解析器
import re import urllib from urllib.parse import urlparse from bs4 import BeautifulSoup class HtmlParser(object): def _get_new_urls(self, page_url, soup): new_urls = set() #/view/123.htm links = soup.find_all('a', href=re.compile(r'/item/.*?')) for link in links: new_url = link['href'] new_full_url = urllib.parse.urljoin(page_url, new_url) new_urls.add(new_full_url) return new_urls #獲取標(biāo)題、摘要 def _get_new_data(self, page_url, soup): #新建字典 res_data = {} #url res_data['url'] = page_url #<dd class="lemmaWgt-lemmaTitle-title"><h1>Python</h1>獲得標(biāo)題標(biāo)簽 title_node = soup.find('dd', class_="lemmaWgt-lemmaTitle-title").find('h1') print(str(title_node.get_text())) res_data['title'] = str(title_node.get_text()) #<div class="lemma-summary" label-module="lemmaSummary"> summary_node = soup.find('div', class_="lemma-summary") res_data['summary'] = summary_node.get_text() return res_data def parse(self, page_url, html_content): if page_url is None or html_content is None: return None soup = BeautifulSoup(html_content, 'html.parser', from_encoding='utf-8') new_urls = self._get_new_urls(page_url, soup) new_data = self._get_new_data(page_url, soup) return new_urls, new_data
網(wǎng)頁(yè)輸出器
class HtmlOutputer(object): def __init__(self): self.datas = [] def collect_data(self, data): if data is None: return self.datas.append(data ) def output_html(self): fout = open('maya.html', 'w', encoding='utf-8') fout.write("<head><meta http-equiv='content-type' content='text/html;charset=utf-8'></head>") fout.write('<html>') fout.write('<body>') fout.write('<table border="1">') # <th width="5%">Url</th> fout.write('''<tr style="color:red" width="90%"> <th>Theme</th> <th width="80%">Content</th> </tr>''') for data in self.datas: fout.write('<tr>\n') # fout.write('\t<td>%s</td>' % data['url']) fout.write('\t<td align="center"><a href=\'%s\'>%s</td>' % (data['url'], data['title'])) fout.write('\t<td>%s</td>\n' % data['summary']) fout.write('</tr>\n') fout.write('</table>') fout.write('</body>') fout.write('</html>') fout.close()
運(yùn)行結(jié)果
附:完整代碼
以上就是本文的全部?jī)?nèi)容,希望對(duì)大家的學(xué)習(xí)有所幫助,也希望大家多多支持腳本之家。
相關(guān)文章
Python PyQt5模塊實(shí)現(xiàn)一個(gè)瀏覽器的示例代碼
在項(xiàng)目開(kāi)發(fā)中,有的應(yīng)用程序可以運(yùn)行在web瀏覽器,本文主要介紹了Python PyQt5模塊實(shí)現(xiàn)一個(gè)瀏覽器的示例代碼,分享給大家,感興趣的可以了解一下2021-07-07Pytorch backward報(bào)錯(cuò)2次訪(fǎng)問(wèn)計(jì)算圖需要retain_graph=True的情況詳解
這篇文章主要介紹了Pytorch backward報(bào)錯(cuò)2次訪(fǎng)問(wèn)計(jì)算圖需要retain_graph=True的情況,具有很好的參考價(jià)值,希望對(duì)大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-02-02Python通用函數(shù)實(shí)現(xiàn)數(shù)組計(jì)算的方法
數(shù)組的運(yùn)算可以進(jìn)行加減乘除,同時(shí)也可以將這些算數(shù)運(yùn)算符進(jìn)行任意的組合已達(dá)到效果。這篇文章主要介紹了Python通用函數(shù)實(shí)現(xiàn)數(shù)組計(jì)算的代碼,非常不錯(cuò),具有一定的參考借鑒價(jià)值,需要的朋友參考下吧2019-06-06Jmeter如何使用BeanShell取樣器調(diào)用Python腳本
這篇文章主要介紹了Jmeter使用BeanShell取樣器調(diào)用Python腳本,文章圍繞Jmeter調(diào)用Python腳本的相關(guān)詳情展開(kāi)標(biāo)題內(nèi)容,需要的小伙伴可以參考一下2022-03-03keras實(shí)現(xiàn)圖像預(yù)處理并生成一個(gè)generator的案例
這篇文章主要介紹了keras實(shí)現(xiàn)圖像預(yù)處理并生成一個(gè)generator的案例,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2020-06-06Numpy數(shù)組轉(zhuǎn)置的實(shí)現(xiàn)
本文主要介紹了Numpy數(shù)組轉(zhuǎn)置的實(shí)現(xiàn),文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2023-02-02