Scrapy框架介紹之Puppeteer渲染的使用
1、Scrapy框架
Scrapy是用純Python實現一個為了爬取網站數據、提取結構性數據而編寫的應用框架,用途非常廣泛。
框架的力量,用戶只需要定制開發(fā)幾個模塊就可以輕松的實現一個爬蟲,用來抓取網頁內容以及各種圖片,非常之方便。
Scrapy 使用了 Twisted'twɪstɪd異步網絡框架來處理網絡通訊,可以加快我們的下載速度,不用自己去實現異步框架,并且包含了各種中間件接口,可以靈活的完成各種需求。
- Scrapy Engine(引擎): 負責Spider、ItemPipeline、Downloader、Scheduler中間的通訊,信號、數據傳遞等。
- Scheduler(調度器): 它負責接受引擎發(fā)送過來的Request請求,并按照一定的方式進行整理排列,入隊,當引擎需要時,交還給引擎。
- Downloader(下載器):負責下載Scrapy Engine(引擎)發(fā)送的所有Requests請求,并將其獲取到的Responses交還給Scrapy Engine(引擎),由引擎交給Spider來處理,
- Spider(爬蟲):它負責處理所有Responses,從中分析提取數據,獲取Item字段需要的數據,并將需要跟進的URL提交給引擎,再次進入Scheduler(調度器),
- Item Pipeline(管道):它負責處理Spider中獲取到的Item,并進行進行后期處理(詳細分析、過濾、存儲等)的地方.
- Downloader Middlewares(下載中間件):你可以當作是一個可以自定義擴展下載功能的組件。
- Spider Middlewares(Spider中間件):你可以理解為是一個可以自定擴展和操作引擎和Spider中間通信的功能組件(比如進入Spider的Responses;和從Spider出去的Requests)
2、Puppeteer渲染
Puppeteer 是 Chrome 開發(fā)團隊在 2017 年發(fā)布的一個 Node.js 包,用來模擬 Chrome 瀏覽器的運行。
為了爬取js渲染的html頁面,我們需要用瀏覽器來解析js后生成html。在scrapy中可以利用pyppeteer來實現對應功能。
完整代碼 📎scrapy-pyppeteer.zip
我們需要新建項目中middlewares.py文件(./項目名/middlewares.py)
import websockets from scrapy.http import HtmlResponse from logging import getLogger import asyncio import pyppeteer import logging from concurrent.futures._base import TimeoutError import base64 import sys import random pyppeteer_level = logging.WARNING logging.getLogger('websockets.protocol').setLevel(pyppeteer_level) logging.getLogger('pyppeteer').setLevel(pyppeteer_level) PY3 = sys.version_info[0] >= 3 def base64ify(bytes_or_str): if PY3 and isinstance(bytes_or_str, str): input_bytes = bytes_or_str.encode('utf8') else: input_bytes = bytes_or_str output_bytes = base64.urlsafe_b64encode(input_bytes) if PY3: return output_bytes.decode('ascii') else: return output_bytes class ProxyMiddleware(object): USER_AGENT = open('useragents.txt').readlines() def process_request(self, request, spider): # 代理服務器 proxyHost = "t.16yun.cn" proxyPort = "31111" # 代理隧道驗證信息 proxyUser = "username" proxyPass = "password" request.meta['proxy'] = "http://{0}:{1}".format(proxyHost, proxyPort) # 添加驗證頭 encoded_user_pass = base64ify(proxyUser + ":" + proxyPass) request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass # 設置IP切換頭(根據需求) tunnel = random.randint(1, 10000) request.headers['Proxy-Tunnel'] = str(tunnel) request.headers['User-Agent'] = random.choice(self.USER_AGENT) class PyppeteerMiddleware(object): def __init__(self, **args): """ init logger, loop, browser :param args: """ self.logger = getLogger(__name__) self.loop = asyncio.get_event_loop() self.browser = self.loop.run_until_complete( pyppeteer.launch(headless=True)) self.args = args def __del__(self): """ close loop :return: """ self.loop.close() def render(self, url, retries=1, script=None, wait=0.3, scrolldown=False, sleep=0, timeout=8.0, keep_page=False): """ render page with pyppeteer :param url: page url :param retries: max retry times :param script: js script to evaluate :param wait: number of seconds to wait before loading the page, preventing timeouts :param scrolldown: how many times to page down :param sleep: how many long to sleep after initial render :param timeout: the longest wait time, otherwise raise timeout error :param keep_page: keep page not to be closed, browser object needed :param browser: pyppetter browser object :param with_result: return with js evaluation result :return: content, [result] """ # define async render async def async_render(url, script, scrolldown, sleep, wait, timeout, keep_page): try: # basic render page = await self.browser.newPage() await asyncio.sleep(wait) response = await page.goto(url, options={'timeout': int(timeout * 1000)}) if response.status != 200: return None, None, response.status result = None # evaluate with script if script: result = await page.evaluate(script) # scroll down for {scrolldown} times if scrolldown: for _ in range(scrolldown): await page._keyboard.down('PageDown') await asyncio.sleep(sleep) else: await asyncio.sleep(sleep) if scrolldown: await page._keyboard.up('PageDown') # get html of page content = await page.content() return content, result, response.status except TimeoutError: return None, None, 500 finally: # if keep page, do not close it if not keep_page: await page.close() content, result, status = [None] * 3 # retry for {retries} times for i in range(retries): if not content: content, result, status = self.loop.run_until_complete( async_render(url=url, script=script, sleep=sleep, wait=wait, scrolldown=scrolldown, timeout=timeout, keep_page=keep_page)) else: break # if need to return js evaluation result return content, result, status def process_request(self, request, spider): """ :param request: request object :param spider: spider object :return: HtmlResponse """ if request.meta.get('render'): try: self.logger.debug('rendering %s', request.url) html, result, status = self.render(request.url) return HtmlResponse(url=request.url, body=html, request=request, encoding='utf-8', status=status) except websockets.exceptions.ConnectionClosed: pass @classmethod def from_crawler(cls, crawler): return cls(**crawler.settings.get('PYPPETEER_ARGS', {}))
然后修改項目配置文件 (./項目名/settings.py)
DOWNLOADER_MIDDLEWARES = { 'scrapypyppeteer.middlewares.PyppeteerMiddleware': 543, 'scrapypyppeteer.middlewares.ProxyMiddleware': 100, }
然后我們運行程序
到此這篇關于Scrapy框架介紹之Puppeteer渲染的使用的文章就介紹到這了,更多相關Scrapy Puppeteer渲染內容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關文章希望大家以后多多支持腳本之家!
相關文章
對python opencv 添加文字 cv2.putText 的各參數介紹
今天小編就為大家分享一篇對python opencv 添加文字 cv2.putText 的各參數介紹,具有很好的參考價值,希望對大家有所幫助。一起跟隨小編過來看看吧2018-12-12