欧美bbbwbbbw肥妇,免费乱码人妻系列日韩,一级黄片

Scrapy爬蟲(chóng)實(shí)例講解_校花網(wǎng)

 更新時(shí)間：2017年10月23日 09:06:45   作者：Python學(xué)習(xí)者  
下面小編就為大家?guī)?lái)一篇Scrapy爬蟲(chóng)實(shí)例講解_校花網(wǎng)。小編覺(jué)得挺不錯(cuò)的，現(xiàn)在就分享給大家，也給大家做個(gè)參考。一起跟隨小編過(guò)來(lái)看看吧

學(xué)習(xí)爬蟲(chóng)有一段時(shí)間了，今天使用Scrapy框架將校花網(wǎng)的圖片爬取到本地。Scrapy爬蟲(chóng)框架相對(duì)于使用requests庫(kù)進(jìn)行網(wǎng)頁(yè)的爬取，擁有更高的性能。

Scrapy官方定義：Scrapy是用于抓取網(wǎng)站并提取結(jié)構(gòu)化數(shù)據(jù)的應(yīng)用程序框架，可用于廣泛的有用應(yīng)用程序，如數(shù)據(jù)挖掘，信息處理或歷史存檔。

建立Scrapy爬蟲(chóng)工程

在安裝好Scrapy框架后，直接使用命令行進(jìn)行項(xiàng)目的創(chuàng)建：

E:\ScrapyDemo>scrapy startproject xiaohuar
New Scrapy project 'xiaohuar', using template directory 'c:\\users\\lei\\appdata\\local\\programs\\python\\python35\\lib
\\site-packages\\scrapy\\templates\\project', created in:
 E:\ScrapyDemo\xiaohuar

You can start your first spider with:
 cd xiaohuar
 scrapy genspider example example.com

創(chuàng)建一個(gè)Scrapy爬蟲(chóng)

創(chuàng)建工程的時(shí)候，會(huì)自動(dòng)創(chuàng)建一個(gè)與工程同名的目錄，進(jìn)入到目錄中執(zhí)行如下命令：

E:\ScrapyDemo\xiaohuar>scrapy genspider -t basic xiaohua xiaohuar.com
Created spider 'xiaohua' using template 'basic' in module:
xiaohuar.spiders.xiaohua命令中"xiaohua"

是生成Spider中*.py文件的文件名，"xiaohuar.com"是將要爬取網(wǎng)站的URL，可以在程序中更改。

編寫(xiě)Spider代碼

編寫(xiě)E:\ScrapyDemo\xiaohuar\xiaohuar\spiders中的xiaohua.py文件。主要是配置URL和對(duì)請(qǐng)求到的頁(yè)面的解析方式。

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request
import re

class XiaohuaSpider(scrapy.Spider):
 name = 'xiaohua'
 allowed_domains = ['xiaohuar.com']
 start_urls = []
 for i in range(43):
  url = "http://www.xiaohuar.com/list-1-%s.html" %i
  start_urls.append(url)

 def parse(self, response):
  if "www.xiaohuar.com/list-1" in response.url:
   # 下載的html源代碼
   html = response.text
   # 網(wǎng)頁(yè)中圖片存儲(chǔ)地址：src="/d/file/20160126/905e563421921adf9b6fb4408ec4e72f.jpg"
   # 通過(guò)正則匹配到所有的圖片
   # 獲取的是圖片的相對(duì)路徑的列表
   img_urls = re.findall(r'/d/file/\d+/\w+\.jpg',html)
   
   # 使用循環(huán)對(duì)圖片頁(yè)進(jìn)行請(qǐng)求
   for img_url in img_urls:
    # 將圖片的URL補(bǔ)全
    if "http://" not in img_url:
     img_url = "http://www.xiaohuar.com%s" %img_url
    
    # 回調(diào)，返回response
    yield Request(img_url)
  else:
   # 下載圖片 
   url = response.url
   # 保存的圖片文件名
   title = re.findall(r'\w*.jpg',url)[0]
   # 保存圖片
   with open('E:\\xiaohua_img\\%s' % title, 'wb') as f:
    f.write(response.body)

這里使用正則表達(dá)式對(duì)圖片的地址進(jìn)行匹配，其他網(wǎng)頁(yè)也都大同小異，需要根據(jù)具體的網(wǎng)頁(yè)源代碼進(jìn)行分析。

運(yùn)行爬蟲(chóng)

E:\ScrapyDemo\xiaohuar>scrapy crawl xiaohua
2017-10-22 22:30:11 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: xiaohuar)
2017-10-22 22:30:11 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'xiaohuar', 'SPIDER_MODULES': ['xiaohuar.
spiders'], 'ROBOTSTXT_OBEY': True, 'NEWSPIDER_MODULE': 'xiaohuar.spiders'}
2017-10-22 22:30:11 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logstats.LogStats']
2017-10-22 22:30:12 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-10-22 22:30:12 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-10-22 22:30:12 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-10-22 22:30:12 [scrapy.core.engine] INFO: Spider opened
2017-10-22 22:30:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min
)
2017-10-22 22:30:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-10-22 22:30:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/robots.txt> (referer: None)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/list-1-0.html> (referer: None
)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170721/cb96f1b106b3d
b4a6bfcf3d2e880dea0.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170824/dcc166b0eba6a
37e05424cfc29023121.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170916/7f78145b1ca16
2eb814fbc03ad24fbc1.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170919/2f728d0f110a2
1fea95ce13e0b010d06.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170819/9c3dfeef7e08c
c0303ce233e4ddafa7f.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170917/715515e7fe1f1
cb9fd388bbbb00467c2.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170628/f3d06ef49965a
edbe18286a2f221fd9f.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170513/6121e3e90ff3b
a4c9398121bda1dd582.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170516/6e295fe48c332
45be858c40d37fb5ee6.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170707/f7ca636f73937
e33836e765b7261f036.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170528/b352258c83776
b9a2462277dec375d0c.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170527/4a7a7f1e6b69f
126292b981c90110d0a.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170715/61110ba027f00
4fb503ff09cdee44d0c.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170520/dd21a21751e24
a8f161792b66011688c.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170529/8140c4ad797ca
01f5e99d09c82dd8a42.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170603/e55f77fb3aa3c
7f118a46eeef5c0fbbf.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170529/e5902d4d3e408
29f9a0d30f7488eab84.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170604/ec3794d0d42b5
38bf4461a84dac32509.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170603/c34b29f68e8f9
6d44c63fe29bf4a66b8.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170701/fb18711a6af87
f30942d6a19f6da6b3e.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170619/e0456729d4dcb
ea569a1acbc6a47ab69.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xiaohuar.com/d/file/20170626/0ab1d89f54c90
df477a90aa533ceea36.jpg> (referer: http://www.xiaohuar.com/list-1-0.html)
2017-10-22 22:30:15 [scrapy.core.engine] INFO: Closing spider (finished)
2017-10-22 22:30:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 8785,
 'downloader/request_count': 24,
 'downloader/request_method_count/GET': 24,
 'downloader/response_bytes': 2278896,
 'downloader/response_count': 24,
 'downloader/response_status_count/200': 24,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 10, 22, 14, 30, 15, 892287),
 'log_count/DEBUG': 25,
 'log_count/INFO': 7,
 'request_depth_max': 1,
 'response_received_count': 24,
 'scheduler/dequeued': 23,
 'scheduler/dequeued/memory': 23,
 'scheduler/enqueued': 23,
 'scheduler/enqueued/memory': 23,
 'start_time': datetime.datetime(2017, 10, 22, 14, 30, 12, 698874)}
2017-10-22 22:30:15 [scrapy.core.engine] INFO: Spider closed (finished)
scrapy crawl xiaohua

圖片保存

在圖片保存過(guò)程中"\"需要進(jìn)行轉(zhuǎn)義。

>>> r = requests.get("https://timgsa.baidu.com/timg?image&quality=80&size=b9999_10000&sec=1508693697147&di=23eb655d8e450f84cf39453bc1029bc0&imgtype=0&src=http%3A%2F%2Fb.hiphotos.baidu.com%2Fimage%2Fpic%2Fitem%2Fc9fcc3cec3fdfc038b027f7bde3f8794a5c226fe.jpg")
>>> open("E:\xiaohua_img\01.jpg",'wb').write(r.content)
 File "<stdin>", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode by
>>> open("E:\\xiaohua_img\1.jpg",'wb').write(r.content)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument: 'E:\\xiaohua_img\x01.jpg'
>>> open("E:\\xiaohua_img\\1.jpg",'wb').write(r.content)

以上這篇Scrapy爬蟲(chóng)實(shí)例講解_?；ňW(wǎng)就是小編分享給大家的全部?jī)?nèi)容了，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。

相關(guān)文章

  • 在Ubuntu 20.04中安裝Pycharm 2020.1的圖文教程

    在Ubuntu 20.04中安裝Pycharm 2020.1的圖文教程

    這篇文章主要介紹了在Ubuntu 20.04中安裝Pycharm 2020.1的圖文教程，本文通過(guò)圖文并茂的形式給大家介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值,需要的朋友可以參考下
    2020-04-04
  • Python3實(shí)現(xiàn)的簡(jiǎn)單三級(jí)菜單功能示例

    Python3實(shí)現(xiàn)的簡(jiǎn)單三級(jí)菜單功能示例

    這篇文章主要介紹了Python3實(shí)現(xiàn)的簡(jiǎn)單三級(jí)菜單功能,涉及Python用戶交互以及針對(duì)json格式數(shù)據(jù)的遍歷、讀取、判斷等相關(guān)操作技巧,需要的朋友可以參考下
    2019-03-03
  • Python文件打開(kāi)方式實(shí)例詳解【a、a+、r+、w+區(qū)別】

    Python文件打開(kāi)方式實(shí)例詳解【a、a+、r+、w+區(qū)別】

    這篇文章主要介紹了Python文件打開(kāi)方式,結(jié)合實(shí)例形式詳細(xì)分析了Python文件打開(kāi)函數(shù)的使用及a、a+、r+、w+等屬性功能、使用區(qū)別與相關(guān)操作注意事項(xiàng),需要的朋友可以參考下
    2019-03-03
  • python之pkl文件的用法及說(shuō)明

    python之pkl文件的用法及說(shuō)明

    這篇文章主要介紹了python之pkl文件的用法及說(shuō)明，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。如有錯(cuò)誤或未考慮完全的地方，望不吝賜教
    2023-06-06
  • python中列表添加元素的幾種方式(+、append()、extend())

    python中列表添加元素的幾種方式(+、append()、extend())

    本文主要介紹了python中列表添加元素的幾種方式(+、append()、extend())，文中通過(guò)示例代碼介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧
    2022-08-08
  • python實(shí)現(xiàn)簡(jiǎn)單的TCP代理服務(wù)器

    python實(shí)現(xiàn)簡(jiǎn)單的TCP代理服務(wù)器

    這篇文章主要介紹了python實(shí)現(xiàn)簡(jiǎn)單的TCP代理服務(wù)器,包含了完整的實(shí)現(xiàn)過(guò)程及對(duì)應(yīng)的源碼與說(shuō)明文檔下載,非常具有參考借鑒價(jià)值,需要的朋友可以參考下
    2014-10-10
  • Python+logging輸出到屏幕將log日志寫(xiě)入文件

    Python+logging輸出到屏幕將log日志寫(xiě)入文件

    這篇文章主要給大家介紹了關(guān)于Python+logging輸出到屏幕將log日志寫(xiě)入文件的相關(guān)資料，文中通過(guò)示例代碼介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧
    2020-11-11
  • Python利用PyPDF2快速拆分PDF文檔

    Python利用PyPDF2快速拆分PDF文檔

    很多時(shí)候需要拆分PDF，但是大部分需要付費(fèi)，免費(fèi)的限制很多，本文就使用Python快速拆分PDF文檔，具有一定的參考價(jià)值，感興趣的可以了解一下
    2021-07-07
  • python實(shí)現(xiàn)雙人五子棋（終端版）

    python實(shí)現(xiàn)雙人五子棋（終端版）

    這篇文章主要為大家詳細(xì)介紹了python實(shí)現(xiàn)終端版的雙人五子棋，文中示例代碼介紹的非常詳細(xì)，具有一定的參考價(jià)值，感興趣的小伙伴們可以參考一下
    2020-12-12
  • pytorch 在網(wǎng)絡(luò)中添加可訓(xùn)練參數(shù),修改預(yù)訓(xùn)練權(quán)重文件的方法

    pytorch 在網(wǎng)絡(luò)中添加可訓(xùn)練參數(shù),修改預(yù)訓(xùn)練權(quán)重文件的方法

    今天小編就為大家分享一篇pytorch 在網(wǎng)絡(luò)中添加可訓(xùn)練參數(shù),修改預(yù)訓(xùn)練權(quán)重文件的方法，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧
    2019-08-08

最新評(píng)論