腳本之家服務(wù)器常用軟件

快捷導(dǎo)航

Python中五種不同解析庫的應(yīng)用與提取速度對比

更新時間：2025年05月30日 10:27:27 作者：失敗又激情的man

這篇文章主要為大家詳細(xì)介紹了Python中五種不同解析庫的應(yīng)用與提取速度對比,文中的示例代碼講解詳細(xì),感興趣的小伙伴可以跟隨小編一起學(xué)習(xí)一下

一、概述

1.scrapy自帶Css選擇

2.scrapy自帶xpath

3.lxml自帶xpath from lxml import etree

4.bs4自帶美味的湯 from bs4 import BeautifulSoup

5.re 正則表達(dá)式

以這個文本為例，在循環(huán)解析鏈接標(biāo)題時間情況，再循環(huán)499次并打印輸出，約循環(huán)7500次，為了避免誤差，考慮到CPU資源波動，那么Python程序能夠得到的CPU時間就會或多或少，從而導(dǎo)致運行時間變長或變短，所以每種方式運行三次。

當(dāng)CPU爆滿時

運行時間非常慢

當(dāng)CPU充足時，同樣是程序就會非?？?/p>

本程序都在CPU充足情況下運行

text ='''<div class="news_list zy_listbox">
                        <!-- 展示15條數(shù)據(jù)后出現(xiàn)分頁 -->
                        <ul>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2565" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-010號</a><span>2024-07-18</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2564" rel="external nofollow"  rel="external nofollow" >單一來源采購公示</a><span>2024-07-15</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2563" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-009號（第二次）</a><span>2024-07-10</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2562" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-009號</a><span>2024-07-05</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2561" rel="external nofollow"  rel="external nofollow" >等保測評詢價函</a><span>2024-05-28</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2560" rel="external nofollow"  rel="external nofollow" >單一來源采購公示</a><span>2024-05-24</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2559" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-007號（第二次）</a><span>2024-05-20</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2558" rel="external nofollow"  rel="external nofollow" >單一來源采購公示</a><span>2024-05-16</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2557" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-008號</a><span>2024-05-06</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2556" rel="external nofollow"  rel="external nofollow" >詢價函</a><span>2024-04-30</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2555" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-007號</a><span>2024-04-22</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2554" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2023-004號（第二次）</a><span>2024-04-12</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2553" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-006號</a><span>2024-03-29</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2552" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-005號</a><span>2024-03-20</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2551" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-004號</a><span>2024-03-16</span></li>
                            
                        </ul>
                    </div>'''

二、探討CSS選擇器解析速度

t= time.time()
selector = Selector(text=text)
for i in range(1,500):
    for l in selector.css('div.news_list>ul>li'):
        print(l.css('a::text').get())
        print(l.css('span::text').get())
        print(l.css('a::attr(href)').get())
css_parse = time.time()-t
print('css_parse用時：',css_parse)

第一次

第二次

第三次

三、探討scrapy里的xpath解析速度

t= time.time()
selector = Selector(text=text)
for i in range(1,500):
    for l in selector.xpath('//div[@class = "news_list zy_listbox"]/ul/li'):
        print(l.xpath('.//a/text()').get())
        print(l.xpath('.//span/text()').get())
        print(l.xpath('.//a/@href').get())
scrapy_xpath_parse = time.time()-t
print('scrapy_xpath_parse用時：',scrapy_xpath_parse)

第一次

第二次

第三次

四、探討lxml里的xpath解析速度

t= time.time()
selector = etree.HTML(text)
for i in range(1,500):
    for l in selector.xpath('//div[@class = "news_list zy_listbox"]/ul/li'):
        print(l.xpath('.//a/text()')[0])
        print(l.xpath('.//span/text()')[0])
        print(l.xpath('.//a/@href')[0])
xpath_parse = time.time()-t
print('xpath_parse用時：',xpath_parse)

第一次

第二次

第三次

五、探討bs4自帶BeautifulSoup解析速度

t = time.time()
soup = BeautifulSoup(text, 'html.parser')
news_list = soup.select('div.news_list.zy_listbox ul li')
for i in range(1,500):
    for l in news_list:
        print( l.select_one('a').get_text(strip=True))
        print(l.select_one('span').get_text(strip=True))
        print(l.select_one('a')['href'])
bs_parse = time.time() - t
print('bs_parse用時：',bs_parse )

第一次

第二次

第三次

六、探討re正則解析速度

t = time.time()
li = re.findall(' <li><b>·</b>.*?</li>',text)
for l in range(1,500):
    for i in li:
        print(re.search('>(.*?)</a>',i).group(1))
        print(re.search('<span>(.*?)</span>',i).group(1))
        print(re.search('<a href="(.*?)" rel="external nofollow"  rel="external nofollow" >',i).group(1))
re_time = time.time()-t
print('re：',re_time )

第一次

第二次

第三次

七、總結(jié)

經(jīng)上述程序?qū)嶒炞C明，速度排序:

re正則 > lxml xpath > bs4 BeautifulSoup > Scrapy css ≈ Scrapy xpath

在本實驗中，并不嚴(yán)謹(jǐn)，除CPU波動外，只選取了一種特定的HTML網(wǎng)頁進(jìn)行速度對比，沒有考慮到不同類型和結(jié)構(gòu)的網(wǎng)頁對解析速度的影響。例如，網(wǎng)頁中若包含大量嵌套的表單、復(fù)雜的JavaScript生成的動態(tài)內(nèi)容等，甚至有個html藏在動態(tài)接口的json里面，不同解析工具的表現(xiàn)可能會大不相同，這使得實驗結(jié)果存在一定的局限性。實際應(yīng)用中還需要根據(jù)具體的HTML網(wǎng)頁結(jié)構(gòu)、解析需求以及性能指標(biāo)等多方面因素綜合考量，選擇最合適的解析工具。

總程序

from lxml import etree
from scrapy import Selector
from bs4 import BeautifulSoup
import re
import requests
import time
text ='''<div class="news_list zy_listbox">
                        <!-- 展示15條數(shù)據(jù)后出現(xiàn)分頁 -->
                        <ul>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2565" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-010號</a><span>2024-07-18</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2564" rel="external nofollow"  rel="external nofollow" >單一來源采購公示</a><span>2024-07-15</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2563" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-009號（第二次）</a><span>2024-07-10</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2562" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-009號</a><span>2024-07-05</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2561" rel="external nofollow"  rel="external nofollow" >等保測評詢價函</a><span>2024-05-28</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2560" rel="external nofollow"  rel="external nofollow" >單一來源采購公示</a><span>2024-05-24</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2559" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-007號（第二次）</a><span>2024-05-20</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2558" rel="external nofollow"  rel="external nofollow" >單一來源采購公示</a><span>2024-05-16</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2557" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-008號</a><span>2024-05-06</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2556" rel="external nofollow"  rel="external nofollow" >詢價函</a><span>2024-04-30</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2555" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-007號</a><span>2024-04-22</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2554" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2023-004號（第二次）</a><span>2024-04-12</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2553" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-006號</a><span>2024-03-29</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2552" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-005號</a><span>2024-03-20</span></li>
                                                        <li><b>·</b><a href="/index/Article/detail.html?id=2551" rel="external nofollow"  rel="external nofollow" >青海省生態(tài)環(huán)境監(jiān)測中心自行采購2024-004號</a><span>2024-03-16</span></li>
                            
                        </ul>
                    </div>'''
t= time.time()
selector = Selector(text=text)
for i in range(1,500):
    for l in selector.css('div.news_list>ul>li'):
        print(l.css('a::text').get())
        print(l.css('span::text').get())
        print(l.css('a::attr(href)').get())
css_parse = time.time()-t
t= time.time()
selector = Selector(text=text)
for i in range(1,500):
    for l in selector.xpath('//div[@class = "news_list zy_listbox"]/ul/li'):
        print(l.xpath('.//a/text()').get())
        print(l.xpath('.//span/text()').get())
        print(l.xpath('.//a/@href').get())
scrapy_xpath_parse = time.time()-t
t= time.time()
selector = etree.HTML(text)
for i in range(1,500):
    for l in selector.xpath('//div[@class = "news_list zy_listbox"]/ul/li'):
        print(l.xpath('.//a/text()')[0])
        print(l.xpath('.//span/text()')[0])
        print(l.xpath('.//a/@href')[0])
xpath_parse = time.time()-t
t = time.time()
soup = BeautifulSoup(text, 'html.parser')
news_list = soup.select('div.news_list.zy_listbox ul li')
for i in range(1,500):
    for l in news_list:
        print( l.select_one('a').get_text(strip=True))
        print(l.select_one('span').get_text(strip=True))
        print(l.select_one('a')['href'])
bs_parse = time.time() - t
t = time.time()
li = re.findall(' <li><b>·</b>.*?</li>',text)
for l in range(1,500):
    for i in li:
        print(re.search('>(.*?)</a>',i).group(1))
        print(re.search('<span>(.*?)</span>',i).group(1))
        print(re.search('<a href="(.*?)" rel="external nofollow"  rel="external nofollow" >',i).group(1))
re_time = time.time()-t
print('css_parse用時：',css_parse)
print('scrapy_xpath_parse用時：',scrapy_xpath_parse)
print('xpath_parse用時：',xpath_parse)
print('bs_parse用時：',bs_parse )
print('re用時：',re_time )

各時間對比