Python HTML解析器BeautifulSoup用法實例詳解【爬蟲解析器】

更新時間：2019年04月05日 11:10:14 作者：薔薇Nina

這篇文章主要介紹了Python HTML解析器BeautifulSoup用法,結(jié)合實例形式詳細分析了第三方庫BeautifulSoup實現(xiàn)的爬蟲解析器功能具體操作技巧,需要的朋友可以參考下

本文實例講述了Python HTML解析器BeautifulSoup用法。分享給大家供大家參考，具體如下：

BeautifulSoup簡介

我們知道，Python擁有出色的內(nèi)置HTML解析器模塊——HTMLParser，然而還有一個功能更為強大的HTML或XML解析工具——BeautifulSoup（美味的湯），它是一個第三方庫。簡單來說，BeautifulSoup最主要的功能是從網(wǎng)頁抓取數(shù)據(jù)。本文我們來感受一下BeautifulSoup的優(yōu)雅而強大的功能吧！

BeautifulSoup安裝

BeautifulSoup3 目前已經(jīng)停止開發(fā)，推薦在現(xiàn)在的項目中使用BeautifulSoup4，不過它已經(jīng)被移植到bs4了，也就是說導入時我們需要 import bs4 ?？梢岳?pip 或者 easy_install 兩種方法來安裝。下面采用pip安裝。

pip install beautifulsoup4
pip install lxml

建議同時安裝"lxml"模塊，BeautifulSoup支持Python標準庫中的HTML解析器（HTMLParser），還支持一些第三方的解析器，如果我們不安裝它，則 Python 會使用 Python默認的解析器，lxml 解析器更加強大，速度更快，推薦安裝。

創(chuàng)建對象

安裝后，創(chuàng)建對象：

soup = BeautifulSoup(markup='html文件', 'lxml')

格式化輸出：

soup.prettify()

BeautifulSoup四大對象類型

BeautifulSoup將復(fù)雜HTML文檔轉(zhuǎn)換成一個復(fù)雜的樹形結(jié)構(gòu),每個節(jié)點都是Python對象,所有對象可以歸納為4種:

Tag（標簽）
NavigableString（內(nèi)容）
BeautifulSoup（文檔）
Comment（注釋）

1.Tag類型

即HTML的整個標簽，如獲取<title>標簽：

print soup.title
#<title>The Dormouse's story</title>

Tag有兩個重要屬性：name，attrs。

name

即HTML的標簽名稱：

print soup.name
#[document]
print soup.head.name
#head

attrs

即HTML的標簽屬性字典：

print soup.p.attrs
#{'class': ['title'], 'name': 'dromouse'}

如果想要單獨獲取某個屬性：

print soup.p['class']
#['title']

2.NavigableString類型

既然我們已經(jīng)得到了整個標簽，那么問題來了，我們要想獲取標簽內(nèi)部的文字內(nèi)容怎么辦呢？很簡單，用 string 即可：

print soup.p.string
#The Dormouse's story

3.BeautifulSoup類型

BeautifulSoup 對象表示的是一個文檔的全部內(nèi)容.：

print soup.name
# [document]

4.Comment類型

HTML的注釋內(nèi)容，注意的是，不包含注釋符號。我們首先判斷它的類型，是否為 Comment 類型，然后再進行其他操作，如打印輸出：

if type(soup.a.string)==bs4.element.Comment:
  print soup.a.string
#<!-- Elsie -->

遍歷文檔樹

1.子節(jié)點

contents

獲取所有子節(jié)點，返回列表：

print soup.head.contents
#[<title>The Dormouse's story</title>]

children

獲取所有子節(jié)點，返回列表生成器：

print soup.head.children
#<listiterator object at 0x7f71457f5710>
## 需要遍歷
for child in soup.body.children:
  print child
## 結(jié)果
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>,
<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a> and
<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>

2.節(jié)點內(nèi)容

string

返回單個文本內(nèi)容。如果一個標簽里面沒有標簽了，那么 string 就會返回標簽里面的內(nèi)容。如果標簽里面只有唯一的一個標簽了，那么 string 也會返回最里面的內(nèi)容。如果tag包含了多個子節(jié)點,tag就無法確定，string 方法應(yīng)該調(diào)用哪個子節(jié)點的內(nèi)容，string 的輸出結(jié)果是 None。例如：

print soup.head.string
print soup.title.string
#The Dormouse's story
#The Dormouse's story
print soup.html.string
# None

strings

返回多個文本內(nèi)容，且包含空行和空格。

stripped_strings

返回多個文本內(nèi)容，且不包含空行和空格：

for string in soup.stripped_strings:
  print(repr(string))
  # u"The Dormouse's story"
  # u"The Dormouse's story"
  # u'Once upon a time there were three little sisters; and their names were'
  # u'Elsie'
  # u','
  # u'Lacie'
  # u'and'
  # u'Tillie'
  # u';\nand they lived at the bottom of a well.'
  # u'...'

get_text()方法

返回當前節(jié)點和子節(jié)點的文本內(nèi)容。

from bs4 import BeautifulSoup
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
  <p class="title"><b>The Dormouse's story</b></p>
  <p class="story">Once upon a time there were three little sisters; and their names were
    <a  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister1" id="link1">Elsie</a>,
    <a  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister2" id="link2">Lacie</a> and
    <a  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister3" id="link3">Tillie</a>;
    and they lived at the bottom of a well.
  </p>
  <p class="story">...</p>
</body>
</html>
"""
soup = BeautifulSoup(markup=html_doc,features='lxml')
node_p_text=soup.find('p',class_='story').get_text()　　　　# 注意class_帶下劃線
print(node_p_text)
# 結(jié)果
Once upon a time there were three little sisters; and their names were
    Elsie,
    Lacie and
    Tillie;
    and they lived at the bottom of a well.

3.父節(jié)點

parent

返回某節(jié)點的直接父節(jié)點：

p = soup.p
print p.parent.name
#body

parents

返回某節(jié)點的所有父輩及以上輩的節(jié)點：

content = soup.head.title.string
for parent in content.parents:
  print parent.name
## 結(jié)果
title
head
html
[document]

4.兄弟節(jié)點

next_sibling

next_sibling 屬性獲取該節(jié)點的下一個兄弟節(jié)點，結(jié)果通常是字符串或空白，因為空白或者換行也可以被視作一個節(jié)點。

previous_sibling

previous_sibling 屬性獲取該節(jié)點的上一個兄弟節(jié)點。

print soup.p.next_sibling
#    實際該處為空白
print soup.p.prev_sibling
#None  沒有前一個兄弟節(jié)點，返回 None
print soup.p.next_sibling.next_sibling
#<p class="story">Once upon a time there were three little sisters; and their names were
#<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>,
#<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a> and
#<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>;
#and they lived at the bottom of a well.</p>
#下一個節(jié)點的下一個兄弟節(jié)點是我們可以看到的節(jié)點

next_siblings、previous_siblings

迭代獲取全部兄弟節(jié)點。

5.前后節(jié)點

next_element、previous_element

不是針對于兄弟節(jié)點，而是在于所有節(jié)點，不分層次的前一個和后一個節(jié)點。

next_elements、previous_elements

迭代獲取所有前和后節(jié)點。

搜索文檔樹

1.find_all(name=None, attrs={}, recursive=True, text=None, limit=None, **kwargs)

find_all()方法搜索當前tag的所有tag子節(jié)點，并判斷是否符合過濾器的條件。

參數(shù)說明

name參數(shù)

name參數(shù)很強大，可以傳多種方式的參數(shù)，查找所有名字為 name 的tag，字符串對象會被自動忽略掉。

（a）傳標簽名

最簡單的過濾器是標簽名。在搜索方法中傳入一個標簽名參數(shù)，BeautifulSoup會查找與標簽名完整匹配的內(nèi)容，下面的例子用于查找文檔中所有的<a>標簽：

print soup.find_all('a')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

返回結(jié)果列表中的元素仍然是BeautifulSoup對象。

（b）傳正則表達式

如果傳入正則表達式作為參數(shù)，BeautifulSoup會通過正則表達式的 match() 來匹配內(nèi)容。下面例子中找出所有以b開頭的標簽,這表示<body>和<b>標簽都應(yīng)該被找到：

import re
for tag in soup.find_all(re.compile("^b")):
  print(tag.name)
# body
# b

（c）傳列表

如果傳入列表參數(shù)，BeautifulSoup會將與列表中任一元素匹配的內(nèi)容返回。下面代碼找到文檔中所有<a>標簽和<b>標簽：

soup.find_all(["a", "b"])
# [<b>The Dormouse's story</b>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

（d）傳True

True 可以匹配任何值，下面代碼查找到所有的tag，但是不會返回字符串節(jié)點：

for tag in soup.find_all(True):
  print(tag.name)
# html
# head
# title
# body
# p
# b
# p
# a
# a

（e）傳函數(shù)

如果沒有合適過濾器，那么還可以定義一個方法，方法只接受一個元素參數(shù)。如果這個方法返回 True 表示當前元素匹配并且被找到，如果不是則反回 False：

def has_class_but_no_id(tag):
  return tag.has_attr('class') and not tag.has_attr('id')
soup.find_all(has_class_but_no_id)
# [<p class="title"><b>The Dormouse's story</b></p>,
# <p class="story">Once upon a time there were...</p>,
# <p class="story">...</p>]

keyword參數(shù)

注意的是，如果一個指定名字的參數(shù)不是搜索內(nèi)置的參數(shù)名，搜索時會把該參數(shù)當作指定名字tag的屬性來搜索，如果包含一個名字為 id 的參數(shù),BeautifulSoup會搜索每個tag的”id”屬性：

soup.find_all(id='link2')
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>]

如果傳入 href 參數(shù)，Beautiful Soup會搜索每個tag的"href"屬性：

soup.find_all(href=re.compile("elsie"))
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>]

使用多個指定名字的參數(shù)可以同時過濾tag的多個屬性:

soup.find_all(href=re.compile("elsie"), id='link1')
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">three</a>]

在這里我們想用 class 過濾，不過 class 是 python 的關(guān)鍵詞，這怎么辦？加個下劃線就可以：

soup.find_all("a", class_="sister")
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

attrs參數(shù)

有些tag屬性在搜索不能使用,比如HTML5中的 " data-* " 自定義屬性：

data_soup = BeautifulSoup('<div data-foo="value">foo!</div>')
data_soup.find_all(data-foo="value")
# SyntaxError: keyword can't be an expression
## 但是可以通過 find_all() 方法的 attrs 參數(shù)定義一個字典參數(shù)來搜索包含特殊屬性的tag
data_soup.find_all(attrs={"data-foo": "value"})
# [<div data-foo="value">foo!</div>]

text參數(shù)

通過 text 參數(shù)可以搜搜文檔中的字符串內(nèi)容。與 name 參數(shù)的可選值一樣，text 參數(shù)接受字符串、正則表達式、列表、True。

soup.find_all(text="Elsie")
# [u'Elsie']
soup.find_all(text=["Tillie", "Elsie", "Lacie"])
# [u'Elsie', u'Lacie', u'Tillie']
soup.find_all(text=re.compile("Dormouse"))　　# 模糊查找
[u"The Dormouse's story", u"The Dormouse's story"]

limit參數(shù)

find_all() 方法返回全部的搜索結(jié)構(gòu)，如果文檔樹很大那么搜索會很慢。如果我們不需要全部結(jié)果，可以使用 limit 參數(shù)限制返回結(jié)果的數(shù)量。效果與SQL中的limit關(guān)鍵字類似，當搜索到的結(jié)果數(shù)量達到 limit 的限制時，就停止搜索返回結(jié)果。

soup.find_all("a", limit=2)
# [<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
# <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>]

recursive參數(shù)

調(diào)用tag的 find_all() 方法時，BeautifulSoup會檢索當前tag的所有子孫節(jié)點，如果只想搜索tag的直接子節(jié)點，可以使用參數(shù) recursive=False。

soup.html.find_all("title")
# [<title>The Dormouse's story</title>]
soup.html.find_all("title", recursive=False)
# []

2.find( name , attrs , recursive , text , **kwargs )

它與 find_all() 方法唯一的區(qū)別是 find_all() 方法的返回結(jié)果是值包含一個元素的列表，而 find() 方法直接返回結(jié)果。

3.find_parents() 和 find_parent()

find_all() 和 find() 只搜索當前節(jié)點的所有子節(jié)點，孫子節(jié)點等。find_parents() 和 find_parent() 用來搜索當前節(jié)點的父輩節(jié)點，搜索方法與普通tag的搜索方法相同，搜索文檔搜索文檔包含的內(nèi)容。

4.find_next_siblings() 和 find_next_sibling()　　

這2個方法通過 .next_siblings 屬性對當 tag 的所有后面解析的兄弟 tag 節(jié)點進行迭代， find_next_siblings() 方法返回所有符合條件的后面的兄弟節(jié)點，find_next_sibling() 只返回符合條件的后面的第一個tag節(jié)點。

5.find_previous_siblings() 和 find_previous_sibling()

這2個方法通過 .previous_siblings 屬性對當前 tag 的前面解析的兄弟 tag 節(jié)點進行迭代， find_previous_siblings() 方法返回所有符合條件的前面的兄弟節(jié)點，find_previous_sibling() 方法返回第一個符合條件的前面的兄弟節(jié)點。

6.find_all_next() 和 find_next()

這2個方法通過 .next_elements 屬性對當前 tag 的之后的 tag 和字符串進行迭代， find_all_next() 方法返回所有符合條件的節(jié)點， find_next() 方法返回第一個符合條件的節(jié)點。

7.find_all_previous() 和 find_previous()

這2個方法通過 .previous_elements 屬性對當前節(jié)點前面的 tag 和字符串進行迭代，find_all_previous() 方法返回所有符合條件的節(jié)點， find_previous()方法返回第一個符合條件的節(jié)點。

CSS選擇器

我們在寫 CSS 時，標簽名不加任何修飾，類名前加點，id名前加 #，在這里我們也可以利用類似的方法來篩選元素，用到的方法是 soup.select()，返回類型是 list。

通過標簽名查找

print soup.select('title')
#[<title>The Dormouse's story</title>]
print soup.select('a')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]
print soup.select('b')
#[<b>The Dormouse's story</b>]

通過類名查找

print soup.select('.sister')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

通過 id 名查找

print soup.select('#link1')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

組合查找

組合查找即和寫 class 文件時，標簽名與類名、id名進行的組合原理是一樣的，例如查找 p 標簽中，id 等于 link1的內(nèi)容，二者需要用空格分開。

print soup.select('p #link1')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

直接子標簽查找：

print soup.select("head > title")
#[<title>The Dormouse's story</title>]

屬性查找

查找時還可以加入屬性元素，屬性需要用中括號括起來，注意屬性和標簽屬于同一節(jié)點，所以中間不能加空格，否則會無法匹配到。

print soup.select('a[class="sister"]')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]
print soup.select('a[ rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ]')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

同樣，屬性仍然可以與上述查找方式組合，不在同一節(jié)點的空格隔開，同一節(jié)點的不加空格：

print soup.select('p a[ rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ]')
#[<a class="sister"  rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1"><!-- Elsie --></a>]

以上的 select 方法返回的結(jié)果都是列表形式，可以遍歷形式輸出，然后用 string或get_text() 方法來獲取它的內(nèi)容：

soup = BeautifulSoup(html, 'lxml')
print type(soup.select('title'))
print soup.select('title')[0].get_text()
for title in soup.select('title'):
  print title.get_text()

更多關(guān)于Python相關(guān)內(nèi)容可查看本站專題：《Python Socket編程技巧總結(jié)》、《Python正則表達式用法總結(jié)》、《Python數(shù)據(jù)結(jié)構(gòu)與算法教程》、《Python函數(shù)使用技巧總結(jié)》、《Python字符串操作技巧匯總》、《Python入門與進階經(jīng)典教程》及《Python文件與目錄操作技巧匯總》

希望本文所述對大家Python程序設(shè)計有所幫助。

您可能感興趣的文章: