scrapy爬蟲:scrapy.FormRequest中formdata參數(shù)詳解

更新時間：2020年04月30日 14:34:14 作者：Kosmoo

這篇文章主要介紹了scrapy爬蟲:scrapy.FormRequest中formdata參數(shù)詳解，具有很好的參考價值，希望對大家有所幫助。一起跟隨小編過來看看吧

1. 背景

在網(wǎng)頁爬取的時候，有時候會使用scrapy.FormRequest向目標(biāo)網(wǎng)站提交數(shù)據(jù)（表單提交）。參照scrapy官方文檔的標(biāo)準(zhǔn)寫法是：

# header信息
unicornHeader = {
  'Host': 'www.example.com',
  'Referer': 'http://www.example.com/',
}

# 表單需要提交的數(shù)據(jù)
myFormData = {'name': 'John Doe', 'age': '27'}

# 自定義信息，向下層響應(yīng)(response)傳遞下去
customerData = {'key1': 'value1', 'key2': 'value2'}

yield scrapy.FormRequest(url = "http://www.example.com/post/action",
             headers = unicornHeader,
             method = 'POST',       # GET or POST
             formdata = myFormData,    # 表單提交的數(shù)據(jù)
             meta = customerData,    # 自定義，向response傳遞數(shù)據(jù)
             callback = self.after_post,
             errback = self.error_handle,
             # 如果需要多次提交表單，且url一樣，那么就必須加此參數(shù)dont_filter，防止被當(dāng)成重復(fù)網(wǎng)頁過濾掉了
             dont_filter = True   
             )

但是，當(dāng)表單提交數(shù)據(jù)myFormData 是形如字典內(nèi)嵌字典的形式，又該如何寫？

2. 案例 — 參數(shù)為字典

在做亞馬遜網(wǎng)站爬取時，當(dāng)進(jìn)入商家店鋪，爬取店鋪內(nèi)商品列表時，發(fā)現(xiàn)采取的方式是ajax請求，返回的是json數(shù)據(jù)。

請求信息如下：

響應(yīng)信息如下：

如上圖所示，F(xiàn)rom Data中的數(shù)據(jù)包含一個字典：

marketplaceID:ATVPDKIKX0DER
seller:A2FE6D62A4WM6Q
productSearchRequestData:{"marketplace":"ATVPDKIKX0DER","seller":"A2FE6D62A4WM6Q","url":"/sp/ajax/products","pageSize":12,"searchKeyword":"","extraRestrictions":{},"pageNumber":"1"}

# formDate 必須構(gòu)造如下：
myFormData = {
  'marketplaceID' : 'ATVPDKIKX0DER',
  'seller' : 'A2FE6D62A4WM6Q',
  # 注意下面這一行，內(nèi)部字典是作為一個字符串的形式
  'productSearchRequestData' :'{"marketplace":"ATVPDKIKX0DER","seller":"A2FE6D62A4WM6Q","url":"/sp/ajax/products","pageSize":12,"searchKeyword":"","extraRestrictions":{},"pageNumber":"1"}'
}

在amazon中實際使用的構(gòu)造方法如下：

def sendRequestForProducts(response):
  ajaxParam = response.meta
  for pageIdx in range(1, ajaxParam['totalPageNum']+1):
    ajaxParam['isFirstAjax'] = False
    ajaxParam['pageNumber'] = pageIdx

    unicornHeader = {
      'Host': 'www.amazon.com',
      'Origin': 'https://www.amazon.com',
      'Referer': ajaxParam['referUrl'],
    }

    '''
    marketplaceID:ATVPDKIKX0DER
    seller:AYZQAQRQKEXRP
    productSearchRequestData:{"marketplace":"ATVPDKIKX0DER","seller":"AYZQAQRQKEXRP","url":"/sp/ajax/products","pageSize":12,"searchKeyword":"","extraRestrictions":{},"pageNumber":1}
    '''

    productSearchRequestData = '{"marketplace": "ATVPDKIKX0DER", "seller": "' + f'{ajaxParam["sellerID"]}' + '","url": "/sp/ajax/products", "pageSize": 12, "searchKeyword": "","extraRestrictions": {}, "pageNumber": "' + str(pageIdx) + '"}'
    formdataProduct = {
      'marketplaceID': ajaxParam['marketplaceID'],
      'seller': ajaxParam['sellerID'],
      'productSearchRequestData': productSearchRequestData
    }
    productAjaxMeta = ajaxParam
    # 請求店鋪商品列表
    yield scrapy.FormRequest(
      url = 'https://www.amazon.com/sp/ajax/products',
      headers = unicornHeader,
      formdata = formdataProduct,
      func = 'POST',
      meta = productAjaxMeta,
      callback = self.solderProductAjax,
      errback = self.error, # 處理http error
      dont_filter = True,  # 需要加此參數(shù)的
    )

3. 原理分析

舉例來說，目前有如下一筆數(shù)據(jù)：

formdata = {
      'Field': {"pageIdx":99, "size":"10"},
      'func': 'nextPage',
      }

從網(wǎng)頁上，可以看到請求數(shù)據(jù)如下：

Field=%7B%22pageIdx%22%3A99%2C%22size%22%3A%2210%22%7D&func=nextPage

第一種，按照如下方式發(fā)出請求，結(jié)果如下（正確）：

yield scrapy.FormRequest(
      url = 'https://www.example.com/sp/ajax',
      headers = unicornHeader,
      formdata = {
            'Field': '{"pageIdx":99, "size":"10"}',
            'func': 'nextPage',
            },
      func = 'POST',
      callback = self.handleFunc,
    )
# 請求數(shù)據(jù)為：Field=%7B%22pageIdx%22%3A99%2C%22size%22%3A%2210%22%7D&func=nextPage

第二種，按照如下方式發(fā)出請求，結(jié)果如下（錯誤，無法獲取到正確的數(shù)據(jù)）：

yield scrapy.FormRequest(
      url = 'https://www.example.com/sp/ajax',
      headers = unicornHeader,
      formdata = {
            'Field': {"pageIdx":99, "size":"10"},
            'func': 'nextPage',
            },
      func = 'POST',
      callback = self.handleFunc,
    )
# 經(jīng)過錯誤的編碼之后，發(fā)送的請求為：Field=size&Field=pageIdx&func=nextPage

我們跟蹤看一下scrapy中的源碼：

# E:/Miniconda/Lib/site-packages/scrapy/http/request/form.py

# FormRequest
class FormRequest(Request):

  def __init__(self, *args, **kwargs):
    formdata = kwargs.pop('formdata', None)
    if formdata and kwargs.get('func') is None:
      kwargs['func'] = 'POST'

    super(FormRequest, self).__init__(*args, **kwargs)

    if formdata:
      items = formdata.items() if isinstance(formdata, dict) else formdata
      querystr = _urlencode(items, self.encoding)
      if self.func == 'POST':
        self.headers.setdefault(b'Content-Type', b'application/x-www-form-urlencoded')
        self._set_body(querystr)
      else:
        self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)

  # 關(guān)鍵函數(shù) _urlencode
  def _urlencode(seq, enc):
    values = [(to_bytes(k, enc), to_bytes(v, enc))
         for k, vs in seq
         for v in (vs if is_listlike(vs) else [vs])]
    return urlencode(values, doseq=1)

分析過程如下：

# 第一步：items = formdata.items() if isinstance(formdata, dict) else formdata
# 第一步結(jié)果：經(jīng)過items()方法執(zhí)行后，原始的dict格式變成如下列表形式：
   dict_items([('func', 'nextPage'), ('Field', {'size': '10', 'pageIdx': 99})])

# 第二步：再經(jīng)過后面的 _urlencode方法將items轉(zhuǎn)換成如下：
  [(b'func', b'nextPage'), (b'Field', b'size'), (b'Field', b'pageIdx')]

# 可以看到就是在調(diào)用 _urlencode方法的時候出現(xiàn)了問題，上面的方法執(zhí)行過后，會使字典形式的數(shù)據(jù)只保留了keys（value是字典的情況下，只保留了value字典中的key）.

解決方案： 就是將字典當(dāng)成普通的字符串，然后編碼（轉(zhuǎn)換成bytes），進(jìn)行傳輸，到達(dá)服務(wù)器端之后，服務(wù)器會反過來進(jìn)行解碼，得到這個字典字符串。然后服務(wù)器按照Dict進(jìn)行解析。

拓展：對于其他特殊類型的數(shù)據(jù)，都按照這種方式打包成字符串進(jìn)行傳遞。

4. 補(bǔ)充1 ——參數(shù)類型

formdata的參數(shù)值必須是unicode , str 或者 bytes object，不能是整數(shù)。

案例：

yield FormRequest(
  url = 'https://www.amztracker.com/unicorn.php',
  headers = unicornHeader,
  # formdata 的參數(shù)必須是字符串
  formdata={'rank': 10, 'category': productDetailInfo['topCategory']},
  method = 'GET',
  meta = {'productDetailInfo': productDetailInfo},
  callback = self.amztrackerSale,
  errback = self.error, # 本項目中這里觸發(fā)errback占絕大多數(shù)
  dont_filter = True, # 按理來說是不需要加此參數(shù)的
)

# 提示如下ERROR：
Traceback (most recent call last):
 File "E:\Miniconda\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
  yield next(it)
 File "E:\Miniconda\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
  for x in result:
 File "E:\Miniconda\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
  return (_set_referer(r) for r in result or ())
 File "E:\Miniconda\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
  return (r for r in result or () if _filter(r))
 File "E:\Miniconda\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
  return (r for r in result or () if _filter(r))
 File "E:\PyCharmCode\categorySelectorAmazon1\categorySelectorAmazon1\spiders\categorySelectorAmazon1Clawer.py", line 224, in parseProductDetail
  dont_filter = True,
 File "E:\Miniconda\lib\site-packages\scrapy\http\request\form.py", line 31, in __init__
  querystr = _urlencode(items, self.encoding)
 File "E:\Miniconda\lib\site-packages\scrapy\http\request\form.py", line 66, in _urlencode
  for k, vs in seq
 File "E:\Miniconda\lib\site-packages\scrapy\http\request\form.py", line 67, in <listcomp>
  for v in (vs if is_listlike(vs) else [vs])]
 File "E:\Miniconda\lib\site-packages\scrapy\utils\python.py", line 117, in to_bytes
  'object, got %s' % type(text).__name__)
TypeError: to_bytes must receive a unicode, str or bytes object, got int

# 正確寫法：
formdata = {'rank': str(productDetailInfo['topRank']), 'category': productDetailInfo['topCategory']},

原理部分（源代碼）：

# 第一階段: 字典分解為items
    if formdata:
      items = formdata.items() if isinstance(formdata, dict) else formdata
      querystr = _urlencode(items, self.encoding)

# 第二階段: 對value，調(diào)用 to_bytes 編碼
def _urlencode(seq, enc):
  values = [(to_bytes(k, enc), to_bytes(v, enc))
       for k, vs in seq
       for v in (vs if is_listlike(vs) else [vs])]
  return urlencode(values, doseq=1)

# 第三階段: 執(zhí)行 to_bytes ，參數(shù)要求是bytes, str
def to_bytes(text, encoding=None, errors='strict'):
  """Return the binary representation of `text`. If `text`
  is already a bytes object, return it as-is."""
  if isinstance(text, bytes):
    return text
  if not isinstance(text, six.string_types):
    raise TypeError('to_bytes must receive a unicode, str or bytes '
            'object, got %s' % type(text).__name__)

5. 補(bǔ)充2 ——參數(shù)為中文

formdata的參數(shù)值必須是unicode , str 或者 bytes object，不能是整數(shù)。

以1688網(wǎng)站搜索產(chǎn)品為案例：

搜索信息如下（搜索關(guān)鍵詞為：動漫周邊）：

可以看到動漫周邊 == %B6%AF%C2%FE%D6%DC%B1%DF

# scrapy中這個請求的構(gòu)造如下

# python3 所有的字符串都是unicode
unicornHeaders = {
  ':authority': 's.1688.com',
  'Referer': 'https://www.1688.com/',
}

# python3 所有的字符串都是unicode
# 動漫周邊 tobyte為：%B6%AF%C2%FE%D6%DC%B1%DF
formatStr = "動漫周邊".encode('gbk')

print(f"formatStr = {formatStr}")
yield FormRequest(
  url = 'https://s.1688.com/selloffer/offer_search.htm',
  headers = unicornHeaders,
  formdata = {'keywords': formatStr, 'n': 'y', 'spm': 'a260k.635.1998096057.d1'},
  method = 'GET',
  meta={},
  callback = self.parseCategoryPage,
  errback = self.error, # 本項目中這里觸發(fā)errback占絕大多數(shù)
  dont_filter = True, # 按理來說是不需要加此參數(shù)的
)

# 日志如下：
formatStr = b'\xb6\xaf\xc2\xfe\xd6\xdc\xb1\xdf'
2017-11-16 15:11:02 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://sec.1688.com/query.htm?smApp=searchweb2&smPolicy=searchweb2-selloffer-anti_Spider-seo-html-checklogin&smCharset=GBK&smTag=MTE1LjIxNi4xNjAuNDYsLDU5OWQ1NWIyZTk0NDQ1Y2E5ZDAzODRlOGM1MDI2OTZj&smReturn=https%3A%2F%2Fs.1688.com%2Fselloffer%2Foffer_search.htm%3Fkeywords%3D%25B6%25AF%25C2%25FE%25D6%25DC%25B1%25DF%26n%3Dy%26spm%3Da260k.635.1998096057.d1&smSign=05U0%2BJXfKLQmSbsnce55Yw%3D%3D> from <GET https://s.1688.com/selloffer/offer_search.htm?keywords=%B6%AF%C2%FE%D6%DC%B1%DF&n=y&spm=a260k.635.1998096057.d1>
# https://s.1688.com/selloffer/offer_search.htm?keywords=%B6%AF%C2%FE%D6%DC%B1%DF&n=y&spm=a260k.635.1998096057.d1

以上這篇scrapy爬蟲:scrapy.FormRequest中formdata參數(shù)詳解就是小編分享給大家的全部內(nèi)容了，希望能給大家一個參考，也希望大家多多支持腳本之家。

您可能感興趣的文章: