快捷導(dǎo)航

解決Python爬蟲錯(cuò)誤之twisted.web.error.SchemeNotSupported: Unsupported scheme: b''

更新時(shí)間：2024年02月08日 09:50:22 作者：流云淺暮

這篇文章主要介紹了解決Python爬蟲錯(cuò)誤之twisted.web.error.SchemeNotSupported: Unsupported scheme: b''問題,具有很好的參考價(jià)值,希望對大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教

Python爬蟲錯(cuò)誤twisted.web.error.SchemeNotSupported: Unsupported scheme: b''

在使用scrapy爬蟲時(shí)，出現(xiàn)了下面了的錯(cuò)誤

Traceback (most recent call last):
File "E:\project\venv\lib\site-packages\twisted\internet\defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "E:\project\venv\lib\site-packages\twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "E:\project\venv\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
File "E:\project\venv\lib\site-packages\scrapy\utils\defer.py", line 45, in mustbe_deferred
result = f(*args, **kw)
File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 65, in download_request
return handler.download_request(request, spider)
File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 67, in download_request
return agent.download_request(request)
File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 331, in download_request
method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 252, in request
proxyEndpoint = self._getEndpoint(self._proxyURI)
File "E:\project\venv\lib\site-packages\twisted\web\client.py", line 1635, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "E:\project\venv\lib\site-packages\twisted\web\client.py", line 1513, in endpointForURI
raise SchemeNotSupported("Unsupported scheme: %r" % (uri.scheme,))
twisted.web.error.SchemeNotSupported: Unsupported scheme: b''

原因

經(jīng)調(diào)查是在下載中間件中設(shè)置代理是出現(xiàn)了錯(cuò)誤：

class IpProxyDownloadMiddleware(object):
    PROXIES = ['110.52.235.131:9999','110.52.235.249:9999','112.17.38.141:3128']
 
    def process_request(self,request,spider):
        proxy = random.choice(self.PROXIES)
        request.meta['proxy'] = proxy

解決

在上面的下載中間件中設(shè)置代理時(shí)，需要添加協(xié)議名稱，“http://”或者“https://”。

如下：

class IpProxyDownloadMiddleware(object):
    PROXIES = ['110.52.235.131:9999','110.52.235.249:9999','112.17.38.141:3128']
 
    def process_request(self,request,spider):
        proxy = random.choice(self.PROXIES)
        request.meta['proxy'] = 'http://'+proxy

問題成功解決。