Python協(xié)程異步爬取數(shù)據(jù)(asyncio+aiohttp)實例

更新時間：2023年08月21日 09:37:55 作者：YiYa_咿呀

這篇文章主要為大家介紹了Python協(xié)程異步爬取數(shù)據(jù)(asyncio+aiohttp)實現(xiàn)示例,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進步,早日升職加薪

使用asyncio+aiohttp異步爬取一部小說

思路

里面涉及到異步文件的讀寫aiofiles,同時在發(fā)送請求時涉及到將字典轉(zhuǎn)化為字符串，接受響應時將字符串轉(zhuǎn)化為字典，故這個里面涉及到json庫,同時在請求下載鏈接的cid和title時使用的是同步獲取一條請求的響應，故其為同步操作，使用requests庫

代碼

import requests
import aiohttp
import asyncio
import json
import aiofiles
# url = 'http://dushu.baidu.com/api/pc/getCatalog?data={%22book_id%22:%224306063500%22}'
# bookid = 'http://dushu.baidu.com/api/pc/getChapterContent?data={%22book_id%22:%224306063500%22,%22cid%22:%224306063500|1569782244%22,%22need_bookinfo%22:1'
async def downloadNovel(cid,title,bid):
    data2 = {
        "book_id": bid,
        "cid": f"{bid}|{cid}",
        "need_bookinfo": 1
    }
    # 將字典轉(zhuǎn)化為字符串
    data2 = json.dumps(data2)
    # 創(chuàng)建請求鏈接
    bookurl = f'http://dushu.baidu.com/api/pc/getChapterContent?data={data2}'
    # 這里老是忘記打括號 aiohttp.ClientSession()
    async with aiohttp.ClientSession() as session:
        async with session.get(bookurl) as resp:
            # 等待結(jié)果返回
            dic = await resp.json()
            # 設置編碼格式encoding='utf-8'
            async with aiofiles.open(f'./articles/{title}',mode = 'w',encoding='utf-8') as f:
                # 異步將內(nèi)容寫入文件
                await f.write(dic['data']['novel']['content'])
async def getCataList(url):
    # 同步爬取所有的章節(jié)相關(guān)信息
    resp = requests.get(url)
    # 將返回的字符轉(zhuǎn)化為字典形式
    dic = resp.json()
    # print(dic)
    # 創(chuàng)建一個空對象用于存儲異步任務
    tasks = []
    # 循環(huán)創(chuàng)建異步任務并且添加至tasks中
    for item in dic['data']['novel']['items']:
        title = item['title']
        cid = item['cid']
        tasks.append(asyncio.create_task(downloadNovel(cid,title,bid)))
        print(title,cid)
        # 執(zhí)行異步任務
    await asyncio.wait(tasks)
if __name__ == '__main__':
    bid = "4306063500"
    url = 'http://dushu.baidu.com/api/pc/getCatalog?data={"book_id":"' + bid + '"}'
    print(url)
    asyncio.run(getCataList(url))