Python實(shí)現(xiàn)多線程下載文件的代碼實(shí)例
實(shí)現(xiàn)簡單的多線程下載,需要關(guān)注如下幾點(diǎn):
1.文件的大?。嚎梢詮膔eponse header中提取,如“Content-Length:911”表示大小是911字節(jié)
2.任務(wù)拆分:指定各個線程下載的文件的哪一塊,可以通過request header中添加“Range: bytes=300-400”(表示下載300~400byte的內(nèi)容),注意可以請求的文件的range是[0, size-1]字節(jié)的。
3.下載文件的聚合:各個線程將自己下載的文件塊保存為臨時文件,所有線程都完成后,再將這些臨時文件按順序聚合寫入到最終的一個文件中。
實(shí)現(xiàn)代碼:
#!/usr/bin/python
# -*- coding: utf-8 -*-
# filename: paxel.py
# FROM: http://jb51.net/code/view/58/full/
# Jay modified it a little and save for further potential usage.
'''It is a multi-thread downloading tool
It was developed following axel.
Author: volans
E-mail: volansw [at] gmail.com
'''
import sys
import os
import time
import urllib
from threading import Thread
# in case you want to use http_proxy
local_proxies = {'http': 'http://131.139.58.200:8080'}
class AxelPython(Thread, urllib.FancyURLopener):
'''Multi-thread downloading class.
run() is a vitural method of Thread.
'''
def __init__(self, threadname, url, filename, ranges=0, proxies={}):
Thread.__init__(self, name=threadname)
urllib.FancyURLopener.__init__(self, proxies)
self.name = threadname
self.url = url
self.filename = filename
self.ranges = ranges
self.downloaded = 0
def run(self):
'''vertual function in Thread'''
try:
self.downloaded = os.path.getsize(self.filename)
except OSError:
#print 'never downloaded'
self.downloaded = 0
# rebuild start poind
self.startpoint = self.ranges[0] + self.downloaded
# This part is completed
if self.startpoint >= self.ranges[1]:
print 'Part %s has been downloaded over.' % self.filename
return
self.oneTimeSize = 16384 # 16kByte/time
print 'task %s will download from %d to %d' % (self.name, self.startpoint, self.ranges[1])
self.addheader("Range", "bytes=%d-%d" % (self.startpoint, self.ranges[1]))
self.urlhandle = self.open(self.url)
data = self.urlhandle.read(self.oneTimeSize)
while data:
filehandle = open(self.filename, 'ab+')
filehandle.write(data)
filehandle.close()
self.downloaded += len(data)
#print "%s" % (self.name)
#progress = u'\r...'
data = self.urlhandle.read(self.oneTimeSize)
def GetUrlFileSize(url, proxies={}):
urlHandler = urllib.urlopen(url, proxies=proxies)
headers = urlHandler.info().headers
length = 0
for header in headers:
if header.find('Length') != -1:
length = header.split(':')[-1].strip()
length = int(length)
return length
def SpliteBlocks(totalsize, blocknumber):
blocksize = totalsize / blocknumber
ranges = []
for i in range(0, blocknumber - 1):
ranges.append((i * blocksize, i * blocksize + blocksize - 1))
ranges.append((blocksize * (blocknumber - 1), totalsize - 1))
return ranges
def islive(tasks):
for task in tasks:
if task.isAlive():
return True
return False
def paxel(url, output, blocks=6, proxies=local_proxies):
''' paxel
'''
size = GetUrlFileSize(url, proxies)
ranges = SpliteBlocks(size, blocks)
threadname = ["thread_%d" % i for i in range(0, blocks)]
filename = ["tmpfile_%d" % i for i in range(0, blocks)]
tasks = []
for i in range(0, blocks):
task = AxelPython(threadname[i], url, filename[i], ranges[i])
task.setDaemon(True)
task.start()
tasks.append(task)
time.sleep(2)
while islive(tasks):
downloaded = sum([task.downloaded for task in tasks])
process = downloaded / float(size) * 100
show = u'\rFilesize:%d Downloaded:%d Completed:%.2f%%' % (size, downloaded, process)
sys.stdout.write(show)
sys.stdout.flush()
time.sleep(0.5)
filehandle = open(output, 'wb+')
for i in filename:
f = open(i, 'rb')
filehandle.write(f.read())
f.close()
try:
os.remove(i)
pass
except:
pass
filehandle.close()
if __name__ == '__main__':
url = 'http://dldir1.qq.com/qqfile/QQforMac/QQ_V3.1.1.dmg'
output = 'download.file'
paxel(url, output, blocks=4, proxies={})
- python實(shí)現(xiàn)單線程多任務(wù)非阻塞TCP服務(wù)端
- Python實(shí)現(xiàn)簡單的多任務(wù)mysql轉(zhuǎn)xml的方法
- python多任務(wù)及返回值的處理方法
- python多任務(wù)之協(xié)程的使用詳解
- Python協(xié)程操作之gevent(yield阻塞,greenlet),協(xié)程實(shí)現(xiàn)多任務(wù)(有規(guī)律的交替協(xié)作執(zhí)行)用法詳解
- python實(shí)現(xiàn)通過隊(duì)列完成進(jìn)程間的多任務(wù)功能示例
- 對Python多線程讀寫文件加鎖的實(shí)例詳解
- Python多線程下載文件的方法
- Python多線程同步---文件讀寫控制方法
- python 多線程將大文件分開下載后在合并的實(shí)例
- python開發(fā)之基于thread線程搜索本地文件的方法
- python多線程案例之多任務(wù)copy文件完整實(shí)例
相關(guān)文章
解讀調(diào)用jupyter?notebook文件內(nèi)的函數(shù)一種簡單方法
這篇文章主要介紹了解讀調(diào)用jupyter?notebook文件內(nèi)的函數(shù)一種簡單方法,具有很好的參考價值,希望對大家有所幫助。如有錯誤或未考慮完全的地方,望不吝賜教2023-01-01Python+PyQt5實(shí)現(xiàn)美劇爬蟲可視工具的方法
這篇文章主要介紹了Python+PyQt5實(shí)現(xiàn)美劇爬蟲可視工具的方法,文中通過示例代碼介紹的非常詳細(xì),對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價值,需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2019-04-04Python 過濾字符串的技巧,map與itertools.imap
Python中的map函數(shù)非常有用,在字符轉(zhuǎn)換和字符遍歷兩節(jié)都出現(xiàn)過,現(xiàn)在,它又出現(xiàn)了,會給我們帶來什么樣的驚喜呢?是不是要告訴我們,map是非常棒的,以后要多找它玩呢?2008-09-09Python隨機(jī)數(shù)種子(random seed)的使用
在科學(xué)技術(shù)和機(jī)器學(xué)習(xí)等其他算法相關(guān)任務(wù)中,我們經(jīng)常需要用到隨機(jī)數(shù),本文就詳細(xì)的介紹一下Python隨機(jī)數(shù)種子,具有一定的參考價值,感興趣的小伙伴們可以參考一下2021-07-07python 通過麥克風(fēng)錄音 生成wav文件的方法
今天小編就為大家分享一篇python 通過麥克風(fēng)錄音 生成wav文件的方法,具有很好的參考價值,希望對大家有所幫助。一起跟隨小編過來看看吧2019-01-01基于Django contrib Comments 評論模塊(詳解)
下面小編就為大家分享一篇基于Django contrib Comments 評論模塊(詳解),具有很好的參考價值,希望對大家有所幫助。一起跟隨小編過來看看吧2017-12-12