python中threading和queue庫實(shí)現(xiàn)多線程編程
摘要
本文主要介紹了利用python的 threading和queue庫實(shí)現(xiàn)多線程編程,并封裝為一個(gè)類,方便讀者嵌入自己的業(yè)務(wù)邏輯。最后以機(jī)器學(xué)習(xí)的一個(gè)超參數(shù)選擇為例進(jìn)行演示。
多線程實(shí)現(xiàn)邏輯封裝
實(shí)例化該類后,在.object_func函數(shù)中加入自己的業(yè)務(wù)邏輯,再調(diào)用.run方法即可。
# -*- coding: utf-8 -*- # @Time : 2021/2/4 14:36 # @Author : CyrusMay WJ # @FileName: run.py # @Software: PyCharm # @Blog :https://blog.csdn.net/Cyrus_May import queue import threading class CyrusThread(object): def __init__(self,num_thread = 10,logger=None): """ :param num_thread: 線程數(shù) :param logger: 日志對象 """ self.num_thread = num_thread self.logger = logger def object_func(self,args_queue,max_q): while 1: try: arg = args_queue.get_nowait() step = args_queue.qsize() self.logger.info("progress:{}\{}".format(max_q,step)) except: self.logger.info("no more arg for args_queue!") break """ 此處加入自己的業(yè)務(wù)邏輯代碼 """ def run(self,args): args_queue = queue.Queue() for value in args: args_queue.put(value) threads = [] for i in range(self.num_thread): threads.append(threading.Thread(target=self.object_func,args = args_queue)) for t in threads: t.start() for t in threads: t.join()
模型參數(shù)選擇實(shí)例
# -*- coding: utf-8 -*- # @Time : 2021/2/4 14:36 # @Author : CyrusMay WJ # @FileName: run.py # @Software: PyCharm # @Blog :https://blog.csdn.net/Cyrus_May import queue import threading import numpy as np from sklearn.datasets import load_boston from sklearn.svm import SVR import logging import sys class CyrusThread(object): def __init__(self,num_thread = 10,logger=None): """ :param num_thread: 線程數(shù) :param logger: 日志對象 """ self.num_thread = num_thread self.logger = logger def object_func(self,args_queue,max_q): while 1: try: arg = args_queue.get_nowait() step = args_queue.qsize() self.logger.info("progress:{}\{}".format(max_q,max_q-step)) except: self.logger.info("no more arg for args_queue!") break # 業(yè)務(wù)代碼 C, epsilon, gamma = arg[0], arg[1], arg[2] svr_model = SVR(C=C, epsilon=epsilon, gamma=gamma) x, y = load_boston()["data"], load_boston()["target"] svr_model.fit(x, y) self.logger.info("score:{}".format(svr_model.score(x,y))) def run(self,args): args_queue = queue.Queue() max_q = 0 for value in args: args_queue.put(value) max_q += 1 threads = [] for i in range(self.num_thread): threads.append(threading.Thread(target=self.object_func,args = (args_queue,max_q))) for t in threads: t.start() for t in threads: t.join() # 創(chuàng)建日志對象 logger = logging.getLogger() logger.setLevel(logging.INFO) screen_handler = logging.StreamHandler(sys.stdout) screen_handler.setLevel(logging.INFO) formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(message)s') screen_handler.setFormatter(formatter) logger.addHandler(screen_handler) # 創(chuàng)建需要調(diào)整參數(shù)的集合 args = [] for C in [i for i in np.arange(0.01,1,0.01)]: for epsilon in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]: for gamma in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]: args.append([C,epsilon,gamma]) # 創(chuàng)建多線程工具 threading_tool = CyrusThread(num_thread=20,logger=logger) threading_tool.run(args)
運(yùn)行結(jié)果
2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\1
2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\2
2021-02-04 20:52:22,826 - run.object_func:31 - INFO - progress:1176219\3
2021-02-04 20:52:22,833 - run.object_func:31 - INFO - progress:1176219\4
2021-02-04 20:52:22,837 - run.object_func:31 - INFO - progress:1176219\5
2021-02-04 20:52:22,838 - run.object_func:31 - INFO - progress:1176219\6
2021-02-04 20:52:22,841 - run.object_func:31 - INFO - progress:1176219\7
2021-02-04 20:52:22,862 - run.object_func:31 - INFO - progress:1176219\8
2021-02-04 20:52:22,873 - run.object_func:31 - INFO - progress:1176219\9
2021-02-04 20:52:22,884 - run.object_func:31 - INFO - progress:1176219\10
2021-02-04 20:52:22,885 - run.object_func:31 - INFO - progress:1176219\11
2021-02-04 20:52:22,897 - run.object_func:31 - INFO - progress:1176219\12
2021-02-04 20:52:22,900 - run.object_func:31 - INFO - progress:1176219\13
2021-02-04 20:52:22,904 - run.object_func:31 - INFO - progress:1176219\14
2021-02-04 20:52:22,912 - run.object_func:31 - INFO - progress:1176219\15
2021-02-04 20:52:22,920 - run.object_func:31 - INFO - progress:1176219\16
2021-02-04 20:52:22,920 - run.object_func:39 - INFO - score:-0.01674283914287855
2021-02-04 20:52:22,929 - run.object_func:31 - INFO - progress:1176219\17
2021-02-04 20:52:22,932 - run.object_func:39 - INFO - score:-0.007992354170952565
2021-02-04 20:52:22,932 - run.object_func:31 - INFO - progress:1176219\18
2021-02-04 20:52:22,945 - run.object_func:31 - INFO - progress:1176219\19
2021-02-04 20:52:22,954 - run.object_func:31 - INFO - progress:1176219\20
2021-02-04 20:52:22,978 - run.object_func:31 - INFO - progress:1176219\21
2021-02-04 20:52:22,984 - run.object_func:39 - INFO - score:-0.018769934807246536
2021-02-04 20:52:22,985 - run.object_func:31 - INFO - progress:1176219\22
到此這篇關(guān)于python中threading和queue庫實(shí)現(xiàn)多線程編程的文章就介紹到這了,更多相關(guān)python 多線程編程內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!
- Python中多線程thread與threading的實(shí)現(xiàn)方法
- python基于queue和threading實(shí)現(xiàn)多線程下載實(shí)例
- Python用threading實(shí)現(xiàn)多線程詳解
- python使用threading獲取線程函數(shù)返回值的實(shí)現(xiàn)方法
- Python 使用threading+Queue實(shí)現(xiàn)線程池示例
- Python線程協(xié)作threading.Condition實(shí)現(xiàn)過程解析
- Python3 socket即時(shí)通訊腳本實(shí)現(xiàn)代碼實(shí)例(threading多線程)
- Python中threading庫實(shí)現(xiàn)線程鎖與釋放鎖
- Python?threading和Thread模塊及線程的實(shí)現(xiàn)
相關(guān)文章
pyqt5 QScrollArea設(shè)置在自定義側(cè)(任何位置)
這篇文章主要介紹了pyqt5 QScrollArea設(shè)置在自定義側(cè)(任何位置),文中通過示例代碼介紹的非常詳細(xì),對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2019-09-09深入探索Python解碼神器Chardet自動(dòng)檢測文本編碼
Chardet,洞察編碼的清晰水晶球,一個(gè)讓你與編碼不再“失聯(lián)”的神器,本文帶大家走近這個(gè)隱藏在Python工具箱中的小寶貝,探索它的秘密2024-01-01Python?copy()與deepcopy()方法之間有什么區(qū)別
這篇文章主要介紹了Python中的copy()和deepcopy(),下面詳細(xì)介紹該內(nèi)容并附上詳細(xì)代碼,需要的朋友可以參考一下文章的具體內(nèi)容,希望對你有所幫助2022-10-10python+opencv圖像分割實(shí)現(xiàn)分割不規(guī)則ROI區(qū)域方法匯總
這篇文章主要介紹了python+opencv圖像分割實(shí)現(xiàn)分割不規(guī)則ROI區(qū)域方法匯總,文中通過示例代碼介紹的非常詳細(xì),對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2021-04-04