python 串行執(zhí)行和并行執(zhí)行實(shí)例
我就廢話不多說(shuō)了,大家還是直接看代碼吧!
#coding=utf-8 import threading import time import cx_Oracle from pprint import pprint import csv print time.asctime() table_name = "dbtest.csv" f = open(table_name + ".csv", "w") conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() def query01(): tname = threading.current_thread() aa=10 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query02(): tname = threading.current_thread() aa=20 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query03(): tname = threading.current_thread() aa=30 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query04(): tname = threading.current_thread() aa=40 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row def query05(): tname = threading.current_thread() aa=50 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query06(): tname = threading.current_thread() aa=60 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query07(): tname = threading.current_thread() aa=70 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() return row cursor.close() def query08(): tname = threading.current_thread() aa=80 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query09(): tname = threading.current_thread() aa=90 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() def query10(): tname = threading.current_thread() aa=100 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() threads = [] t1 = threading.Thread(target=query01, name='query01') threads.append(t1) t2 = threading.Thread(target=query02, name='query02') threads.append(t2) t2 = threading.Thread(target=query03, name='query03') threads.append(t2) t2 = threading.Thread(target=query04, name='query04') threads.append(t2) t2 = threading.Thread(target=query05, name='query05') threads.append(t2) t2 = threading.Thread(target=query06, name='query06') threads.append(t2) t2 = threading.Thread(target=query07, name='query07') threads.append(t2) t2 = threading.Thread(target=query08, name='query08') threads.append(t2) t2 = threading.Thread(target=query09, name='query09') threads.append(t2) t2 = threading.Thread(target=query10, name='query10') threads.append(t2) if __name__ == '__main__': for t in threads: # t.setDaemon(True) t.start() # t.run() # t.start() # print '3333333' print threading.current_thread() # print t.is_alive() # print '3333333' t.join() print "all over " print time.asctime() C:\Python27\python.exe C:/Users/tlcb/PycharmProjects/untitled/a2.py Wed Mar 28 11:08:19 2018 <_MainThread(MainThread, started 18744)> [(10, '10boobook10', '10aaaaaaaaaaaa10', '10bbbbbbbbbbbbbbbbb10'), (10, '10sssssssss10', 'tlcb', 'tlcb'), (10, '10boobook10', '10aaaaaaaaaaaa10', '10bbbbbbbbbbbbbbbbb10')] <_MainThread(MainThread, started 18744)> [(20, '20boobook20', '20aaaaaaaaaaaa20', '20bbbbbbbbbbbbbbbbb20'), (20, '20boobook20', '20aaaaaaaaaaaa20', '20bbbbbbbbbbbbbbbbb20'), (20, '20boobook20', '20aaaaaaaaaaaa20', '20bbbbbbbbbbbbbbbbb20')] <_MainThread(MainThread, started 18744)> [(30, '30boobook30', '30aaaaaaaaaaaa30', '30bbbbbbbbbbbbbbbbb30'), (30, '30boobook30', '30aaaaaaaaaaaa30', '30bbbbbbbbbbbbbbbbb30'), (30, '30boobook30', '30aaaaaaaaaaaa30', '30bbbbbbbbbbbbbbbbb30')] <_MainThread(MainThread, started 18744)> [(40, '40boobook40', '40aaaaaaaaaaaa40', '40bbbbbbbbbbbbbbbbb40'), (40, '40boobook40', '40aaaaaaaaaaaa40', '40bbbbbbbbbbbbbbbbb40'), (40, '40boobook40', '40aaaaaaaaaaaa40', '40bbbbbbbbbbbbbbbbb40')] <_MainThread(MainThread, started 18744)> [(50, '50boobook50', '50aaaaaaaaaaaa50', '50bbbbbbbbbbbbbbbbb50'), (50, '50boobook50', '50aaaaaaaaaaaa50', '50bbbbbbbbbbbbbbbbb50'), (50, '50boobook50', '50aaaaaaaaaaaa50', '50bbbbbbbbbbbbbbbbb50')] <_MainThread(MainThread, started 18744)> [(60, '60boobook60', '60aaaaaaaaaaaa60', '60bbbbbbbbbbbbbbbbb60'), (60, '60boobook60', '60aaaaaaaaaaaa60', '60bbbbbbbbbbbbbbbbb60'), (60, '60boobook60', '60aaaaaaaaaaaa60', '60bbbbbbbbbbbbbbbbb60')] <_MainThread(MainThread, started 18744)> <_MainThread(MainThread, started 18744)> [(80, '80boobook80', '80aaaaaaaaaaaa80', '80bbbbbbbbbbbbbbbbb80'), (80, '80boobook80', '80aaaaaaaaaaaa80', '80bbbbbbbbbbbbbbbbb80'), (80, '80boobook80', '80aaaaaaaaaaaa80', '80bbbbbbbbbbbbbbbbb80')] <_MainThread(MainThread, started 18744)> [(90, '90boobook90', '90aaaaaaaaaaaa90', '90bbbbbbbbbbbbbbbbb90'), (90, '90boobook90', '90aaaaaaaaaaaa90', '90bbbbbbbbbbbbbbbbb90'), (90, '90boobook90', '90aaaaaaaaaaaa90', '90bbbbbbbbbbbbbbbbb90')] <_MainThread(MainThread, started 18744)> [(100, '100boobook100', '100aaaaaaaaaaaa100', '100bbbbbbbbbbbbbbbbb100'), (100, '100boobook100', '100aaaaaaaaaaaa100', '100bbbbbbbbbbbbbbbbb100'), (100, '100boobook100', '100aaaaaaaaaaaa100', '100bbbbbbbbbbbbbbbbb100')] all over Wed Mar 28 11:08:34 2018 Process finished with exit code 0 這個(gè)時(shí)候是串行 花費(fèi)了15秒 多線程跑: #coding=utf-8 import threading import time import cx_Oracle from pprint import pprint import csv print time.asctime() table_name = "dbtest.csv" f = open(table_name + ".csv", "w") def query01(): tname = threading.current_thread() aa=10 conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query02(): tname = threading.current_thread() aa=20 conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query03(): tname = threading.current_thread() aa=30 conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query04(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=40 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query05(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=50 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query06(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=60 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query07(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=70 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() return row cursor.close() conn.close() def query08(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=80 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query09(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=90 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() def query10(): conn = cx_Oracle.connect('tlcbuser/tlcbuser@20.5.101.31/tlyy') cursor = conn.cursor() tname = threading.current_thread() aa=100 # cursor.execute("SELECT * FROM TEST100 WHERE ID = %s", [aa]) cursor.prepare("""SELECT * FROM TEST100 WHERE ID=:id""") cursor.execute(None,{'id':aa}) row = cursor.fetchall() print row return row cursor.close() conn.close() threads = [] t1 = threading.Thread(target=query01, name='query01') threads.append(t1) t2 = threading.Thread(target=query02, name='query02') threads.append(t2) t2 = threading.Thread(target=query03, name='query03') threads.append(t2) t2 = threading.Thread(target=query04, name='query04') threads.append(t2) t2 = threading.Thread(target=query05, name='query05') threads.append(t2) t2 = threading.Thread(target=query06, name='query06') threads.append(t2) t2 = threading.Thread(target=query07, name='query07') threads.append(t2) t2 = threading.Thread(target=query08, name='query08') threads.append(t2) t2 = threading.Thread(target=query09, name='query09') threads.append(t2) t2 = threading.Thread(target=query10, name='query10') threads.append(t2) if __name__ == '__main__': for t in threads: # t.setDaemon(True) t.start() # t.run() # t.start() # print '3333333' print threading.current_thread() # print t.is_alive() # print '3333333' t.join() print "all over " print time.asctime() C:\Python27\python.exe C:/Users/tlcb/PycharmProjects/untitled/a2.py Wed Mar 28 11:12:47 2018 <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> <_MainThread(MainThread, started 22500)> [(40, '40boobook40', '40aaaaaaaaaaaa40', '40bbbbbbbbbbbbbbbbb40'), (40, '40boobook40', '40aaaaaaaaaaaa40', '40bbbbbbbbbbbbbbbbb40'), (40, '40boobook40', '40aaaaaaaaaaaa40', '40bbbbbbbbbbbbbbbbb40')] [(60, '60boobook60', '60aaaaaaaaaaaa60', '60bbbbbbbbbbbbbbbbb60'), (60, '60boobook60', '60aaaaaaaaaaaa60', '60bbbbbbbbbbbbbbbbb60'), (60, '60boobook60', '60aaaaaaaaaaaa60', '60bbbbbbbbbbbbbbbbb60')] [(80, '80boobook80', '80aaaaaaaaaaaa80', '80bbbbbbbbbbbbbbbbb80'), (80, '80boobook80', '80aaaaaaaaaaaa80', '80bbbbbbbbbbbbbbbbb80'), (80, '80boobook80', '80aaaaaaaaaaaa80', '80bbbbbbbbbbbbbbbbb80')] [(50, '50boobook50', '50aaaaaaaaaaaa50', '50bbbbbbbbbbbbbbbbb50'), (50, '50boobook50', '50aaaaaaaaaaaa50', '50bbbbbbbbbbbbbbbbb50'), (50, '50boobook50', '50aaaaaaaaaaaa50', '50bbbbbbbbbbbbbbbbb50')] [(10, '10boobook10', '10aaaaaaaaaaaa10', '10bbbbbbbbbbbbbbbbb10'), (10, '10sssssssss10', 'tlcb', 'tlcb'), (10, '10boobook10', '10aaaaaaaaaaaa10', '10bbbbbbbbbbbbbbbbb10')] [(20, '20boobook20', '20aaaaaaaaaaaa20', '20bbbbbbbbbbbbbbbbb20'), (20, '20boobook20', '20aaaaaaaaaaaa20', '20bbbbbbbbbbbbbbbbb20'), (20, '20boobook20', '20aaaaaaaaaaaa20', '20bbbbbbbbbbbbbbbbb20')] [(30, '30boobook30', '30aaaaaaaaaaaa30', '30bbbbbbbbbbbbbbbbb30'), (30, '30boobook30', '30aaaaaaaaaaaa30', '30bbbbbbbbbbbbbbbbb30'), (30, '30boobook30', '30aaaaaaaaaaaa30', '30bbbbbbbbbbbbbbbbb30')] [(100, '100boobook100', '100aaaaaaaaaaaa100', '100bbbbbbbbbbbbbbbbb100'), (100, '100boobook100', '100aaaaaaaaaaaa100', '100bbbbbbbbbbbbbbbbb100'), (100, '100boobook100', '100aaaaaaaaaaaa100', '100bbbbbbbbbbbbbbbbb100')] [(90, '90boobook90', '90aaaaaaaaaaaa90', '90bbbbbbbbbbbbbbbbb90'), (90, '90boobook90', '90aaaaaaaaaaaa90', '90bbbbbbbbbbbbbbbbb90'), (90, '90boobook90', '90aaaaaaaaaaaa90', '90bbbbbbbbbbbbbbbbb90')] all over Wed Mar 28 11:12:55 2018 Process finished with exit code 0 此時(shí)花了 8秒
補(bǔ)充知識(shí):python logging定制logstash的json日志格式
最近一直在折騰日志的收集,現(xiàn)在算是收尾了。 寫一篇算python優(yōu)化logstash的方案。
其實(shí)大家都知道logstash調(diào)用grok來(lái)解析日志的話,是要消耗cpu的成本的,畢竟是需要正則的匹配的。
根據(jù)logstash調(diào)優(yōu)的方案,咱們可以預(yù)先生成json的格式。 我這邊基本是python的程序,怎么搞尼 ?
有兩種方法,第一種方法是生成json后,直接打入logstash的端口。 還有一種是生成json寫入文件,讓logstash做tail操作的時(shí)候,把一行的日志數(shù)據(jù)直接載入json就可以了。
python下的日志調(diào)試用得時(shí)logging,改成json也是很好改得。 另外不少老外已經(jīng)考慮到這樣的需求,已經(jīng)做了python logstash的模塊。
import logging import logstash import sys host = 'localhost' test_logger = logging.getLogger('python-logstash-logger') test_logger.setLevel(logging.INFO) test_logger.addHandler(logstash.LogstashHandler(host, 5959, version=1)) # test_logger.addHandler(logstash.TCPLogstashHandler(host, 5959, version=1)) test_logger.error('python-logstash: test logstash error message.') test_logger.info('python-logstash: test logstash info message.') test_logger.warning('python-logstash: test logstash warning message.') # add extra field to logstash message extra = { 'test_string': 'python version: ' + repr(sys.version_info), 'test_boolean': True, 'test_dict': {'a': 1, 'b': 'c'}, 'test_float': 1.23, 'test_integer': 123, 'test_list': [1, 2, '3'], } test_logger.info('python-logstash: test extra fields', extra=extra)
python-logstash自帶了amqp的方案
import logging import logstash # AMQP parameters host = 'localhost' username = 'guest' password= 'guest' exchange = 'logstash.py' # get a logger and set logging level test_logger = logging.getLogger('python-logstash-logger') test_logger.setLevel(logging.INFO) # add the handler test_logger.addHandler(logstash.AMQPLogstashHandler(version=1, host=host, durable=True, username=username, password=password, exchange=exchange)) # log test_logger.error('python-logstash: test logstash error message.') test_logger.info('python-logstash: test logstash info message.') test_logger.warning('python-logstash: test logstash warning message.') try: 1/0 except: test_logger.exception('python-logstash: test logstash exception with stack trace')
不管怎么說(shuō),最后生成的格式是這樣就可以了。
{ "@source"=>"unknown", "@type"=>"nginx", "@tags"=>[], "@fields"=>{ "remote_addr"=>"192.168.0.1", "remote_user"=>"-", "body_bytes_sent"=>"13988", "request_time"=>"0.122", "status"=>"200", "request"=>"GET /some/url HTTP/1.1", "request_method"=>"GET", "http_referrer"=>"http://www.example.org/some/url", "http_user_agent"=>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.79 Safari/537.1" }, "@timestamp"=>"2012-08-23T10:49:14+02:00" }
我這里簡(jiǎn)單提一下,這個(gè)模塊用的不是很滿意,我在python下把日志打成了json字符串,我原本以為會(huì)像grok那樣,在Es里面,我的這條日志是個(gè)字段的結(jié)構(gòu),而不是這個(gè)日志都在message里面…. 我想大家應(yīng)該明白了我的意思,這樣很是不容易在kibana的搜索…
在kibana搜索,我經(jīng)常上 source:xxx AND level:INFO 結(jié)果正像上面描述的那樣,整條日志,都在@message里面。
以上這篇python 串行執(zhí)行和并行執(zhí)行實(shí)例就是小編分享給大家的全部?jī)?nèi)容了,希望能給大家一個(gè)參考,也希望大家多多支持腳本之家。
相關(guān)文章
基于keras中訓(xùn)練數(shù)據(jù)的幾種方式對(duì)比(fit和fit_generator)
這篇文章主要介紹了keras中訓(xùn)練數(shù)據(jù)的幾種方式對(duì)比(fit和fit_generator),具有很好的參考價(jià)值,希望對(duì)大家有所幫助。如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2021-05-05一個(gè)基于flask的web應(yīng)用誕生(1)
這篇文章主要為大家詳細(xì)介紹了基于flask的web應(yīng)用誕生,具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2017-04-04Python face_recognition實(shí)現(xiàn)AI識(shí)別圖片中的人物
最近碰到了照片識(shí)別的場(chǎng)景,正好使用了face_recognition項(xiàng)目,給大家分享分享。face_recognition項(xiàng)目能做的很多,人臉檢測(cè)功能也是有的,是一個(gè)比較成熟的項(xiàng)目。感興趣的可以了解一下2022-01-01python實(shí)現(xiàn)大文本文件分割成多個(gè)小文件
這篇文章主要為大家詳細(xì)介紹了python實(shí)現(xiàn)大文本文件分割成多個(gè)小文件,文中示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2021-04-04python 文件的基本操作 菜中菜功能的實(shí)例代碼
這篇文章主要介紹了python 文件的基本操作 菜中菜功能,本文通過(guò)實(shí)例代碼給大家介紹的非常詳細(xì),具有一定的參考借鑒價(jià)值 ,需要的朋友可以參考下2019-07-07Pytorch之nn.Upsample()和nn.ConvTranspose2d()用法詳解
nn.Upsample和nn.ConvTranspose2d是PyTorch中用于上采樣的兩種主要方法,nn.Upsample通過(guò)不同的插值方法(如nearest、bilinear)執(zhí)行上采樣,沒(méi)有可學(xué)習(xí)的參數(shù),適合快速簡(jiǎn)單的尺寸增加,而nn.ConvTranspose2d通過(guò)可學(xué)習(xí)的轉(zhuǎn)置卷積核進(jìn)行上采樣2024-10-10Python驗(yàn)證的50個(gè)常見(jiàn)正則表達(dá)式
這篇文章主要給大家介紹了關(guān)于利用Python驗(yàn)證的50個(gè)常見(jiàn)正則表達(dá)式的相關(guān)資料,文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2021-03-03