pymongo insert_many 批量插入的實(shí)例
我就廢話不多說(shuō)了,大家還是直接看代碼吧~
lt=client.fangjia.district_stat_all_0416 dl = dt.find(query) bf=[] for m in dl: bf.append(m) if len(bf)==20000: lt.insert_many(bf) bf=[] lt.insert_many(bf)
補(bǔ)充知識(shí):Python操作Mongodb插入數(shù)據(jù)的兩種方法:insert_one()與insert_many()
代碼說(shuō)明:
將mysql中table的數(shù)據(jù)插入到mongodb里
sys.setdefaultencoding('utf8') import web from pymongo import MongoClient class getPltfList(object): def __init__(self): self.db1 = web.database(dbn='mysql', db = 'episode', user = 'root', pw= 'abc111--', host = '127.0.0.1') self.db1.printing = False self.db2 = web.database(dbn='mysql', db = 'episode_soc', user = 'root', pw= 'abc111--', host = '127.0.0.1') self.db2.printing = False self.db3 = web.database(dbn='mysql', db = 'cl_episode', user = 'root', pw= 'abc111--', host = '127.0.0.1') self.db3.printing = False self.db4 = web.database(dbn='mysql', db = 'cl_episode_soc', user = 'root', pw= 'abc111--', host = '127.0.0.1') self.db4.printing = False def __call__(self): return self.createPltfList() def createPltfList(self): self.ckpltfList = list(self.db1.query('select name, ip from EPISODE_PLTF_INFO order by id DESC')) self.ckpltfList += list(self.db2.query('select name, ip from EPISODE_PLTF_INFO order by id DESC')) self.clpltfList = list(self.db3.query('select name, ip from EPISODE_PLTF_INFO order by id DESC')) self.clpltfList += list(self.db4.query('select name, ip from EPISODE_PLTF_INFO order by id DESC')) return self.ckpltfList,self.clpltfList if __name__ == '__main__' : mpList = list() flag = 0 tmp = 0 sum1 = 0 sum2 = 0 pltfList = getPltfList()() client = MongoClient("127.0.0.1", 27017) mdb = client.episode collection = mdb.pltf_basic_info # print (type(pltfList[1])) #list result= list() for pltf_my in pltfList[0]: pltf_mog = collection.find() for record in pltf_mog: IP = record.get('Cfg').get('Debug_IP') Name = record.get('Cfg').get('Register_Name') # print IP, Name if IP == pltf_my['ip'] and Name == pltf_my['name'] : flag = 1 # print IP, Name break else: flag = 0 if flag == 0 : data1 = {"Cfg" : {"Debug_IP" : pltf_my['ip'],"Register_Name":pltf_my['name'], "Site":"SH-CK"} } # print data1 result.append((data1)) # collection.insert_one(data1) # collection.delete_one(data) sum1 = sum1+1 # print len(result) # collection.insert_many(result) for pltf_my in pltfList[1]: pltf_mog = collection.find() for record in pltf_mog: IP = record.get('Cfg').get('Debug_IP') Name = record.get('Cfg').get('Register_Name') if pltf_my['ip'].encode("utf-8") == IP.encode("utf-8") and pltf_my['name'].encode("utf-8") == Name.encode("utf-8") : tmp = 1 # print IP, Name break else: tmp = 0 if tmp == 0 : data2 = {"Cfg" : {"Debug_IP":pltf_my['ip'],"Register_Name":pltf_my['name'], "Site":"SH-CL"} } # print data2 result.append((data2)) # collection.insert_one(data2) # collection.delete_one(data) sum2 = sum2+1 collection.insert_many(result) print sum1,sum2
剛開始的時(shí)候我使用的是insert_one()方法,一條一條的插入到mongodb的集合里,但是計(jì)算出的sum有出入。
在調(diào)試的過(guò)程中我發(fā)現(xiàn):
注釋掉 # collection.insert_one(data1) # collection.insert_one(data2)
計(jì)算出來(lái)的sum1 = 193 sum2 = 222 這是合理的,因?yàn)閏k_mysql里有193條記錄,cl_mysql里有234條記錄,mongod里有總共有15條,但是12條是與cl_mysql重復(fù),所以正確。
但是當(dāng)我去掉注釋使用 collection.insert_one(data1) collection.insert_one(data2)時(shí),打印出的sum1=181 sum2 = 213
也就是說(shuō)少了幾個(gè)數(shù)據(jù),我不知道去哪了。
想了好多辦法,于是采用insert_many()的方法插入。先定義一個(gè)list(),將每個(gè)數(shù)據(jù)(數(shù)據(jù)的類型是dict)追加到list里:reslult.append(data1/data2),最后result里就會(huì)含有所有的數(shù)據(jù),一起插入。
于是乎,問(wèn)題解決了。
可是我還是很困惑insert_one()哪里出問(wèn)題了?。。。。。?/p>
以上這篇pymongo insert_many 批量插入的實(shí)例就是小編分享給大家的全部?jī)?nèi)容了,希望能給大家一個(gè)參考,也希望大家多多支持腳本之家。
相關(guān)文章
一文帶你掌握Python中enumerate函數(shù)和for循環(huán)的對(duì)比
在Python編程中,循環(huán)是一項(xiàng)常見的任務(wù),而for循環(huán)是最常見的一種,然而,Python提供了enumerate函數(shù),它允許在迭代過(guò)程中訪問(wèn)元素的同時(shí)獲得它們的索引,下面我們就來(lái)學(xué)習(xí)一下二者的區(qū)別吧2023-11-11pytorch transform數(shù)據(jù)處理轉(zhuǎn)c++問(wèn)題
這篇文章主要介紹了pytorch transform數(shù)據(jù)處理轉(zhuǎn)c++問(wèn)題,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2023-02-02python+selenium實(shí)現(xiàn)登錄賬戶后自動(dòng)點(diǎn)擊的示例
本篇文章主要介紹了python+selenium實(shí)現(xiàn)登錄賬戶后自動(dòng)點(diǎn)擊的示例,小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,也給大家做個(gè)參考。一起跟隨小編過(guò)來(lái)看看吧2017-12-12解決pip install psycopg2出錯(cuò)問(wèn)題
這篇文章主要介紹了解決pip install psycopg2出錯(cuò)問(wèn)題。具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2020-07-07Python?SQLAlchemy庫(kù)的實(shí)現(xiàn)示例
SQLAlchemy庫(kù)是一個(gè)強(qiáng)大的工具,為開發(fā)人員提供了便捷的方式來(lái)處理與數(shù)據(jù)庫(kù)的交互,本文主要介紹了Python?SQLAlchemy庫(kù)的實(shí)現(xiàn)示例,具有一定的參考價(jià)值,感興趣的可以了解一下2024-06-06