pymongo insert_many 批量插入的實(shí)例
我就廢話不多說(shuō)了,大家還是直接看代碼吧~
lt=client.fangjia.district_stat_all_0416
dl = dt.find(query)
bf=[]
for m in dl:
bf.append(m)
if len(bf)==20000:
lt.insert_many(bf)
bf=[]
lt.insert_many(bf)
補(bǔ)充知識(shí):Python操作Mongodb插入數(shù)據(jù)的兩種方法:insert_one()與insert_many()
代碼說(shuō)明:
將mysql中table的數(shù)據(jù)插入到mongodb里
sys.setdefaultencoding('utf8')
import web
from pymongo import MongoClient
class getPltfList(object):
def __init__(self):
self.db1 = web.database(dbn='mysql', db = 'episode', user = 'root', pw= 'abc111--', host = '127.0.0.1')
self.db1.printing = False
self.db2 = web.database(dbn='mysql', db = 'episode_soc', user = 'root', pw= 'abc111--', host = '127.0.0.1')
self.db2.printing = False
self.db3 = web.database(dbn='mysql', db = 'cl_episode', user = 'root', pw= 'abc111--', host = '127.0.0.1')
self.db3.printing = False
self.db4 = web.database(dbn='mysql', db = 'cl_episode_soc', user = 'root', pw= 'abc111--', host = '127.0.0.1')
self.db4.printing = False
def __call__(self):
return self.createPltfList()
def createPltfList(self):
self.ckpltfList = list(self.db1.query('select name, ip from EPISODE_PLTF_INFO order by id DESC'))
self.ckpltfList += list(self.db2.query('select name, ip from EPISODE_PLTF_INFO order by id DESC'))
self.clpltfList = list(self.db3.query('select name, ip from EPISODE_PLTF_INFO order by id DESC'))
self.clpltfList += list(self.db4.query('select name, ip from EPISODE_PLTF_INFO order by id DESC'))
return self.ckpltfList,self.clpltfList
if __name__ == '__main__' :
mpList = list()
flag = 0
tmp = 0
sum1 = 0
sum2 = 0
pltfList = getPltfList()()
client = MongoClient("127.0.0.1", 27017)
mdb = client.episode
collection = mdb.pltf_basic_info
# print (type(pltfList[1])) #list
result= list()
for pltf_my in pltfList[0]:
pltf_mog = collection.find()
for record in pltf_mog:
IP = record.get('Cfg').get('Debug_IP')
Name = record.get('Cfg').get('Register_Name')
# print IP, Name
if IP == pltf_my['ip'] and Name == pltf_my['name'] :
flag = 1
# print IP, Name
break
else:
flag = 0
if flag == 0 :
data1 = {"Cfg" : {"Debug_IP" : pltf_my['ip'],"Register_Name":pltf_my['name'], "Site":"SH-CK"} }
# print data1
result.append((data1))
# collection.insert_one(data1)
# collection.delete_one(data)
sum1 = sum1+1
# print len(result)
# collection.insert_many(result)
for pltf_my in pltfList[1]:
pltf_mog = collection.find()
for record in pltf_mog:
IP = record.get('Cfg').get('Debug_IP')
Name = record.get('Cfg').get('Register_Name')
if pltf_my['ip'].encode("utf-8") == IP.encode("utf-8") and pltf_my['name'].encode("utf-8") == Name.encode("utf-8") :
tmp = 1
# print IP, Name
break
else:
tmp = 0
if tmp == 0 :
data2 = {"Cfg" : {"Debug_IP":pltf_my['ip'],"Register_Name":pltf_my['name'], "Site":"SH-CL"} }
# print data2
result.append((data2))
# collection.insert_one(data2)
# collection.delete_one(data)
sum2 = sum2+1
collection.insert_many(result)
print sum1,sum2
剛開(kāi)始的時(shí)候我使用的是insert_one()方法,一條一條的插入到mongodb的集合里,但是計(jì)算出的sum有出入。
在調(diào)試的過(guò)程中我發(fā)現(xiàn):
注釋掉 # collection.insert_one(data1) # collection.insert_one(data2)
計(jì)算出來(lái)的sum1 = 193 sum2 = 222 這是合理的,因?yàn)閏k_mysql里有193條記錄,cl_mysql里有234條記錄,mongod里有總共有15條,但是12條是與cl_mysql重復(fù),所以正確。
但是當(dāng)我去掉注釋使用 collection.insert_one(data1) collection.insert_one(data2)時(shí),打印出的sum1=181 sum2 = 213
也就是說(shuō)少了幾個(gè)數(shù)據(jù),我不知道去哪了。
想了好多辦法,于是采用insert_many()的方法插入。先定義一個(gè)list(),將每個(gè)數(shù)據(jù)(數(shù)據(jù)的類型是dict)追加到list里:reslult.append(data1/data2),最后result里就會(huì)含有所有的數(shù)據(jù),一起插入。
于是乎,問(wèn)題解決了。
可是我還是很困惑insert_one()哪里出問(wèn)題了?。。。。?!
以上這篇pymongo insert_many 批量插入的實(shí)例就是小編分享給大家的全部?jī)?nèi)容了,希望能給大家一個(gè)參考,也希望大家多多支持腳本之家。
相關(guān)文章
一文帶你掌握Python中enumerate函數(shù)和for循環(huán)的對(duì)比
在Python編程中,循環(huán)是一項(xiàng)常見(jiàn)的任務(wù),而for循環(huán)是最常見(jiàn)的一種,然而,Python提供了enumerate函數(shù),它允許在迭代過(guò)程中訪問(wèn)元素的同時(shí)獲得它們的索引,下面我們就來(lái)學(xué)習(xí)一下二者的區(qū)別吧2023-11-11
pytorch transform數(shù)據(jù)處理轉(zhuǎn)c++問(wèn)題
這篇文章主要介紹了pytorch transform數(shù)據(jù)處理轉(zhuǎn)c++問(wèn)題,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2023-02-02
python+selenium實(shí)現(xiàn)登錄賬戶后自動(dòng)點(diǎn)擊的示例
本篇文章主要介紹了python+selenium實(shí)現(xiàn)登錄賬戶后自動(dòng)點(diǎn)擊的示例,小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,也給大家做個(gè)參考。一起跟隨小編過(guò)來(lái)看看吧2017-12-12
解決pip install psycopg2出錯(cuò)問(wèn)題
這篇文章主要介紹了解決pip install psycopg2出錯(cuò)問(wèn)題。具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2020-07-07
Python?SQLAlchemy庫(kù)的實(shí)現(xiàn)示例
SQLAlchemy庫(kù)是一個(gè)強(qiáng)大的工具,為開(kāi)發(fā)人員提供了便捷的方式來(lái)處理與數(shù)據(jù)庫(kù)的交互,本文主要介紹了Python?SQLAlchemy庫(kù)的實(shí)現(xiàn)示例,具有一定的參考價(jià)值,感興趣的可以了解一下2024-06-06

