Python神器之使用watchdog監(jiān)控文件變化
一、官方文檔
需要細節(jié)選擇去官網(wǎng),需要了解和應用范例看本文即可
https://pythonhosted.org/watchdog/官網(wǎng)上的:User’s Guide,主要看API相關這里即可
- 安裝
- 快速開始
- API 相關
二、watchdog安裝
watchdog需要在python2.6以上的版本工作,如果使用的是Linux//FreeBSD/Mac OS X 系統(tǒng),python已經(jīng)安裝了,但是需要保證python的版本至少是python2.7(學習用最好還是使用3.6以上的版本吧)
下面有三種不同的安裝方法:選擇第一種最簡單的就行了,可以簡便的不要自求麻煩。
1. Installing from PyPI using pip
pip install watchdog
2. Installing from source tarballs
$ wget -c http://pypi.python.org/packages/source/w/watchdog/watchdog-0.8.2.tar.gz $ tar zxvf watchdog-0.8.2.tar.gz $ cd watchdog-0.8.2 $ python setup.py install
3. Installing from the code repository
$ git clone --recursive git://github.com/gorakhargosh/watchdog.git $ cd watchdog $ python setup.py install
在官方還說需要下載一些依賴,但是你使用第一種pip安裝的方法基本都不會有這種需求。
三、快速開始(官方范例)
下面我們提供一個簡單的示例,該示例以遞歸方式監(jiān)視當前目錄(這意味著它將遍歷所有子目錄)以檢測更改。 這是我們將要使用的API:
1.創(chuàng)建watchdog.observers.Observer線程類的實例
2.實現(xiàn)watchdog.events.FileSystemEventHandler的子類(或者在我們的例子中,我們將使用內(nèi)置的watchdog.events.LoggingEventHandler)。
3.通過附加事件處理程序的觀察者實例來計劃監(jiān)視一些路徑。
4.啟動觀察者線程并等待它生成事件而不會阻塞我們的主線程。
import sys
import time
import logging
from watchdog.observers import Observer # 監(jiān)控
from watchdog.events import LoggingEventHandler # 觸發(fā)事件
?
if __name__=='__main__':
logging.basicConfig(level=logging.INFO,
format='%(asctime)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
path = sys.argv[1] if len(sys.argv) > 1 else '.'
event_handler = LoggingEventHandler() # 監(jiān)控處理事件的方法,這個類非常重要,可以根據(jù)自己的需要重寫
observer = Observer() # 定義監(jiān)控類,多線程類 thread class
observer.schedule(event_handler,'./monitor_folder_1', recursive=True) # 指定監(jiān)控路徑/觸發(fā)對應的監(jiān)控事件類
observer.start()# 將observer運行在同一個線程之內(nèi),不阻塞主進程運行,可以調度observer來停止該線程
try:
while True:
time.sleep(1) # 監(jiān)控頻率(1s1次,根據(jù)自己的需求進行監(jiān)控)
except KeyboardInterrupt:
observer.stop()
observer.join()四、API Reference(翻譯官網(wǎng))
觀察者模型主要有三個角色:observer、event_handler、被監(jiān)控的文件夾。三者原本是獨立的,主要通過observer.schedule函數(shù)將三者串起來,意思為observer不斷檢測調用平臺依賴代碼對監(jiān)控文件夾進行變動檢測,當發(fā)現(xiàn)改變時,通知event_handler處理。
watchdog.event
模型:watchdog.events簡介:文件系統(tǒng)事件和時間處理的程序原作者: yesudeep@google.com (Yesudeep Mangalapilly)
1. Event Class
首先考慮一下關于文件和文件夾的相關操作有那些?無外乎:
- 創(chuàng)建(FileCreatedEvent/DirCreateEvent)
- 移動(FileMovedEvent/DirMovedEvent)
- 刪除(FildDeletedEvent/DirDeletedEvent)
- 修改(FileModifiedEvent/DirModifiedEvent)
對文件/文件夾的監(jiān)控也就無非以上的操作了,watchdog對文件夾的監(jiān)控也就是針對上述需求進行設計的Event Classes也就包含了大概八個類(都分別依賴于watchdog.events.FileSystemEvent 和watchdog.events.FileSystemMovedEvent):
- class watchdog.events.FileMovedEvent(src_path, dest_path)
- class watchdog.events.DirMovedEvent(src_path, dest_path)
- class watchdog.events.FileModifiedEvent(src_path)
- class watchdog.events.DirModifiedEvent(src_path)
- class watchdog.events.FileCreatedEvent(src_path)
- class watchdog.events.DirCreatedEvent(src_path)
- class watchdog.events.FileDeletedEvent(src_path)
- class watchdog.events.DirDeletedEvent(src_path)
2. Event handler Classes事件處理類
官方文件中對Event Handler Classes的說明:
class watchdog.events.FileSystemEventHandler[source]
Bases: object
Base file system event handler that you can override methods from.
dispatch(event)[source]
Dispatches events to the appropriate methods.
Parameters:event (FileSystemEvent) – The event object representing the file system event.
on_any_event(event)[source]
Catch-all event handler.
Parameters:event (FileSystemEvent) – The event object representing the file system event.
on_created(event)[source]
Called when a file or directory is created.
Parameters:event (DirCreatedEvent or FileCreatedEvent) – Event representing file/directory creation.
on_deleted(event)[source]
Called when a file or directory is deleted.
Parameters:event (DirDeletedEvent or FileDeletedEvent) – Event representing file/directory deletion.
on_modified(event)[source]
Called when a file or directory is modified.
Parameters:event (DirModifiedEvent or FileModifiedEvent) – Event representing file/directory modification.
on_moved(event)[source]
Called when a file or a directory is moved or renamed.
Parameters:event (DirMovedEvent or FileMovedEvent) – Event representing file/directory movement.
上述也就說明了watchdog的觸發(fā)事件的方法為:
- dispatch
- on_any_event
- on_create
- on_delete
- on_modified
- on_moved
3. watchdog.observers.api
class watchdog.observers.api.ObservedWatch(path, recursive)
Bases: object
An scheduled watch.
Parameters:
path – Path string.
recursive:
True if watch is recursive;
False
otherwise.is_recursive:Determines whether subdirectories are watched for the path.
path:The path that this watch monitors
Observer用以監(jiān)控文件的變化,根據(jù)觸發(fā)事件調用相關聯(lián)的事件處理方法來執(zhí)行。
observer是threading.Thread的子類,通過observer.start()使之運行在一個線程之中,不阻塞主線程的運行,然后可以調用observer.stop()來停止該進程.
observer.schedule(event_handler,path,recursive=False) # event_handler:觸發(fā)事件的操作方法 # path:監(jiān)控的文件或者目錄 # recursive:如果監(jiān)控的是文件則需要設置為True
五、linux中使用watchdog監(jiān)控文件夾各種操作會觸發(fā)的監(jiān)控事件
1. 移除文件
文件刪除:File_deleted
文件夾修改 Directory_modified
2. 添加文件
文件新建:File_created
文件夾修改:Directory_modified
3.修改文件
兩次文件修改:File_modified(顯示針對一個備份文件,然后是針對原始文件)
刪除備份文件:File_deleted
文件夾修改:Directory_modified
4.文件改名
文件移動:File_moved
文件夾修改:Directory_modified
5.使用cp命令復制文件到監(jiān)控目錄
文件新建:File_created
一大堆文件修改操作:Directory_modified
文件夾修改操作:File_modified
六、應用實例
1.監(jiān)控文件夾中是否有后綴為".segy"的文件進入
#! /home/liufeng/project
# -*- coding:utf-8 -*-
# Created by jerry_liufeng
?
# 本模塊的功能:<檢測文件夾變化>
?
# 導入watchdog對應模塊
from watchdog.observers import Observer
from watchdog.events import *
# 導入時間模塊
import time
?
class FileEventHandler(FileSystemEventHandler):
# 初始化魔術方法
def __init__(self):
FileSystemEventHandler.__init__(self)
self.fileNameList = [] # 文件名列表(存儲文件名)
?
# 文件或文件夾移動
def on_moved(self, event):
if event.is_directory:
print("directory moved from {0} to {1}".format(event.src_path,event.dest_path))
else:
print("file moved from {0} to {1}".format(event.src_path,event.dest_path))
?
# 創(chuàng)建文件或文件夾
def on_created(self, event):
if event.is_directory:
print("directory created:{0}".format(event.src_path))
else:
print("file created:{0}".format(event.src_path))
fileAllName =str(event.src_path.split('/')[-1])
if fileAllName.endswith('.segy'):
self.fileNameList.append(fileAllName)
print(self.fileNameList)
# 刪除文件或文件夾
def on_deleted(self, event):
if event.is_directory:
print("directory deleted:{0}".format(event.src_path))
else:
print("file deleted:{0}".format(event.src_path))
?
# # 修改文件或文件夾
# def on_modified(self, event):
# if event.is_directory:
# print("directory modified:{0}".format(event.src_path))
# else:
# print("file modified:{0}".format(event.src_path))
?
if __name__ == "__main__":
# 實例化Observer對象
observer = Observer()
event_handler = FileEventHandler()
# 設置監(jiān)聽目錄
dis_dir = "./03_seis_data/"
observer.schedule(event_handler,dis_dir,True)
observer.start()
try:
while True:
# 設置監(jiān)聽頻率(間隔周期時間)
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()說明:之所以注釋掉modified這段代碼是因為我關心的是文件的進入,而非文件內(nèi)容的改變,太過頻繁的modified操作會影響我對新增文件數(shù)量的判斷。
2.監(jiān)控多個文件夾
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import time
?
class FileEventHandler(FileSystemEventHandler):
def __init__(self):
FileSystemEventHandler.__init__(self)
def on_moved(self, event):
if event.is_directory:
print("directory moved from {0} to {1}".format(event.src_path,event.dest_path))
else:
print("file moved from {0} to {1}".format(event.src_path,event.dest_path))
def on_created(self, event):
if event.is_directory:
print("directory created:{0}".format(event.src_path))
else:
print("file created:{0}".format(event.src_path))
def on_deleted(self, event):
if event.is_directory:
print("directory deleted:{0}".format(event.src_path))
else:
print("file deleted:{0}".format(event.src_path))
def on_modified(self, event):
if event.is_directory:
print("directory modified:{0}".format(event.src_path))
else:
print("file modified:{0}".format(event.src_path))
?
if __name__ == "__main__":
observer = Observer()
dirs = [r'./monitor_folder_1', r'./monitor_folder_1']
# dirs = [ r'./monitor_folder_1']
for dir in dirs:
event_handler = FileEventHandler()
observer.schedule(event_handler, dir, True)
observer.start()3.文件系統(tǒng)一次操作多次觸發(fā)各類事件問題解決
原文參考:Python中WatchDog的使用經(jīng)驗總結
問題:
文件操作引發(fā)的事件比我們想象的多了不少,而且難以在事件函數(shù)中做出針對性處理。
在 windows 平臺上每一次的文件修改引發(fā)兩次 modified 事件?在 windows 平臺上,由于 watchdog 封裝的是 windows 系統(tǒng)的 FileSystemWatcher Events,處理文件的過程中執(zhí)行了多次文件系統(tǒng)操作,無法避免地觸發(fā)了多次事件。
改進方案:
如果對監(jiān)視文件的實時性要求不高,又懶得處理一大堆事件,那么,比較事件前后的文件夾快照就是一個值得嘗試的改進方案。實現(xiàn)這個思路,有三個前提條件:
- 快速獲取文件夾快照。幸運的是,watchdog 模塊為我們提供了 DirectorySnapshot 功能
- 可以接受200毫秒的延時。文件操作引發(fā)的一大堆事件會集中在一個較短的時間內(nèi),一般情況下,在文件操作之后200毫秒獲取文件夾快照,是一個不錯的間隔
- 快速比較文件夾快照。這也不是問題,watchdog 模塊有 DirectorySnapshotDiff 子模塊
改進思路:設置一個定時器, 200毫秒后抓取快照,并與上一張快照比較。每當有事件發(fā)生時,檢查定時器是否已經(jīng)啟動。如果未啟動,則直接啟動定時器;否則,說明該事件距離上次事件不足200毫秒,可視為是同一組事件,此時終止定時器,再次重啟。具體代碼如下:
import time
import os, threading
from watchdog.observers import Observer
from watchdog.events import *
from watchdog.utils.dirsnapshot import DirectorySnapshot, DirectorySnapshotDiff
?
class FileEventHandler(FileSystemEventHandler):
def __init__(self, aim_path):
FileSystemEventHandler.__init__(self)
self.aim_path = aim_path
self.timer = None
self.snapshot = DirectorySnapshot(self.aim_path)
def on_any_event(self, event):
if self.timer:
self.timer.cancel()
self.timer = threading.Timer(0.2, self.checkSnapshot)
self.timer.start()
def checkSnapshot(self):
snapshot = DirectorySnapshot(self.aim_path)
diff = DirectorySnapshotDiff(self.snapshot, snapshot)
self.snapshot = snapshot
self.timer = None
print("files_created:", diff.files_created)
print("files_deleted:", diff.files_deleted)
print("files_modified:", diff.files_modified)
print("files_moved:", diff.files_moved)
print("dirs_modified:", diff.dirs_modified)
print("dirs_moved:", diff.dirs_moved)
print("dirs_deleted:", diff.dirs_deleted)
print("dirs_created:", diff.dirs_created)
# def on_moved(self, event):
# if event.is_directory:
# print("directory moved from {0} to {1}".format(event.src_path,event.dest_path))
# else:
# print("file moved from {0} to {1}".format(event.src_path,event.dest_path))
# def on_created(self, event):
# if event.is_directory:
# print("directory created:{0}".format(event.src_path))
# else:
# print("file created:{0}".format(event.src_path))
# def on_deleted(self, event):
# if event.is_directory:
# print("directory deleted:{0}".format(event.src_path))
# else:
# print("file deleted:{0}".format(event.src_path))
# def on_modified(self, event):
# if event.is_directory:
# print("directory modified:{0}".format(event.src_path))
# else:
# print("file modified:{0}".format(event.src_path))
?
?
class DirMonitor(object):
"""文件夾監(jiān)視類"""
def __init__(self, aim_path):
"""構造函數(shù)"""
self.aim_path= aim_path
self.observer = Observer()
def start(self):
"""啟動"""
event_handler = FileEventHandler(self.aim_path)
self.observer.schedule(event_handler, self.aim_path, True)
self.observer.start()
def stop(self):
"""停止"""
self.observer.stop()
if __name__ == "__main__":
monitor = DirMonitor(r"./monitor_folder_1")
monitor.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
monitor.stop()以上就是Python神器之使用watchdog監(jiān)控文件變化的詳細內(nèi)容,更多關于Python watchdog監(jiān)控文件變化的資料請關注腳本之家其它相關文章!
相關文章
pyecharts在數(shù)據(jù)可視化中的應用詳解
這篇文章主要介紹了pyecharts在數(shù)據(jù)可視化中的應用詳解,文中通過示例代碼介紹的非常詳細,對大家的學習或者工作具有一定的參考學習價值,需要的朋友們下面隨著小編來一起學習學習吧2020-06-06

