Monitor your file system with Python watchdog

The Python monitoring library makes it easy to monitor your file system and respond to these changes automatically. Watchdog is a cross-platform API that allows you to execute commands in response to any changes to the file system being monitored. We can set triggers on multiple events, such as file creation, modification, deletion, and movement, and then respond to these changes with our custom scripts.

Guard dog setup

You will need two modules to get started:

• Guard dog: Run this command below in terminal to install watchdog.

• Login: It is a built-in Python module, so there is no need to install it externally.

Basic use

Let's create a simple script 'main.py' that monitors a directory and prints a message every time a file is created, modified or deleted.

Step 1: Import the necessary modules

First, import the necessary modules from the monitoring library:

import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

Step 2: Define the event handler class

We define a class MyHandler that inherits from FileSystemEventHandler. This class overrides methods such as on_modified, on_created, and on_deleted to specify what to do when these events occur. The event handler object will be notified when any changes occur to the file system.

class MyHandler(FileSystemEventHandler):
    def on_modified(self, event):
        print(f'File {event.src_path} has been modified')

    def on_created(self, event):
        print(f'File {event.src_path} has been created')

    def on_deleted(self, event):
        print(f'File {event.src_path} has been deleted')

Some useful methods of FileSystemEventHandler are explained below.

• on_any_event: Executed for any event.
• on_created: Executed when creating a new file or directory.
• en_modified: It is executed after modifying a file or when changing the name of a directory.
• on_deleted: Triggered when deleting a file or directory.
• in_moved: Triggered when a file or directory is relocated.

Step 3: Initialize and run the observer

The Observer class is responsible for tracking the file system to detect any changes and subsequently notifying the event handler. Continuously monitors file system activities for updates.

if __name__ == "__main__":
    event_handler = MyHandler()
    observer = Observer()
    observer.schedule(event_handler, path=".", recursive=True)
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

We start the observer and use a loop to keep it running. When you want to stop it, you can interrupt with a keyboard signal. (Ctrl+C).

Step 4: Run the script

Finally, run the script with the following command.

Production:

File .\File1.txt has been modified
File .\New Text Document (2).txt has been created
File .\New Text Document (2).txt has been deleted
File .\New Text Document.txt has been deleted

The above code will log all directory changes in terminal if any file/folder is created, modified or deleted.

Advanced use

In the following example, we will explore how to set up a system that detects any changes to Python files and automatically runs tests. We need to install pytest with the following command.

Step 1 – Create a Simple Python Project with Tests

First, set up the basic structure of your project:

my_project/
│
├── src/
│   ├── __init__.py
│   └── example.py
│
├── tests/
│   ├── __init__.py
│   └── test_example.py
│
└── watchdog_test_runner.py

Step 2: Write code in an example Python file

Create a simple Python module in src/example.py:

def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

Step 3 – Write the test cases

Next, write test cases for functions in tests/test_example.py:

import pytest
from src.example import add, subtract

def test_add():
    assert add(1, 2) == 3
    assert add(-1, 1) == 0
    assert add(-1, -1) == -2

def test_subtract():
    assert subtract(2, 1) == 1
    assert subtract(1, 1) == 0
    assert subtract(1, -1) == 2

Step 4 – Write the Surveillance Script

Now, create the watchdog_test_runner.py script to monitor changes to Python files and run tests automatically:

import time
import subprocess
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class TestRunnerHandler(FileSystemEventHandler):
    def on_modified(self, event):
        if event.src_path.endswith('.py'):
            self.run_tests()

    def run_tests(self):
        try:
            result = subprocess.run(('pytest'), check=False, capture_output=True, text=True)
            print(result.stdout)
            print(result.stderr)
            if result.returncode == 0:
                print("Tests passed successfully.")
            else:
                print("Some tests failed.")
        except subprocess.CalledProcessError as e:
            print(f"Error running tests: {e}")

if __name__ == "__main__":
    path = "."  # Directory to watch
    event_handler = TestRunnerHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    
    observer.start()
    print(f"Watching for changes in {path}...")

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    
    observer.join()

Step 5: Run the watch script

In the end, open a terminal, navigate to your project directory (my_project) and run the watch script:

python watchdog_test_runner.py

Production:

Watching for changes in ....
========================= test session starts =============================
platform win32 -- Python 3.9.13, pytest-8.2.1, pluggy-1.5.0
rootdir: F:\Web Dev\watchdog
plugins: anyio-3.7.1
collected 2 items

tests\test_example.py ..                                                 (100%)

========================== 2 passed in 0.04s ==============================

Tests passed successfully.

This output shows that all test cases are passed after making changes to the example.py file.

summarizing

The Python monitoring library is a powerful tool for monitoring your file system. Whether you're automating tasks, syncing files, or creating more responsive applications, the watchdog mechanism makes it easy to react to file system changes in real time. With just a few lines of code, you can start monitoring directories and handling events to optimize your workflow.

Kanwal Mehreen Kanwal is a machine learning engineer and technical writer with a deep passion for data science and the intersection of ai with medicine. She is the co-author of the eBook “Maximize Productivity with ChatGPT.” As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She is also recognized as a Teradata Diversity in tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is a passionate advocate for change and founded FEMCodes to empower women in STEM fields.