This Python app is containerised with Docker Compose for a modular and cloud native deployment that fits in any microservice architecture.
It does the following:
- Use the Python libraries, Requests and Beautiful Soup, to download a copy of the web page of interest and any JavaScript files that are referred to in the page;
- Use the hashlib module to compute a SHA 256-bit checksum with the downloaded contents; and
- Compare the checksum with a previous build, if one exists, and on mismatch save the checksum and a copy of the downloaded contents, before using the smtplib module to send a notification email to the predefined recipient if it is required. Otherwise if no previous checksum exists, save the checksum the rest of the downloaded contents for future matching and references.
A detailed walk-through is available here.
Get started in three simple steps:
- Download a copy of the app;
- Create the environment variables for email notification if needed and modify the crontab; and
- Docker Compose or build and run the image manually to start the app, or alternatively run the Python script as a standalone service.
Download a copy of the script with git clone
. Be sure to pass the --recurse-submodules
argument to initialise and update each submodule in the repository.
$ git clone --recurse-submodules https://github.com/kurtcms/web-js-monitor /app/web-js-monitor/
If email notification is required, the app expects the SMTPS port number, SMTP server address, the notification receiver email address, the notification sender email address and password as environment variables in a .env
file in the same directory.
Create the .env
file.
$ nano /app/web-js-monitor/.env
And define the variables accordingly.
EMAIL_SSL_PORT = 465
EMAIL_SMTP_SERVER = 'smtp.kurtcms.org'
EMAIL_SENDER = 'alert@kurtcms.org'
EMAIL_RECEIVER = 'noc@kurtcms.org'
EMAIL_SENDER_PASSWORD = '(redacted)'
By default the app is scheduled with cron to pull a copy of the web page and check for changes every 15 minutes, with stdout
and stderr
redirected to the main process for Docker logs
.
Modify the crontab
to feed in the URL of interest, and signify to the Python script whether email notification is needed. Change to a different schedule if required.
$ nano /app/web-js-monitor/crontab
And define the variables accordingly.
*/15 * * * * /usr/bin/python3 /app/web-js-monitor/web-js-monitor.py -u https://lookingglass.pccwglobal.com/ > /proc/1/fd/1 2>/proc/1/fd/2
#
# Usage: web-js-monitor.py [-e] -u <url>
#
# Option:
# -h
# Display usage
# -e, --email
# Send email notification on changes
# -u, --url
# The URL of interest
Packaged as a container, the app is a standalone, executable package that may be run on Docker Engine. Be sure to have Docker installed.
With Docker Compose, the app may be provisioned with a single command.
Install Docker and Docker Compose with the Bash script that comes with app.
$ chmod +x /app/web-js-monitor/docker-compose/docker-compose.sh \
&& /app/web-js-monitor/docker-compose/docker-compose.sh
Start the containers with Docker Compose.
$ docker-compose -f /app/web-js-monitor/docker-compose.yml up -d
Stopping the container is as simple as a single command.
$ docker-compose -f /app/web-js-monitor/docker-compose.yml down
Otherwise the Docker image can also be built manually.
$ docker build -t web-js-monitor /app/web-js-monitor/
Run the image with Docker once it is ready.
$ docker run -it --rm --name web-js-monitor web-js-monitor
Alternatively the web-js-monitor.py
script may be deployed as a standalone service.
In which case be sure to install the following required libraries:
Install them with pip3
:
$ pip3 install requests python-dotenv bs4
The script reads the URL of interest from the -u
argument and sends email notification on changes if the -e
option is given.
Usage: web-js-monitor.py [-e] -u <url>
Option:
-h
Display usage
-e, --email
Send email notification on changes
-u, --url
The URL of interest
It may then be executed with a task scheduler such as cron that runs it once every 15 minutes for example.
$ (crontab -l; echo "*/15 * * * * /usr/bin/python3 /app/web-js-monitor/web-js-monitor.py -u https://lookingglass.pccwglobal.com/") | crontab -
The SHA 256-bit checksum and the rest of the downloaded contents will be saved as separate files on a Docker volume
that is mounted in the same directory of the docker-compose.yml
file on the Docker host. If the Python script is run as a standalone service, the files will be in the same directory of the script instead.
In any case, the files are stored under a directory by the URL of the web page of interest, with the checksum stored in a file directly under it and the rest of the downloaded contents in a nested directory named by the full date and time of the download to ease access.
.
└── url/
├── url-sha256hash
└── YYYY-MM-DD-HH-MM-SS/
├── index.html
├── javaScript1.js
├── javaScript2.js
└── javaScript3.js