Introduction¶
urlwatch monitors the output of webpages or arbitrary shell commands.
Every time you run urlwatch, it:
- retrieves the output and processes it
- compares it with the version retrieved the previous time (“diffing”)
- if it finds any differences, generates a summary “report” that can be displayed or sent via one or more methods, such as email
Jobs¶
Each website or shell command to be monitored constitutes a “job”.
The instructions for each such job are contained in a config file in the YAML format, accessible with the urlwatch --edit
command.
If you get an error, set your $EDITOR
(or $VISUAL
) environment
variable in your shell with a command such as export EDITOR=/bin/nano
.
Typically, the first entry (“key”) in a job is a name
, which can be anything you want and helps you identify what you’re monitoring.
The second key is one of either url
, navigate
or command
:
url
retrieves what is served by the web server,navigate
handles more web pages requiring JavaScript to display the content to be monitored, andcommand
runs a shell command.
You can then use optional keys to finely control various job’s parameters.
Finally, you often use the filter
key to select one or more filters to apply to the data after it is retrieved, to:
- select HTML:
css
,xpath
,element-by-class
,element-by-id
,element-by-style
,element-by-tag
- make HTML more readable:
html2text
,beautify
- make PDFs readable:
pdf2text
- make JSON more readable:
format-json
- make iCal more readable:
ical2text
- make binary readable:
hexdump
- just detect changes:
sha1sum
- edit text:
grep
,grepi
,strip
,sort
These filters can be chained. As an example, after retrieving an HTML document by using the url
key, you can extract a selection with the xpath
filter, convert this to text with html2text
, use grep
to extract only lines matching a specific regular expression, and then sort
them:
name: "Sample urlwatch job definition"
url: "https://example.dummy/"
https_proxy: "http://dummy.proxy/"
max_tries: 2
filter:
- xpath: '//section[@role="main"]'
- html2text:
method: pyhtml2text
unicode_snob: true
body_width: 0
inline_links: false
ignore_links: true
ignore_images: true
pad_tables: false
single_line_break: true
- grep: "lines I care about"
- sort:
---
If you have more than one job, per YAML specifications, you separate them with a line containing only ---
.
Reporters¶
urlwatch can be configured to do something with its report besides (or in addition to) the default of displaying it on the console, such as one or more of:
email
(using SMTP)- email using
mailgun
slack
discord
pushbullet
telegram
matrix
pushover
stdout
xmpp
Reporters are configured in a separate file, see Configuration.