Modules are dynamically fed by the system logger (rsyslog or syslog-ng). The modules analyze logging messages in real-time,  keep track of attackers in databases, and feed the firewall autonomously with DROPs as 'bandits' reach configurable limits of attacking your server system.

Why Bash & Co.

Watcher is fully based on 'onboard tools' (bash, awk, grep, etc.; 'coreutils' et al.) that are common for every unixoid operating system and these are present out-of-the-box after system installation on a server system.

• Bash is fast! Due to 'builtins' for so-called 'transient programs' on disk use of Bash prevents superfluous disk accesses which increases processing speed profoundly.
  
  
• No compilation is needed.
  
  
• Watcher does not need any special runtime environments for fancy 3rd-party programming languages like PERL, JAVA, PYTHON, LUA, RUBY, etc.
  So there is no need to install and maintain development tools or runtime environments for these onto a server system.
  
  
• AWK (a.k.a. "the chain saw for text") is used for long 'list files' since AWK can be up to 100 times faster than BASH if it comes to traversing and processing long list files.
  .
• All programs & tools are 'self-aware'; i.e. every component always 'knows' where it was started in relation to the install path (MASTER_PATH). So the entire program system can be installed wherever you like or can even be moved around to your taste and/or needs.
  
  
• The modules are self-cleaning through automated 'database expiration' along with a consecutive firewall cleanup. This automatic housekeeping also keeps maintenance tasks away from you.

What's the audience for Watcher?

Everyone, who has to operate a Linux-based 'root server' (physical or VPS)  in a data center with 'rental servers' from a provider.

Such server systems are usually running as 'exposed hosts' and are not protected by any external firewalling appliance. The provider leaves the protection up to you since the provider cannot know which type and range of protection is needed for the purpose for which you operate your server.

Also, servers that are running in the DMZ of a usual data center situation can benefit from Watcher; e.g. mail servers, which are the primary targets for attackers,  SPAM spreaders, and hijackers.

What does Watcher do for you?

• Watcher cuts down the attack rate of certain services on your server by a factor of 100 or even more. So it protects your server (and the services in particular) from [D]DoS ([Distributed] Denial-of-Service) attacks very efficiently.
  
  
• Frequently downloading the drop & edrop lists from 'SpamHaus' and integrating them into your firewall prevents your server from 'hijacking attacks' and cases of 'IP spoofing' continuously conducted by known professional attackers.
  
  
• The module WatchLG automatically tracks break-in attempts to the console login service (usually 'sshd') in real-time in a database at login level and prevents your system from being brought under the control of attackers.
  
  
• Using the module WatchMX for a mail server tracks your mail server log in real-time and prevents the mail server from dealing with SPAM spreaders so that the mail server (e.g. PostFix) and mailbox services (DBmail, DoveCot, ...) can do their job undisturbed and with best performance.  WatchMX comes with a companion process WatchMB that takes care of malicious mailbox access and prevents further access attempts to mailboxes if a configurable number of 'affairs' are exceeded.
  
  
• The module WatchWB (since Q3/2021 along with Watcher 1.3) tracks the logging of your web server instances (vhosts) for malicious access attempts and in particular for flooding requests by 'crawlers' and/or 'robots' - 'bots' for short. WatchWB is the most complex module of the three as a web server can service multiple website instances (customers) on a single physical server system. So the tracking must include mechanisms, that take care of which 'instance' was attacked by which IP address. For security reasons, this information must be kept separate in reports and statistics information that goes to several customers.

What else?

• "Watcher-Report" provides you with frequent reports of the firewall state, dynloader provisioning, and the database state of the fully dynamic modules that are installed.
  
  
• For long-term reporting separate 'statistics reports' from the 'Stat tools'  of the module databases will get you an overlook of the efficiency of all the measures that you have taken. The statistics data is provided in CSV format so you can simply load it into a spreadsheet of your choice.
  
  
• Frequent database expiration (configurable) in the modules keeps the firewall entries at a reasonable level since a firewall that is flooded with pointless and hypothetic IP addresses can slow down your network traffic.
  
  
• The modules store only IP addresses for firewall DROPs of IP addresses, that are really attacking your system. So flooding your firewall with lists of hypothetical attacker assumptions from elsewhere on the Internet is not necessary.

Watcher construction

Starting with Watcher release 1.0 the system was modularized; i.e. broken up into:

• Watcher master (framework & library for dynloaders & modules)
• WatchLG (login tracker module)
• WatchMX (mail services tracking module with a companion process WatchMB that takes care of break-in affairs to mailbox accounts [POP & IMAP] and mail transport requests that are protected with credentials; e.g. by SASL)
• WatchWB (protecting WEB services (HTTP server) [along with Watcher release 1.3]
• The DynLoaders (CRON controlled) are from the early days of the Watcher project and are reworked for better integration with the modularized concept.
  They are included in the master package as they are really handy for basic mail server protection and hijacking prevention by known professional attackers.

What makes the modules so different?

• Each module runs autonomously if once started by the master.
• Modules are directly (synchronously) fed by the system logger and track reported complaints from the related service in real-time with instant measure; i.e. a DROP in the firewall if a configurable limit of 'affairs' is exceeded. So detection & reaction happen within milliseconds at the moment of the attack.
• Modules store the attack event and measure it in a database to fully restore the firewall within seconds after start-up & reboot.
  Due to the enormous load speed, it makes sense to restart 'watcher' from crontab on a regular basis; e.g. once a day.
  This can keep the 'in memory' firewall clean and compact for the best performance.
• Each module provides an 'open inject interface' provided as a FIFO (named pipe) in $FIFO_BASE/WatchXX (where XX is the module token 'LG' (for login services), 'MX' (for mail services) or 'WB' (for WEB services/httpd) respectively)

In principle, all modules have the same architecture ...

Each module has an exclusive FIFO in the system, that is fed with a replicated logging line as it is going to the log file in /var/log/...

This must be established in the system logger's (rsyslog or syslog-ng [recommended]) configuration file.

Example for syslog-ng in /etc/syslog-ng/syslog-ng.conf for the LG module:

destination d_auth { file("/var/log/secure");
                     pipe("/var/log/.pipes/WatchLG"
                     flush_lines (0));
};

Module-related filter rules are read from the 'rules' sub-directory of the module (e.g. ../modules/WatchLG/rules/..).

On a match with a filter rule the 'injector' is called and registers the attacker in the database and counts the attacks in the database as well until a configurable limit is reached. If the limit is reached a DROP action for the firewall will be flagged in the database and with the next attempt the attacker will be shot by establishing a DROP in the firewall.

IP addresses are not just IP addresses ...

In the Watcher concept IP addresses get 'classified' by the way in which they relate to forward and reverse DNS resolution. An IP address that does not resolve to a DNS name from a DNS request (with 'nslookup', 'host', or 'dig') is a so-called NXDOMAIN (Non-eXistend domain): i.e. the IP address is not registered by any DNS - which makes the address suspicious.

If a DNS name could be resolved but the reverse resolution does not resolve to the requesting IP address this is a case of a FAKEHOST which is even more suspicious and in all probability a case of 'IP spoofing'.

Only if an IP address can be resolved to a DNS name and the reverse resolution of the name resolved to the IP address of the request the IP address is classified as TRUEHOST - which is still not a sign of trust since a TRUEHOST might have been hacked and brought under the control of an attacker ...

Requests from NXDOMAINs & FAKEHOSTs get their event counters preset to MAXAFFAIRS-1; i.e. they are shot by a firewall DROP on 2nd affair.

NXDOMAINs don't have any legal business to do on legal mail servers (MTAs) anyway, since MTAs are not 'open relays' for anyone to send in email messages. And FAKEHOSTs are suspicious all over the place and get the same treatment as NXDOMAINs: Dead on 2nd attempt ...

 
(excerpt from a module log; here WatchLG)

Dynamic rule system

With Watcher Version 1.2 the 'dynamic rule system' was introduced.
The dynamic rule system allows rules to be defined for practically any service that provides log lines to a module.

The rules are no longer 'hard coded' in a module's code but are provided in separate "*.rule" files in a sub-directory named  'rules'.
These files are 'assembled' into a 'filter' block every time the module is [re]started. This way -and in particular the MX module- can be adapted to any MTA (postfix, exim, qmail, etc.) and mailbox service (DBmail, dovecot, etc.) of choice.

The supplied rule sets in the distribution are ready-made for:

• sshd                    (LG module)
• Postfix & DBmail (MX/MB module)
• Apache 2.x          (WB module)  

For other services, the rules can easily be changed by just modifying the 'Pattern' variable in the rule's heading:

RULE="NXdomain"
Pattern=': connect from unknown\['
#--------- Don't change below --------------------------------
          result=`echo "$REPLY" | grep -E "($Pattern)"`
          if [ ! -z "$result" ]
          then : echo "------- Matched rule $RULE --------"
                 inject
                 return $?
          fi
#-------------------------------------------------------------

For Watcher release 1.3 the 'dynamic rule system' was consolidated and simplified for speed optimization.

It now uses an internal BASH command that avoids calls to grep and compares the log message in $REPLY directly 'in memory' instead of transporting the log message by 'echo'. This leads to a tremendous speed-up in traversing the compiled 'rope ladder' of rules.

• 'Pattern' can now be a simple string and does not need the escaping of special REGEX characters in the definition.
• The 'decision block' is now formulated as a 'one-liner', that can be easier copied.

What is needed to run Watcher ??

As said in the introduction Watcher works based on 'on-board tools' and is targeted to Linux flavors of UNIX systems. So the GNU renditions of the system tools must be available. Namely: bash, awk, grep, sort & sed are essential to operate Watcher.

Also, the firewall management is based on 'iptables' and 'iptables-services' packages. Since Watcher revision 1.2 'ipset' is involved.

Note: Watcher does not support "firewalld', since firewalld is just a Python wrapper around 'nftables' and the user-space program 'nft'.

In addition 'ipcalc' is used for IP address validation on RHEL systems to suppress that junk gets stored for an attacker in the databases. (On Debian-style systems 'ipcalc' is automatically mimicked, since 'ipcalc' in Debianish systems [Debian/Ubuntu/Mint,...] is incompatible with the original 'ipcalc' found in RHEL systems) 

To run any of the modules 'sqlite3' is needed to run a module since modules count attack events in a database and store event type, event date, IP address, and other information in the database for fast storage and retrieval during processing.

How can you get Watcher??

Watcher does not have a price - it is given away for a small 'donation' and is available in the online shop.

1. In the first step register as a user on the main page https://comserve-it-services.de
2. Watch your mailbox and reply to the confirmation email to finish the registration process
3. Log in to the site and then simply order your Watcher product straight through the online shop by pretending a usual order process.  After payment of the 'donation' you will receive a usual invoice with the download link(s) for the chosen products in the invoice. Just click on the links in the invoice to accomplish the download ...
4. We recommend using PayPal since this will speed up the processing so that it just takes seconds and is fully automatic.
   (If you have chosen 'bank transfer' it takes the time we need to recognize and manually confirm the payment)

With the payment of the 'donation', you will also get access to the support area at https://watcher.comserve-it-services.de 

In the support area you can find the following:

• a Forum to discuss your experience with other users and you may ask questions to get help and advice.
• an Issue tracking system where you can drop a ticket that describes any problems you have experienced.

Have any questions ??

If anything is unclear to you just use the contact form in the menu  (under "Info > Anfrage" in the German section or "Contact form" in the English section) and provide us with your request. We will answer your request as soon as possible.

For registered users, a support site is set up under https://watcher.comserve-it-services.de
Here users have an opportunity to:

• Read the FAQ
• Read the support WiKi
• Read and write articles to share their own ideas
• Read and write in the forum to:
  • Discuss their experiences
  • Ask for help from other users and offer help to other users
  • Make proposals
• Drop a ticket in case of troubles and/or unexpected issues with Watcher.