Modules are dynamically fed by the system logger (rsyslog or syslog-ng). The modules analyse logging messages in real-time,  keep track of attackers in databases and feed the firewall autonomously with DROPs as 'bandits' reach configurable limits of attacking your server system.

Why Bash & Co.

Watcher is fully based on 'onboard tools' (bash, awk, grep, etc.; 'coreutils' et al.) that are common for every unixoid operating system and these are present out-of-the-box after system installation on a server system.

• Bash is fast! Due to 'builtins' for so-called 'transient programs' on disk use of Bash prevents for superflous disk accesses which increases processing speed profoundly.
  
  
• No compilation is needed.
  
  
• Watcher does not need any special runtime environments for fancy 3rd-party programing languages like PERL, JAVA, PYTHON, LUA, RUBY, etc. So there is no need to install and maintain development tools, runtime environments onto a server system.
  
  
• AWK (a.k.a. "the chain saw for text") is used for long 'list files', since AWK can be up to a 100 times faster than BASH if it comes to traverse and process long list files.
  .
• All programs & tools are 'self-aware'; i.e. every component always 'knows' where it was started in relation to the install path (MASTER_PATH). So the entire program-system can be installed wherever you like or can even be moved around  to your taste and/or needs.
  
  
• The modules are self-cleaning through automated 'database expiration' along with a consecutive firewall cleanup. This automatic housekeeping also keeps maintenance tasks away from you.

What's the audience  for Watcher?

Everyone who has to operate a Linux-based 'root server' (physical or VPS)  in a data center with 'rental servers' from a provider.

Such server systems  are usually running as 'exposed hosts' and are not protected by any external firewalling appliance. The provider leaves the protection up to you, since the provider cannot know which type and range of protection is needed for the purpose that you operate your server for.

Also servers that are running in the DMZ of a usual data center situation can take benefit from Watcher; e.g. mail servers, that are the primary targets for attackers,  SPAM spreaders and hijackers.

What does Watcher do for you?

• Watcher cuts down the attack rate of certain services on your server by a factor of 100 or even more. So it protects your server (and the services in particular) from [D]DoS ([Distributed] Denial-of-Service) attacks very efficiently.
  
  
• Frequently downloading the drop & edrop lists from 'SpamHaus' and integrating them into your firewall prevents your server from 'hijacking attacks' and cases of 'IP spoofing' continuously conducted by known professional attackers.
  
  
• The module WatchLG automatically tracks break-in attempts to the console login service (usually 'sshd') in real-time in a database at login level and prevents your system from being brought under control of attackers.
  
  
• Using the module WatchMX for a mail server tracks your mail server log in real-time and prevents the mailserver from dealing with SPAM spreaders, so that the mail server (e.g. PostFix) and mailbox services (DBmail, DoveCot, ...) can do their job undisturbed and with best performance.  WatchMX comes with a companion process WatchMB that takes care of malicious mailbox access and prevents futher access attempts to mailboxes, if a configurable number of 'affairs' are exeeded.
  
  
• The module WatchWB (since Q3/2021 along with Watcher 1.3) tracks the logging of your web server instances (vhosts) for malicious access attempts and in particular for flooding requests by 'crawlers' and/or 'robots' - 'bots' for short. WatchWB is the most complex module of the three as a web server can service multiple web site instances (customers) on a single physical server system. So the tracking must include mechanisms, that take care which 'instance' was attacked by which IP address. For security reasons this information must be kept separated in reports and statistics information that goes to several customers.

What else?

• "Watcher-Report" provides you with frequent reports of the firewall state, dynloader provisioning and the database state of the fully dynamic modules that are installed.
  
  
• For long-term reporting separate 'statistics reports' from the 'Stat tools'  of the module databases will get you an overlook of the efficiency of all the measures that you have taken. The statistics data is provided in CSV format so can you can simply load it into a spreadsheet of your choice.
  
  
• Frequent database expiration (configurable) in the modules keeps the firewall entries at a reasonable level, since a firewall that is flooded with pointless and hypothetic IP addresses can slow down your network traffic.
  
  
• The modules store only IP addresses for firewall DROPs of IP addresses, that are really attacking your system. So flooding your firewall with lists of hypthetical attacker assumptions from elsewhere on the Internet is not neccesary.

Watcher construction

Starting with Watcher release 1.0 the system was modularized; i.e. broken up into:

• Watcher master (framework & library for dynloaders & modules)
• WatchLG (login tracker module)
• WatchMX (mail services tracking module with a companion process WatchMB that takes care of break-in affairs to mailbox accounts [POP & IMAP] and mail transport requests that are protected with credetials; e.g. by SASL)
• WatchWB (protecting WEB services (http server) [under contruction; coming in Q3 along with Watcher release 1.3)]
• The DynLoaders (CRON controlled) are from the early days of the Watcher project and are re-worked for better integration with the modularized concept. They are included in the master package as they are really handy for basic mail server protection and hijacking prevention by known professional attackers.

What makes the modules so different?

• Each module runs autonomously if once started by the master.
• Modules are directly (syncronously) fed by the system logger and track reported complains from the related service in real-time with instant measure; i.e. a DROP in the firewall if a configurable limit is exceeded. So detection & reaction happen within milliseconds at the moment of the attack.
• Modules store the attack event and measure in a database to fully restore the firewall within seconds after start-up & reboot.
  Due to the enormous load speed it makes sense to restart 'watcher' from crontab on a regular basis; e.g. once a day or even every hour.
  This can keep the 'in memory' firewall clean and compact for best performance.
• Each module provides an 'open inject interface' provided as a FIFO (named pipe) in $FIFOBASE/WatchXX (where XX is the module token 'LG' (for login services), 'MX' (for mailservices) or 'WB' (for WEB services/httpd) respectively)

In principle all modules have the same architecture ...

Each module has an exclusive FIFO in the system, that is fed with a replicated logging line as it going to the log file in /var/log/...

This must be established in the system-logger's (rsyslog or syslog-ng [recommended]) configuration file.

Example for syslog-ng in /etc/syslog-ng/syslog-ng.conf for the LG module:

destination d_auth { file("/var/log/secure");
                     pipe("/var/log/.pipes/WatchLG"
                     flush_lines (0));
};

Module related filter rules are read from 'rules' sub-directory of the module (e.g. ../modules/WatchLG/rules/..).

On a match with a filter rule the 'injector' is called that registers the attacker in the database and counts the attacks in the database as well until a configurable limit is reached. It the limit is reached a DROP action for the firewall will be flagged in the database and with the next attempt the attacker will be shot by establishing a DROP in the firewall.

IP addresses are not just IP addresses ...

In the Watcher concept IP addresses get 'classified' by the way of which they relate to forward and reverse DNS-Resolution. An IP address that does not resolve to a DNS name from a DNS request (with 'nslookup', 'host', or 'dig') is a so-called NXDOMAIN (Non-eXistend domain): i.e. the IP address is not registered by any DNS - which makes the address suspicious.

If a DNS name could be resolved but the reverse resolution does not resolve to the requesting IP address this is a case of a FAKEHOST which is even more suspicious and in all probability a case of 'IP spoofing'.

Only if an IP address can be resolved to a DNS name and the reverse resolution of the name resolved to the IP address of the request then the IP address is classified as TRUEHOST - which is still not a sign of trust, since a TRUEHOST might have  been hacked and  brought under control of an attacker ...

Requests from NXDOMAINs & FAKEHOSTs get their event counters preset to MAXAFFAIRS-1; i.e. they are shot by a firewall DROP on 2nd affair.

NXDOMAINs don't have any legal business to do on legal mail servers (MTAs) anyway, since MTAs are not 'open relays' for anyone to send-in email messages. And FAKEHOSTs are suspicious all over the place and get the same treatment than NXDOMAINs: Dead on 2nd attempt ...

 
(excerpt from a module log; here WatchLG)

With Watcher Version 1.2 the 'dynamic rule system' was introduced.

The rules are no longer 'hard coded' in the module's code but are provided by a sub-directory named  'rules' in separate "*.rule" files. These files are 'assembled' into a 'filter block' every time the module is [re]started. This way -and in particular the MX module- can be adapted to any MTA (postfix, exim, qmail, etc.) and mailbox service (DBmail, dovecot, etc.) of choice.

The supplied rule-sets in the distribution are ready-made for:

• sshd                    (LG module)
• postfix & DBmail (MX/MB module)
• apache 2.x          (WB module)  

For other services the rules can easily be changed by just modifying the 'Pattern' variable in the rule's heading:

RULE="NXdomain"
Pattern=': connect from unknown\['
#--------- Don't change below --------------------------------
          result=`echo "$REPLY" | grep -E "($Pattern)"`
          if [ ! -z "$result" ]
          then : echo "------- Matched rule $RULE --------"
                 inject
                 return $?
          fi
#-------------------------------------------------------------

For Watcher release 1.3 the 'dynamic rule system' was consolidated and simplified for speed optimizaition.

It now uses an internal BASH command that avoids calls to grep and compares the log message in $REPLY directly 'in memory' instead of transporting the log message by 'echo'. This leads to a tremendous speed-up in traversing the compiled 'rope ladder' of rules.

• 'Pattern' can now be a simple string and does not need escaping of special REGEX characters in the definition.
• The 'decision block' is now formulated as a 'one liner', that can be easier copied.

What is needed to run Watcher ??

As said in the introduction Watcher works based on 'on-board tools' and is targeted to Linux flavours of UNIX systems. So the GNU renditions of the system tools must be availalble. Namely: bash, awk, grep, sort & sed are essential to operate Watcher.

Also the firewall management is based on 'iptables' and 'iptables-services' package. Since Watcher revision 1.2 'ipset' is involved.

Note: Watcher does not support "firewalld', since firewalld is just a Python wrapper around 'nftables' and the user-space program 'nft'.

In addition 'ipcalc' is used for IP address validation on RHEL systems to supress that junk gets stored for an attacker in the databases. (On Debian-style systems 'ipcalc' is automatically mimiced, since 'ipcalc' in Debianish systems [Debian/Ubuntu/Mint,...] is incompatible with the original 'ipcalc' found in RHEL systems) 

To run any of the modules 'sqlite3' is needed to run a module as modules count attack events in a database and store event-type, event-date, ip address and other information in the database for fast storage and retrieval during processing.

How can you get Watcher??

Watcher does not have a price - it is given away for a small 'donation' and is available in the online-shop.

1. In the first step register as a user on the main page https://comserve-it-services.de
2. Watch your mailbox and reply to the confirmation email to finish the registration process
3. Log in to the site and then simply order your Watcher product straight through the online shop by pretending a usual order process.  After payment of the 'donation' you will receive a usual invoice with the download link(s) for the chosen products in the invoice. Just click on the links in invoice to accomplish the download ...
4. We recommend to use PayPal, since this will speed-up the processing so that it just takes seconds and is fully automatic.
   (If you have chosen 'bank transfer' it takes the time that we need to recognize and manually confirm the payment)

With payment of the 'donation' you will also get access to the support area at https://watcher.comserve-it-services.de 

In the support area you can find:

• a Forum to discuss your experience with other users and you may ask questions to get help and advice.
• an Issue tracking system where you can drop a ticket that describes any problems you have got.

Have any questions ??

If anything is unclear to you just use the contact form in the menu  (under "Info > Anfrage" in the German section or "Contact form" in the English section) and provide us with your request. We will answer your request as soon as possible.

For registered users a support site is set up under https://watcher.comserve-it-services.de
Here users have an opportunity to:

• Read the FAQ
• Read the support WiKi
• Read and write articles to share their own ideas
• Read and write in the forum to:
  • Discuss their experiences
  • Ask for help by other users and offer help to other users
  • Make proposals
• Drop a ticket in case of troubles and/or unexpected issues with Watcher.