... the real-time intrusion detection & prevention system and superordinate firewall manager
 
Watcher Base logo
 
Why wait until attackers get
 
through to your services?
 

What is Watcher? ... and what is it NOT?

 
Watcher is a security toolset for networked Linux server systems that provides real-time intrusion detection and prevention and efficiently locks out malicious attackers from accessing the server by putting firewall DROPs into the firewall.
 
Watcher is in particular designed for so-called 'root servers' that run million-wise in data centers as 'exposed hosts' on the open Internet. This is the same situation as servers, that run in the DMZ of regular company datacenters for which Watcher can also be used.

Watcher is not a network analysis tool-set to analyze ethernet packages in all detail that are going in and out from the server system because this is absolutely not needed for the purpose. Much analysis takes much processing time and when facing attacks like 'brute-force attacks' where attackers flood a service on a server in milli-seconds, processing time is a precious resource if proactive reaction in realtime is the demand.

"Do one thing and do it well... is the secret to performance and reliability.
 
Latest release:
  • Watcher 1.3 is now available in the online shop.
  • It will come along with the WatchWB module for WEB server protection.
  • Watcher 1.3 is coming with adaption for Debian-style & SuSE-style Linux distributions.
 
 
Nightly package builts since the release and for the ongoing development of Watcher 1.4 can be found in the public repository at no charge.

 
 

Motivation ...


Attacking server systems turned into a kind of new "public sport" during the last 5 years.

From 2015 until the end of 2019, the reported number of malicious IP addresses that are attacking server systems on the open Internet has grown from about 1 million to 55 million attacks identified by IP addresses which makes up an increase of about 300% by every year.
This is exponential growth ...

The majority of attacks go into account of mail services (mail transport & mailbox access; ~60%) followed by console access services (login/sshd; ~25%). The remaining 15% goes into account for other services like (chiefly) WEB services and other informational services.

Spreaders of SPAM, SCAM, Phishing email, etc. earn about 10 to 15 thousand US dollars per week and a number in 2005 says, that 5.4 billion US dollars were made by spreading SPAM - 15 years ago ... today's numbers can be expected tremendously higher.

 
 
Atacken 2010 2019 
(Source: AbuseIPDB)
 
 
In 2020 the reported number of server attacks "only" doubled to ~118  million compared to ~56 million in year 2019 ...
 Atacken 2010 2020
 (Source: AbuseIPDB)
 
... but the "quality" by which internet bandits conduct their attacks has increased as well.
Although the number of attacks with such a rate of nearly 300% is decreasing there is still an increase of over 30% per year in the number of attacks.
 
Attacken 2010 2022
 
 (Source: AbuseIPDB)

The remedy ...


Detect and react ...

The only effective countermeasure is a real-time intrusion detection system (RIDS) with instant measures by firewall DROPs to get attacking IP addresses under control before these can mess with the services on a server system. Nowadays attackers have 'tools' to deploy brute-force-attacks that can fire over 20 attacks per second against a server system. Spammers cause, that over 85% (283 billion) of all email sent in a day (330 billion) is SPAM, which in turn must be processed somehow by mail servers.

With 'traditional' log file scanning it is not even closely possible to detect such an aggressive behavior ... not to speak of the defense ...

Watcher fills your firewall lightning-fast at machine start-up or after reboot from:

  • Static lists (individual 'blacklist' and 'whitelist' files; manually maintained)

  • DynLoaders (from common and reliable public resources like 'SpamHaus' & 'NixSpam'.
    Dynloaders for Spamhaus & NixSpam are included with the master package.
    Instructions on how you can build your individual dynloader can be found in the Watcher-Master manual.

  • Watcher modules for:

    • Login [sshd]

    • Mail transport [MTA] and mailbox access [POP, IMAP],

    • WEB server [httpd]

Modules are working continuously and fully dynamic and provide intrusion detection & prevention in real-time. The attacking events are tracked and stored in the exclusive databases of each module for rapid retrieval with predictable processing times.


Modules are dynamically fed by the system logger (rsyslog or syslog-ng). The modules analyze logging messages in real-time,  keep track of attackers in databases, and feed the firewall autonomously with DROPs as 'bandits' reach configurable limits of attacking your server system.


Watcher 1.3 start
(Start-up screen if manually [re]started)

 

On start-up, Watcher can flush-load your firewall at rates of 10,000 DROPs per second or far more depending on the power and/or load state of your server machine. I have seen load rates of over 22,000 DROPs per second on a VPS (i7, 2 cores) in phases of low system load.

 

 

Why Bash & Co.


Watcher is fully based on 'onboard tools' (bash, awk, grep, etc.; 'coreutils' et al.) that are common for every unixoid operating system and these are present out-of-the-box after system installation on a server system.

  • Bash is fast! Due to 'builtins' for so-called 'transient programs' on disk use of Bash prevents superfluous disk accesses which increases processing speed profoundly.

  • No compilation is needed.

  • Watcher does not need any special runtime environments for fancy 3rd-party programming languages like PERL, JAVA, PYTHON, LUA, RUBY, etc.
    So there is no need to install and maintain development tools or runtime environments for these onto a server system.

  • AWK (a.k.a. "the chain saw for text") is used for long 'list files' since AWK can be up to 100 times faster than BASH if it comes to traversing and processing long list files.
    .
  • All programs & tools are 'self-aware'; i.e. every component always 'knows' where it was started in relation to the install path (MASTER_PATH). So the entire program system can be installed wherever you like or can even be moved around to your taste and/or needs.

  • The modules are self-cleaning through automated 'database expiration' along with a consecutive firewall cleanup. This automatic housekeeping also keeps maintenance tasks away from you.

 

What's the audience for Watcher?


Everyone, who has to operate a Linux-based 'root server' (physical or VPS)  in a data center with 'rental servers' from a provider.

Such server systems are usually running as 'exposed hosts' and are not protected by any external firewalling appliance. The provider leaves the protection up to you since the provider cannot know which type and range of protection is needed for the purpose for which you operate your server.

Also, servers that are running in the DMZ of a usual data center situation can benefit from Watcher; e.g. mail servers, which are the primary targets for attackers,  SPAM spreaders, and hijackers.

 

What does Watcher do for you?


  • Watcher cuts down the attack rate of certain services on your server by a factor of 100 or even more. So it protects your server (and the services in particular) from [D]DoS ([Distributed] Denial-of-Service) attacks very efficiently.

  • Frequently downloading the drop & edrop lists from 'SpamHaus' and integrating them into your firewall prevents your server from 'hijacking attacks' and cases of 'IP spoofing' continuously conducted by known professional attackers.

  • The module WatchLG automatically tracks break-in attempts to the console login service (usually 'sshd') in real-time in a database at login level and prevents your system from being brought under the control of attackers.

  • Using the module WatchMX for a mail server tracks your mail server log in real-time and prevents the mail server from dealing with SPAM spreaders so that the mail server (e.g. PostFix) and mailbox services (DBmail, DoveCot, ...) can do their job undisturbed and with best performance.  WatchMX comes with a companion process WatchMB that takes care of malicious mailbox access and prevents further access attempts to mailboxes if a configurable number of 'affairs' are exceeded.

  • The module WatchWB (since Q3/2021 along with Watcher 1.3) tracks the logging of your web server instances (vhosts) for malicious access attempts and in particular for flooding requests by 'crawlers' and/or 'robots' - 'bots' for short. WatchWB is the most complex module of the three as a web server can service multiple website instances (customers) on a single physical server system. So the tracking must include mechanisms, that take care of which 'instance' was attacked by which IP address. For security reasons, this information must be kept separate in reports and statistics information that goes to several customers.

What else?


  • "Watcher-Report" provides you with frequent reports of the firewall state, dynloader provisioning, and the database state of the fully dynamic modules that are installed.

  • For long-term reporting separate 'statistics reports' from the 'Stat tools'  of the module databases will get you an overlook of the efficiency of all the measures that you have taken. The statistics data is provided in CSV format so you can simply load it into a spreadsheet of your choice.

  • Frequent database expiration (configurable) in the modules keeps the firewall entries at a reasonable level since a firewall that is flooded with pointless and hypothetic IP addresses can slow down your network traffic.

  • The modules store only IP addresses for firewall DROPs of IP addresses, that are really attacking your system. So flooding your firewall with lists of hypothetical attacker assumptions from elsewhere on the Internet is not necessary.

 

Watcher construction


Starting with Watcher release 1.0 the system was modularized; i.e. broken up into:

  • Watcher master (framework & library for dynloaders & modules)
  • WatchLG (login tracker module)
  • WatchMX (mail services tracking module with a companion process WatchMB that takes care of break-in affairs to mailbox accounts [POP & IMAP] and mail transport requests that are protected with credentials; e.g. by SASL)
  • WatchWB (protecting WEB services (HTTP server) [along with Watcher release 1.3]
  • The DynLoaders (CRON controlled) are from the early days of the Watcher project and are reworked for better integration with the modularized concept.
    They are included in the master package as they are really handy for basic mail server protection and hijacking prevention by known professional attackers.

 

Overall Arch 1.3
 

 

What makes the modules so different?


  • Each module runs autonomously if once started by the master.
  • Modules are directly (synchronously) fed by the system logger and track reported complaints from the related service in real-time with instant measure; i.e. a DROP in the firewall if a configurable limit of 'affairs' is exceeded. So detection & reaction happen within milliseconds at the moment of the attack.
  • Modules store the attack event and measure it in a database to fully restore the firewall within seconds after start-up & reboot.
    Due to the enormous load speed, it makes sense to restart 'watcher' from crontab on a regular basis; e.g. once a day.
    This can keep the 'in memory' firewall clean and compact for the best performance.
  • Each module provides an 'open inject interface' provided as a FIFO (named pipe) in $FIFO_BASE/WatchXX (where XX is the module token 'LG' (for login services), 'MX' (for mail services) or 'WB' (for WEB services/httpd) respectively)

 

In principle, all modules have the same architecture ...

 

Module Architecture

Each module has an exclusive FIFO in the system, that is fed with a replicated logging line as it is going to the log file in /var/log/...

This must be established in the system logger's (rsyslog or syslog-ng [recommended]) configuration file.

Example for syslog-ng in /etc/syslog-ng/syslog-ng.conf for the LG module:


destination d_auth { file("/var/log/secure");
                     pipe("/var/log/.pipes/WatchLG"
                     flush_lines (0));
};


Module-related filter rules are read from the 'rules' sub-directory of the module (e.g. ../modules/WatchLG/rules/..).

On a match with a filter rule the 'injector' is called and registers the attacker in the database and counts the attacks in the database as well until a configurable limit is reached. If the limit is reached a DROP action for the firewall will be flagged in the database and with the next attempt the attacker will be shot by establishing a DROP in the firewall.

 


IP addresses are not just IP addresses ...

In the Watcher concept IP addresses get 'classified' by the way in which they relate to forward and reverse DNS resolution. An IP address that does not resolve to a DNS name from a DNS request (with 'nslookup', 'host', or 'dig') is a so-called NXDOMAIN (Non-eXistend domain): i.e. the IP address is not registered by any DNS - which makes the address suspicious.

If a DNS name could be resolved but the reverse resolution does not resolve to the requesting IP address this is a case of a FAKEHOST which is even more suspicious and in all probability a case of 'IP spoofing'.

Only if an IP address can be resolved to a DNS name and the reverse resolution of the name resolved to the IP address of the request the IP address is classified as TRUEHOST - which is still not a sign of trust since a TRUEHOST might have been hacked and brought under the control of an attacker ...

Requests from NXDOMAINs & FAKEHOSTs get their event counters preset to MAXAFFAIRS-1; i.e. they are shot by a firewall DROP on 2nd affair.

NXDOMAINs don't have any legal business to do on legal mail servers (MTAs) anyway, since MTAs are not 'open relays' for anyone to send in email messages. And FAKEHOSTs are suspicious all over the place and get the same treatment as NXDOMAINs: Dead on 2nd attempt ...

Log excerpt 
(excerpt from a module log; here WatchLG)

 


Dynamic rule system

With Watcher Version 1.2 the 'dynamic rule system' was introduced.
The dynamic rule system allows rules to be defined for practically any service that provides log lines to a module.

The rules are no longer 'hard coded' in a module's code but are provided in separate "*.rule" files in a sub-directory named  'rules'.
These files are 'assembled' into a 'filter' block every time the module is [re]started. This way -and in particular the MX module- can be adapted to any MTA (postfix, exim, qmail, etc.) and mailbox service (DBmail, dovecot, etc.) of choice.

The supplied rule sets in the distribution are ready-made for:

  • sshd                    (LG module)
  • Postfix & DBmail (MX/MB module)
  • Apache 2.x          (WB module)  

For other services, the rules can easily be changed by just modifying the 'Pattern' variable in the rule's heading:

RULE="NXdomain"
Pattern=': connect from unknown\['
#--------- Don't change below --------------------------------
result=`echo "$REPLY" | grep -E "($Pattern)"`
if [ ! -z "$result" ]
then : echo "------- Matched rule $RULE --------"
inject
return $?
fi
#-------------------------------------------------------------

 

For Watcher release 1.3 the 'dynamic rule system' was consolidated and simplified for speed optimization.

It now uses an internal BASH command that avoids calls to grep and compares the log message in $REPLY directly 'in memory' instead of transporting the log message by 'echo'. This leads to a tremendous speed-up in traversing the compiled 'rope ladder' of rules.

  • 'Pattern' can now be a simple string and does not need the escaping of special REGEX characters in the definition.
  • The 'decision block' is now formulated as a 'one-liner', that can be easier copied.

RULE="NXdomain"
Pattern=': connect from unknown['
#--------- Decision --------------------------------
if [[ "$REPLY" =~ "$Pattern" ]]; then inject; return $?; fi

 

 

What is needed to run Watcher ??

As said in the introduction Watcher works based on 'on-board tools' and is targeted to Linux flavors of UNIX systems. So the GNU renditions of the system tools must be available. Namely: bash, awk, grep, sort & sed are essential to operate Watcher.

Also, the firewall management is based on 'iptables' and 'iptables-services' packages. Since Watcher revision 1.2 'ipset' is involved.

Note: Watcher does not support "firewalld', since firewalld is just a Python wrapper around 'nftables' and the user-space program 'nft'.

In addition 'ipcalc' is used for IP address validation on RHEL systems to suppress that junk gets stored for an attacker in the databases. (On Debian-style systems 'ipcalc' is automatically mimicked, since 'ipcalc' in Debianish systems [Debian/Ubuntu/Mint,...] is incompatible with the original 'ipcalc' found in RHEL systems) 

To run any of the modules 'sqlite3' is needed to run a module since modules count attack events in a database and store event type, event date, IP address, and other information in the database for fast storage and retrieval during processing.

 

How can you get Watcher??

Watcher does not have a price - it is given away for a small 'donation' and is available in the online shop.

  1. In the first step register as a user on the main page https://comserve-it-services.de
  2. Watch your mailbox and reply to the confirmation email to finish the registration process
  3. Log in to the site and then simply order your Watcher product straight through the online shop by pretending a usual order process.  After payment of the 'donation' you will receive a usual invoice with the download link(s) for the chosen products in the invoice. Just click on the links in the invoice to accomplish the download ...
  4. We recommend using PayPal since this will speed up the processing so that it just takes seconds and is fully automatic.
    (If you have chosen 'bank transfer' it takes the time we need to recognize and manually confirm the payment)

With the payment of the 'donation', you will also get access to the support area at https://watcher.comserve-it-services.de 

In the support area you can find the following:

  • a Forum to discuss your experience with other users and you may ask questions to get help and advice.
  • an Issue tracking system where you can drop a ticket that describes any problems you have experienced.

 

Have any questions ??

If anything is unclear to you just use the contact form in the menu  (under "Info > Anfrage" in the German section or "Contact form" in the English section) and provide us with your request. We will answer your request as soon as possible.


For registered users, a support site is set up under https://watcher.comserve-it-services.de
Here users have an opportunity to:

  • Read the FAQ
  • Read the support WiKi
  • Read and write articles to share their own ideas
  • Read and write in the forum to:
    • Discuss their experiences
    • Ask for help from other users and offer help to other users
    • Make proposals
  • Drop a ticket in case of troubles and/or unexpected issues with Watcher.

 

This system uses cookies, since it is essential to operate an online-shop. I am aware that cookies will store notes on my computer about the contact with the visited WEB site and I accept this.