... the real-time intrusion detection & prevention system and superordinate firewall manager
 
Watcher Base logo
 
Why wait until attackers get
 
through to your services ?
 

What is Watcher? ... and what it is NOT?

 
Watcher is a security tool-set for networked Linux server systems that provides real-time intrusion detection and prevention and efficiently locks out malicious attackers from access to the server by putting firewall DROPs into the firewall.

Watcher is not a network analysis tool-set to analyse ethernet packages in all detail that are going in and out from the server system as this is not needed for the purpose. Much analysis takes much processing time and when facing attacks like 'brute-force attacks' where attackers flood a service on a server in milli-seconds processing time is a preciuos resource if proactive reaction in real-time is the demand.

"Do one thing and do it well... is the secret for performance and reliability.
 
Latest release:
  • Watcher 1.3 is now available in the online shop.
  • It will come along with the WatchWB module for WEB server protection.
  • Watcher 1.3 is coming with adaption for Debian-style & SuSE-style Linux distributions.
 
 
Nightly package builts since the release and for the ongoing development of Watcher 1.4 can be found in the public repository at no charge.

 
 

Motivation ...


Attacking server systems turned into a kind of new "public sports" during the last 5 years.

From 2015 until end of 2019 the reported numbers of malicious IP addresses that are attacking server systems on the open Internet has grown from about 1 million to 55 million attacks identified by IP addresses that makes up an increase of about 300% by every year.
This is exponential grows ...

The majority of attacks goes into account of mail services (mail transport & mailbox access; ~60%) followed by console access services (login/sshd; ~25%). The remaining 15% go into account of other services like chiefly WEB services and other informational services.

Spreaders of SPAM, SCAM & Phishing email etc. earn about 10 to 15 thousands US dollars per week and a number of 2005 says, that 5.4 billion US dollars were made with spreading SPAM - 15 years ago ... today's numbers can be expected tremendously higher.

 
 
Atacken 2010 2019 
(Source: AbuseIPDB)
 
 
In 2020 the reported number of server attacks "only" doubled to ~118  million compared to ~56 million in year 2019 ...
 Atacken 2010 2020
 (Source: AbuseIPDB)
 
... but the "quality" by which internet bandits conduct their attacks has increased as well.
 

The remedy ...


Detect and react ...

The only effective counter measure is a realtime intrusion detection system (RIDS) with instant measure by firewall DROPs to get attacking IP addresses under control before these can mess with the services on a server system. Nowadays attackers have 'tools' to deploy brute-force-attacks that can fire over 20 attacks per second against a server system. With 'traditional' log file scanning it is not even closely possible to detect such an aggressive behavior ... not to speak of the defense ...

Watcher fills your firewall lightning-fast at machine start-up or after reboot from:

  • Static lists (individual blacklists and a whitelist; manually maintained)

  • DynLoaders (from common and reliable public resources like 'SpamHaus' & 'NixSpam'; Dynloaders for SpamHaus & NixSpam are included with the master-package at no charge. Instructions on how you can build your individual dynloader can be found in the Watcher-Master manual.

  • Optional Watcher modules for:

    • Login [sshd]

    • Mail transport [MTA] & Mailbox access [POP, IMAP],

    • WEB SERVER [httpd]

are working fully dynamic and provide intrusion detection & prevention in real-time. The attacking events are tracked and stored in the exclusive databases of  each module for rapid retrieval with predictable processing times.


Modules are dynamically fed by the system logger (rsyslog or syslog-ng). The modules analyse logging messages in real-time,  keep track of attackers in databases and feed the firewall autonomously with DROPs as 'bandits' reach configurable limits of attacking your server system.


Watcher 1.3 start
(Start-up screen if manually [re]started)

 

On start-up Watcher can flush-load your firewall at rates of 10,000 DROPs per second or far more depending on the power and/or load state of your server machine. I have seen loadrates of over 22.000 DROPs per second on a VPS (i7, 2 cores) in phases of low system load.

 

 

Why Bash & Co.


Watcher is fully based on 'onboard tools' (bash, awk, grep, etc.; 'coreutils' et al.) that are common for every unixoid operating system and these are present out-of-the-box after system installation on a server system.

  • Bash is fast! Due to 'builtins' for so-called 'transient programs' on disk use of Bash prevents for superflous disk accesses which increases processing speed profoundly.

  • No compilation is needed.

  • Watcher does not need any special runtime environments for fancy 3rd-party programing languages like PERL, JAVA, PYTHON, LUA, RUBY, etc. So there is no need to install and maintain development tools, runtime environments onto a server system.

  • AWK (a.k.a. "the chain saw for text") is used for long 'list files', since AWK can be up to a 100 times faster than BASH if it comes to traverse and process long list files.
    .
  • All programs & tools are 'self-aware'; i.e. every component always 'knows' where it was started in relation to the install path (MASTER_PATH). So the entire program-system can be installed wherever you like or can even be moved around  to your taste and/or needs.

  • The modules are self-cleaning through automated 'database expiration' along with a consecutive firewall cleanup. This automatic housekeeping also keeps maintenance tasks away from you.

 

What's the audience  for Watcher?


Everyone who has to operate a Linux-based 'root server' (physical or VPS)  in a data center with 'rental servers' from a provider.

Such server systems  are usually running as 'exposed hosts' and are not protected by any external firewalling appliance. The provider leaves the protection up to you, since the provider cannot know which type and range of protection is needed for the purpose that you operate your server for.

Also servers that are running in the DMZ of a usual data center situation can take benefit from Watcher; e.g. mail servers, that are the primary targets for attackers,  SPAM spreaders and hijackers.

 

What does Watcher do for you?


  • Watcher cuts down the attack rate of certain services on your server by a factor of 100 or even more. So it protects your server (and the services in particular) from [D]DoS ([Distributed] Denial-of-Service) attacks very efficiently.

  • Frequently downloading the drop & edrop lists from 'SpamHaus' and integrating them into your firewall prevents your server from 'hijacking attacks' and cases of 'IP spoofing' continuously conducted by known professional attackers.

  • The module WatchLG automatically tracks break-in attempts to the console login service (usually 'sshd') in real-time in a database at login level and prevents your system from being brought under control of attackers.

  • Using the module WatchMX for a mail server tracks your mail server log in real-time and prevents the mailserver from dealing with SPAM spreaders, so that the mail server (e.g. PostFix) and mailbox services (DBmail, DoveCot, ...) can do their job undisturbed and with best performance.  WatchMX comes with a companion process WatchMB that takes care of malicious mailbox access and prevents futher access attempts to mailboxes, if a configurable number of 'affairs' are exeeded.

  • The module WatchWB (since Q3/2021 along with Watcher 1.3) tracks the logging of your web server instances (vhosts) for malicious access attempts and in particular for flooding requests by 'crawlers' and/or 'robots' - 'bots' for short. WatchWB is the most complex module of the three as a web server can service multiple web site instances (customers) on a single physical server system. So the tracking must include mechanisms, that take care which 'instance' was attacked by which IP address. For security reasons this information must be kept separated in reports and statistics information that goes to several customers.

What else?


  • "Watcher-Report" provides you with frequent reports of the firewall state, dynloader provisioning and the database state of the fully dynamic modules that are installed.

  • For long-term reporting separate 'statistics reports' from the 'Stat tools'  of the module databases will get you an overlook of the efficiency of all the measures that you have taken. The statistics data is provided in CSV format so can you can simply load it into a spreadsheet of your choice.

  • Frequent database expiration (configurable) in the modules keeps the firewall entries at a reasonable level, since a firewall that is flooded with pointless and hypothetic IP addresses can slow down your network traffic.

  • The modules store only IP addresses for firewall DROPs of IP addresses, that are really attacking your system. So flooding your firewall with lists of hypthetical attacker assumptions from elsewhere on the Internet is not neccesary.

 

Watcher construction


Starting with Watcher release 1.0 the system was modularized; i.e. broken up into:

  • Watcher master (framework & library for dynloaders & modules)
  • WatchLG (login tracker module)
  • WatchMX (mail services tracking module with a companion process WatchMB that takes care of break-in affairs to mailbox accounts [POP & IMAP] and mail transport requests that are protected with credetials; e.g. by SASL)
  • WatchWB (protecting WEB services (http server) [under contruction; coming in Q3 along with Watcher release 1.3)]
  • The DynLoaders (CRON controlled) are from the early days of the Watcher project and are re-worked for better integration with the modularized concept. They are included in the master package as they are really handy for basic mail server protection and hijacking prevention by known professional attackers.

 

Overall Arch 1.3
 

 

What makes the modules so different?


  • Each module runs autonomously if once started by the master.
  • Modules are directly (syncronously) fed by the system logger and track reported complains from the related service in real-time with instant measure; i.e. a DROP in the firewall if a configurable limit is exceeded. So detection & reaction happen within milliseconds at the moment of the attack.
  • Modules store the attack event and measure in a database to fully restore the firewall within seconds after start-up & reboot.
    Due to the enormous load speed it makes sense to restart 'watcher' from crontab on a regular basis; e.g. once a day or even every hour.
    This can keep the 'in memory' firewall clean and compact for best performance.
  • Each module provides an 'open inject interface' provided as a FIFO (named pipe) in $FIFOBASE/WatchXX (where XX is the module token 'LG' (for login services), 'MX' (for mailservices) or 'WB' (for WEB services/httpd) respectively)

 

In principle all modules have the same architecture ...

 

Module Architecture

Each module has an exclusive FIFO in the system, that is fed with a replicated logging line as it going to the log file in /var/log/...

This must be established in the system-logger's (rsyslog or syslog-ng [recommended]) configuration file.

Example for syslog-ng in /etc/syslog-ng/syslog-ng.conf for the LG module:


destination d_auth { file("/var/log/secure");
                     pipe("/var/log/.pipes/WatchLG"
                     flush_lines (0));
};


Module related filter rules are read from 'rules' sub-directory of the module (e.g. ../modules/WatchLG/rules/..).

On a match with a filter rule the 'injector' is called that registers the attacker in the database and counts the attacks in the database as well until a configurable limit is reached. It the limit is reached a DROP action for the firewall will be flagged in the database and with the next attempt the attacker will be shot by establishing a DROP in the firewall.

 


IP addresses are not just IP addresses ...

In the Watcher concept IP addresses get 'classified' by the way of which they relate to forward and reverse DNS-Resolution. An IP address that does not resolve to a DNS name from a DNS request (with 'nslookup', 'host', or 'dig') is a so-called NXDOMAIN (Non-eXistend domain): i.e. the IP address is not registered by any DNS - which makes the address suspicious.

If a DNS name could be resolved but the reverse resolution does not resolve to the requesting IP address this is a case of a FAKEHOST which is even more suspicious and in all probability a case of 'IP spoofing'.

Only if an IP address can be resolved to a DNS name and the reverse resolution of the name resolved to the IP address of the request then the IP address is classified as TRUEHOST - which is still not a sign of trust, since a TRUEHOST might have  been hacked and  brought under control of an attacker ...

Requests from NXDOMAINs & FAKEHOSTs get their event counters preset to MAXAFFAIRS-1; i.e. they are shot by a firewall DROP on 2nd affair.

NXDOMAINs don't have any legal business to do on legal mail servers (MTAs) anyway, since MTAs are not 'open relays' for anyone to send-in email messages. And FAKEHOSTs are suspicious all over the place and get the same treatment than NXDOMAINs: Dead on 2nd attempt ...

Log excerpt 
(excerpt from a module log; here WatchLG)

 


With Watcher Version 1.2 the 'dynamic rule system' was introduced.

The rules are no longer 'hard coded' in the module's code but are provided by a sub-directory named  'rules' in separate "*.rule" files. These files are 'assembled' into a 'filter block' every time the module is [re]started. This way -and in particular the MX module- can be adapted to any MTA (postfix, exim, qmail, etc.) and mailbox service (DBmail, dovecot, etc.) of choice.

The supplied rule-sets in the distribution are ready-made for:

  • sshd                    (LG module)
  • postfix & DBmail (MX/MB module)
  • apache 2.x          (WB module)  

For other services the rules can easily be changed by just modifying the 'Pattern' variable in the rule's heading:

RULE="NXdomain"
Pattern=': connect from unknown\['
#--------- Don't change below --------------------------------
result=`echo "$REPLY" | grep -E "($Pattern)"`
if [ ! -z "$result" ]
then : echo "------- Matched rule $RULE --------"
inject
return $?
fi
#-------------------------------------------------------------

 

For Watcher release 1.3 the 'dynamic rule system' was consolidated and simplified for speed optimizaition.

It now uses an internal BASH command that avoids calls to grep and compares the log message in $REPLY directly 'in memory' instead of transporting the log message by 'echo'. This leads to a tremendous speed-up in traversing the compiled 'rope ladder' of rules.

  • 'Pattern' can now be a simple string and does not need escaping of special REGEX characters in the definition.
  • The 'decision block' is now formulated as a 'one liner', that can be easier copied.

RULE="NXdomain"
Pattern=': connect from unknown['
#--------- Decision --------------------------------
if [[ "$REPLY" =~ "$Pattern" ]]; then inject; return $?; fi

 

 

What is needed to run Watcher ??

As said in the introduction Watcher works based on 'on-board tools' and is targeted to Linux flavours of UNIX systems. So the GNU renditions of the system tools must be availalble. Namely: bash, awk, grep, sort & sed are essential to operate Watcher.

Also the firewall management is based on 'iptables' and 'iptables-services' package. Since Watcher revision 1.2 'ipset' is involved.

Note: Watcher does not support "firewalld', since firewalld is just a Python wrapper around 'nftables' and the user-space program 'nft'.

In addition 'ipcalc' is used for IP address validation on RHEL systems to supress that junk gets stored for an attacker in the databases. (On Debian-style systems 'ipcalc' is automatically mimiced, since 'ipcalc' in Debianish systems [Debian/Ubuntu/Mint,...] is incompatible with the original 'ipcalc' found in RHEL systems) 

To run any of the modules 'sqlite3' is needed to run a module as modules count attack events in a database and store event-type, event-date, ip address and other information in the database for fast storage and retrieval during processing.

 

How can you get Watcher??

Watcher does not have a price - it is given away for a small 'donation' and is available in the online-shop.

  1. In the first step register as a user on the main page https://comserve-it-services.de
  2. Watch your mailbox and reply to the confirmation email to finish the registration process
  3. Log in to the site and then simply order your Watcher product straight through the online shop by pretending a usual order process.  After payment of the 'donation' you will receive a usual invoice with the download link(s) for the chosen products in the invoice. Just click on the links in invoice to accomplish the download ...
  4. We recommend to use PayPal, since this will speed-up the processing so that it just takes seconds and is fully automatic.
    (If you have chosen 'bank transfer' it takes the time that we need to recognize and manually confirm the payment)

With payment of the 'donation' you will also get access to the support area at https://watcher.comserve-it-services.de 

In the support area you can find:

  • a Forum to discuss your experience with other users and you may ask questions to get help and advice.
  • an Issue tracking system where you can drop a ticket that describes any problems you have got.

 

Have any questions ??

If anything is unclear to you just use the contact form in the menu  (under "Info > Anfrage" in the German section or "Contact form" in the English section) and provide us with your request. We will answer your request as soon as possible.


For registered users a support site is set up under https://watcher.comserve-it-services.de
Here users have an opportunity to:

  • Read the FAQ
  • Read the support WiKi
  • Read and write articles to share their own ideas
  • Read and write in the forum to:
    • Discuss their experiences
    • Ask for help by other users and offer help to other users
    • Make proposals
  • Drop a ticket in case of troubles and/or unexpected issues with Watcher.

 

This system uses cookies, since it is essential to operate an online-shop. I am aware that cookies will store notes on my computer about the contact with the visited WEB site and I accept this.