What is new in Watcher revision 1.4?
With revision 1.4 Watcher introduces 'geological blocking' & 'geological tracking'; i.e. blocking and/or tracking 'by country' of registration.
Furthermore, when detecting a malicious country, not just a single IP address is blocked, but the entire subnet (CIDR) is blocked.
Geo-blocking & tracking come in two flavors.
- Dynloader 'geo'
- Module 'GeoTrack'
Find the documentation in: https://watcher.comserve-it-services.de/repo/Docs
Dynloader 'geo' ...
As usual, a dynloader (dynamic loader) picks up data from external resources and transforms the information into a loadable file to be loaded into an ipset of the firewall. The creation of the 'match-set' in 'iptables' and the creation of the corresponding 'ipset' is 'on-the-fly' for all dynloaders.
Like all other dynloaders the dynloader 'geo' is located in $MASTER_PATH/dynload/geo with the executable by the same name below this path; i.e., $MASTER_PATH/dynload/geo/geo
The 'geo' dynloader picks up the excellent data provided by 'ipdeny.com' updated once every 24 hours (at about 12:30 [PM]).
The update is managed by a crontab entry (as usual for dynloaders) shortly after the update time:
| ( ... crontab excerpt of user 'root'...) # * * * * * /usr/local/sbin/MyRouter >/dev/null 2>&1 #----------Dynloaders------------------------------------ |
'geo' then downloads the configured *.zone' files and transforms these into loadable files for 'ipset' according to the 'geo.conf' file in its working directory.
|
# - geo.conf - WGET='wget -q --no-check-certificate' |
As a result, load files for 'ipset' get created and are flushed into the corresponding ipset.
root@vmd28527 Watcher]# ipset -t list | grep -E "geo-"
Name: geo-cn
Name: geo-ru
Name: geo-kp
Name: geo-by
Name: geo-er
...
With the 'Watcher-Report' tool, the filling state can be easily viewed:
|
[root@vmd28527 Watcher]# Watcher-Report _____ Geo blockings ___________________________________________________________ |
Notice that the shown records are NOT always single IP addresses.
Often, these are CIDRs and thus complete subnets that get blocked if the '*-aggregated.zone' files are downloaded.
This can result in millions of blocked IP addresses (xIPs) for a given country (zone).
If you want or need IP addresses or entire subnets excluded from blocking, just put these addresses in your 'whitelist' file.
This works since the ipset whitelist is always checked first with a chain exit of ACCEPT.
|
[root@vmd28527 ~]# iptables -nL | head -6 |
Module 'GeoTrack' ...
The (pseudo-)module GeoTrack does not rely on any 3rd-party data. (e.g. AbuseIPdb and such).
Instead, it requests the data from official WHOIS servers (RIRs; Regional Internet Registries).
IANA (Internet Assigned Numbers Authority; http://iana.org; Los Angeles/USA) manages in a central place all assigned IP address ranges and so knows which RIR is responsible for a particular IP address range. From their own WHOIS server, IANA responds with the 'whois server' that has the regional responsibility for an IP address range.
|
[root@vmd28527 bin]# whois -h whois.iana.org 178.238.228.100 refer: whois.ripe.net inetnum: 178.0.0.0 - 178.255.255.255 whois: whois.ripe.net changed: 2009-01 |
GeoTrack architecture
The GeoTrack module is actually a pseudo-module as it does not serve a particular service (login, mail, web,...)
Like all other modules, it has its own database and reads its own exclusive FiFo ('named pipe'). But the pipe is fed by the other regular modules from the 'initial' section in the 'inject' function; i.e. if an attacker is initially registered to the module's database the attacker's IP address is also passed to the pipe of GeoTrack for further determination and processing.
The reason for this is that for geolocation determination, a very time-consuming request to 'whois' databases on the Internet must be conducted. Doing these 'whois' requests inside the scanner loops of the regular modules would tremendously slow down the working speed of the scanner in the regular modules. Instead, the IP address and the module name of the origin are simply passed to the GeoTrack (pseudo-)module for processing.
|
(excerpt from the WatchLG module) ... ACTION="Initial $AFFAIRS/$MAX_AFFAIRS" if [ ! -z "$GEOTRACK" ] ... |
With the 'GEOTRACK' configuration variable in the module's configuration file, geotracking can be turned 'on' or 'off' in general for a regular module. To turn it 'off' just set GEOTRACK to an empty string in the regular module; i.e. 'GEOTRACK='
In addition, a function in the common.bashlib checks that the GeoTrack pseudo-module was started and thus has established its listening pipe as '$FIFO_BASE/GeoTrack'.
| do_geotrack() { if [ -p $FIFO_BASE/GeoTrack ] then echo $BANDIT $ME >> $FIFO_BASE/GeoTrack fi } |
GeoTrack always determines a CIDR from the information on whois-servers and stores this in the ipset 'custody'. If a prefix bitmask cannot be determined, a fake-CIDR will be recorded in the GeoTrack database (geotrack.db) as '<IP_ADDR>/32' which will at least result in a 'point-to-point' blocking between the server and the attacking IP address.
The geotrack database only holds little information.
|
[root@vmd28527 GeoTrack]# sqlite --table geotrack.db |
Note: 'route' is a CIDR (complete sub-net) and the primary key in the table and thus the entry in the table is unique, 'ip' is just the IP address that has triggered the entry in the database.
So a mask length of '/14' (in the example above) will block 2^(32-14) = 2^18 = 262144 IP addresses below the network's base address '111.72.0.0' ... not just the one that has triggered the entry.
How GeoTrack differs from regular modules ...
The pseudo-module GeoTrack works without 'rules'. It just uses the 'ZONES=...' configured in the GeoTrack.conf configuration file; e.g.: ZONES='cn ru'
GeoTrack has an 'ENABLED=...' configuration variable that must be set to a non-empty value to enable the (pseudo-)module GeoTrack. If GeoTrack is not enabled, its listening pipe $FIFO_BASE/GeoTrack won't be established, and thus, the regular modules won't write to the GeoTrack listening pipe. (see 'do_geotrack' function above)
|
# # Set to a non-empty string to enable the GeoTrack (pseudo-)module # The list of zones to be blocked … |
Scenarios ...
The dynloader 'geo' and/or the (pseudo-)module 'GeoTrack' can be used 'standalone' or in combination (recommended).
If only using the 'geo' dynloader the information can be incomplete as there can be quite a couple of changes in the registration of IP address ranges for a particular country within the 24-hour update cycle. These are then not covered by the *.zone files, that ipdeny.com provides. As a result, attackers can slip through the door by frequently changing the IP address (ranges) they are using.
As typical for modules -and thus for the (pseudo-)module 'GeoTrack'- only IP addresses that are really attacking are detected by complaints from their corresponding log-information of the service that the module covers. So, GeoTrack has to 'learn' all these IP addresses for the countries to be blocked at first. Without the information from the dynloader 'geo', the GeoTrack database can grow to several thousands of entries.
The best scenario is using both:
- The dynloader 'geo'
- The module 'Geotrack'
in combination.
The registration changes, that the 'geo' dynloader does not cover within a 24-hour update cycle can so be caught by the GeoTrack module. The EXPIRATION value for the GeoTrack database can then be set to '1' (for one day) so that the database gets cleared from entries, that are older than 24 hours.
Alternatively one may call the 'Expire' routine of GeoTrack as 'Expire 1' from the crontab entry just a minute after the 'geo' dynloader has picked up fresh information from the Internet.
| (excerpt of the crontab for 'root')
#============ Watcher ========================================= ..... |
Measuring & statistics ...
Just collecting data makes no sense if the data isn't used to measure the success of the endeavors.
GeoTrack has a small companion named GeoCount that is implemented as a BASH coprocess and is fired up by GeoTrack when the process starts.
GeoCount gets all detected country codes passed from GeoTrack and counts (asynchronously) in the background all accesses from all regular Watcher modules in its exclisive table in the database.
| (code snippet from the 'GeoCount' code)
... TABLE=geocount ... $SQL "update $TABLE |
As simple as this is, nice statistics can be drawn and in fact, this is provided by the Stat component, that all regular modules have - so GeoTrack has a Stat component as well.
[root@vmd28527 GeoTrack]# pwd |
Calling Stat will generate a CSV file (or a bunch of CSV files) and then sends it/these to the configured $REPORT_MAIL address as an email message with the CSV file(s) in the attachment. From the CSV data, very nice statistics can be created with your favorite Spreadsheet software (EXCEL, LibreOffice CALC, ...) to be shown in meetings or to report the situation to your boss ... for instance.
Conclusion ...
The regular Watcher modules (Watch{LG|MX/MB|WB]) with support from the dynloaders will cut down the 'attack rate' from milliseconds to several minutes. In some cases, I have seen time lags of over an hour between attacks ...
From the experience during testing 'geo blocking' of just the two most aggressive countries (Russia & China) has cut down about 60% of all the remaining attacks, that were not already detected by the regular modules.
The overall amount of attacks dropped by 85% compared to a situation when Watcher is not running.
Additions & Changes in Revision 1.4
Additions
Dynamic (re)load of rules for modules
With revision 1.4 it is no longer needed to restart the entire Watcher service if rules or the 'superflous_map' are changed.
A module will now reload its rules and/or 'superflous_map' by sending it a HUP signal. A new utility program in the Watcher toolbox called 'Refresh'' triggers the reload of rules and superflous_map, if it is called with the module token as an argument: e.g:
# Refresh mx
This will transparently reload the rules and superflous_map for the WatchMX module.
Dynamic restart for modules
With Watcher 1.4 it is no longer needed to restart the entire Watcher process (restart/reload).
In the Watcher toolbox there is now a 'Restart' command that can restart a module selectively:
# Restart <module token>
'Loop rate' in traces
Traces now show the elapsed time between the previous call to 'filter' and the current call to show the load of the scanner.
|
(Snippet from a 'Trace mx')
…
2023-07-20T14:06:12.313 WatchMX[10396]: [inject] Finished for rule ['NXdomain'], 154/141 ms 2023-07-20T15:00:17.273 WatchMX[10396]: « Loop rate: 0 d, 00:54:04.954 » 2023-07-20T15:00:17.275 WatchMX[10396]: [Loop: 3917] 'Jul 20 15:00:17 vmd28527 postfix/smtps/smtpd[18374]: connect from unknown[107.170.255.51]' … |
From this, one gets a clue on how effective the firewall does its job.
Rule action 'kickoff'
The standard action on a rule match for rules was 'inject', which does the whole job of IP classification (NXDOMAIN, FAKEHOST, TRUEHOST), counting 'affairs' of an IP in the database, and all the other determination tasks. But there situations where this is not needed due to other conditions that denote the request of an IP addrress being an affair.
In such cases 'kickoff' can be called (instead of 'inject') to shorten the process to register the IP address and DROP it immediately and permanently.with much less processing. This will speed up the processing rate of the scanner tremendously.
Changes
Injector penalty ('affairs' preset)
The injector ('inject' function in rules) can now take a parameter as a 'penalty'. This penalty is a preset for AFFAIRS in the 'initial' stage (i.e., if an IP address is registered) so that the counter for affairs does not start from '1'.
| RULE=root-login Pattern=': Failed password for root' #--------------- if [[ "$REPLY" =~ "$Pattern" ]]; then inject 3; return $?; fi |
This does not affect the penalty for IP addresses of class 'NXDOMAIN' and/or 'FAKEHOST'. These still get a preset of 'MAX_AFFAIRS-1' by the 'inject' function regardless of which penalty was specified as a parameter to 'inject'.
Variable TIME_SLICE in the module's configuration files
The time slice for the timed ipset 'tarpit', which blocks an ill-behaving IP address with each affair for an increasing amount of time, is now made variable. Previously, it was hard-coded as '60' (seconds) in the module's code. Now, a variable TIME_SLICE can be configured in the module's configuration file, which affects the timeout calculation in the module's code:
Timeout calculation in the module code:
| Old: | (( 2affairs * 60 )) |
| New: | (( 2affairs * TIME_SLICE )) |
The default of TIME_SLICE in the configuration file is '60'. If TIME_SLICE is not set, (probably deleted or definition garbled) it is internally forced by the module to be '60' to prevent a division-by-zero error.
The factor is still logarithmic by power of 2; i.e. it doubles with each 'affair' until a DROP is applied to a particular IP address.
That results in the following timeout table for 60, 300, and 600 seconds for TIME_SLICE with increasing affairs:
| Time slice ⇓ Affairs ⇒ |
1; (21)=2 |
2; (22)=4 |
3; (23)=8 |
4; (24)=16 |
5; (25)=32 |
| 60 seconds ~ 1 Minute |
120 | 240 | 480 | 960 | 1920 |
| 300 seconds ~ 5 minutes |
600 | 1200 | 2400 | 4800 | 9600 |
| 600 seconds ~ 10 minutes | 1200 | 2400 | 4800 | 9600 | 19200 |
