Securing with Brute Force Monitor
Detecting and preventing brute-force attacks with DirectAdmin's Brute Force Monitor (BFM)
A common method of gaining access over a server is to use a technique called a brute force attack, or dictionary attack. What the attacker will do, is use a script to try and login to an account with every possible password combination. This tends to require tens of thousands of login attempts, but eventually, the right combination will be found, and they can login normally.
To prevent this, we can use a brute force login detection system.
The original Brute Force Monitor feature was created in DA 1.25.5, and would detect and block login attempts on DA itself (port 2222 only):
http://www.directadmin.com/features.php?id=573
The Brute Force Monitor has since come along way from its original implementation to include the ability to enable the following features (in addition to its ability to protect the DirectAdmin login from bruteforce attacks):
- direct CSF firewall integration for blocking bruteforcing IPs (implemented in DA version 1.61.0)
- scanning for WordPress attacks, including:
- xmlrpc.php
- wp-login.php
- scanning service logs, such as:
- Apache
- Dovecot
- Exim
- ProFTPD/Pure-FTPd
- SSHd
- Roundcube
- SquirrelMail
- PHPMyAdmin
- the ability to truncate its own logs
- the ability to send bruteforce attack notifications
- via the message system
- via email
- one can specify the email to send notifications to
- whitelists/blacklists
- hook scripts
Enabling and Configuring DirectAdmin's Brute Force Monitor with IP blocking capabilities
First, you should consider a firewall to actually block the attacker IPs. CSF is recommended for this as DirectAdmin integrates with it so nicely. Additionally, CSF/LFD consists of 2 components, one of which is the Login Failure Daemon. Thus, by enabling CSF/LFD, you will have yet another layer of protection to combat bruteforce attacks against your services.
You can install CSF during a fresh installation or for existing servers. Instructions for both situations are outlined below.
Enabling CSF & BFM during a fresh installation
If installation isn't customized to disable CSF, it will result in both CSF and BFM being enabled and configured automatically to work in tandem upon successful completion of the installation script.
Enabling CSF for an existing Server
The following steps should work regardless of your existing implementation, whether you're using an older BFM-firewall integration or none at all, as long as you are using DirectAdmin version 1.61,0 or newer.
- Make sure the CSF is installed, it can be done with commands:
da build set csf yes
da build csf
Open a new terminal window and ensure that you can log into the server via SSH as the root user before you close any established terminal connections.
Alternatively, one could use the official installation instructions for CSF here. Confirm that you have SSH access to the server via another terminal, then proceed to set TESTING = "0"
in /etc/csf/csf.conf
, and restart the firewall:
csf -ra
Be sure to confirm you can log into the server via SSH before closing any existing SSH sessions!
- Confirm your version of DirectAdmin
DirectAdmin integrated with CSF directly in Version 1.61.0 (prior versions utilized hook scripts for the integration). Make sure you are using this version before proceeding with these instructions, otherwise, you'll need to either upgrade or use a deprecated method of CSF-BFM integration.
/usr/local/directadmin/directadmin version
- If using DirectAdmin Version 1.61.0, make sure that the following hook scripts and files are removed and/or do not exist on your system:
/usr/local/directadmin/scripts/custom/block_ip.sh
/usr/local/directadmin/scripts/custom/brute_force_notice_ip.sh
/usr/local/directadmin/scripts/custom/show_blocked_ips.sh
/usr/local/directadmin/scripts/custom/unblock_ip.sh
/root/blocked_ips.txt
/root/exempt_ips.txt
Note: Though the DA csf_install.sh
script may install these files (these files will likely be removed from the script after further testing and allowing sufficient time for users to update their DA versions), these files are not needed with DA version 1.61.0 and later.
Now that the firewall is installed and enabled, you will want to proceed to set up BFM.
Enabling and Configuring BFM
To enable this feature, go to Admin Level -> Admin Settings -> Blacklist IPs for excessive login attempts under the Security Settings tab. Other settings can also be configured here or from directly within the directadmin.conf file.
Admin Settings > Security Tab BFM Settings, their corresponding directadmin.conf values, and descriptions
Blacklist IPs for excessive login attempts (checkbox enabled):
bruteforce=1
Global enable/disable switch for a Brute Force Monitor service.
Blacklist IPs for excessive DA login attempts after X login attempts (where XX = 20).
brutecount=20
Number of login attempts to DirectAdmin panel after which IP address will be blacklisted by BFM (Brute Force Monitor).
Blacklist IPs after XXX unauthorized connections (where XXX = 100).
brute_dos_count=100
Number of attempts on loading DirectAdmin login page after which IP address will be blacklisted by BFM (Brute Force Monitor).
Prevent 127.0.0.1 from being Blacklisted
exempt_local_block=0
Set to 1 to prevent 127.0.0.1 from being blacklisted
Time before failed login count resets XXXX seconds after last attempt (where XXXX = 1200).
brute_force_time_limit=1200
The time window for which the attempts (either failed logins or unauthorized connections) must pass with no activity before the count is reset.
Related: brute_dos_count
Remove an IP from the blacklist after X minutes (where X = 0 and 0 = never).
clear_blacklist_ip_time=0
Number of minutes after which the blacklisted IP address will be removed automatically.
Parse service logs for brute force attacks
brute_force_log_scanner=1
Turns ON ability to have DirectAdmin scan service logs for any brute force login attempts on a server (dovecot, exim, proftpd, sshd).
Notify Admins after an IP has XXX login failures on any account (where XXX = 100).
user_brutecount=100
The BruteForceMonitor can scan how many times a specific IP attacks a server, but also how many times a specific User is attacked from any IP. Sometimes the Admin might not want to bother with the number of attacks on a specific User, so you can set the option user_brutecount to 0, which will disable DA's count on specific Users. Setting to 0 will likely improve the loading time of the Brute Force Monitor page.
Notify Admins after an User has XXX login failures from any IP (where XXX = 100).
ip_brutecount=100
Number of bruteforce attempts per IP to trigger sending a notification to admins.
Remove an IP from the BF blacklist after XXXXX minutes (where XXXXX = 86400).
unblock_brute_ip_time=86400
A number of minutes after which the IP is automatically unblocked by Brute Force Monitoring.
Reset count of IP/User failed attempts XX hours after last attempt (where XX = 24).
clear_brute_log_time=24
Number of hours the failed login attempts to be checked within. If ip_brutecount is set to 100 then an ip can have 100 failed attempts within 48 hours before all Admins are notified. If the IP has 99 failed attempts, waits 24 hours, then makes 99 more attempts, no notifications will be sent.
Clear failed login attempts from log X days after entry was made (where X = 4).
clear_brute_log_entry_time=4
The number of days how long to keep brute-force incidents (in /usr/local/directadmin/data/admin/brute_log_entries.list file).
Scan for WordPress attacks
brute_force_scan_apache_logs=0
A Brute Force Monitor can scan apache domain logs for WordPress wp-login.php attacks.
- All logs ( 2 : DirectAdmin itself will create a list of all logs to form the /usr/local/directadmin/data/admin/brute.conf.)
- Manual ( 1 : Scan apache logs but only those specified in /usr/local/directadmin/data/admin/brute.conf file, the string should end with "equals" sign. Example adding procedure: echo "/var/log/httpd/domains/domain.com.log=" >> /usr/local/directadmin/data/admin/brute.conf )
- No ( 0 : Disable scanning of apache logs by BFM )
Related: brute_force_apache_log_list_update_interval=10
Other directadmin.conf settings relating to BFM
brute_force_apache_log_list_update_interval=10
Number of minutes between the refresh of apache log list, used if brute_force_scan_apache_logs set to 2. Missing logs are always removed from the list, but new logs won't start scanning for this amount of time.
ip_blacklist=/usr/local/directadmin/data/admin/ip_blacklist
A path of blacklisted IPs to be used in Brute Force Monitor.
ip_whitelist=/usr/local/directadmin/data/admin/ip_whitelist
A path of whitelisted IPs to be used in Brute Force Monitor.
brute_force_ignore_attempts_on_suspended=1
To ignore all attempts on suspended accounts by Brute Force Monitor.
brute_force_log_scanner=1
Turns ON ability to have DirectAdmin scan service logs for any brute force login attempts on a server (dovecot, exim, proftpd, sshd).
brute_force_notifications_email_only=0
Ability to send email notifications only without flooding a DirectAdmin panel message system. The email will contain the details of the attack, with a link to server/BFM panel to react quickly. Requires that the BFM messages be enabled, and not hidden (hide_brute_force_notifications=0).
- 0 : BFM will create a notification in DA Message System
- 1 : BFM will not create a ticket in DA Message System, but will only send an email notification to admin
hide_brute_force_notifications=0
Change to 1 to prevent sending brute-force notifications by email.
include_directadmin_port_in_brute_firewall=0
Option to include 2222 failed attempt in BFM blocks (CSF).
brute_force_ignore_attempts_on_suspended=1
To ignore all attempts on suspended accounts by Brute Force Monitor.
BFM Scanned Logs' Location Settings via the directadmin.conf file
brute_force_exim_log=/var/log/exim/mainlog
A path to exim mainlog file to be scanned by Brute Force Monitor.
brute_force_exim_reject_log=/var/log/exim/rejectlog
A path to exim rejectlog file to be scanned by Brute Force Monitor.
brute_force_mail_log=/var/log/maillog
A path to main dovecot log file to be scanned by Brute Force Monitor.
brute_force_messages_log=/var/log/messages
A path to main system messages log file to be scanned by Brute Force Monitor.
brute_force_mysql_log=/var/lib/mysql/server.hostname.tld.err
A path to main mysql log file to be scanned by Brute Force Monitor (for server with hostname 'server.hostname.tld').
brute_force_pma_log=/var/www/html/phpMyAdmin/log/auth.log
A path to PHPMyAdmin authentication log file to be scanned by Brute Force Monitor.
brute_force_pureftpd_log=/var/log/pureftpd.log
A path to pureftpd log file to be scanned by Brute Force Monitor.
brute_force_roundcube_log=/var/www/html/roundcube/logs/errors
A path to RoundCube log file to be scanned by Brute Force Monitor.
brute_force_secure_log=/var/log/secure
A path to OS secure log file to be scanned by Brute Force Monitor.
brute_force_squirrelmail_log=/var/www/html/squirrelmail/data/squirrelmail_access_log
A path to SquirrelMail log file to be scanned by Brute Force Monitor.
/usr/local/directadmin/data/admin/ BFM Files
The following discusses the purposes of the files and directories under /usr/local/directadmin/data/admin/
as related to BFM:
brute_skip.list
ip_whitelist
ip_blacklist
brute.conf
brute_ip.data
brute_log_entries.list
brute_user.data
ip_access
ip_access/*/unauthorized_connections
ip_access/*/failed_logins
brute.conf(.lock)
ip_blacklists, ip_whitelists, and brute_skip.list
- ip_whitelists if for whitelisting IPs for DirectAdmin logins only (port 2222 by default), whereas brute_skip.list is a whitelist for all services monitored by BFM.
- ip_blacklist is where IPs bruteforcing the DA login are blocked.
brute.conf
- The brute.conf files stores the logs' last write and parse data, to prevent redundant parsing by BFM.
brute_ip.data
- This contains the IPs blocked for bruteforcing as shown in the Brute Force Monitor interface.
brute_log_entries.list
- This file is not part of the Brute Force Monitor's counting, as it's only used to show you what each attack was in the BFM GUI interface.
brute_user.data
- This file is used for the BFM GUI interface to log and display the most recent usernames that have been bruteforced.
ip_access
- BFM will use this directory for storing
brute_dos_count
andbrutecount
IPs:
- BFM will use this directory for storing
/usr/local/directadmin/data/admin/ip_access
- for each IP, e.g.,
/usr/local/directadmin/data/admin/ip_access/1.2.3.4/unauthorized_connections
/usr/local/directadmin/data/admin/ip_access/1.2.3.4/failed_logins
The above 2 files will store 1 byte per attempt, so the size of those files would be the number of attempts.
This makes tracking much quicker, and non-racing, in that it just appends a "1" to the end of each file for the given case.
Once login is successful, the failed_logins are counted and added to the Users logins.list, as before, and the 1.2.3.4 directory is removed.
- brute.conf(.lock)
- A lock file to prevent dataskq from starting brute force scanning if the preceding scanning a minute prior has not yet completed
An issue arose where very large logs caused the dataskq to take more than one minute to scan them. The issue is that the dataskq is run again the next minute, and starts to scan the same logs from the beginning.
Each time this happens, the system gets slower, and prevents any of the scans from finishing. A simple lock has been added on the file /usr/local/directadmin/data/admin/brute.conf (.lock)
for the entire scanning process. If a lock is present on that file, the brute force scan is skipped and the other normal dataskq activities can resume. The lock is allowed to live 3 hours, before the dataskq assumes something went wrong and removes it (allowing the next scan to happen).
Viewing BFM Blocks in DirectAdmin
The brute force monitor (BFM) page can be viewed at Admin Level -> Brute Force Monitor.
The interface consists of these tabs:
Failed Logins
Failed Logins: IP List
Failed Logins: Username List
Skip List
Block List
Failed Logins This tab shows the size of the file brute_log_entries.list and a list of users and IPs failiing authentication and blocked for it. The BFM will use the /etc/csf/csf.deny to show the listing of blocked IPs and comments/dates.
**Failed Logins: IP List ** Shows the IPs that have been blocked for bruteforcing. This data is from
/usr/local/directadmin/data/admin/brute_ip.data
.Failed Logins: Username List Shows the usernames that have been used for bruteforcing. This data is from
/usr/local/directadmin/data/admin/brute_user.data
.Skip List CSF does have its own skip list.
The Brute Force Manager also has its own skip list:
/usr/local/directadmin/data/admin/brute_skip.list
With integration, when adding an IP to the skip list through the BFM, DA will add it to both brute_skip.list and to CSF's csf.ignore.
Adding ranges in the format: 1.2.3.4-5 will only be added to the BFM.
DA will not accept 1.2.3.0/24, but CSF will. Adding the 1.2.3.0/24 style ranges should be done in CSF, as even if DA decides to try and block an IP, the CSF csf.ignore would override the block, so the CSF allow has priority.
Listing all Skip value in DA will show the contents of both lists. Deleting a value from DA ONLY deletes it from the skip.list, not from CSF's allow list (the value would remain showing).
- Block List This is where you can submit IPs for blocking. There is an empty field to enter the IP and a button next to it to "Block IP".
Then a list of IPs that have been blocked and by which service follows beneath the "Block IP" button.
Apache WordPress log scanning in BFM
Brute Force Monitor can scan Apache domain logs for WordPress wp-login.php
attacks.
Internal default directadmin.conf setting:
brute_force_scan_apache_logs=0
There are 2 ways of using this feature:
Options 1: scan logs only for specific domains
To scan logs only for certain domains:
- set the following option
brute_force_scan_apache_logs=1
- add domain logs for domains that should be scanned to
/usr/local/directadmin/data/admin/brute.conf
. Entries should look like the following format:
/var/log/httpd/domains/<domain.com>.log=
where <domain.com>
is the name of the domain to be scanned.
When brute_force_scan_apache_logs=1
is set, dataskq
will look for logs that start with /var/log/httpd/domains
(or whatever other value is set with the apachelogdir
option).
Note that existing entries inside /usr/local/directadmin/data/admin/brute.conf
have pos=
and size=
values appended to them. There is no need to add these manually. Directadmin will add them automatically.
Options 2: scan logs for all domains
This is the preferred method for most systems. However, if you are able to keep track of which websites are using WordPress, then use the 1st option, to help reduce load caused by log parsing.
To scan logs of all domains, set the following option:
brute_force_scan_apache_logs=2
DA will then manage brute.conf
for you by adding all of the domain log entries from /var/log/httpd/domains/*.log
automatically (excluding logs that end in error.log
).
When brute_force_scan_apache_logs=2
this variable takes effect:
brute_force_apache_log_list_update_interval=10
it sets an interval for apache log list renewal in minutes.
Missing logs are always removed from the list in brute.conf
, but new logs won't start scanning for this amount of time.
IMPORTANT
Note that apache does not know if a wp-login.php
entry was a login failure or not. All it knows is that a POST
request was made to wp-login.php
.
Because of this, Brute Force Monitor considers each POST
request to wp-login.php
a failed login, even if the correct password was used. Keep this fact in mind when deciding what IP block limit to set.
If you log in and out many times in a row, then DA will consider this an attack, even if the login was successful.
Find which domains are being attacked via wp-login.php attacks
Resulting list of afflicted domains is sorted by count after using the command below:
cd /var/log/httpd/domains
grep -c wp-login.php /var/log/httpd/domains/*.log | grep -v :0 | awk -F':' '{print $2,$1}' | sort -n
If you want to monitor only domains that are most affected by wp-login.php
attacks, you can use the command below. It will get top 20 most affected domains and add them to brute.conf
. This can be changed by substituting the count set via argument -n 20
.
Make sure to confirm that the output from the commands above is satisfactory before using commands below:
cp /usr/local/directadmin/data/admin/brute.conf /usr/local/directadmin/data/admin/brute.conf.backup
grep -c wp-login.php /var/log/httpd/domains/*.log | grep -v :0 | awk -F':' '{print $2,$1"="}' | sort -n | tail -n 20 | cut -d\\ -f2 >> /usr/local/directadmin/data/admin/brute.conf
For more info, see original changelog entry here
Troubleshooting
I can't connect to DirectAdmin due to error message "Your IP is blacklisted"
If BFM is blocking your IP, your IP will be listed in the file /usr/local/directadmin/data/admin/ip_blacklist
. This is controlled by the "brute force login detection" in Admin Settings. If you get the message:
Your IP is blacklisted
this means that you've had too many failed login attempts or accesses to port 2222 without being logged in.
You can find your current IP like so:
curl myip.directadmin.com
To remove yourself from the blacklist, log into your server as root via SSH and remove your IP from the file:
/usr/local/directadmin/data/admin/ip_blacklist
If you wish to ensure you never get blacklisted, you can create the file below and add your IP to it:
/usr/local/directadmin/data/admin/ip_whitelist
Issues Caused by Excessively Large BFM Files
High CPU
A large brute_log_entries.list
file can cause the dataskq to work hard to parse the data, and thus cause higher CPU usage.
If the brute_log_entries.list
continues to grow and you'd like it to keep itself smaller, go to Admin Level -> Admin Settings and lower the value for:
Clear failed login attempts from log X days after entry was made.
to something around 2 days.
Also, increasing the values for:
Notify Admins after an IP has X login failures on any account.
Notify Admins after a User has X login failures from any IP.
will reduce the number of entries made into the brute_log_entries.list file.
Timeouts
If you get a timeout message similar to the one below when accessing the Brute Force Monitor (BMF) CMD_BRUTE_FORCE_MONITOR:
Your connection has timed out.
Either your request was invalid or the program hasn't completed your request.
Please notify the server admin.
This is usually caused by an over-sized file /usr/local/directadmin/data/admin/brute_log_entries.list
. Check this file to see if it's large. If it is, you shrink it down, but only keeping the entries at the end of the file (end of the file has the newest entries).
cd /usr/local/directadmin/data/admin
mv brute_log_entries.list brute_log_entries.list.backup
tail -n 1000 brute_log_entries.list.backup > brute_log_entries.list
chown diradmin:diradmin brute_log_entries.list
Where you can increase or decrease the 1000 lines as needed.
Other possible affected file are the brute_user.data
and brute_ip.data
. Of these 2 files, it's more likely the brute_user.data
is causing the slowdown, because attacks typically happen on many Users from 1 IP address. If you need to empty the brute_user.data
file, type:
cd /usr/local/directadmin/data/admin
mv brute_user.data brute_user.data.backup
touch brute_user.data
chown diradmin:diradmin brute_user.data
We wouldn't recommend emptying the brute_ip.data
because this is what counts the attacks per IP, and is what is used to determine if an IP should be blocked.
BFM Customizations
I want to customize BFM filtering
This will require the feature log scanning feature to be enabled (brute_force_log_scanner=1
) along with BFM (bruteforce=1
).
There will be a set of filter definitions (multiple definitions for each service) stored in:
/usr/local/directadmin/data/templates/brute_filter.list
where you can also create a custom version here:
/usr/local/directadmin/data/templates/custom/brute_filter.list
The /usr/local/directadmin/data/admin/brute.conf
will be created if it doesn't exist, and will store the logs' last write and parse data, to prevent redundant parsing and improve efficiency.
For each filter line, there are multiple filter item definitions:
filtername=binary=b&text=t&ip_after=ia&ip_until=iu&user_after=ua&user_until=uu&attempts_after=aa&attempts_until=au
Note that the entry values will be URL encoded (really, just needed for = and & to prevent the url string from breaking).
A sample line will look like:
dovecot1=attempts_after=(auth failed,%20&attempts_until=%20attempts)&binary=dovecot&ip_after=rip%3D&ip_until=,&text=(auth failed&user_after=user%3D<&user_until=>
which would catch the log entry:
Jun 11 00:49:05 hostname dovecot: imap-login: Aborted login (auth failed, 2 attempts): user=<user@test.com>, method=PLAIN, rip=1.2.3.4, lip=4.3.2.1, secured
The filtername will be in the format of the service name, followed by a filter number.
The value of the number (e.g., dovecot1) will affect the order in which the filters are run on the log entries, which does matter in a few cases (where the text is a subset of another filter's text).
The text value is the string in the entry that will tell us if it's a failed login or not, and is a required value.
The following values are optional:
- attempts_after
- attempts_until
- user_after
- user_until
- binary
The optional binary is only for exim, where it's not a shared log... /var/log/exim/mainlog
is exclusively for exim, and exim[1234] doesn't show up after the date in the log format, so we don't scan for it.
The binary variable is used to see if the log entry in the given log matches the filter... for example, we don't care about spamd lines when looking for dovecot.
Using this information, one could copy the brute_filter.list
to the custom path and make the desired edits in order to customize the brute_filter.list
as desired. We recommend testing any customizations thoroughly and ensuring that you have an alternate whitelisted IP from which you can gain access to the server if your accidentally get your current IP blocked.
I want to send BFM notifications to alternate email account(s)
You can now manually add a field to your user.conf
(possible GUI to be added later), e.g.,
alternate_email=attacks@domain.com
such that the brute force monitor notices will go to that email, if the field exists.
The file can support multiple values too, e.g.,
alternate_email=attacks@domain.com,other@email.com
This will free up your inbox from getting flooded, while still being able to get notified with a different email of the attacks.
I want to Prevent BFM from reporting a specific email account's login failures
If you're using the Brute Force Monitor (BFM) and have an email address that is repeatedly checking your server, but that account is invalid (either wrong password, or a non-existent account), and you know that the person doing it is not malicious, but only stubborn for not updating their settings, then you can tell the BFM to skip that email address from being checked.
To do this, create the /usr/local/directadmin/scripts/custom/brute_force_notify_pre.sh
script and add the following code:
#!/bin/sh
if [ "$type" = "User" ]; then
if [ "$value" = "your@email.com" ]; then
echo "exempt your@email.com";
exit 1;
fi
fi
exit 0;
Make it executable:
chmod 755 /usr/local/directadmin/scripts/custom/brute_force_notify_pre.sh
Note that this will only prevent reports for users/emails being reported. It will not prevent the blocking IPs, as the IP could be "attacking" some other account as well.
I want to block certain countries via BFM
The CSF configuration file (/etc/csf/csf.conf
) contains the following:
CC_IGNORE = "CA,US"
which lists country codes to be ignored based on the location of the IP.
If you have CSF installed, the Brute Force Manager will now respect this list, using the command:
csf -i 1.2.3.4
to get the required info about the IP address, thus allowing that country to be skipped.
NOTE Please be aware that blocking entire countries can require a lot of resources and can potentially lock up the firewall by overloading it, so you'll want to use this carefully and sparingly. Do make sure that you can access your server via SSH before you close out your current session after configuring Country Code blocking, especially if you've added more than a few countries to the list.