Directadmin Docs

Troubleshooting

Websites are not opening

If you see apache running however websites are not opening a few things could cause this:

  1. Check if firewall is not closing ports 80/443 .

  2. Check if MaxRequestWorkers not reached:

grep MaxRequestWorkers /var/log/httpd/error_log

If you see anything like 'server reached MaxRequestWorkers setting, consider raising' you have to increase it in extra/httpd-mpm.conf , properly customizing the config with customize main apache config and copying configure/ap2/conf/extra/httpd-mpm.conf to custom/ap2/conf/extra/ , modifying it and doing rewriting configs.

  1. Semaphores limit reached:
tail -f /var/log/httpd/error_log

And if you see the 'No space left on device' error, check semaphores:

ipcs -a  | grep apache

Remove them with:

for i in `ipcs -s | grep apache | awk {'print $2'}`; do (ipcrm -s $i); done

And start apache.

You may want to automate it.

Automating the removal of semaphores

If you routinely need to remove apache semaphores with the ipcs/ipcrm tool, and if you cannot sort out why they keep building up, then use a hook script that can be called for downed service where we will use the script below, after confirming the "$service" is "httpd".

  1. Create the file /usr/local/directadmin/scripts/custom/service_down_notice.sh

  2. Open it and paste following code:

#!/bin/sh
if [ "$service" != "httpd" ]; then

EMAIL=your@email.com
MAX_SEMAPHORES=15

IPCS=/usr/bin/ipcs
IPCRM=/usr/bin/ipcrm
MAIL=/bin/mail

COUNT=`${IPCS} | grep apache | wc -l`

if [ "$COUNT" -le $MAX_SEMAPHORES ]; then
       #all is well, there are no semaphore build-ups.
       exit 0;
fi

#we have more than MAX_SEMAPHORES, so clear them out and restart Apache.

LIST=/root/sem.txt

${IPCS} | grep apache | awk '{print $2}' > ${LIST}
for i in `cat ${LIST}`; do
{
       ${IPCRM} -s $i;
};
done;

/etc/init.d/httpd restart

TXT="${COUNT} semaphores cleared for apache for `hostname`"
echo "${TXT}" | ${MAIL} -s "${TXT}" ${EMAIL}


       exit 1;
fi
  1. And make it executable:
chmod 755 /usr/local/directadmin/scripts/custom/service_down_notice.sh

How to track which site is producing high load

If your server is overloaded with apache processes you may track down which sites are causing it with [https://httpd.apache.org/docs/2.4/mod/mod_status.html](server status module). It is already compiled into apache, just all IPs are restricted to access it.

If you need to monitor server load just once feel free to modify /etc/httpd/conf/extra/httpd-info.conf directly and change the Require ip to your desktop outgoing IP address (12.34.56.78 in below example):

<Location /server-status>
    SetHandler server-status
    Require host .example.com
    Require ip 12.34.56.78
</Location>

Or remove all Require ip to allow all but change the /server-status to something hard-to guess since we don't want just anyone viewing this page.

<Location /secret-server-status>
    SetHandler server-status
</Location>

Then open http://server_ip/secret-server-status page.

Also change uncomment the #ExtendedStatus by removing the # character to view more details:

ExtendedStatus On

If you want to keep server-status page for longer period and not overwritten with DirectAdmin you have to [/webservices/apache/customizing/](customize it) accordingly.

How to mitigate a slow-loris attack

A slow loris attack is one where an IP will connect to your apache server and clog up all child processes with it's specially formed requests (I won't get into the details as to how).

Apache is compiled with the reqtimeout module, which can help mitigate such attacks. To check, type:

httpd -M | grep reqtimeout

And make sure you see the reqtimeout_module in the output. The /etc/httpd/conf/extra/httpd-default.conf file contains some default values already:

<IfModule reqtimeout_module>
  RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>

Which means wait for the first byte of headers for 20 seconds. If some data arrives, increase the timeout corresponding to a data rate of 500 bytes/s, but not above 40 seconds.

Do not restart apache during the nightly tally

By default, DirectAdmin always issues graceful restarts, which means that no connections are interrupted, which is the safest method.

An actual restart is required to rotate logs and close file descriptors. We do not recommend disabling log rotation, which the directadmin.conf variable 'rotation=1' controls. A switch to restart apache after the tally runs exists, as well: 'restart_apache_after_tally=1'.

pcfg_openfile: unable to check htaccess file

When viewing your domain, if you see the error "Forbidden", then check your apache logs:

tail -f /var/log/httpd/error_log /var/log/httpd/domains/domain.com.error.log

and see the error:

[crit] [client 12.34.56.78] (13)Permission denied: /home/username/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

This would imply that apache does not have read access on your /home/username directory. The cause for this can be permissions related, config related, or ownership related.

Apache runs as user 'apache' thus the directory must be +x for that user or group. Full directory structure of the public_html for testuser1 as example :

dr-xr-xr-x. 19 root root  /
drwx--x--x. 16 root root  /home
drwx--x--- 7 testusr access  /home/testusr
drwx--x--x 4 testusr testusr  /home/testusr/domains
drwx--x--x 7 testusr testusr  /home/testusr/domains/example.com
drwxr-xr-x 3 testusr testusr  /home/testusr/domains/example.com/public_html

Permissions are most often 644 for files and 755 for directories and owned by the user, but these can vary depending on the php handler and webserver setup and by CMS.

How to trace down what exactly apache/php are doing

Trying to find a slowdown in your website and not sure where it is? Example: WordPress loads slowly and you don't know why. Use strace!

The idea is that strace can dump all binary calls to a log, so we'll dump them to disk and sift through them later. You can also use this method for other processes, just swap "httpd" with some other process name in the command below.

Note: this is a fairly high-level debug method for advanced administrators. Root ssh access is required.

  1. Run strace to log:
cd /root
mkdir straces
cd straces
ps ax | grep httpd | grep -v grep | awk '{ print "strace -f -s500 -o strace."$1".log -p "$1"&" }' | sh

  1. Trigger the slow case. Load the website that is slow, however you need to duplicate the issue. Note, it might be best to highlight the URL bar and hit enter, instead of using F5. The reason is we only want to load the site, and not the included images/css/js, etc.. and F5 or ctrl-F5 may reload everything, causing the logs to fill with more info, making the cause harder to find.

  2. End the strace, open a 2nd console and run:

killall -9 strace

You'll now have several strace.*.log files, one for each httpd process that was running. Note, if you're debugging a slowdown (or possibly a socket timeout), hit ctrl-c before the timeout happens, but allow enough time where you can let it process things for a bit. This way, the last "chunks" of code that caused the slowdown will be near the end of the log, making it easier to track.

  1. You'll need to find which PID handled your request, so grep your IP from the logs to find the PID. Change 1.2.3.4 with your actual IP address and use the command below:
grep 1.2.3.4 *.log

This will dump some messy code, but the far left of the output should display which files the code originated from. Let's say that it showed the output in the strace.29622.log (your PID number will be different from 29622). Also there could be multiple files, depending on how many requests were made and how apache handled them. We're only concerned with the "slow" code, and not the other things like images, so you'll need to go through each one to figure out which is which to be able to analyze the relevant bits you're after.

  1. From there, you can go through the logs, looking at what was happening at that time, eg:
less strace.29622.log

This is going to show you quite a lot of code, but you can also search with the / character.

You can alternatively "grep" for specific information, eg:

grep somethingspecific strace.29622.log

Which only shows the lines you're looking for.

From this point, what you're looking for, how you find it, and what you do with it would be up to you.

How to debug an apache segfault

Debugging Apache segfaults can be tricky, in an attempt to find out which module may be causing it. You might see something like this in the /var/log/httpd/error_log:

[Mon Aug 07 23:56:18.309463 2017] [core:notice] [pid 17630] AH00052: child pid 18187 exit signal Aborted (6)

  1. You can debug it using the gdb option, like this:
service httpd stop
gdb /usr/sbin/httpd

and quickly (before the dataskq starts it up again), run:

run -X -d /etc/httpd

which should let you trigger one request in the foreground, so do it quickly, before any other connections arrive to your server.

  1. Hopefully this has triggered your segfault, in which case it might show:
Thread 1 "httpd" received signal SIGABRT, Aborted.

along with other info about exactly where it crashed.

  1. To see more details about the function tree used to get to that point, call a backtrace like this:
bt full

Which should give you the list of functions called, where it started lower down, and ended at the top of the output. Somewhere in there might show you which module triggered it.

  1. To quit gdb, just run:
quit
Last Updated: