What is the dataskq processor?

DirectAdmin processes many different tasks, some of them are immediate while others are scheduled to be processed by an internal processor - the task queue manager (/usr/local/directadmin/dataskq). It is executed by cron every minute from a file /etc/cron.d/directadmin_cron:

* * * * * root /usr/local/directadmin/dataskq
1

Similar tasks are grouped and executed only once. For example, adding new domain, subdomain, enabling SSL requires a reload of the WWW server. If there are multiple tasks queued to reload WWW server, reload would be executed only once. This means that you have to wait at least 1 minute after doing some vague changes to the server before the dataskq processor is run.

The task queue is stored in the /usr/local/directadmin/data/task.queue file. When queue processing is finished, the file is removed. It is normal to find this file absent on the system.

Tasks can be input into the task.queue file manually to be processed by dataskq. It is just the syntax that should be known for this. Most common examples:

  • restart directadmin: echo "action=directadmin&value=restart" >> /usr/local/directadmin/data/task.queue
  • rewrite user's WWW server configuration: echo "action=rewrite&value=httpd" >> /usr/local/directadmin/data/task.queue
  • update DirectAdmin service: echo "action=update&value=program" >> /usr/local/directadmin/data/task.queue
  • process nightly tally: echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

Dataskq could be launched manually without waiting for the cronjob to process it. It also accepts different debug levels:

/usr/local/directadmin/dataskq d80
1

Where d80 is debug level. Commonly used levels are d80, d400, d800, d2000.

Is my task.queue processed?

  1. Check if the /usr/local/directadmin/data/task.queue file exists. It should be deleted every minute. If it exists, check the content in it. If there are more than just 1 or 2 lines, then the dataskq is not running.

  2. Check /var/log/cron (/var/log/syslog on Debian). See if the dataskq is running every minute (logged every minute). If not, try running the following commands:

chmod 644 /etc/cron.d/directadmin_cron
/sbin/service crond restart
1
2
  1. Make sure crond is running ("cron" on FreeBSD):
ps ax | grep cron
1
  1. Try restarting cron:
service crond restart
1

Check /var/log/cron and /var/log/messages for possible problems with the /etc/cron.d/directadmin_cron file (/etc/crontab on FreeBSD). If you don't have crond (RPM-based systems), type:

yum -y install vixie-cron cronie
service crond start
1
2
  1. If all of the above are checked, but the task.queue is still not being processed, try running it manually:
/usr/local/directadmin/dataskq d80
1

What is the task.queue.cb file

There could be another task.queue file /usr/local/directadmin/data/task.queue.cb which is always parsed and mostly is filled by the custombuild script. Could be used same way like the 'real' task.queue.

The "feature" part is that the dataskq can now be called like this:

/usr/local/directadmin/dataskq d2000 --custombuild
1

With the --custombuild option passed to the dataskq (with or without debug mode), this tells the dataskq to skip all other tasks, and only process the task.queue.cb file.

This means:

  1. The task.queue.cb file is executed.
  2. No other task.queue or task.queue.da file is executed
  3. Services are not checked if they're running
  4. Partition usage is not checked
  5. Brute force scanner is not run
  6. The DA login history counter is not checked (for port 2222)

Only one instance of the dataskq at a time

Every minute, the dataskq is run via cronjob. If things like backups are issued at different times, say 5 minutes after the first run of a backup, then you'll end up having 2 backups being created at the same time, which can generate slow performance because of the drive being forced to bounce between 2 locations on disk simultaneously.

The solution for that is to only have 1 instance of the dataskq ever running at one time. Note that the consequence to that is that any actions that needs to be run while a backup is executing will not be processed. E.g., service monitoring, service restarts, or other task.queue commands. They will be stored in the task.queue file and executed once the first instance of the dataskq has finished.

The easiest way to ensure there is only 1 instance running at a time, is to run a check of the system before executing the dataskq every minute. The way you can do that is to edit /etc/cron.d/directadmin_cron (or /etc/crontab on FreeBSD) and change the code that says:

* * * * * root /usr/local/directadmin/dataskq
1

to become:

* * * * * root if [ "`ps ax | grep -v grep | grep -c dataskq`" -eq 0 ]; then /usr/local/directadmin/dataskq; fi;
1

then restart your cron program:

service crond restart
1

I want the dataskq to run more than once per minute

If you're looking for a quicker response in terms of service restarts, backup starts, and Brute Force Monitoring, etc., you can add another cron such that the dataskq runs every 30 seconds, rather than once every minute.

Edit /etc/cron.d/directadmin_cron and add 2nd call to the dataskq, making the 2 lines look like:

* * * * * root /usr/local/directadmin/dataskq
* * * * * root sleep 30; /usr/local/directadmin/dataskq
1
2

and then reload crond:

service crond restart
1

The dataskq is causing 100% CPU

If you're noticing your system load is on the rise, and you can see that the "dataskq" binary is running at the top of the list (in "top"), this guide will help you debug what it's doing.

  1. Check /var/log/directadmin/system.log to see if the tally is being run. If Users continue to be added to the log (the logs are doing something), then it's likely just the nightly tally, which is normal (assuming the log continues to grow and isn't just stuck on one User).

  2. The first thing to do, is to simply ask the dataskq what it's up to. To do this, type:

killall -USR1 dataskq
tail -n 10 /var/log/directadmin/errortaskq.log
1
2

What this will do is dump the running dataskq's current process location to the errortaskq.log. What that log output says will determine what to do next.

  1. If the output makes any reference to Maildir along with a path, then what it likely means is that the mentioned path contains an over-sized inbox. Check that inbox and delete the messages, if the email user doesn't seem to be deleting them.

  2. If the output makes any reference to brute_force or some related file, then the cause is likely the dataskq chewing on the system logs with many entries.

  • First, ensure you're using the latest version of DirectAdmin.
  • Then check:
cd /usr/local/directadmin/data/admin
ls -la brute_force*
1
2

We're checking for a large brute_log_entries.list file. If it seems to be "large" (say, over a Meg in size), then that can slow things down. You can safely delete "brute_log_entries.list", as it's not part of the Brute Force Monitor's (BFM's) counting...it's only for your own purposes, showing you what each attack was. DA will recreate this file automatically.

  • If the brute_log_entries.list continues to grow and you'd like it to keep itself smaller, go to Admin Level -> Admin Settings and lower the value for Clear failed login attempts from log X days after entry was made. to something around 2 days.
  • Also, increasing the values for Notify Admins after an IP has X login failures on any account and Notify Admins after a User has X login failures from any IP. will reduce the number of entries made into the brute_log_entries.list file.

Slow down disk access to the dataskq

If the tally is using up too much disk I/O, you can slow down the dataskq using the ionice binary, if you have it.

To limit the dataskq, edit /etc/cron.d/directadmin_cron (/etc/crontab on FreeBSD).

Find the line:

* * * * * root /usr/local/directadmin/dataskq
1

and change it to:

* * * * * root /usr/bin/ionice -c2 -n7 /usr/local/directadmin/dataskq
1

Of course, you'll need to ensure that you actually have the /usr/bin/ionice binary. Install it if you don't. Reload crond (or cron) after you make the change:

service crond restart
1

For backups, you can have the dataskq insert the ionice command in front of all tar, zip and unzip commands:
http://www.directadmin.com/features.php?id=1423open in new window

My monthly bandwidth is not resetting

The monthly bandwidth reset is done at 4:20am on the first day of the month. If you find that this did not happen (allow about an hour for the dataskq to run, pending the number of sites), then there are a few steps you can take to find out why, and reset it manually.

The first thing to check is the cron logs to ensure that the reset was actually issued. Check the /var/log/cron for the 4:20am timestamp for the 1st day of the month. You should see something like:

echo 'action=reset&value=all' >> /usr/local/directadmin/data/task.queue
1

If you don't see that command, then you'll want to check this guide to ensure that crond is running and the /etc/cron.d/directadmin_cron file is working correctly.

If you do see the above command in your cron log, then the next step is to check /var/log/directadmin/system.log for the entry that looks something like this:

2019:04:01-04:20:01: Reset all command Received
1

If you don't see that log entry, then you'd again want to use the same guide as above to debug the issue.

If you do see the above log entry, then scroll just below it to confirm that each user is mentioned. If not, then likely the users are not all set in the users.list file. You can use this guide to reset all users.list files.

At this stage, hopefully you would have figured out why the reset didn't happen, and have fixed it so it does reset all users' bandwidth next time. However, the reset still wouldn't have happened for the past reset. So to do that, you'd use the cleanreset task.queue command. Type:

echo "action=cleanreset&value=all" >> /usr/local/directadmin/data/task.queue
/usr/local/directadmin/dataskq d
echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue
1
2
3

which should reset things from the start of the month without losing data. Manually doing the regular reset mid-month will lose current data that you still want, hence we use the cleanreset, which scans all datestamps to ensure it doesn't delete anything it shouldn't.

If you didn't figure out why the reset didn't happen, you may also want to check for other competing tasks. For example, if the server is rebooted at the time of the reset, that could prevent it from working. Also, the dataskq is also used by other tasks such as backups, so check to see if you've got any large backups running near the same time as the reset, and check /var/log/directadmin/errortaskq.log for any errors relating to them.

Make the bandwidth update more often

Cron is the tool used to control the frequency of the stats tally. You can easily change how often it will happen by simply modifying the cron job file /etc/cron.d/directadmin_cron (/etc/crontab on FreeBSD). Locate the line that looks like the following:

10 0 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
1

That means that the tally will be run 10 minutes after midnight every day.

If you'd like to (for example) run the tally every 4 hours, you'd change it to:

10 */4 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
1

One important thing to remember is that for very large servers, with upwards of 1000 domains, the tally can take a significantly longer time to run (up to 2 hours), so if you run a large server, you may not want to run the tally too often.

Once you've set the value you want, save the file and reload the cron daemon:

service crond restart
1

WWW server graceful restarts after the nightly tally

After the nightly tally runs, the dataskq will issue a graceful restart to the WWW server to both re-open logs and reload configs in case accounts have been suspended/unsuspend:
http://www.directadmin.com/features.php?id=980open in new window
If you wish to disable this (not recommended), you may use this:
http://www.directadmin.com/features.php?id=1020open in new window
Use the tally_post.sh to take some special action that you want (note that it's called just before the graceful restart, as the restart is sent to the task.queue to be processed in the next minute/dataskq run).

How to delete domains from the command line

Domain deletion could be passed to the dataskq process with this command:

echo "action=delete&value=domain&requestedby=admin&select0=domain.com(&select1=domain2.com...)" >> /usr/local/directadmin/data/task.queue
1

Where options are:

  • requestedby=admin : controls where the result Message is sent to within the Message System, where the result is either success or fail. It does not have to be the owner of the domain, but can be. The actual domain owner is looked up for each domain.
  • select0=domain.com : the domain to be deleted.
  • select1=domain2.com : optional other domains, for any User, does not need to be under the same User. (select2=, select3=, etc.)

Related to interface Admin Level -> IP Manager -> click on an IP -> Link IP, this can now be done through the task.queue.

Sample:

echo "action=linked_ips&ip_action=add&ip=1.2.3.4&ip_to_link=1.2.3.5&apache=yes&dns=yes&apply=yes" >> /usr/local/directadmin/data/task.queue
1

The command is the same as the form in CMD_IP_MANAGER_DETAILS?ip=1.2.3.4, except we move the form action=add to ip_action=add, as the dataskq action needs to be broader.

How to Message Users From the Command Line

To message a specific account, list of accounts or all accounts (Admins+Resellers+Users), the task.queue could be used. Format is as follows:

action=notify
value=users
users=all|select1=fred&select2=george&select3=gary
subject=Your Subject
message=Content of your Message
1
2
3
4
5

where the "users" variable can be "all" (meaning DA will message all accounts on the server) or can be a list of accounts that are url-encoded with select1, select2.

For a single account, just use:

users=select1=gary
1

Note that the = and & characters in the users list must be url encoded with hex (%3D and %26, respectively). Also, if you want to include a newline in the message, use %0A instead.

A sample echo command is:

echo 'action=notify&value=users&subject=Hello World&message=Hello,%0A%0A  This is a message.&users=select1%3Dadmin%26select2%3Dgary' >> /usr/local/directadmin/data/task.queue
1

How to send notification to the Admin Message System

This is a very simple new task.queue command to send Admin notices.

action=notify
value=admin
subject=Your Subject
message=Content of your Message
1
2
3
4

It should be URL encoded (mainly if you want newline characters: %0A) so that it can be posted to the task.queue.

Here's an example:

echo "action=notify&value=admin&subject=hello this is my subject&message=this is the notification%0Awith a new line." >> data/task.queue.cb; ./dataskq d10 --custombuild
1

Note: the above sends to the task.queue.cb file, and calls the dataskq with the --custombuild option.

This skips all other typical dataskq commands, so you don't accidentally process a backup or the tally.

How to run bandwidth only tally

You may force dataskq to run bandwidth only tally using the following command:

echo "action=bandwidthtally&value=all" >> /usr/local/directadmin/data/task.queue
1

Alternatively to process per-user:

echo "action=bandwidthtally&value=all&value=user&type=username" >> /usr/local/directadmin/data/task.queue
1

How to run a quota only tally

When you delete a domain, the disk usage and totals are recounted for the domain as the deletion actually runs a 'mini' tally for quota only, like:

echo "action=quotatally&value=bob&type=user" >> /usr/local/directadmin/data/task.queue
1

You may recalculate the 'quotatally' for all users like so:

echo "action=quotatally&value=all" >> /usr/local/directadmin/data/task.queue
1

In the system.log, the usual tally entries will show up, but with this, it will show the mask, e.g.,

Tally User admin Begin: mask 20
1

This lets you know what the tally is actually looking at. If no mask is present, e.g.,

Tally User admin Begin
1

then everything is checked.

The mask layout is as follows:

#define TLY_MASK_WEBALIZER 1
#define TLY_MASK_BANDWIDTH 2
#define TLY_MASK_DISKUSAGE 4
#define TLY_MASK_LOGROTATE 8
#define TLY_MASK_TOTALS 16
#define TLY_MASK_SUSPEND 32
1
2
3
4
5
6

so, that means a mask of 20 = 16+4, so we have totals + quotas in the above example.

How to set quota limits for users

This command will go through each user, and set the quota limits in the system quotas. Users with unlimited quotas will also have their quota limit set to 0 (unlimited):

echo "action=rewrite&value=quota" >> /usr/local/directadmin/data/task.queue
1

Or per user:

echo "action=rewrite&value=quota&user=bob" >> /usr/local/directadmin/data/task.queue
1

How to rewrite a DNS zone

The domain key could be used to rewrite a DNS zone for only that domain:

echo "action=rewrite&value=named&domain=domain.com" >> /usr/local/directadmin/data/task.queue
1

Also, an optional key could be set to not update the DNS zone's serial number:

update_serial=no
1

If update_serial is not passed, yes is assumed.

echo "action=rewrite&value=named&domain=domain.com&update_serial=no" >> /usr/local/directadmin/data/task.queue
1

How to recreate spambox settings for mailboxes

When you make change in your SpamAssassin settings, each time you click "Save", DA will recreate/set the ownership/permissions on each spambox file.

If you have a large number of pop accounts, this can take long enough that the timeout is reached. Moving this task to the task.queue file to be run in the background will allow things to run quicker in the foreground in DA.

The actual task.queue command:

echo "action=defaultspam&domain=domain.com&username=user" >> /usr/local/directadmin/data/task.queue
1

A command to rewrite spam filters is:

echo "action=rewrite&value=filter" >> /usr/local/directadmin/data/task.queue
1

How to reset monthly stats

In the case that the monthly reset did not run, an admin can run:

echo "action=cleanreset&value=all" >> /usr/local/directadmin/data/task.queue
1

in order to only remove the previous month's data.

This will do a standard reset of bandwidth.tally files and user.usage files, but will not follow with a tally.

You'd need to run the full tally command afterwards to recount the correct stats for Reseller Level and Admin Level counting:

echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue
1

How to rewrite virtual user files

To rewrite/convert the /etc/virtual/domain.com/passwd files, you may use following command:

echo "action=rewrite&value=email_passwd&user=fred" >> /usr/local/directadmin/data/task.queue
1

Where fred is desired user that owns domain.com.

Or, to rewrite all, run:

echo "action=rewrite&value=email_passwd" >> /usr/local/directadmin/data/task.queue
1

How to rewrite apache/nginx configs for single user

Instead of rewriting configs for all users, save time by rewriting a single account's config as follows:

echo "action=rewrite&value=httpd&user=fred" >> /usr/local/directadmin/data/task.queue
1

List of other taskq commands

A list of other useful taskq commands that one could echo into the task.queue file are outlined below.

Regenerate vacation config files, first for all, and then for a single domain:

action=vacation&value=all
action=vacation&value=steve.com&type=domain
1
2

Regenerate email filters, first for all, and then for a single user:

action=rewrite&value=filter
action=rewrite&value=filter&user=bob
1
2

Regenerate email passwd files, first for all, and then for a single user:

action=rewrite&value=email_passwd
action=rewrite&value=email_passwd&user=fred
1
2

Regenerate email aliases, first for all, and then for a single user:

action=rewrite&value=email_aliases
action=rewrite&value=email_aliases&user=fred
1
2

Regenerate /etc/virtual/domainips and helo_data, first for all, and then for a single domain/ip:

action=rewrite&value=domainips
action=rewrite&value=domainips&domain=domain.com
action=rewrite&value=helo_data
action=rewrite&value=helo_data&ip=1.2.3.4
1
2
3
4

Run sync to cluster with 10 retries (in case one server is unreachable):

action=sync&type=cluster&value=domain.com&count=10
1

Run sync to cluster email data for the user fred to host 1.2.3.4 (failed host), with 10 retries and the longrequest URL encoded POST from the original client request:

action=sync&type=cluster&value=email&username=fred&host=1.2.3.4&count=10&request=long%request%to%set%email%account
1

Run domain verification process of /etc/virtual/domainowners to /usr/local/directadmin/data/users/fred/domains/:

action=confirm&type=domains.list&user=fred
1

Check if user/file exists and ensure the lines in Apache configs are correct based on that:

action=confirm&type=httpd.conf&value=/usr/local/directadmin/data/users/fred/httpd.conf
1

Regenerate the users' pureftpd database:

action=rewrite&value=pureftp_db
1

Regenerate dkim 1) for all domains, 2) for a single domain, and 3) for a single domain and add the key to the domain's DNS:

action=rewrite&value=dkim
action=rewrite&value=dkim&domain=domain.com
action=rewrite&value=dkim&domain=domain.com&dns=yes
1
2
3

Forcefully run BFM to scan Apache logs 1) for all domains, 2) for a list of users, and 3) for a list of domains:

action=rewrite&value=brute_force_scan_apache_logs
action=rewrite&value=brute_force_scan_apache_logs&users=...userslist...
action=rewrite&value=brute_force_scan_apache_logs&domains=...domainslist...
1
2
3

Update/check the panel license:

action=update&value=license
action=check&value=license
1
2

Update the DirectAdmin panel itself:

action=update&value=program
1

Creating backups:

action=backup&type=user&value=john
action=backup&owner=reseller&type=reseller&value=multiple&when=now&where=local&who=all
action=backup&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=george&select2=eddy.. [ftp_ip=1.2.3.4&ftp_username=asdf&ftp_password=base64enc&ftp_path=/]
action=backup&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=buger
action=backup&value=now&type=system
action=backup&type=sitebackup&value=bob&password=pass&select0=ftp&select1=email, etc (backup options)
1
2
3
4
5
6

Restoring from a backup:

action=restore&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=george.tar.gz&select2=eddy.tar.gz.. [ftp_ip=1.2.3.4&ftp_username=asdf&ftp_password=base64enc&ftp_path=/]
1

Notify user 'bob' about blacklisted IP '1.2.3.4':

action=notify&value=ip_blacklist&ip=1.2.3.4&time=12345678&username=bob
1

Regenerate the caches used throughout the DirectAdmin panel:

action=cache&type=popquota
action=cache&type=popquota&value=domain.com
action=cache&value=showallusers
action=cache&value=showallusers&user=bob
action=cache&value=showallusers&select0=bob&select1=fred&...
action=cache&value=safemode
action=cache&value=safemode&domain.com=username&domain2.com=user2
1
2
3
4
5
6
7

Add or delete DNS records:

action=dns&do=add&domain=es5.com&type=A&name=taskqrec&value=1.2.3.4&ttl=500
action=dns&do=delete&domain=domain.com&type=A&name=some.thing.&value=1.2.3.4
action=dns&do=delete&domain=domain.com&type=A&name=some.thing.&value=* //deletes any value.
action=dns&do=delete_zone&domain=domain.com
1
2
3
4

Delete old mbox files (nowadays Maildir format is in use):

action=delete&value=mbox
1

Convert mailboxes to use new mail partitioning:

action=convert&value=mail_partition
action=convert&value=mail_partition&smart=yes
action=convert&value=mail_partition&user=fred
action=convert&value=mail_partition&user=fred&smart=yes
action=delete&value=old_mail_partition
action=delete&value=old_mail_partition&user=fred
1
2
3
4
5
6

Delete users, delete domains, and lastly, delete databases:

action=delete&value=account&requestedby=admin&select0=bob(&select1=fred...)
action=delete&value=domain&requestedby=admin&select0=domain.com(&select1=domain2.com...)
action=delete&value=test_dbs
1
2
3

Switch from one IP address to another:

action=ipswap&old=1.2.3.4&new=4.3.2.1
1

Update plugin (CustomBuild GUI plugin example):

action=plugin&value=update&username=admin&select0=custombuild
1
Last Updated: 6/23/2021, 9:36:08 PM