Home Server Dspam filter

Index

Introduction.
Installation.
Configure MySQL.
Configure dspam.
Configure Apache web interface.
Configure Nginx web interface.
Maintenance.
Download.

26-Jun-2011: first release.
28-Aug-2011: updated directory permissions, logrotate.
18-Jan-2012: add nginx configuration.
07-Jan-2013: adjusted for Slackware.

Introduction.

DSPAM is a scalable and open-source content-based spam filter designed for multi-user enterprise systems. The filter does this by “learning” from it’s mistakes. In our setup dspam is installed between postfix and dovecot-lda. Dspam is not run as a daemon, but spawned by postfix every time a email is processed. This is fine for our small home server. The overview of the total setup is like:

[Postfix] ---> [DSPAM] ----> [dovecot-lda] ---> (Users mailbox)
                       \
                        \--> [Quarantine]
[End user] ----------------> [Web interface]

Users can choose to use the Quarantine for spam mail, or let it pass and let the dovecot sieve filter put the mail in the users local Spam mailbox. These settings can be configured in the web interface.

Installation.

Build dspam using a Slackbuild and install the package.

Configure MySQL.

Create a database user for dspam:

root@homsrv:~# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.13 Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE database dspam;
Query OK, 1 row affected (0.08 sec)

mysql> GRANT ALL PRIVILEGES ON dspam.* TO dspam@'localhost' IDENTIFIED BY 'dspam';
Query OK, 0 rows affected (0.04 sec)

mysql> quit;
Bye
root@homsrv:~#

The script /var/lib/dspam/mysql/mysql_objects-speed.sql can create most tables for dspam, but the syntax in that script is not correct for the current MySQL version. So edit that script with vi (or other editor of your choice) and global replace type= with ENGINE=. After the script is fixed, create the tables and add one other table from the command line:

root@homsrv:~# mysql -u dspam dspam -p \
 < /var/lib/dspam/mysql/mysql_objects-speed.sql
Enter password:
root@homsrv:~# echo "create table dspam_virtual_uids ( \ 
 uid smallint(5) unsigned NOT NULL AUTO_INCREMENT, \
 username varchar(128) DEFAULT NULL, PRIMARY KEY (uid), \
 UNIQUE KEY id_virtual_uids_01 (username) ) \
 ENGINE=MyISAM;" | mysql -u dspam -p dspam

There should now be five tables in the database dspam.

Configure dspam.

First add the user dspam to the group users.

root@homsrv:~# usermod -a -G users dspam
root@homsrv:~#

Then, edit /etc/dspam/dspam.conf (diff -u is shown):

--- dspam.conf.orig     2011-06-26 13:10:09.000000000 +0200
+++ dspam.conf  2011-06-26 13:17:27.000000000 +0200
@@ -45,7 +45,7 @@
 #TrustedDeliveryAgent "/bin/maildrop"          # Maildrop
 #TrustedDeliveryAgent "/usr/local/sbin/exim -oMr spam-scanned" # Exim
 #
-TrustedDeliveryAgent "/usr/bin/procmail"
+TrustedDeliveryAgent "/usr/libexec/dovecot/dovecot-lda"

 #
 # Untrusted Delivery Agent: Specifies the local delivery agent and arguments
@@ -55,7 +55,7 @@
 # the user DSPAM is processing mail for. This configuration parameter is only
 # necessary if you plan on allowing untrusted processing.
 #
-#UntrustedDeliveryAgent "/usr/bin/procmail -d %u"
+UntrustedDeliveryAgent "/usr/libexec/dovecot/dovecot-lda -d %u"

 #
 # SMTP or LMTP Delivery: Alternatively, you may wish to use SMTP or LMTP
@@ -129,11 +129,11 @@
 #
 Trust root
 Trust dspam
-Trust apache
-Trust mail
-Trust mailnull
-Trust smmsp
-Trust daemon
+Trust http
+#Trust mail
+#Trust mailnull
+#Trust smmsp
+#Trust daemon
 #Trust nobody
 #Trust majordomo

@@ -293,6 +293,7 @@
 Preference "optOutClamAV=off"          # { on | off } -> default:off
 Preference "ignoreRBLLookups=off"      # { on | off } -> default:off
 Preference "RBLInoculate=off"          # { on | off } -> default:off
+Preference "notifications=off"         # { on | off } -> default:off

 #
 # Overrides: Specifies the user preferences which may override configuration
@@ -323,6 +324,7 @@
 AllowOverride trainingMode
 AllowOverride whitelistThreshold
 AllowOverride dailyQuarantineSummary
+AllowOverride notifications

 # --- MySQL ---

@@ -330,12 +332,12 @@
 # Storage driver settings: Specific to a particular storage driver. Uncomment
 # the configuration specific to your installation, if applicable.
 #
-#MySQLServer           /tmp/mysql.sock
+MySQLServer            /var/run/mysqld/mysqld.sock
 #MySQLPort
-#MySQLUser             dspam
-#MySQLPass             changeme
-#MySQLDb               dspam
-#MySQLCompress         true
+MySQLUser              dspam
+MySQLPass              dspam
+MySQLDb                dspam
+MySQLCompress          true
 #MySQLReconnect                true

 # If you are using replication for clustering, you can also specify a separate
@@ -381,7 +383,7 @@
 # (pointing to some arbitrary user), and the uid in the signature will
 # switch to the correct user. Result: you need only one spam alias

-#MySQLUIDInSignature   on
+MySQLUIDInSignature    on

 # --- PostgreSQL ---

@@ -623,7 +625,7 @@
 # Notifications: Enable the sending of notification emails to users (first
 # message, quarantine full, etc.)
 #
-Notifications  off
+Notifications  on

 #
 # Purge configuration: Set dspam_clean purge default options, if not otherwise
@@ -814,7 +816,7 @@
 # you are running the client and server on the same machine, as it eliminates
 # much of the bandwidth overhead.
 #
-ServerDomainSocketPath "/var/run/dspam/dspam.sock"
+#ServerDomainSocketPath        "/var/run/dspam/dspam.sock"

 #
 # Client Mode: If you are running DSPAM in client/server mode, uncomment and

Then edit /etc/postfix-in/main.cf and disable dovecot-lda and enable dspam, so that the delivery part now looks like:

mailbox_command = /usr/bin/dspam --deliver=innocent --user $USER -- -d %u
#mailbox_command = /usr/lib/dovecot/dovecot-lda -d "$USER"

To use dspam between postfix and dovecot-lda some file permissions need to be changed:

root@homsrv:~# cd /etc/dspam/
root@homsrv:/srv/http/dspam# chmod 644 *
root@homsrv:/srv/http/dspam# cd /var/lib/dspam/
root@homsrv:/srv/http/dspam# mkdir data
root@homsrv:/srv/http/dspam# chown dspam:users data
root@homsrv:/srv/http/dspam# chmod 770 data
root@homsrv:/srv/http/dspam# chown dspam:users .
root@homsrv:/srv/http/dspam# chmod 775 .
root@homsrv:/srv/http/dspam# ln -s /etc/dspam/txt/
root@homsrv:/srv/http/dspam# ls -la
total 24
drwxrwxr-x  5 dspam  users 4096 Jun 26 14:17 .
drwxr-xr-x 20 root   root  4096 Jun 26 12:37 ..
drwxrwx---  2 dspam  users 4096 Jun 26 14:17 data
drwxr-xr-x  2 root   root  4096 Jun 26 12:48 mysql
drwxr-xr-x  2 root   root  4096 Apr  4 14:35 pgsql
lrwxrwxrwx  1 root   root    15 Jun 26 14:34 txt -> /etc/dspam/txt/
root@homsrv:/srv/http/dspam#

In the directory etc/dspam/txt you will find three text files. It are templates for emails that are sent to a user when he/she gets the first mail or spam seen by dspam. If you like just rewrite and translate them. Finally, restart postfix:

root@homsrv:~# /etc/rc.d/rc.sendmail restart
postfix-in/postfix-script: stopping the Postfix mail system
postfix/postfix-script: stopping the Postfix mail system
postfix/postfix-script: starting the Postfix mail system
postfix-in/postfix-script: starting the Postfix mail system
root@homsrv:~#

Configure Apache web interface.

Skip to the next paragraph if you use nginx as web server. The dspam web interface needs the apache suexec module to run. This means that the directory permissions of the installed web interface must be changed, the files in the cgi-bin directory of dspam must be owned by dspam, else suexec will not work. Then all users that want to make use of the web interface, need a user name and password. So if you’re server has Samba too, you will have three places where passwords are stored. In the next session, the permissions are set, a password is created for just one user, and extra perl modules are installed so that the web interface knows how to draw graphics.

root@homsrv:~# cd /srv/http/dspam/
root@homsrv:/srv/http/dspam# mkdir etc
root@homsrv:/srv/http/dspam# cd etc
root@homsrv:/srv/http/dspam/etc# htpasswd -c passwd mbroek (note: -c only the first time)
New password:
Re-type new password:
Adding password for user mbroek
root@homsrv:/srv/http/dspam/etc# cd ..
root@homsrv:/srv/http/dspam# chown -R dspam:dspam cgi-bin
root@homsrv:/srv/http/dspam#

In /srv/http/dspam/cgi-bin you find the file admins. This file has the names of all users that are allowed to use the admin functions of the web interface. Currently only root is in that file. You may add normal users (like yourself) in that file, each user on a new line.

We use the dspam web interface as a virtual host, that means that when we connect to http://spam.wpl.ym and we log in we are in fact in the directory /srv/http/dspam/htdocs and there is no index.html. Put the following file as index.php in that directory so that users will be moved to the dspam.cgi as soon as they are logged in.

<?php
header( "Location: http://spam.wpl.ym/cgi-bin/dspam.cgi" );

exit( 0 );
?>

Add the following virtual host to /etc/httpd/conf/extra/httpd-vhosts.conf:

#
# spam.wpl.ym
<VirtualHost *:80>
 ServerAdmin admin@wpl.uk
 DocumentRoot /srv/http/dspam/htdocs
 ServerName spam.wpl.ym
 SuexecUserGroup dspam dspam
 ScriptAlias /cgi-bin/ /srv/http/dspam/cgi-bin/
 <Directory /srv/http/dspam>
 Options FollowSymLinks ExecCGI
 Order deny,allow
 Deny from all
 Require valid-user
 AuthType Basic
 AuthName "DSPAM Control Center"
 Satisfy Any
 AuthUserFile /srv/http/dspam/etc/passwd
 Options +SymlinksIfOwnerMatch +Includes
 </Directory>
</VirtualHost>

And add a “spam CNAME homsrv” line to /var/named/int/db.wpl.ym so that the nameserver resolves the hostname spam to the IP address of our home server.

Configure Nginx web interface.

The dspam web interface needs to run the CGI scripts as user dspam. In the nginx setup article you can see how we setup the standard fcgiwrap daemon. For this web interface we need to start a second fcgiwrap as user dspam. Edit /etc/fcgiwrap.conf to do this:

#
# A second one for a different user
#
FCGI_ADDRESS[1]='127.0.0.1'
FCGI_PORT[1]='9002'
FCGI_USER[1]='dspam'
FCGI_GROUP[1]='dspam'
FCGI_EXTRA_OPTIONS[1]=''

Start the new fcgiwrap:

root@homsrv:~# /etc/rc.d/init.d/fcgiwrap restart
Stopping fcgiwrap daemon ..
Starting fcgiwrap for user: apache dspam 
root@homsrv:~#

Now we have a CGI wrapper that runs as the dspam user, and also the original apache user wrapper still exists.

All users that want to make use of the web interface, need a user name and password. So if you’re server has Samba too, you will have three places where user passwords are stored. In the next session a password is created for just one user, and extra perl modules are installed so that the web interface knows how to draw graphics.

root@homsrv:~# cd /srv/http/dspam/
root@homsrv:/srv/http/dspam# mkdir etc
root@homsrv:/srv/http/dspam# cd etc
root@homsrv:/srv/http/dspam/etc# htpasswd -c passwd mbroek (note: -c only the first time)
New password:
Re-type new password:
Adding password for user mbroek
root@homsrv:/srv/http/dspam/etc#

<?php
header( "Location: http://spam.wpl.ym/cgi-bin/dspam.cgi" );

exit( 0 );
?>

Add the following virtual host to /etc/nginx/conf/nginx.conf and restart nginx:

    # spam.wpl.ym
    server {
        listen                  10.126.160.253:80;
        listen                  [2001:1af8:fecf:7ea0::201]:80;
        server_name             spam.wpl.ym;
        root                    /srv/http/dspam/htdocs;
        access_log              logs/spam_access_log;
        error_log               logs/spam_error_log;
        auth_basic              "DSPAM Access";
        auth_basic_user_file    /srv/http/dspam/etc/passwd;
        index                   index.php;

        location ~ \.php$ {
            fastcgi_pass        php;
            include             fastcgi.conf;
        }

        location ~ ^/cgi-bin/(.*\.cgi)$ {
            root                /srv/http/dspam/cgi-bin/;
            rewrite             ^/cgi-bin/(.*)\.cgi /$1.cgi break;
            fastcgi_param       AUTH_USER $remote_user;
            fastcgi_param       REMOTE_USER $remote_user;
            include             fastcgi.conf;
            fastcgi_pass        127.0.0.1:9002;
        }
    }

And add a “spam CNAME homsrv” line to /var/named/int/db.wpl.ym so that the nameserver resolves the hostname spam to the IP address of our home server.

Maintenance.

Because after each processed email, new data is added to the MySQL database. A maintenance script /etc/cron.daily/dspam must be used to purge old data from the database. This script will also take care of log rotation, the file /etc/logrotate.d/dspam is not good for our purpose and must be removed.

After an update of the dspam package, /etc/logrorate.d/dspam is there again but you should get mail that it doesn’t work so you can remove it again.

#!/bin/sh
#
# /etc/cron.daily/dspam
#
# Michiel Broek, Arch Linux version (based on a Gentoo script).

LOGROTATE_AGE=30
SIGNATURE_AGE=90

PURGE_SCRIPT=/var/lib/dspam/mysql/purge-4.1.sql

# Idea from Gentoo script.
clean_mysql_drv()
{
 if [ ! -e "/usr/bin/mysql" ]
 then
  echo "Can not run MySQL purge script: /usr/bin/mysql does not exist"
  return 1
 fi

 if [ ! -f "$PURGE_SCRIPT" ]
 then
  echo "Can not run MySQL purge script: $PURGE_SCRIPT does not exist"
  return 1
 fi

 # Get parameters from the config file
 SQLSERVER=$(grep ^MySQLServer /etc/dspam/dspam.conf | cut -f 3)
 SQLPORT=$(grep ^MySQLPort /etc/dspam/dspam.conf | cut -f 3)
 SQLUSER=$(grep ^MySQLUser /etc/dspam/dspam.conf | cut -f 3)
 SQLPASS=$(grep ^MySQLPass /etc/dspam/dspam.conf | cut -f 3)
 SQLDB=$(grep ^MySQLDb /etc/dspam/dspam.conf | cut -f 3)

 if [ ! -z "$(echo $SQLSERVER | grep ^/var)" ]
 then
  SQLCMD="/usr/bin/mysql --socket=${SQLSERVER} --user=${SQLUSER} \
          --password=${SQLPASS}"
 else
  if [ -z "$SQLPORT" ]
  then
   SQLCMD="/usr/bin/mysql --host=${SQLSERVER} --user=${SQLUSER} \
           --password=${SQLPASS}"
  else
   SQLCMD="/usr/bin/mysql --host=${SQLSERVER} --port=${SQLPORT} \
           --user=${SQLUSER} --password=${SQLPASS}"
  fi
 fi

 $SQLCMD ${SQLDB} < $PURGE_SCRIPT >/dev/null
}

run_dspam_clean()
{
 if [ ! -e "/usr/bin/dspam_clean" ]
 then
  echo "Can not run DSPAM clean application:"
  echo "  /usr/bin/dspam_clean does not exist"
  return 1
 else
  /usr/bin/dspam_clean -s${SIGNATURE_AGE} -p${SIGNATURE_AGE} \
          -u${SIGNATURE_AGE},${SIGNATURE_AGE},${SIGNATURE_AGE},${SIGNATURE_AGE} \
          >/dev/null 2>&1
  return $?
 fi
}

clean_mysql_drv
run_dspam_clean

# Rotate logs
su - dspam -c "umask 002 ; /usr/bin/dspam_logrotate \
     -a ${LOGROTATE_AGE} -d /var/lib/dspam"

Download.

See the download page for the script and configuration files.