Print this Page

Home Server Dspam filter

Index

26-Jun-2011: first release.
28-Aug-2011: updated directory permissions, logrotate.
18-Jan-2012: add nginx configuration.

 

Introduction.

DSPAM is a scalable and open-source content-based spam filter designed for multi-user enterprise systems. The filter does this by “learning” from it’s mistakes. In our setup dspam is installed between postfix and dovecot-lda. Dspam is not run as a daemon, but spawned by postfix every time a email is processed. This is fine for our small home server. The overview of the total setup is like:

[Postfix] ---> [DSPAM] ----> [dovecot-lda] ---> (Users mailbox)
                       \
                        \--> [Quarantine]
[End user] ----------------> [Web interface]

Users can choose to use the Quarantine for spam mail, or let it pass and let the dovecot sieve filter put the mail in the users local Spam mailbox. These settings can be configured in the web interface.

 

Installation.

Install dspam using pacman:

[root@homsrv ~]# pacman -S dspam
resolving dependencies...
looking for inter-conflicts...

Targets (1): dspam-3.9.0-4

Total Download Size:    0.27 MB
Total Installed Size:   1.81 MB

Proceed with installation? [Y/n]
:: Retrieving packages from community...
 dspam-3.9.0-4-i686      281.2K  710.0K/s 00:00:00 [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) installing dspam                             [######################] 100%
adding dspam system group... done.
adding dspam system user... done.
>>> To populate the DSPAM database, you need to follow several steps.
>>> First create a database. Login to the mysql command prompt.
 $ mysql -u root -p
 mysql> CREATE database dspam;
>>> Next, you need to create a dspam user. At the same MySQL prompt:
 mysql> GRANT ALL PRIVILEGES ON dspam.* TO dspam@'localhost' IDENTIFIED BY 'passwd';
>>> Replacing passwd with your chosen password.
>>> If you want a space optimized db do:
 $ mysql -u dspam dspam -p < /var/lib/dspam/mysql/mysql_objects-space.sql
>>> If you want a speed optimized db do:
 $ mysql -u dspam dspam -p < /var/lib/dspam/mysql/mysql_objects-speed.sql
>>> Enter the password you set in the previous step, and the database should be
    populated.

>>> Remember to edit /etc/dspam/dspam.conf accordenly
>>> If you want to use the postgresql, sqlite3 or Berekely DB4 backends,
>>> read the documentation
>>>
Optional dependencies for dspam
 libmysqlclient: MySQL support
 postgresql-libs: PostgreSQL support
 db: BerkeleyDB support
 sqlite3: SQLite support
[root@homsrv ~]#

 

 

Configure MySQL.

Create a database user for dspam:

[root@homsrv ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.13 Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE database dspam;
Query OK, 1 row affected (0.08 sec)

mysql> GRANT ALL PRIVILEGES ON dspam.* TO dspam@'localhost' IDENTIFIED BY 'dspam';
Query OK, 0 rows affected (0.04 sec)

mysql> quit;
Bye
[root@homsrv ~]#

The script /var/lib/dspam/mysql/mysql_objects-speed.sql can create most tables for dspam, but the syntax in that script is not correct for the current MySQL version. So edit that script with vi (or other editor of your choice) and global replace type= with ENGINE=. After the script is fixed, create the tables and add one other table from the command line:

[root@homsrv ~]# mysql -u dspam dspam -p \
 < /var/lib/dspam/mysql/mysql_objects-speed.sql
Enter password:
[root@homsrv ~]# echo "create table dspam_virtual_uids ( \ 
 uid smallint(5) unsigned NOT NULL AUTO_INCREMENT, \
 username varchar(128) DEFAULT NULL, PRIMARY KEY (uid), \
 UNIQUE KEY id_virtual_uids_01 (username) ) \
 ENGINE=MyISAM;" | mysql -u dspam -p dspam

There should now be five tables in the database dspam.

 

Configure dspam.

First add the user dspam to the group users.

[root@homsrv ~]# usermod -a -G users dspam
[root@homsrv ~]#

Then, edit /etc/dspam/dspam.conf (diff -u is shown):

--- dspam.conf.orig     2011-06-26 13:10:09.000000000 +0200
+++ dspam.conf  2011-06-26 13:17:27.000000000 +0200
@@ -45,7 +45,7 @@
 #TrustedDeliveryAgent "/bin/maildrop"          # Maildrop
 #TrustedDeliveryAgent "/usr/local/sbin/exim -oMr spam-scanned" # Exim
 #
-TrustedDeliveryAgent "/usr/bin/procmail"
+TrustedDeliveryAgent "/usr/lib/dovecot/dovecot-lda"

 #
 # Untrusted Delivery Agent: Specifies the local delivery agent and arguments
@@ -55,7 +55,7 @@
 # the user DSPAM is processing mail for. This configuration parameter is only
 # necessary if you plan on allowing untrusted processing.
 #
-#UntrustedDeliveryAgent "/usr/bin/procmail -d %u"
+UntrustedDeliveryAgent "/usr/lib/dovecot/dovecot-lda -d %u"

 #
 # SMTP or LMTP Delivery: Alternatively, you may wish to use SMTP or LMTP
@@ -129,11 +129,11 @@
 #
 Trust root
 Trust dspam
-Trust apache
-Trust mail
-Trust mailnull
-Trust smmsp
-Trust daemon
+Trust http
+#Trust mail
+#Trust mailnull
+#Trust smmsp
+#Trust daemon
 #Trust nobody
 #Trust majordomo

@@ -293,6 +293,7 @@
 Preference "optOutClamAV=off"          # { on | off } -> default:off
 Preference "ignoreRBLLookups=off"      # { on | off } -> default:off
 Preference "RBLInoculate=off"          # { on | off } -> default:off
+Preference "notifications=off"         # { on | off } -> default:off

 #
 # Overrides: Specifies the user preferences which may override configuration
@@ -323,6 +324,7 @@
 AllowOverride trainingMode
 AllowOverride whitelistThreshold
 AllowOverride dailyQuarantineSummary
+AllowOverride notifications

 # --- MySQL ---

@@ -330,12 +332,12 @@
 # Storage driver settings: Specific to a particular storage driver. Uncomment
 # the configuration specific to your installation, if applicable.
 #
-#MySQLServer           /tmp/mysql.sock
+MySQLServer            /var/run/mysqld/mysqld.sock
 #MySQLPort
-#MySQLUser             dspam
-#MySQLPass             changeme
-#MySQLDb               dspam
-#MySQLCompress         true
+MySQLUser              dspam
+MySQLPass              dspam
+MySQLDb                dspam
+MySQLCompress          true
 #MySQLReconnect                true

 # If you are using replication for clustering, you can also specify a separate
@@ -381,7 +383,7 @@
 # (pointing to some arbitrary user), and the uid in the signature will
 # switch to the correct user. Result: you need only one spam alias

-#MySQLUIDInSignature   on
+MySQLUIDInSignature    on

 # --- PostgreSQL ---

@@ -623,7 +625,7 @@
 # Notifications: Enable the sending of notification emails to users (first
 # message, quarantine full, etc.)
 #
-Notifications  off
+Notifications  on

 #
 # Purge configuration: Set dspam_clean purge default options, if not otherwise
@@ -814,7 +816,7 @@
 # you are running the client and server on the same machine, as it eliminates
 # much of the bandwidth overhead.
 #
-ServerDomainSocketPath "/var/run/dspam/dspam.sock"
+#ServerDomainSocketPath        "/var/run/dspam/dspam.sock"

 #
 # Client Mode: If you are running DSPAM in client/server mode, uncomment and

Then edit /etc/postfix-in/main.cf and disable dovecot-lda and enable dspam, so that the delivery part now looks like:

mailbox_command = /usr/bin/dspam --deliver=innocent --user $USER -- -d %u
#mailbox_command = /usr/lib/dovecot/dovecot-lda -d "$USER"

To use dspam between postfix and dovecot-lda some file permissions need to be changed:

[root@homsrv ~]# cd /etc/dspam/
[root@homsrv dspam]# chmod 644 *
[root@homsrv dspam]# cd /var/lib/dspam/
[root@homsrv dspam]# mkdir data
[root@homsrv dspam]# chown dspam:users data
[root@homsrv dspam]# chmod 770 data
[root@homsrv dspam]# chown dspam:users .
[root@homsrv dspam]# chmod 775 .
[root@homsrv dspam]# ln -s /etc/dspam/txt/
[root@homsrv dspam]# ls -la
total 24
drwxrwxr-x  5 dspam  users 4096 Jun 26 14:17 .
drwxr-xr-x 20 root   root  4096 Jun 26 12:37 ..
drwxrwx---  2 dspam  users 4096 Jun 26 14:17 data
drwxr-xr-x  2 root   root  4096 Jun 26 12:48 mysql
drwxr-xr-x  2 root   root  4096 Apr  4 14:35 pgsql
lrwxrwxrwx  1 root   root    15 Jun 26 14:34 txt -> /etc/dspam/txt/
[root@homsrv dspam]#

In the support archive (see download) in the directory examples/homsrv/etc/dspam/txt you will find three text files, put these in a /etc/dspam/txt and edit these files. You may translate them too if you like. It are templates for emails that are sent to a user when he/she gets the first mail or spam seen by dspam. For some reason these files are not shipped with Arch Linux.

Dspam writes the logfile to /var/lib.dspam so edit /etc/logrotate.d/dspam.logrotated:

/var/lib/dspam/system.log {
        weekly
        compress
        su dspam users
        create 0664 dspam users
}

Finally, restart postfix:

[root@homsrv ~]# /etc/rc.d/postfix restart
:: Stopping postfix daemon                                               [DONE]
:: Starting postfix daemon                                               [DONE]
[root@homsrv ~]#

 

Configure Apache web interface.

Skip to the next paragraph if you use nginx as web server. The dspam web interface needs the apache suexec module to run. This means that the directory permissions of the installed web interface must be changed, the files in the cgi-bin directory of dspam must be owned by dspam, else suexec will not work. Then all users that want to make use of the web interface, need a user name and password. So if you’re server has Samba too, you will have three places where passwords are stored. In the next session, the permissions are set, a password is created for just one user, and extra perl modules are installed so that the web interface knows how to draw graphics.

[root@homsrv ~]# pacman -S perl-gdgraph
resolving dependencies...
looking for inter-conflicts...

Targets (4): gd-2.0.36RC1-3  perl-gd-2.44-3  perl-gdtextutil-0.86-2
 perl-gdgraph-1.44-2

Total Download Size:    0.40 MB
Total Installed Size:   1.57 MB

Proceed with installation? [Y/n]
:: Retrieving packages from extra...
 gd-2.0.36RC1-3-i686     177.1K  555.9K/s 00:00:00 [######################] 100%
:: Retrieving packages from community...
 perl-gd-2.44-3-i686     129.4K  468.7K/s 00:00:00 [######################] 100%
 perl-gdtextutil-0....    24.4K  219.3K/s 00:00:00 [######################] 100%
 perl-gdgraph-1.44-...    76.0K  348.9K/s 00:00:00 [######################] 100%
(4/4) checking package integrity                   [######################] 100%
(4/4) checking for file conflicts                  [######################] 100%
(1/4) installing gd                                [######################] 100%
(2/4) installing perl-gd                           [######################] 100%
(3/4) installing perl-gdtextutil                   [######################] 100%
(4/4) installing perl-gdgraph                      [######################] 100%
[root@homsrv ~]# cd /srv/http/dspam/
[root@homsrv dspam]# mkdir etc
[root@homsrv dspam]# cd etc
[root@homsrv etc]# htpasswd -c passwd mbroek (note: -c only the first time)
New password:
Re-type new password:
Adding password for user mbroek
[root@homsrv etc]# cd ..
[root@homsrv dspam]# chown -R dspam:dspam cgi-bin
[root@homsrv dspam]#

In /srv/http/dspam/cgi-bin you find the file admins. This file has the names of all users that are allowed to use the admin functions of the web interface. Currently only root is in that file. You may add normal users (like yourself) in that file, each user on a new line.

We use the dspam web interface as a virtual host, that means that when we connect to http://spam.wpl.ym and we log in we are in fact in the directory /srv/http/dspam/htdocs and there is no index.html. Put the following file as index.php in that directory so that users will be moved to the dspam.cgi as soon as they are logged in.

<?php
header( "Location: http://spam.wpl.ym/cgi-bin/dspam.cgi" );

exit( 0 );
?>

Add the  following virtual host to  /etc/httpd/conf/extra/httpd-vhosts.conf:

#
# spam.wpl.ym
<VirtualHost *:80>
 ServerAdmin admin@wpl.uk
 DocumentRoot /srv/http/dspam/htdocs
 ServerName spam.wpl.ym
 SuexecUserGroup dspam dspam
 ScriptAlias /cgi-bin/ /srv/http/dspam/cgi-bin/
 <Directory /srv/http/dspam>
 Options FollowSymLinks ExecCGI
 Order deny,allow
 Deny from all
 Require valid-user
 AuthType Basic
 AuthName "DSPAM Control Center"
 Satisfy Any
 AuthUserFile /srv/http/dspam/etc/passwd
 Options +SymlinksIfOwnerMatch +Includes
 </Directory>
</VirtualHost>

And add a “spam   CNAME homsrv” line to /var/named/int/db.wpl.ym so that the nameserver resolves the hostname spam to the IP address of our home server.

 

Configure Nginx web interface.

The dspam web interface needs to run the CGI scripts as user dspam. In the nginx setup article you can see how we setup the standard fcgiwrap daemon. For this web interface we need to start a second fcgiwrap as user dspam. The next session will do this:

[root@homsrv ~]# cd /etc/conf.d/
[root@homsrv conf.d]# cp fcgiwrap fcgiwrap-dspam
[root@homsrv conf.d]# vi fcgiwrap-dspam

In the new /etc/conf.d/fcgiwrap-dspam change the port to 9002 and the user and group into dspam. The next step:

[root@homsrv conf.d]# cd ../rc.d
[root@homsrv rc.d]# cp fcgiwrap fcgiwrap-dspam
[root@homsrv rc.d]# vi fcgiwrap-dspam

Edit the DAEMON variable and save the file:

DAEMON=fcgiwrap-dspam

Start the new fcgiwrap:

[root@homsrv rc.d]# rc.d start fcgiwrap-dspam
:: Starting fcgiwrap-dspam daemon                                        [DONE]
[root@homsrv rc.d]# less fcgiwrap-dspam
[root@homsrv rc.d]#

Now we have a CGI wrapper that runs as the dspam user. Add fcgiwrap-dspam to the DAEMONS() array /etc/rc.conf just before nginx so that the new wrapper starts during system boot.

All users that want to make use of the web interface, need a user name and password. So if you’re server has Samba too, you will have three places where user passwords are stored. In the next session a password is created for just one user, and extra perl modules are installed so that the web interface knows how to draw graphics.

[root@homsrv ~]# pacman -S perl-gdgraph
resolving dependencies...
looking for inter-conflicts...

Targets (4): gd-2.0.36RC1-3  perl-gd-2.44-3  perl-gdtextutil-0.86-2
 perl-gdgraph-1.44-2

Total Download Size:    0.40 MB
Total Installed Size:   1.57 MB

Proceed with installation? [Y/n]
:: Retrieving packages from extra...
 gd-2.0.36RC1-3-i686     177.1K  555.9K/s 00:00:00 [######################] 100%
:: Retrieving packages from community...
 perl-gd-2.44-3-i686     129.4K  468.7K/s 00:00:00 [######################] 100%
 perl-gdtextutil-0....    24.4K  219.3K/s 00:00:00 [######################] 100%
 perl-gdgraph-1.44-...    76.0K  348.9K/s 00:00:00 [######################] 100%
(4/4) checking package integrity                   [######################] 100%
(4/4) checking for file conflicts                  [######################] 100%
(1/4) installing gd                                [######################] 100%
(2/4) installing perl-gd                           [######################] 100%
(3/4) installing perl-gdtextutil                   [######################] 100%
(4/4) installing perl-gdgraph                      [######################] 100%
[root@homsrv ~]# cd /srv/http/dspam/
[root@homsrv dspam]# mkdir etc
[root@homsrv dspam]# cd etc
[root@homsrv etc]# htpasswd -c passwd mbroek (note: -c only the first time)
New password:
Re-type new password:
Adding password for user mbroek
[root@homsrv etc]#

In /srv/http/dspam/cgi-bin you find the file admins. This file has the names of all users that are allowed to use the admin functions of the web interface. Currently only root is in that file. You may add normal users (like yourself) in that file, each user on a new line.

We use the dspam web interface as a virtual host, that means that when we connect to http://spam.wpl.ym and we log in we are in fact in the directory /srv/http/dspam/htdocs and there is no index.html. Put the following file as index.php in that directory so that users will be moved to the dspam.cgi as soon as they are logged in.

<?php
header( "Location: http://spam.wpl.ym/cgi-bin/dspam.cgi" );

exit( 0 );
?>

Add the following virtual host to /etc/nginx/conf/nginx.conf and restart nginx:

    # spam.wpl.ym
    server {
        listen                  10.126.160.253:80;
        listen                  [2001:1af8:fecf:7ea0::201]:80;
        server_name             spam.wpl.ym;
        root                    /srv/http/dspam/htdocs;
        access_log              logs/spam_access_log;
        error_log               logs/spam_error_log;
        auth_basic              "DSPAM Access";
        auth_basic_user_file    /srv/http/dspam/etc/passwd;
        index                   index.php;

        location ~ \.php$ {
            fastcgi_pass        php;
            include             fastcgi.conf;
        }

        location ~ ^/cgi-bin/(.*\.cgi)$ {
            root                /srv/http/dspam/cgi-bin/;
            rewrite             ^/cgi-bin/(.*)\.cgi /$1.cgi break;
            fastcgi_param       AUTH_USER $remote_user;
            fastcgi_param       REMOTE_USER $remote_user;
            include             fastcgi.conf;
            fastcgi_pass        127.0.0.1:9002;
        }
    }

And add a “spam CNAME homsrv” line to /var/named/int/db.wpl.ym so that the nameserver resolves the hostname spam to the IP address of our home server.

 

Maintenance.

Because after each processed email, new data is added to the MySQL database. A maintenance script /etc/cron.daily/dspam must be used to purge old data from the database. This script will also take care of log rotation, the file /etc/logrotate.d/dspam is not good for our purpose and must be removed.

After an update of the dspam package, /etc/logrorate.d/dspam  is there again but you should get mail that it doesn’t work so you can remove it again.

#!/bin/sh
#
# /etc/cron.daily/dspam
#
# Michiel Broek, Arch Linux version (based on a Gentoo script).

LOGROTATE_AGE=30
SIGNATURE_AGE=90

PURGE_SCRIPT=/var/lib/dspam/mysql/purge-4.1.sql

# Idea from Gentoo script.
clean_mysql_drv()
{
 if [ ! -e "/usr/bin/mysql" ]
 then
  echo "Can not run MySQL purge script: /usr/bin/mysql does not exist"
  return 1
 fi

 if [ ! -f "$PURGE_SCRIPT" ]
 then
  echo "Can not run MySQL purge script: $PURGE_SCRIPT does not exist"
  return 1
 fi

 # Get parameters from the config file
 SQLSERVER=$(grep ^MySQLServer /etc/dspam/dspam.conf | cut -f 3)
 SQLPORT=$(grep ^MySQLPort /etc/dspam/dspam.conf | cut -f 3)
 SQLUSER=$(grep ^MySQLUser /etc/dspam/dspam.conf | cut -f 3)
 SQLPASS=$(grep ^MySQLPass /etc/dspam/dspam.conf | cut -f 3)
 SQLDB=$(grep ^MySQLDb /etc/dspam/dspam.conf | cut -f 3)

 if [ ! -z "$(echo $SQLSERVER | grep ^/var)" ]
 then
  SQLCMD="/usr/bin/mysql --socket=${SQLSERVER} --user=${SQLUSER} \
          --password=${SQLPASS}"
 else
  if [ -z "$SQLPORT" ]
  then
   SQLCMD="/usr/bin/mysql --host=${SQLSERVER} --user=${SQLUSER} \
           --password=${SQLPASS}"
  else
   SQLCMD="/usr/bin/mysql --host=${SQLSERVER} --port=${SQLPORT} \
           --user=${SQLUSER} --password=${SQLPASS}"
  fi
 fi

 $SQLCMD ${SQLDB} < $PURGE_SCRIPT >/dev/null
}

run_dspam_clean()
{
 if [ ! -e "/usr/bin/dspam_clean" ]
 then
  echo "Can not run DSPAM clean application:"
  echo "  /usr/bin/dspam_clean does not exist"
  return 1
 else
  /usr/bin/dspam_clean -s${SIGNATURE_AGE} -p${SIGNATURE_AGE} \
          -u${SIGNATURE_AGE},${SIGNATURE_AGE},${SIGNATURE_AGE},${SIGNATURE_AGE} \
          >/dev/null 2>&1
  return $?
 fi
}

clean_mysql_drv
run_dspam_clean

# Rotate logs
su - dspam -c "umask 002 ; /usr/bin/dspam_logrotate \
     -a ${LOGROTATE_AGE} -d /var/lib/dspam"

 

Download.

The complete set of configuration files is available in the next archive:

Linux Home Server complete package
Linux Home Server complete package
homeserver-complete.tar.gz
2.6 MiB
7 Downloads
Details...

 

Permanent link to this article: http://www.mbse.eu/linux/homeserver/mail/dspam/