Configure Detective server

Table of content

Quickstart
1. The Detective config file
2. Start the Detective
  1. Running as daemon
Connect to Postfix
Finetuning
What's next ?

Quickstart

For an example setup, we'll use DeepDNSBL, DKIMVerify and Archive, because they all do not need any third party programs (such as ClamAV or CRM114).

The Detective config file

If you have used the debian package or stuck to the default directory structure, you can find the config file in /etc/decency/detective.yml

Clean it up (empty it), and let's start with a new config. As in the Doorman tutorial, we're setting up the minimal requirements: logging, database, cache, server and then the injection part (where you inject the mails back to postfix for final delivery).

Also, here again the warning about YAML indention: All the config snippets are YAML. So DONT use TABS! I use four spaces, you can go with two if you are a lazy typer - as long as you stay consistent.

If you've read the Doorman tutorial, database, cache and logging are exactly the same! For the sake of having a complete tutorial, i'd copy-and-pasted it here again.

Database

I am going to use a SQLite database, cause it's either already installed or you can install it quite simple (via your distribution installer).

Insert into the (empty) configuration file the following:

---

# Using SQLite database
database:
    type: DBD
    args:
        - 'dbi:SQLite:dbname=/var/spool/decency/database.db'

I think it's quite readable. The 'dbi:SQLite:dbname=' gibberish is a Perl Data Source Name (DSN) - don't bother. You can edit the path to the database if you wish.

Cache

Having this done, let's use a cache. For the example, we'll got with the simplest one: a memory cache. The downside of this cache is, that i will be wiped on server restart and also it can grow large over time. For production i'd rather go with FastMmap or even Memcached.

So, add the following under the database configuration:

# Using Memory cache
cache:
    class: Memory

Logging

Lets use directory logging for the example. Normally, decency writes directory logs in /var/log/decency/, but you can change this as you like. Just make sure the decency user can write to the directory.

Here we go, add the following below the cache settings:

# Directory logging
logging:
    log_level: 1
    directory: /var/log/decency

Server settings

This part sets up the server itself, where it listens, how many mails it can work on in parallel and so on. Also you have to define a spool dir (default /var/spool/decency). The Detective will create some sub directories, such as quarantine/, queue/ and so on, so assure the decency user has write access!

Here we go:


# where to listen
server:
    host: 127.0.0.1
    port: 16000
    instances: 2

# where temporary and other files will be stored
spool_dir: /var/spool/decency

Re-injection settings

This is for re-inject the mail back to postfix, so that he can deliver it to the final destination (eg hand over to the MDA or relay it or whatever).

reinject:
    host: 127.0.0.1
    port: 10250

Modules

The last part is setting up the modules. You could write the module configuration inline, but i am not a big fan of huge config files (better more, each short).

Announcing modules in the server config

The order of the modules is important. So we begin with the Basic checks, then the DNSBL and at last the Greylist.

modules:
    - DeepDNSBL: /etc/decency/detective/deep-dnsbl.yml
    - DKIMVerify: /etc/decency/detective/dkim-verify.yml
    - Archive: /etc/decency/detective/archive.yml

One thing, while we're at the last config entry in the server config file: It should end with a blank link, or the Perl YAML parser will complain..

DeepDNSBL-Module config

DeepDNSBL does basically the same as the DNSBL module from Doorman, but it checks the Received-header for IPs. So if the mail you've received was forwarded by a "good" server (not on a blacklist), it would not have been recognized by DNSBL. As long as the Received-header is still intact (not cut), this module now checks all the other senders (aside from LAN or localhost IPs) for being on a blacklist. Also the IP, which has already been checked from DNSBL in the Doorman will not be rechecked again (that is, if you configured them to communicate).

Here the config in /etc/decency/detective/deep-dnsbl.yml

---

blacklist:
    -
        host: ix.dnsbl.manitu.net
        weight: -80
    -
        host: bl.spamcop.net
        weight: -80
    -
        host: dnsbl.sorbs.net
        weight: -60

Substitute the hosts with blacklists of your choice - for an example this will suffice.

DKIMVerify-Module config

DKIMVerify is, simply put, a technique to validate that an authorized mailserver has send the mail. The mailserver therefore signs some headers of the mail with a private key. Your mailserver (the recipient) then can verify the validation against a public key, deposited in special subdomain TXT zone record of the sender's domain.

Here the config in /etc/decency/detective/dkim-verify.yml

--

weight_pass: 50
weight_fail: -100
weight_invalid: -50
weight_temperror: -10

Archive-Module config

Archive does not fight SPAM in any way, but archives mail in local directories. Depending on what you offer your mailserver users, this can come in handy, as it is very simple to implement.

Here the config in /etc/decency/detective/archive.yml

--

archive_dir: /var/spool/decency/archive/%ymd%/%recipient_domain%/%recipient_prefix%

The above directory format will create/use directories similar to the following:

/var/spool/decency/archive/2011-05-17/somedomain.tld/username/mail-123213123-asd213.eml
/var/spool/decency/archive/2011-05-17/somedomain.tld/otheruser/mail-123213124-asd213.eml
/var/spool/decency/archive/2011-05-17/otherdomain.tld/yadda/mail-123213124-asd213.eml

Make sure the Decency user has write access to the base directory (here: /var/spool/decency/archive).

Setup the database

Only the Detective server might need a database, as you can activate statistics. So better set it up, it'll take a minute:

-- TABLE: stats_detective_results (SQLITE):
CREATE TABLE STATS_DETECTIVE_RESULTS ("calls" integer, "period" varchar(10),
    "last_update" integer, "status" varchar(32), "module" varchar(32),
    "start" integer, id INTEGER PRIMARY KEY);
CREATE INDEX STATS_DETECTIVE_RESULTS_START ON STATS_DETECTIVE_RESULTS ("start");
CREATE UNIQUE INDEX STATS_DETECTIVE_RESULTS_MODULE_PERIOD_STATUS_START
    ON STATS_DETECTIVE_RESULTS ("module", "period", "status", "start");

-- TABLE: stats_detective_performance (SQLITE):
CREATE TABLE STATS_DETECTIVE_PERFORMANCE ("calls" integer, "period"
    varchar(10), "last_update" integer, "score" integer, "runtime" real,
    "module" varchar(32), "start" integer, id INTEGER PRIMARY KEY);
CREATE INDEX STATS_DETECTIVE_PERFORMANCE_START ON
    STATS_DETECTIVE_PERFORMANCE ("start");
CREATE UNIQUE INDEX STATS_DETECTIVE_PERFORMANCE_MODULE_PERIOD_START
    ON STATS_DETECTIVE_PERFORMANCE ("module", "period", "start");

-- TABLE: stats_detective_final_state (SQLITE):
CREATE TABLE STATS_DETECTIVE_FINAL_STATE ("amount" integer,
    "period" varchar(25), "status" varchar(10), "start" integer,
    id INTEGER PRIMARY KEY);
CREATE INDEX STATS_DETECTIVE_FINAL_STATE_START ON
    STATS_DETECTIVE_FINAL_STATE ("start");
CREATE UNIQUE INDEX STATS_DETECTIVE_FINAL_STATE_PERIOD_STATUS_START
    ON STATS_DETECTIVE_FINAL_STATE ("period", "status", "start");

Then create your database out of it:

$ sqlite3 /var/spool/decency/database.db < /tmp/tables.sql

Substitute /var/spool/decency/database.db with the path to your database. Also assure the Decency user has write access to the database:

# assuming your user and group are called "decency"
chown decency:decency /var/spool/decency/database.db

Start the Detective

Thats it! You can now start the Detective. For the first run, it's a good idea not to use the init.d-scripts or daemonize the server, because you will see all errors directly.

# direct
$ /opt/decency/bin/decency.pl -u decency -g decency -a detective -l 6

You can stop the server by pressing "Ctrl+c".

Running as daemon

If everything works well, you can run the Doorman as daemon. Either use the init.d scripts or append the "-d" parameter:

# init scripts
$ /etc/init.d/decency-detective start

# direct
$ /opt/decency/bin/decency.pl -u decency -g decency -a detective -d

Connect to Postfix

This is a bit more complicated as for the Doorman (there is also an easier way, but it implies that really any mail will be filtered - mails you send as well as mails you receive; in a receive-only mailserver, this would be of course no problem.. read the Detective config on how to do this).

So add the following in your smtpd_recipient_restrictions (as they are the last used restriction class) to /etc/postfix/main.cf. Assure that your "outgoing" mails (reads: mails from authenticated users) are passed before, eg via permit_sasl_authenticated ..

smtpd_recipient_restrictions =
    # .. something before
    check_client_access pcre:/etc/postfix/detective-filter.pcre
    # .. something after

Also, you need to disable MIME output conversion, because it would break DKIM verification. So put the following somewhere in the main.cf as well:

disable_mime_output_conversion = yes

Now you need to setup the pcre file in /etc/postfix/detective-filter.pcre:

/./ FILTER decency:[127.0.0.1]:16000

Now you need to add the mentioned decency server at port 16000 in master.cf and setup a new server in /etc/postfix/master.cf for re-injection of the filtered mail. So add:


# Decency server
decency	unix  -       -       n       -       2        smtp
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
    -o smtp_send_xforward_command=yes
    -o disable_mime_output_conversion=yes
    -o smtp_destination_recipient_limit=1

# Re-inject mails from Decency after filtering
127.0.0.1:10250      inet  n       -       -       -       2       smtpd
    -o content_filter= 
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject_unauth_destination,permit
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

As the number of instances in the server config have been set to two, it is a good idea to align the number of processes of the both instances as well.

Phew, thats it. Restart postfix and you are good to go.

Finetuning

Read the module descriptions, how and why they work and get to know at least ClamAV and one solid heuristic SPAM filter (DSPAM, CRM144 or Bogofilter). Following a list of currently available modules:

Archive
Not an anti SPAM filter. Just archives mail.
Bogofilter
Uses the Bogofilter anti SPAM filter to analyze mail contents.
ClamAV
Uses the ClamAV daemon clamd to find viruses.
CRM114
Uses the CRM114 Discriminator, another anti SPAM filter.
Deep DNSBL
An anti SPAM filter checking Received-headers IPs against blacklists.
DKIM Sign
Not an anti SPAM filter. Signs outgoing mails (your send mails) with private DKIM keys.
DKIM Verify
Anti SPAM filter, verifying incoming mails against DKIM keys.
DSPAM
Utilizes the anti SPAM filter DSPAM
Razor
Uses the anti SPAM Vipul's Razor network to determine SPAM.
SpamAssassin
Uses the SpamAssassin anti SPAM filter

What's next ?

Also read the Detective configuration, where you can find out about fine tuning your scoring, default responses and so on.

In the cache and database documentation you can read about the pros and cons about the supported databases and caches (and how to configure them).

Then of course read the Doorman tutorial, if you did not already. Then learn how to link Doorman and Detective together.

Decency

The spam fighting framework