Detective

Table of content

  1. What is a content filter?
    1. What kind of filters are there in Decency ?
      1. External filters
      2. Internal filters
    2. What other content filters are there ?
  2. Configuration
    1. server
    2. spool_dir
    3. accept_scoring
    4. doorman_verify_key
    5. notification_from
    6. reinject
    7. spam
    8. virus
    9. modules
      1. cache, database and logging
      2. Full YAML example
  3. Shared module configuration
    1. Default
      1. max_size
    2. Anti Spam modules
      1. weight_innocent
      2. weight_spam
      3. weight_translate
    3. User module
      1. default_user
      2. cmd_user
  4. Integration in Postfix
    1. master.cf
    2. main.cf
      1. content_filter
      2. Via check*_access

The Detective is a MIME content filter, utilizing internal checks as well as multiple external tools to determine wheter a mail is virus infected or SPAM.

What is a content filter?

A content filter comes in action after the mail is received by the mails server. It has access to the full mail (headers, body) and is therefore able to check the contents against any kind of filters such as DKIM (requires at least all headers), virus filters (require body, especially attachments), SPAM filters (analyzing the words in the body) and so on. In opposite to the policy behavior, the MTA will hand over / deliver the the whole mail to the content filter. The content filter has to re-inject the mail after filtering back to the mail server for delivery. Therefore, you have to have two running mail servers processes (with postfix, you can setup another one in the master.cf).

What kind of filters are there in Decency ?

In general, there are two different kinds: external, which are mostly command line programs or other running servers for which Decency acts as a proxy, and internal, which are filters implemented directly into Decency.

External filters

For external filters there are for now:

Internal filters

In the current release are:

What other content filters are there ?

There are multiple:

Configuration

Most parts of the configuration is similar to the Doorman configuration. Also, you can have multiple Detectives "in a row", on multiple servers, which are able to access previous scorings, as long as they can access the same cache.

In the debian package the configuration file can be found in /etc/decency/detective.yml .

server

Allowed values: Hash
Default: -
Suggested: { host: "127.0.0.1″, port: 16000, instances: 3 }
Required: yes

This configures the TCP settings for the server and sets the amount of instances you want to run. The server host and port settings has to be adjusted in your postfix configuration (below) accordingly.

server:
    host: 127.0.0.1
    port: 16000
    instances: 3

server.host

The hostname or IP address the server should be listen to.

Default: 127.0.0.1 (localhost)

server.port

The port the server should be listen to.

Default: 16000

server.instances

The amount of instances to be run in parallel. The more you run, the more mails can be handled in parallel and the memory will be used.

Default: 3

spool_dir

Allowed values: String (path)
Default: /var/spool/decency
Required: yes

The directory where all processed mails are temporary stored. See the folder structure.

accept_scoring

Allowed values: Bool
Default: 0
Required: no

Whether scoring from external Doorman servers will be accepted. All local, cached scoring informations (if Detective and Doorman server access the same caches) will be accepted automatically. This should only be enabled, if you provide a the verification key ( doorman_verify_key ).

doorman_verify_key

Allowed values: String (path)
Default: -
Required: no

Path to the public part of the verification key. Provide an absolute path or a relative path from the configuration dir (where detective.yml is). This key verifies, that the signed scorings originate from one of your Doorman servers.

How you can generate the forward sign key and more about this matter, read here.

doorman_verify_key: sign.pub

notification_from

Allowed values: String (path)
Default: -
Required: no

From-header for all notifications. There are three possible notifications:

notification_from: 'Postmaster <postmaster@localhost>'

reinject

Allowed values: HashRef or Array of HashRef
Default: { host: "127.0.0.1", port: 10250 }
Required: yes

Reinjection information for all to-be-passed mails (virus could be and so on). Remember to setup a MTA listening on this host/port (eg via your master.cf in postfix).

reinject.host

Can be IP or hostname.

reinject.port

The port for delivery.

reinject.copy

Defaults to 0. Set to 1 to enforce reinjection anyway.

reinject.debug

Optional. Set to 1 to print SMTP on STDERR

reinject.user

Optional. User for reinjection.

reinject.password

Optional. Password for reinjection.

reinject.ssl

Optional. Use SSL. Default 0.

# one single host
reinject:
    host: 127.0.0.1
    port: 10250

# multiple hosts
reinject:
    -
        host: 10.20.30.40
        port: 10250
    
    # failover
    -
        host: 10.20.30.41
        port: 10250
    
    # copy
    -
        host: 10.20.30.42
        port: 25
        copy: 1
        user: someuser
        pass: somepass
        ssl: 1

spam

Allowed values: Hash
Default: -
Required: yes

How to handle SPAM and when to recognize a mail as SPAM.

spam.behavior

There are three behaviors:

spam.threshold

Only for "scoring" behavior. Should be a negative integer. The lower, the more mails will not be recognized as SPAM, the higher (closer to zero), the more mails will be marked as SPAM.

spam.handle

What to do, if a mail is recognized as SPAM (as defined via spam.behavior).

spam.noisy_headers

If this is enabled (1), all SPAM headers will be included on any mail. The following matrix shows which header will be injected in the mail wheter noisy_headers are enabled or not:

MIME-Header enabled disabled
Is SPAM Not SPAM Is SPAM Not SPAM
X-Decency-Result SPAM GOOD SPAM -
X-Decency-Score Number Number Number -
X-Decency-SpamInfo All All - -

Enabling this would be the only possiblity to determine why a certain mail has been marked as SPAM. However, there might be some security concerns.

spam.notify_recipient

Enabling this will send a notification mail to the recipient of the SPAM mail. Of course, should only be used with spam.handle set to delete. Otherise: not suggested.

spam.recipient_template

If notify_recipient is enabled. Path to the notification template. Variables usable in the template are:

spam.notify_subject

If notify_recipient is enabled. Subject of the mail. Default is "SPAM notification".

spam:
    behavior: scoring
    threshold: -50
    handle: tag
    noisy_headers: 1

    # for handle: bounce or delete:
    #notify_recipient: 1
    #recipient_template: 'templates/spam-recipient-notify.tmpl'
    #recipient_subject: 'Spam detection notification'

virus

Allowed values: Hash
Default: -
Required: yes

What to do with recognized infected mails ?

virus.handle

There are four handle types:

virus.notify_sender, virus.notify_recipient

For handle types: bounce , delete or quarantine . Enables notification of the sender / recipient of the mail. It is not recommended to send mails back to the sender!

sender_template, recipient_template

Templates used for the notification mail. Variables usable are:

sender_subject, recipient_subject

Subjected used for the notifications. Defaults to "VIRUS notification"

virus:
    handle: quarantine
    #notify_sender: 1
    notify_recipient: 1
    #sender_template: 'templates/virus-sender-notify.tmpl'
    #sender_subject: 'Virus detection notification'
    recipient_template: 'templates/virus-recipient-notify.tmpl'
    recipient_subject: 'Virus detection notification'

modules

Allowed values: List of Module Hashes or List of Module configuration file paths
Default: none
Required: yes

This is the most important part, because all modules are activated here. You can either use inline configuration or include external files. See the Doorman modules section for an example.

cache, database and logging

Read here

Full YAML example

---

spool_dir: /var/spool/decency

accept_scoring: 1
doorman_verify_key: sign.pub

notification_from: 'Postmaster <postmaster@localhost>'

enable_stats: 1

include:
    - logging.yml
    - database.yml
    - cache.yml

server:
    host: 127.0.0.1
    port: 16000
    instances: 3

reinject:
    host: 127.0.0.1
    port: 10250

spam:
    behavior: scoring
    threshold: -50
    handle: tag
    noisy_headers: 1
    subject_prefix: "SPAM:"

    # for handle: bounce or delete:
    #notify_recipient: 1
    #recipient_template: 'templates/spam-recipient-notify.tmpl'
    #recipient_subject: 'Spam detection notification'

virus:
    handle: bounce

    # for handle: bounce, delete or quarantine
    notify_sender: 1
    notify_recipient: 1
    sender_template: 'templates/virus-sender-notify.tmpl'
    sender_subject: 'Virus detection notification'
    recipient_template: 'templates/virus-recipient-notify.tmpl'
    recipient_subject: 'Virus detection notification'

modules:
    #- MimeAttribs: "detective/mime-attribs.yml"
    - DKIMVerfify: "detective/dkim-verify.yml"
    # - ClamAV: detective/clamav.yml
    - Bogofilter: detective/bogofilter.yml
    #- DSPAM: detective/dspam.yml
    - CRM114: detective/crm114.yml
    - Razor: detective/razor.yml
    - HoneyCollector: detective/honey-collector.yml
    # -
    #     SpamAssassin:
    #         disable: 0
    #         default_user:
    #         weight_translate:
    #             1: -100
    #             -2: 0
    #             -3: 100
    - Archive: detective/archive.yml

Shared module configuration

Default

Can be used in any module. See here for timeout and disable.

max_size

Allowed values: Integer (size in bytes)
Default: 0
Required: no

Can be set in each Detective module. If the mail is bigger than max_size (and max_size is > 0) the mail will not filtered by the module. Most (external) filters might take the longer, the bigger the mail is.

Anti Spam modules

weight_innocent

Default: 10
Allowed values: Integer
Required: yes

The score which should be applied for innocent (good, HAM) mails. Should not be as high as the SPAM weight (in absolute terms).

weight_spam

Default: -50
Allowed values: Integer
Required: yes

The score which should be applied for SPAM mails. The higher, the more weight will this module have.

weight_translate

Default: -
Allowed values: HashRef[Int]
Required: no

Can be set instead of weight_innocent and weight_spam. Translates numeric CRM114 response to a score for Decency.

Example: CRM114 lower than -10 will be translated to -100, scores between -3 and -10 will be translated to -50 and so on ..

weight_translate:
    5: 20
    1: 10
    0: 0
    -2: 0
    -3: -50
    -10: -100

User module

Many modules support a "user" in some kind. For some this is a configuration path (eg CRM114) and for some this is a unix user account. The "user" will be determined in the following order:

default_user

Default: -
Allowed values: String (path)

See the modules docu for further information. If this is set, it will overwrite the recipient. Common scenario: one shared configuration for CRM114, one shared user for DSPAM and so on..

cmd_user

Default: -
Allowed values: String (path to command line)
Required: no

Command line for determining the user. Can be any executable shell script / prog. The program will get the "RCPT TO" in STDIN and has to print out the user. Example:

echo "recipient@domain.tld" | /usr/local/bin/determine-crm-user
# prints: /etc/crm114/local-user/

Integration in Postfix

You have to edit two files: master.cf and main.cf in /etc/postfix.

master.cf

Add the following to the end of your master.cf file:

# the Decency server itself
decency	unix  -       -       n       -       4        smtp
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
    -o smtp_send_xforward_command=yes
    -o disable_mime_output_conversion=yes
    -o smtp_destination_recipient_limit=1

# re-inject mails from Decency for delivery
127.0.0.1:10250      inet  n       -       -       -       -       smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject_unauth_destination,permit
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

main.cf

There are two possible ways you can include the Detective into postfix. The first is via the content_filter directive, the second via the check*access, eg check_client_access, directive.

content_filter

The advantage: it is easy. The disadvantage: all mails (incoming, outgoing) will be filtered. In a one-mail-server-for-all configuration this might be ugly.

# main.cf
content_filter = decency:127.0.0.1:16000

Via check_*_access

# main.cf
smtpd_client_restrictions =
    # ...
    check_client_access = pcre:/etc/postfix/decency-filter, reject
    # ...

Then in the /etc/postfix/decency-filter file (has to be readable by the postfix user):

# /path/to/access
/./ FILTER decency:127.0.0.1:16000