Cache & Database

Table of content

Databases
Caches

Decency requires a database and a cache. The database is used for storing and querying persistent data (eg: greylisted sender->recipient pairs, custom whitelists, custom blacklists, honeypot blacklists and so on..) and caches to increase the performance by keeping results temporary (eg: custom blacklist results, spf results and so on). Within the "warm-up phase" (when the caches are empty) the load on the databases will be much bigger than in a running system. Some caches are persistent over restart (eg Memcached), if you use one of those your server will not suffer from this.

Databases

There is a huge range of databases supported. In the DBD section there are a lot of relational databases (RDBMS) listed, then there is MongoDB..

RDBs

This is only an example of supported database. You have to install the driver yourself. The debian decency-perl-modules package ships with SQLite and MySQL support, anything else is up to you.

SQLite

From the sqlite.org website: "SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine". SQLite is very easy to setup and runs nearly everywhere. However, it is not very fast nor network capable.

database:
    type: DBD
    args:
        - 'dbi:SQLite:dbname=/var/spool/database/decency.db'

MySQL

.. is a very widespread, easy to setup (hard to tune ) relational database. It can be used in a LAN environment.

database:
    type: DBD
    args:
        - 'dbi:mysql:host=localhost;database=decency;port=1234'
        - 'username'
        - 'password'

PostgreSQL

PostgresSQL is a very powerful open source relational database. Widely seen as the open source oracle. Harder to configure than mysql, but also more sophisticated.

database:
    type: DBD
    args:
        - 'dbi:Pg:host=localhost;dbname=decency'
        - 'username'
        - 'password'
        # the following might come in handy, if you need:
        -
            pg_enable_utf8: 1
            name_sep: '.'
            quota_char: '"'
            on_connect_do:
                - 'SET search_path=my_schema'
                - 'SET client_encoding=UTF8'

And more ..

There are a couple more usable databases ..

IBM Solid DB
Oracle Database
for a complete list read here

Whatever of those you prefer, keep in mind: read, read, read performance is what matters.

MongoDB

Then there is MongoDB, which is highly recommended (above any above). It is fast as lightning and very - as in very, very - scalable. Redundancy is no problem if you setup multiple instances. The (memory|cpu) footprint can be kept reasonable small, but you should setup a dedicated machine for it.

One of those new, "fancy" but still beta applications, you conclude - but wait! Before you stop considering MongoDB (or Decency for that matter) have a look at this list of MongoDB users (yes, sourceforge is among them!). Some of them switched from classy symbols of stability as Oracle to MongoDBm, so i figure it's mature enough.

And last but not least: installation is dead easy, too. For most distris there is a binary package, check out for debian and ubuntu, centos and fedora and the rest and here. Install, edit config, start, done.

database:
    type: MongoDB
    server: 'localhost'
    port: 27017
    database: 'decency'

    # or if you have a "left" and "right":
    #server: '123.123.123.123,234.234.234.234'
    #port: '27017,27018'

    # if authentication is required:
    #username: 'yourname'
    #password: 'yourpassword'

LDAP

I've tested only OpenLDAP, but any LDAP speaking server should do.

OpenLDAP

OpenLDAP is an open source implementation of the Lightweight Directory Access Protocol. It is easy to setup and very fast (for an LDAP server). Performance compared to the RDBs (MySQL, PostgreSQL, ..) or MongoDB is rather slow, especially writing, but still sufficient for a mail server handling less than, let's say hundred mails per minute. Benchmarks implied, that it slows down Decency to about half it's possible speed overall (compared against MySQL), in database intensive modules (eg Greylist) it can be about 10 times slower or more. However, it is very easy to replicate, uses small ressources, works well accross multiple defcency instances in the network and easy to setup on single nodes.

database:
    type: LDAP
    host: 127.0.0.1:389
    user: cn=admin,dc=nodomain
    base: dc=nodomain
    password: *******
    scheme: ldap

What's the best database ?

Huh, cannot say. Depends on your environment. If you whole infrastructure relies on PostgreSQL, you probably go best with it. If you require lot's of speed, plan to scale large (or already are) and/or don't want to hassle with tuning your RDBMS then you should give MongoDB a shot (read above). If redundancy via replication and/or small ressource usage is more important than speed, go with OpenLDAP. If you just want to try it out use SQLite, because it is very easy and runs nearly everywhere.

I Could not find Berkeley Database nor CDB in the list.. did i miss it ?

No, you didn't. Berkeley DBs / CDBs are really nice and very fast. However, within perl you mostly deal with forked processes (not threads, but "kind of"). Sharing Berkeley DBs among those in write mode in a save matter is possible, but never satisfying. It took me nearly two days of reading, trial and error and stumbling in the dark to conclude that it probably might or might not work out. And finally, using above RDBs, OpenLDAP and MongoDB allows to store "complex" data, whereas Berkeley goes with plain "key = value" (i already wrote an abstraction, but it was a patchwork). Even if i could figure out how to share it between all writers, it still would not be usable in a network shared between multiple instances - so why the effort? In the end the huge speed of MongoDB made the difference and dropped Berkeley DBs for good.

Caches

Decency supports any Cache from CPAN using the Cache-API (Cache::*). Following i will introduce the four most interesting.

Memcached and Memcached::XS

Speed:fast
Persistent: yes
Network: yes
Use Case: multiple servers

Memcached from Danga Interactive is a very popular and fast network cache server. Probably the way to go, if you have multiple mail servers and/or Decency servers. The ::XS variant is the faster choice, but harder to install (if you don't use the debian package). More info..

cache:
    class: Memcached
    servers:
        - '127.0.0.1:11211'
        - '10.0.20.100:11211'

FastMmap

Speed:fast
Persistent: yes
Network: no
Use Case: single server

Uses a so called mmaped file. Good choice for a single server (also multiple Decency instances on the same machine). More info..

cache:
    class: FastMmap
    share_file: /tmp/decency.mmap
    expire_time: 7d
    cache_size: 128m

See the new-method description on CPAN for more parameters.

File

Speed: slow
Persistent: yes
Network: no
Use Case: testing

As the name implies, it is a cache base on the file system. You should not use it. It is only good for testing. More info ..

cache:
    class: File
    cache_root: /tmp/decency-cache
    default_expires: '86400 sec'

Memory

Speed: very fast
Persistent: no
Network: no
Use Case: testing or stand alone + huge memory

You guessed right, puts everything in the memory. Very fast, very memory extensive and wiped on server shutdown. More info ..

cache:
    class: Memory
    default_expires: '86400 sec'

And more ..

There are more, look for everything named "Cache::XYZ" here, but i think i've covered the most important and fitting.

Decency

The spam fighting framework

Cache & Database

Table of content

Databases

RDBs

SQLite

MySQL

PostgreSQL

And more ..

MongoDB

LDAP

OpenLDAP

What's the best database ?

I Could not find Berkeley Database nor CDB in the list.. did i miss it ?

Caches

Memcached and Memcached::XS

FastMmap

File

Memory

And more ..