Obsolete: this page refers to the pre-2019 Libravatar service. See ivatar setup for the new instructions.

Here are some notes on how to setup the main libravatar.org server after you've installed Debian and the usual server packages.

If you are migrating to a new server, you may be interested in the server migration plan that has been used for the last few server migrations.

Basic setup

Install Debian and tweak a few things
Install rssh and uncomment allowrsync in /etc/rssh.conf
Setup appropriate mail aliases
- Add these to /etc/aliases
```
root: francois
francois: francois@libravatar.org
www-data: root
libravatar-master: root
```
- Run newaliases to update /etc/aliases.db
- Test the aliases by running mail root
Ensure that the only entry for localhost in /etc/hosts is for 127.0.0.1 (if you leave the one for ::1, it will confuse gearman)

Copy the local logcheck rules from the old server and add the following to /etc/logcheck/logcheck.logfiles:

/var/log/libravatar/error-cdn.log
/var/log/libravatar/error-seccdn.log
/var/log/libravatar/error-www.log
/var/log/libravatar/workers.log
/var/log/postgresql/postgresql-9.4-main.log

Create empty log files (for logcheck):

touch /var/log/libravatar/error-{cdn,seccdn,www}.log
chown www-data:www-data /var/log/libravatar/error-{cdn,seccdn,www}.log

Firewall

In /etc/network/iptables.up.rules, put the following:

# Set default policies
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

# Allow unlimited outbound traffic
-A OUTPUT -j ACCEPT

# Annoying attack bots
-A INPUT -s xxx.xxx.xxx.xxx/xx -j DROP

# Accept all other traffic
-A INPUT -j ACCEPT

COMMIT

Enable the rules using iptables-apply

In /etc/network/ip6tables.up.rules, put the following:

# Set default policies
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

# Allow unlimited outbound traffic
-A OUTPUT -j ACCEPT

# Allow unlimited traffic on the loopback interface
-A INPUT -i lo -j ACCEPT

# TODO: ban attackers!

# Accept all other traffic
-A INPUT -j ACCEPT

COMMIT

Enable the rules using ip6tables-apply

Set something like this in /etc/network/interfaces:

auto eth0
iface eth0 inet static
    address x.x.x.x
    netmask 255.255.255.0
    gateway x.x.x.x
    pre-up iptables-restore /etc/network/iptables.up.rules
iface eth0 inet6 static
    address x::x
    netmask 64
    gateway x::x
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

Database

Install Postgres:
- In /etc/postgresql/*/main/pg_hba.conf, change all local connections to trust
- In /etc/postgresql/*/main/postgresql.conf, set these:
```
log_min_duration_statement =  1000
ssl = off
```
- Create postgres users:
```
sudo -u postgres createuser -s francois
createuser -s root
createuser -S -R -D djangouser
```

Install pgbouncer

apt install pgbouncer

Turn it on in /etc/default/pgbouncer
```
START=1
```

Put the following in /etc/pgbouncer/pgbouncer.ini:

[databases]
libravatar = dbname=libravatar

[pgbouncer]
admin_users = postgres

Put the following in /etc/pgbouncer/userlist.txt:
```
"djangouser" ""
"postgres" ""
```
Restart the daemon:
```
systemctl restart pgbouncer
```

Web server

Create a global TLS configuration in /etc/apache2/conf-available/tls.conf:

SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SSLHonorCipherOrder     on
SSLCompression          off

SSLUseStapling          on
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
SSLStaplingCache        shmcb:/var/run/ocsp(128000)

Enable TLS and mod_rewrite:

a2enconf tls
a2enmod ssl
a2enmod rewrite

Setup some Apache vhosts:

/etc/apache2/sites-enabled/000-default.conf:

<VirtualHost *:80>
    ServerAdmin webmaster@libravatar.org
    DocumentRoot /var/www/acme

    <Directory /var/www/acme>
        Options -Indexes
    </Directory>

    RewriteEngine on
    RewriteCond "/var/www/acme%{REQUEST_URI}" !-f
    RewriteRule ^(.*)$ https://www.libravatar.org/ [last,redirect=301]
</VirtualHost>

/etc/apache2/sites-available/default-ssl.conf:

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/libravatar/www.crt
    SSLCertificateKeyFile /etc/libravatar/www.pem
    SSLCertificateChainFile /etc/libravatar/www-chain.pem

    RewriteEngine On
    RewriteRule ^ https://www.libravatar.org [redirect=301,last]
</VirtualHost>

/etc/apache2/sites-available/stats.conf:

<VirtualHost *:443>
    ServerName stats.libravatar.org
    ServerAdmin webmaster@libravatar.org
    DocumentRoot /var/cache/awstats
    Alias /awstats-icon/ /usr/share/awstats/icon/
    Alias /favicon.ico /usr/share/libravatar/libravatar/favicon.ico

    SSLEngine on
    SSLCertificateFile /etc/libravatar/stats.crt
    SSLCertificateKeyFile /etc/libravatar/stats.pem
    SSLCertificateChainFile /etc/libravatar/stats-chain.pem

    Header add Strict-Transport-Security: "max-age=15768000"

    <Location />
        AuthType Basic
        AuthName "Libravatar Stats"
        AuthUserFile /etc/apache2/stats.passwd
        Require valid-user
        Order allow,deny
        allow from all
        Options +indexes
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ServerName stats.libravatar.org
    ServerAdmin webmaster@libravatar.org
    DocumentRoot /var/www/acme
    <Directory /var/www/acme>
        Options -Indexes
    </Directory>
</VirtualHost>

/etc/apache2/sites-available/acme.conf:

<VirtualHost *:80>
    ServerName acme.libravatar.org
    ServerAdmin webmaster@libravatar.org
    DocumentRoot /var/www/acme
    <Directory /var/www/acme>
        Options -Indexes
    </Directory>
</VirtualHost>

/etc/apache2/sites-available/apt.conf:

<VirtualHost *:443>
        ServerName apt.libravatar.org
        ServerAdmin webmaster@libravatar.org

        SSLEngine On
        SSLCertificateFile /etc/libravatar/apt.crt
        SSLCertificateKeyFile /etc/libravatar/apt.pem
        SSLCertificateChainFile /etc/libravatar/apt-chain.pem

        DocumentRoot /var/www/apt
        <Directory /var/www/apt>
                ForceType text/html
                Options -MultiViews -Indexes
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

<VirtualHost *:80>
    ServerName apt.libravatar.org
    ServerAdmin webmaster@libravatar.org
    DocumentRoot /var/www/acme
    <Directory /var/www/acme>
        Options -Indexes
    </Directory>
</VirtualHost>

Copy the stats password file (/etc/apache2/stats.passwd).

Create the empty apt repo:

mkdir /var/www/apt
chown francois:francois /var/www/apt
cd /var/www/apt
ln -s /usr/share/libravatar/libravatar/favicon.ico .
touch index.html sitemap.xml robots.txt

Enable the new vhosts:

mkdir /var/www/acme
a2ensite acme
a2ensite apt
a2ensite default-ssl
a2ensite stats

Install certbot and then create a few certs:

certbot certonly --webroot -w /var/www/acme -d www.libravatar.org -d libravatar.org -d selfoss.libravatar.org -d apt.libravatar.org
certbot certonly --webroot -w /var/www/acme -d stats.libravatar.org
certbot certonly --webroot -w /var/www/acme -d seccdn.libravatar.org
systemctl restart apache2

Symlink the letsencrypt certs in the right place:

cd /etc/libravatar/
ln -s ../letsencrypt/live/seccdn.libravatar.org/privkey.pem seccdn.pem
ln -s ../letsencrypt/live/seccdn.libravatar.org/cert.pem seccdn.crt
ln -s ../letsencrypt/live/seccdn.libravatar.org/chain.pem seccdn-chain.pem
ln -s ../letsencrypt/live/stats.libravatar.org/privkey.pem stats.pem
ln -s ../letsencrypt/live/stats.libravatar.org/cert.pem stats.crt
ln -s ../letsencrypt/live/stats.libravatar.org/chain.pem stats-chain.pem
ln -s ../letsencrypt/live/www.libravatar.org/privkey.pem apt.pem
ln -s ../letsencrypt/live/www.libravatar.org/cert.pem apt.crt
ln -s ../letsencrypt/live/www.libravatar.org/chain.pem apt-chain.pem
ln -s ../letsencrypt/live/www.libravatar.org/privkey.pem www.pem
ln -s ../letsencrypt/live/www.libravatar.org/cert.pem www.crt
ln -s ../letsencrypt/live/www.libravatar.org/chain.pem www-chain.pem
chmod a+rx /etc/letsencrypt/archive /etc/letsencrypt/live

Install a cronjob to automatically renew these certs in /etc/cron.daily/certbot-renew-libravatar:

#!/bin/bash

/usr/bin/certbot renew --quiet --post-hook "/bin/systemctl restart apache2.service"

pushd /etc/ > /dev/null
/usr/bin/git add letsencrypt
DIFFSTAT="$(/usr/bin/git diff --cached --stat)"
if [ -n "$DIFFSTAT" ] ; then
    /usr/bin/git commit --quiet -m "Renewed letsencrypt certs"
    echo "$DIFFSTAT"
fi
popd > /dev/null

Install awstats and add this to /etc/awstats/awstats.conf.local:
```
SiteDomain="libravatar.org"
LogType=W
LogFormat=1
```
Reduce log retention to 10 days by setting the following in /etc/logrotate.d/apache2:
```
rotate 10
```

Backups

Add this script to /usr/local/sbin/libravatar_backups:

#!/bin/sh

DUMP_DIR="/var/backups/libravatar"

# Perform fresh backup
DUMP_FILE="$DUMP_DIR/`date +%Y%m%dT%H%M%S`.pg"
pg_dump -Fc libravatar > $DUMP_FILE
chmod 600 $DUMP_FILE

# Encrypt backup
gpg --recipient 007c98d1 --encrypt $DUMP_FILE
rm $DUMP_FILE

# Purge old backups
find $DUMP_DIR -ctime +7 -delete

Create an empty /var/backups/libravatar directory
Install duplicity:
```
apt install duplicity python-paramiko
```
Add my duplicity backup script to /home/francois/.backup/ and update:
- GPG passhrase

Add the following to /home/francois/.backup/exclude:

/etc/.git
/home/francois/.cache
/home/francois/git-mirrors
/root/.cache

Add the following to /home/francois/.backup/include:

/etc
/home/francois
/usr/local/bin
/usr/local/sbin
/root
/var/backups/libravatar
/var/lib/libravatar
/var/log/libravatar

Create this cronjob in /etc/cron.d/libravatar_backups:

# Local DB backups
4 1,13 * * *       root    ionice -c3 nice -n10 /usr/local/sbin/libravatar_backups

# Full backups to another server
6 14 * * *      root    /home/francois/.backup/backup-selfoss
6 23 * * 3      root    /home/francois/.backup/backup-selfoss --full

Install the appropriate gpg key as root:

gpg --keyserver pgp.mit.edu --recv-key 16281F2E007C98D1

Mark the key as ultimately trusted by using the trust command after typing:
```
gpg --edit-key 16281F2E007C98D1
```
Copy the required id_rsa key into /root/.ssh/.
Configure the port, hostname and username in /root/.ssh/config.

Run an initial backup:

sudo /home/francois/.backup/backup-selfoss

Ensure the expected files have been backed up:

sudo /home/francois/.backup/backup-selfoss --list-current-files > list.txt

Install application

Install Libravatar packages
- Start by installing the apt-transport-https package
- Add the Libravatar repository to /etc/apt/sources.list:
```
deb https://apt.libravatar.org/ jessie main
```
- Then install server packages:
```
apt update
apt install libravatar{,-common,-cdn,-cdn-common,-deployment,-www,-master}
```
- Use the defaults for all of the debconf questions except:
  - 6432 for the database port number
  - a string from pwgen -s 50 for the Django secret

Install gearman-job-server:

apt install --no-install-recommends gearman-job-server

Add your username to the libravatar-deployment group:
```
adduser francois libravatar-deployment
```

Squid proxy

Install squid:
```
apt install squid3
```

Restrict access and outgoing requests by putting the following in /etc/squid3/squid.conf:

acl to_localnet dst 0.0.0.1-0.255.255.255
acl to_localnet dst 10.0.0.0/8
acl to_localnet dst 100.64.0.0/10
acl to_localnet dst 169.254.0.0/16
acl to_localnet dst 172.16.0.0/12
acl to_localnet dst 192.168.0.0/16
acl to_localnet dst fc00::/7
acl to_localnet dst fe80::/10
acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 443
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny manager
http_access deny to_localhost
http_access deny to_localnet
http_access allow localhost
http_access deny all
http_port 127.0.0.1:3128

Force all outgoing connections from Django to go through the proxy by putting the following near the end of /etc/libravatar/django.wsgi:

os.environ['ftp_proxy'] = "http://127.0.0.1:3128"
os.environ['http_proxy'] = "http://127.0.0.1:3128"
os.environ['https_proxy'] = "http://127.0.0.1:3128"

Git mirrors

Install hg:
```
apt install mercurial
```
Enable the convert mercurial extension by putting this in /etc/mercurial/hgrc:
```
[extensions]
convert =
```

Add this script in /usr/local/bin/git-mirror:

#!/bin/bash

DIR=/home/francois/git-mirrors
cd $DIR/git
git fetch --quiet origin 2> /dev/null
git reset --quiet --hard origin/master
git push --quiet github master 2> /dev/null
exit 0

Add this script in /usr/local/bin/hg-mirror:

#!/bin/bash
DIR=/home/francois/git-mirrors
cd $DIR/git
git fetch --quiet origin 2> /dev/null
git reset --quiet --hard origin/master
cd $DIR
hg --quiet convert git libravatar-hg
cd $DIR/libravatar-hg
hg --quiet update
hg --quiet push ssh://bitbucket.org/libravatar/libravatar 2> /dev/null
exit 0

Add this to /home/francois/.ssh/config:

Host bitbucket.org
  User hg
  IdentityFile ~/.ssh/repository_mirrors
Host github.com
  User git
  IdentityFile ~/.ssh/repository_mirrors
  AddressFamily inet

Copy repository_mirrors ssh key from old server into /home/francois/.ssh/ (make sure that the private key has 600 permissions)

Prepare directory for git and mercurial repositories:

mkdir -p /home/francois/git-mirrors/libravatar-hg
cd /home/francois/git-mirrors/
git clone https://git.launchpad.net/~libravatar/libravatar git
cd git
git remote add github git@github.com:libravatar/libravatar.git

Do a test run for each of them (as francois):

/usr/local/bin/git-mirror && echo success
/usr/local/bin/hg-mirror && echo success

Create the following cron jobs in /etc/cron.d/libravatar_git-mirrors:

50 * * * *      francois        /usr/local/bin/git-mirror
55 * * * *      francois        /usr/local/bin/hg-mirror

From the old server

If migrating from one server to another:

Setup the database

Get a DB dump from the old server:
```
pg_dump -Fc libravatar > libravatar.pg
```

Create a new database:

createdb -O djangouser -E utf8 libravatar

Restore the database:

pg_restore -d libravatar < libravatar.pg

Copy the contents of /var/lib/libravatar/ from the old server and fix the file permissions:
```
chown -R root:root /var/lib/libravatar/avatar/* /var/lib/libravatar/user/*
```