Obsolete: this page refers to the pre-2019 Libravatar service. See ivatar setup for the new instructions.

Here are some notes on how to setup the main libravatar.org server after you've installed Debian and the usual server packages.

If you are migrating to a new server, you may be interested in the server migration plan that has been used for the last few server migrations.

Basic setup

  • Install Debian and tweak a few things
  • Install rssh and uncomment allowrsync in /etc/rssh.conf
  • Setup appropriate mail aliases

    • Add these to /etc/aliases

      root: francois
      francois: francois@libravatar.org
      www-data: root
      libravatar-master: root
      
    • Run newaliases to update /etc/aliases.db

    • Test the aliases by running mail root
  • Ensure that the only entry for localhost in /etc/hosts is for 127.0.0.1 (if you leave the one for ::1, it will confuse gearman)

  • Copy the local logcheck rules from the old server and add the following to /etc/logcheck/logcheck.logfiles:

    /var/log/libravatar/error-cdn.log
    /var/log/libravatar/error-seccdn.log
    /var/log/libravatar/error-www.log
    /var/log/libravatar/workers.log
    /var/log/postgresql/postgresql-9.4-main.log
    
  • Create empty log files (for logcheck):

    touch /var/log/libravatar/error-{cdn,seccdn,www}.log
    chown www-data:www-data /var/log/libravatar/error-{cdn,seccdn,www}.log
    

Firewall

  • In /etc/network/iptables.up.rules, put the following:

    # Set default policies
    *filter
    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    
    # Allow unlimited outbound traffic
    -A OUTPUT -j ACCEPT
    
    # Annoying attack bots
    -A INPUT -s xxx.xxx.xxx.xxx/xx -j DROP
    
    # Accept all other traffic
    -A INPUT -j ACCEPT
    
    COMMIT
    
  • Enable the rules using iptables-apply

  • In /etc/network/ip6tables.up.rules, put the following:

    # Set default policies
    *filter
    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    
    # Allow unlimited outbound traffic
    -A OUTPUT -j ACCEPT
    
    # Allow unlimited traffic on the loopback interface
    -A INPUT -i lo -j ACCEPT
    
    # TODO: ban attackers!
    
    # Accept all other traffic
    -A INPUT -j ACCEPT
    
    COMMIT
    
  • Enable the rules using ip6tables-apply

  • Set something like this in /etc/network/interfaces:

    auto eth0
    iface eth0 inet static
        address x.x.x.x
        netmask 255.255.255.0
        gateway x.x.x.x
        pre-up iptables-restore /etc/network/iptables.up.rules
    iface eth0 inet6 static
        address x::x
        netmask 64
        gateway x::x
        pre-up ip6tables-restore /etc/network/ip6tables.up.rules
    

Database

  • Install Postgres:

    • In /etc/postgresql/*/main/pg_hba.conf, change all local connections to trust
    • In /etc/postgresql/*/main/postgresql.conf, set these:

      log_min_duration_statement =  1000
      ssl = off
      
    • Create postgres users:

      sudo -u postgres createuser -s francois
      createuser -s root
      createuser -S -R -D djangouser
      
  • Install pgbouncer

    apt install pgbouncer
    
    • Turn it on in /etc/default/pgbouncer

      START=1
      
    • Put the following in /etc/pgbouncer/pgbouncer.ini:

      [databases]
      libravatar = dbname=libravatar
      
      [pgbouncer]
      admin_users = postgres
      
    • Put the following in /etc/pgbouncer/userlist.txt:

      "djangouser" ""
      "postgres" ""
      
    • Restart the daemon:

      systemctl restart pgbouncer
      

Web server

  • Create a global TLS configuration in /etc/apache2/conf-available/tls.conf:

    SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite          ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    SSLHonorCipherOrder     on
    SSLCompression          off
    
    SSLUseStapling          on
    SSLStaplingResponderTimeout 5
    SSLStaplingReturnResponderErrors off
    SSLStaplingCache        shmcb:/var/run/ocsp(128000)
    
  • Enable TLS and mod_rewrite:

    a2enconf tls
    a2enmod ssl
    a2enmod rewrite
    
  • Setup some Apache vhosts:

    • /etc/apache2/sites-enabled/000-default.conf:

      <VirtualHost *:80>
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
      
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      
          RewriteEngine on
          RewriteCond "/var/www/acme%{REQUEST_URI}" !-f
          RewriteRule ^(.*)$ https://www.libravatar.org/ [last,redirect=301]
      </VirtualHost>
      
    • /etc/apache2/sites-available/default-ssl.conf:

      <VirtualHost *:443>
          SSLEngine on
          SSLCertificateFile /etc/libravatar/www.crt
          SSLCertificateKeyFile /etc/libravatar/www.pem
          SSLCertificateChainFile /etc/libravatar/www-chain.pem
      
          RewriteEngine On
          RewriteRule ^ https://www.libravatar.org [redirect=301,last]
      </VirtualHost>
      
    • /etc/apache2/sites-available/stats.conf:

      <VirtualHost *:443>
          ServerName stats.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/cache/awstats
          Alias /awstats-icon/ /usr/share/awstats/icon/
          Alias /favicon.ico /usr/share/libravatar/libravatar/favicon.ico
      
          SSLEngine on
          SSLCertificateFile /etc/libravatar/stats.crt
          SSLCertificateKeyFile /etc/libravatar/stats.pem
          SSLCertificateChainFile /etc/libravatar/stats-chain.pem
      
          Header add Strict-Transport-Security: "max-age=15768000"
      
          <Location />
              AuthType Basic
              AuthName "Libravatar Stats"
              AuthUserFile /etc/apache2/stats.passwd
              Require valid-user
              Order allow,deny
              allow from all
              Options +indexes
          </Location>
      </VirtualHost>
      
      <VirtualHost *:80>
          ServerName stats.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      </VirtualHost>
      
    • /etc/apache2/sites-available/acme.conf:

      <VirtualHost *:80>
          ServerName acme.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      </VirtualHost>
      
    • /etc/apache2/sites-available/apt.conf:

      <VirtualHost *:443>
              ServerName apt.libravatar.org
              ServerAdmin webmaster@libravatar.org
      
              SSLEngine On
              SSLCertificateFile /etc/libravatar/apt.crt
              SSLCertificateKeyFile /etc/libravatar/apt.pem
              SSLCertificateChainFile /etc/libravatar/apt-chain.pem
      
              DocumentRoot /var/www/apt
              <Directory /var/www/apt>
                      ForceType text/html
                      Options -MultiViews -Indexes
                      AllowOverride None
                      Order allow,deny
                      allow from all
              </Directory>
      
              ErrorLog ${APACHE_LOG_DIR}/error.log
      
              # Possible values include: debug, info, notice, warn, error, crit,
              # alert, emerg.
              LogLevel warn
      
              CustomLog ${APACHE_LOG_DIR}/access.log combined
      </VirtualHost>
      
      <VirtualHost *:80>
          ServerName apt.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      </VirtualHost>
      
  • Copy the stats password file (/etc/apache2/stats.passwd).

  • Create the empty apt repo:

    mkdir /var/www/apt
    chown francois:francois /var/www/apt
    cd /var/www/apt
    ln -s /usr/share/libravatar/libravatar/favicon.ico .
    touch index.html sitemap.xml robots.txt
    
  • Enable the new vhosts:

    mkdir /var/www/acme
    a2ensite acme
    a2ensite apt
    a2ensite default-ssl
    a2ensite stats
    
  • Install certbot and then create a few certs:

    certbot certonly --webroot -w /var/www/acme -d www.libravatar.org -d libravatar.org -d selfoss.libravatar.org -d apt.libravatar.org
    certbot certonly --webroot -w /var/www/acme -d stats.libravatar.org
    certbot certonly --webroot -w /var/www/acme -d seccdn.libravatar.org
    systemctl restart apache2
    
  • Symlink the letsencrypt certs in the right place:

    cd /etc/libravatar/
    ln -s ../letsencrypt/live/seccdn.libravatar.org/privkey.pem seccdn.pem
    ln -s ../letsencrypt/live/seccdn.libravatar.org/cert.pem seccdn.crt
    ln -s ../letsencrypt/live/seccdn.libravatar.org/chain.pem seccdn-chain.pem
    ln -s ../letsencrypt/live/stats.libravatar.org/privkey.pem stats.pem
    ln -s ../letsencrypt/live/stats.libravatar.org/cert.pem stats.crt
    ln -s ../letsencrypt/live/stats.libravatar.org/chain.pem stats-chain.pem
    ln -s ../letsencrypt/live/www.libravatar.org/privkey.pem apt.pem
    ln -s ../letsencrypt/live/www.libravatar.org/cert.pem apt.crt
    ln -s ../letsencrypt/live/www.libravatar.org/chain.pem apt-chain.pem
    ln -s ../letsencrypt/live/www.libravatar.org/privkey.pem www.pem
    ln -s ../letsencrypt/live/www.libravatar.org/cert.pem www.crt
    ln -s ../letsencrypt/live/www.libravatar.org/chain.pem www-chain.pem
    chmod a+rx /etc/letsencrypt/archive /etc/letsencrypt/live
    
  • Install a cronjob to automatically renew these certs in /etc/cron.daily/certbot-renew-libravatar:

    #!/bin/bash
    
    /usr/bin/certbot renew --quiet --post-hook "/bin/systemctl restart apache2.service"
    
    pushd /etc/ > /dev/null
    /usr/bin/git add letsencrypt
    DIFFSTAT="$(/usr/bin/git diff --cached --stat)"
    if [ -n "$DIFFSTAT" ] ; then
        /usr/bin/git commit --quiet -m "Renewed letsencrypt certs"
        echo "$DIFFSTAT"
    fi
    popd > /dev/null
    
  • Install awstats and add this to /etc/awstats/awstats.conf.local:

    SiteDomain="libravatar.org"
    LogType=W
    LogFormat=1
    
  • Reduce log retention to 10 days by setting the following in /etc/logrotate.d/apache2:

    rotate 10
    

Backups

  • Add this script to /usr/local/sbin/libravatar_backups:

    #!/bin/sh
    
    DUMP_DIR="/var/backups/libravatar"
    
    # Perform fresh backup
    DUMP_FILE="$DUMP_DIR/`date +%Y%m%dT%H%M%S`.pg"
    pg_dump -Fc libravatar > $DUMP_FILE
    chmod 600 $DUMP_FILE
    
    # Encrypt backup
    gpg --recipient 007c98d1 --encrypt $DUMP_FILE
    rm $DUMP_FILE
    
    # Purge old backups
    find $DUMP_DIR -ctime +7 -delete
    
  • Create an empty /var/backups/libravatar directory

  • Install duplicity:

    apt install duplicity python-paramiko
    
  • Add my duplicity backup script to /home/francois/.backup/ and update:

    • GPG passhrase
  • Add the following to /home/francois/.backup/exclude:

    /etc/.git
    /home/francois/.cache
    /home/francois/git-mirrors
    /root/.cache
    
  • Add the following to /home/francois/.backup/include:

    /etc
    /home/francois
    /usr/local/bin
    /usr/local/sbin
    /root
    /var/backups/libravatar
    /var/lib/libravatar
    /var/log/libravatar
    
  • Create this cronjob in /etc/cron.d/libravatar_backups:

    # Local DB backups
    4 1,13 * * *       root    ionice -c3 nice -n10 /usr/local/sbin/libravatar_backups
    
    # Full backups to another server
    6 14 * * *      root    /home/francois/.backup/backup-selfoss
    6 23 * * 3      root    /home/francois/.backup/backup-selfoss --full
    
  • Install the appropriate gpg key as root:

    gpg --keyserver pgp.mit.edu --recv-key 16281F2E007C98D1
    
  • Mark the key as ultimately trusted by using the trust command after typing:

    gpg --edit-key 16281F2E007C98D1
    
  • Copy the required id_rsa key into /root/.ssh/.

  • Configure the port, hostname and username in /root/.ssh/config.

  • Run an initial backup:

    sudo /home/francois/.backup/backup-selfoss
    
  • Ensure the expected files have been backed up:

    sudo /home/francois/.backup/backup-selfoss --list-current-files > list.txt
    

Install application

  • Install Libravatar packages

    • Start by installing the apt-transport-https package
    • Add the Libravatar repository to /etc/apt/sources.list:

      deb https://apt.libravatar.org/ jessie main
      
    • Then install server packages:

      apt update
      apt install libravatar{,-common,-cdn,-cdn-common,-deployment,-www,-master}
      
    • Use the defaults for all of the debconf questions except:

      • 6432 for the database port number
      • a string from pwgen -s 50 for the Django secret
  • Install gearman-job-server:

    apt install --no-install-recommends gearman-job-server
    
  • Add your username to the libravatar-deployment group:

    adduser francois libravatar-deployment
    

Squid proxy

  • Install squid:

    apt install squid3
    
  • Restrict access and outgoing requests by putting the following in /etc/squid3/squid.conf:

    acl to_localnet dst 0.0.0.1-0.255.255.255
    acl to_localnet dst 10.0.0.0/8
    acl to_localnet dst 100.64.0.0/10
    acl to_localnet dst 169.254.0.0/16
    acl to_localnet dst 172.16.0.0/12
    acl to_localnet dst 192.168.0.0/16
    acl to_localnet dst fc00::/7
    acl to_localnet dst fe80::/10
    acl SSL_ports port 443
    acl Safe_ports port 80
    acl Safe_ports port 443
    acl CONNECT method CONNECT
    http_access deny !Safe_ports
    http_access deny CONNECT !SSL_ports
    http_access deny manager
    http_access deny to_localhost
    http_access deny to_localnet
    http_access allow localhost
    http_access deny all
    http_port 127.0.0.1:3128
    
  • Force all outgoing connections from Django to go through the proxy by putting the following near the end of /etc/libravatar/django.wsgi:

    os.environ['ftp_proxy'] = "http://127.0.0.1:3128"
    os.environ['http_proxy'] = "http://127.0.0.1:3128"
    os.environ['https_proxy'] = "http://127.0.0.1:3128"
    

Git mirrors

  • Install hg:

    apt install mercurial
    
  • Enable the convert mercurial extension by putting this in /etc/mercurial/hgrc:

    [extensions]
    convert =
    
  • Add this script in /usr/local/bin/git-mirror:

    #!/bin/bash
    
    DIR=/home/francois/git-mirrors
    cd $DIR/git
    git fetch --quiet origin 2> /dev/null
    git reset --quiet --hard origin/master
    git push --quiet github master 2> /dev/null
    exit 0
    
  • Add this script in /usr/local/bin/hg-mirror:

    #!/bin/bash
    DIR=/home/francois/git-mirrors
    cd $DIR/git
    git fetch --quiet origin 2> /dev/null
    git reset --quiet --hard origin/master
    cd $DIR
    hg --quiet convert git libravatar-hg
    cd $DIR/libravatar-hg
    hg --quiet update
    hg --quiet push ssh://bitbucket.org/libravatar/libravatar 2> /dev/null
    exit 0
    
  • Add this to /home/francois/.ssh/config:

    Host bitbucket.org
      User hg
      IdentityFile ~/.ssh/repository_mirrors
    Host github.com
      User git
      IdentityFile ~/.ssh/repository_mirrors
      AddressFamily inet
    
  • Copy repository_mirrors ssh key from old server into /home/francois/.ssh/ (make sure that the private key has 600 permissions)

  • Prepare directory for git and mercurial repositories:

    mkdir -p /home/francois/git-mirrors/libravatar-hg
    cd /home/francois/git-mirrors/
    git clone https://git.launchpad.net/~libravatar/libravatar git
    cd git
    git remote add github git@github.com:libravatar/libravatar.git
    
  • Do a test run for each of them (as francois):

    /usr/local/bin/git-mirror && echo success
    /usr/local/bin/hg-mirror && echo success
    
  • Create the following cron jobs in /etc/cron.d/libravatar_git-mirrors:

    50 * * * *      francois        /usr/local/bin/git-mirror
    55 * * * *      francois        /usr/local/bin/hg-mirror
    

From the old server

If migrating from one server to another:

  • Setup the database

    • Get a DB dump from the old server:

      pg_dump -Fc libravatar > libravatar.pg
      
    • Create a new database:

      createdb -O djangouser -E utf8 libravatar
      
    • Restore the database:

      pg_restore -d libravatar < libravatar.pg
      
  • Copy the contents of /var/lib/libravatar/ from the old server and fix the file permissions:

    chown -R root:root /var/lib/libravatar/avatar/* /var/lib/libravatar/user/*