Here are some notes on how to setup the main libravatar.org server after you've installed Debian and the usual server packages.

If you are migrating to a new server, you may be interested in the server migration plan that has been used for the last few server migrations.

Basic setup

  • Install Debian and tweak a few things
  • Install rssh and uncomment allowrsync in /etc/rssh.conf
  • Setup appropriate mail aliases

    • Add these to /etc/aliases

      root: francois
      francois: francois@libravatar.org
      www-data: root
      libravatar-master: root
      
    • Run newaliases to update /etc/aliases.db

    • Test the aliases by running mail root
  • Ensure that the only entry for localhost in /etc/hosts is for 127.0.0.1 (if you leave the one for ::1, it will confuse gearman)

  • Copy the local logcheck rules from the old server and add the following to /etc/logcheck/logcheck.logfiles:

    /var/log/libravatar/error-cdn.log
    /var/log/libravatar/error-seccdn.log
    /var/log/libravatar/error-www.log
    /var/log/libravatar/workers.log
    /var/log/postgresql/postgresql-9.4-main.log
    
  • Create empty log files (for logcheck):

    touch /var/log/libravatar/error-{cdn,seccdn,www}.log
    chown www-data:www-data /var/log/libravatar/error-{cdn,seccdn,www}.log
    

Firewall

  • In /etc/network/iptables.up.rules, put the following:

    # Set default policies
    *filter
    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    
    # Allow unlimited outbound traffic
    -A OUTPUT -j ACCEPT
    
    # Annoying attack bots
    -A INPUT -s xxx.xxx.xxx.xxx/xx -j DROP
    
    # Accept all other traffic
    -A INPUT -j ACCEPT
    
    COMMIT
    
  • Enable the rules using iptables-apply

  • In /etc/network/ip6tables.up.rules, put the following:

    # Set default policies
    *filter
    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    
    # Allow unlimited outbound traffic
    -A OUTPUT -j ACCEPT
    
    # Allow unlimited traffic on the loopback interface
    -A INPUT -i lo -j ACCEPT
    
    # TODO: ban attackers!
    
    # Accept all other traffic
    -A INPUT -j ACCEPT
    
    COMMIT
    
  • Enable the rules using ip6tables-apply

  • Set something like this in /etc/network/interfaces:

    auto eth0
    iface eth0 inet static
        address x.x.x.x
        netmask 255.255.255.0
        gateway x.x.x.x
        pre-up iptables-restore /etc/network/iptables.up.rules
    iface eth0 inet6 static
        address x::x
        netmask 64
        gateway x::x
        pre-up ip6tables-restore /etc/network/ip6tables.up.rules
    

Database

  • Install Postgres:

    • In /etc/postgresql/*/main/pg_hba.conf, change all local connections to trust
    • In /etc/postgresql/*/main/postgresql.conf, set these:

      log_min_duration_statement =  1000
      ssl = off
      
    • Create postgres users:

      sudo -u postgres createuser -s francois
      createuser -s root
      createuser -S -R -D djangouser
      
  • Install pgbouncer

    apt install pgbouncer
    
    • Turn it on in /etc/default/pgbouncer

      START=1
      
    • Put the following in /etc/pgbouncer/pgbouncer.ini:

      [databases]
      libravatar = dbname=libravatar
      
      [pgbouncer]
      admin_users = postgres
      
    • Put the following in /etc/pgbouncer/userlist.txt:

      "djangouser" ""
      "postgres" ""
      
    • Restart the daemon:

      systemctl restart pgbouncer
      

Web server

  • Create a global TLS configuration in /etc/apache2/conf-available/tls.conf:

    SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite          ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    SSLHonorCipherOrder     on
    SSLCompression          off
    
    SSLUseStapling          on
    SSLStaplingResponderTimeout 5
    SSLStaplingReturnResponderErrors off
    SSLStaplingCache        shmcb:/var/run/ocsp(128000)
    
  • Enable TLS and mod_rewrite:

    a2enconf tls
    a2enmod ssl
    a2enmod rewrite
    
  • Setup some Apache vhosts:

    • /etc/apache2/sites-enabled/000-default.conf:

      <VirtualHost *:80>
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
      
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      
          RewriteEngine on
          RewriteCond "/var/www/acme%{REQUEST_URI}" !-f
          RewriteRule ^(.*)$ https://www.libravatar.org/ [last,redirect=301]
      </VirtualHost>
      
    • /etc/apache2/sites-available/default-ssl.conf:

      <VirtualHost *:443>
          SSLEngine on
          SSLCertificateFile /etc/libravatar/www.crt
          SSLCertificateKeyFile /etc/libravatar/www.pem
          SSLCertificateChainFile /etc/libravatar/www-chain.pem
      
          RewriteEngine On
          RewriteRule ^ https://www.libravatar.org [redirect=301,last]
      </VirtualHost>
      
    • /etc/apache2/sites-available/stats.conf:

      <VirtualHost *:443>
          ServerName stats.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/cache/awstats
          Alias /awstats-icon/ /usr/share/awstats/icon/
          Alias /favicon.ico /usr/share/libravatar/libravatar/favicon.ico
      
          SSLEngine on
          SSLCertificateFile /etc/libravatar/stats.crt
          SSLCertificateKeyFile /etc/libravatar/stats.pem
          SSLCertificateChainFile /etc/libravatar/stats-chain.pem
      
          Header add Strict-Transport-Security: "max-age=15768000"
      
          <Location />
              AuthType Basic
              AuthName "Libravatar Stats"
              AuthUserFile /etc/apache2/stats.passwd
              Require valid-user
              Order allow,deny
              allow from all
              Options +indexes
          </Location>
      </VirtualHost>
      
      <VirtualHost *:80>
          ServerName stats.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      </VirtualHost>
      
    • /etc/apache2/sites-available/acme.conf:

      <VirtualHost *:80>
          ServerName acme.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      </VirtualHost>
      
    • /etc/apache2/sites-available/apt.conf:

      <VirtualHost *:443>
              ServerName apt.libravatar.org
              ServerAdmin webmaster@libravatar.org
      
              SSLEngine On
              SSLCertificateFile /etc/libravatar/apt.crt
              SSLCertificateKeyFile /etc/libravatar/apt.pem
              SSLCertificateChainFile /etc/libravatar/apt-chain.pem
      
              DocumentRoot /var/www/apt
              <Directory /var/www/apt>
                      ForceType text/html
                      Options -MultiViews -Indexes
                      AllowOverride None
                      Order allow,deny
                      allow from all
              </Directory>
      
              ErrorLog ${APACHE_LOG_DIR}/error.log
      
              # Possible values include: debug, info, notice, warn, error, crit,
              # alert, emerg.
              LogLevel warn
      
              CustomLog ${APACHE_LOG_DIR}/access.log combined
      </VirtualHost>
      
      <VirtualHost *:80>
          ServerName apt.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/www/acme
          <Directory /var/www/acme>
              Options -Indexes
          </Directory>
      </VirtualHost>
      
  • Copy the stats password file (/etc/apache2/stats.passwd).

  • Create the empty apt repo:

    mkdir /var/www/apt
    chown francois:francois /var/www/apt
    cd /var/www/apt
    ln -s /usr/share/libravatar/libravatar/favicon.ico .
    touch index.html sitemap.xml robots.txt
    
  • Enable the new vhosts:

    mkdir /var/www/acme
    a2ensite acme
    a2ensite apt
    a2ensite default-ssl
    a2ensite stats
    
  • Install certbot and then create a few certs:

    certbot certonly --webroot -w /var/www/acme -d www.libravatar.org -d libravatar.org -d selfoss.libravatar.org -d apt.libravatar.org
    certbot certonly --webroot -w /var/www/acme -d stats.libravatar.org
    certbot certonly --webroot -w /var/www/acme -d seccdn.libravatar.org
    systemctl restart apache2
    
  • Symlink the letsencrypt certs in the right place:

    cd /etc/libravatar/
    ln -s ../letsencrypt/live/seccdn.libravatar.org/privkey.pem seccdn.pem
    ln -s ../letsencrypt/live/seccdn.libravatar.org/cert.pem seccdn.crt
    ln -s ../letsencrypt/live/seccdn.libravatar.org/chain.pem seccdn-chain.pem
    ln -s ../letsencrypt/live/stats.libravatar.org/privkey.pem stats.pem
    ln -s ../letsencrypt/live/stats.libravatar.org/cert.pem stats.crt
    ln -s ../letsencrypt/live/stats.libravatar.org/chain.pem stats-chain.pem
    ln -s ../letsencrypt/live/www.libravatar.org/privkey.pem apt.pem
    ln -s ../letsencrypt/live/www.libravatar.org/cert.pem apt.crt
    ln -s ../letsencrypt/live/www.libravatar.org/chain.pem apt-chain.pem
    ln -s ../letsencrypt/live/www.libravatar.org/privkey.pem www.pem
    ln -s ../letsencrypt/live/www.libravatar.org/cert.pem www.crt
    ln -s ../letsencrypt/live/www.libravatar.org/chain.pem www-chain.pem
    chmod a+rx /etc/letsencrypt/archive /etc/letsencrypt/live
    
  • Install a cronjob to automatically renew these certs in /etc/cron.daily/certbot-renew-libravatar:

    #!/bin/bash
    
    /usr/bin/certbot renew --quiet --post-hook "/bin/systemctl restart apache2.service"
    
    pushd /etc/ > /dev/null
    /usr/bin/git add letsencrypt
    DIFFSTAT="$(/usr/bin/git diff --cached --stat)"
    if [ -n "$DIFFSTAT" ] ; then
        /usr/bin/git commit --quiet -m "Renewed letsencrypt certs"
        echo "$DIFFSTAT"
    fi
    popd > /dev/null
    
  • Install awstats and add this to /etc/awstats/awstats.conf.local:

    SiteDomain="libravatar.org"
    LogType=W
    LogFormat=1
    
  • Reduce log retention to 10 days by setting the following in /etc/logrotate.d/apache2:

    rotate 10
    

Backups

  • Add this script to /usr/local/sbin/libravatar_backups:

    #!/bin/sh
    
    DUMP_DIR="/var/backups/libravatar"
    
    # Perform fresh backup
    DUMP_FILE="$DUMP_DIR/`date +%Y%m%dT%H%M%S`.pg"
    pg_dump -Fc libravatar > $DUMP_FILE
    chmod 600 $DUMP_FILE
    
    # Encrypt backup
    gpg --recipient 007c98d1 --encrypt $DUMP_FILE
    rm $DUMP_FILE
    
    # Purge old backups
    find $DUMP_DIR -ctime +7 -delete
    
  • Create an empty /var/backups/libravatar directory

  • Install duplicity:

    apt install duplicity python-paramiko
    
  • Add my duplicity backup script to /home/francois/.backup/ and update:

    • GPG passhrase
  • Add the following to /home/francois/.backup/exclude:

    /etc/.git
    /home/francois/.cache
    /home/francois/git-mirrors
    /root/.cache
    
  • Add the following to /home/francois/.backup/include:

    /etc
    /home/francois
    /usr/local/bin
    /usr/local/sbin
    /root
    /var/backups/libravatar
    /var/lib/libravatar
    /var/log/libravatar
    
  • Create this cronjob in /etc/cron.d/libravatar_backups:

    # Local DB backups
    4 1,13 * * *       root    ionice -c3 nice -n10 /usr/local/sbin/libravatar_backups
    
    # Full backups to another server
    6 14 * * *      root    /home/francois/.backup/backup-selfoss
    6 23 * * 3      root    /home/francois/.backup/backup-selfoss --full
    
  • Install the appropriate gpg key as root:

    gpg --keyserver pgp.mit.edu --recv-key 16281F2E007C98D1
    
  • Mark the key as ultimately trusted by using the trust command after typing:

    gpg --edit-key 16281F2E007C98D1
    
  • Copy the required id_rsa key into /root/.ssh/.

  • Configure the port, hostname and username in /root/.ssh/config.

  • Run an initial backup:

    sudo /home/francois/.backup/backup-selfoss
    
  • Ensure the expected files have been backed up:

    sudo /home/francois/.backup/backup-selfoss --list-current-files > list.txt
    

Install application

  • Install Libravatar packages

    • Start by installing the apt-transport-https package
    • Add the Libravatar repository to /etc/apt/sources.list:

      deb https://apt.libravatar.org/ jessie main
      
    • Then install server packages:

      apt update
      apt install libravatar{,-common,-cdn,-cdn-common,-deployment,-www,-master}
      
    • Use the defaults for all of the debconf questions except:

      • 6432 for the database port number
      • a string from pwgen -s 50 for the Django secret
  • Install gearman-job-server:

    apt install --no-install-recommends gearman-job-server
    
  • Add your username to the libravatar-deployment group:

    adduser francois libravatar-deployment
    

Git mirrors

  • Install hg:

    apt install mercurial
    
  • Enable the convert mercurial extension by putting this in /etc/mercurial/hgrc:

    [extensions]
    convert =
    
  • Add this script in /usr/local/bin/git-mirror:

    #!/bin/bash
    
    DIR=/home/francois/git-mirrors
    cd $DIR/git
    git fetch --quiet origin 2> /dev/null
    git reset --quiet --hard origin/master
    git push --quiet github master 2> /dev/null
    exit 0
    
  • Add this script in /usr/local/bin/hg-mirror:

    #!/bin/bash
    DIR=/home/francois/git-mirrors
    cd $DIR/git
    git fetch --quiet origin 2> /dev/null
    git reset --quiet --hard origin/master
    cd $DIR
    hg --quiet convert git libravatar-hg
    cd $DIR/libravatar-hg
    hg --quiet update
    hg --quiet push ssh://bitbucket.org/libravatar/libravatar 2> /dev/null
    exit 0
    
  • Add this to /home/francois/.ssh/config:

    Host bitbucket.org
      User hg
      IdentityFile ~/.ssh/repository_mirrors
    Host github.com
      User git
      IdentityFile ~/.ssh/repository_mirrors
      AddressFamily inet
    
  • Copy repository_mirrors ssh key from old server into /home/francois/.ssh/ (make sure that the private key has 600 permissions)

  • Prepare directory for git and mercurial repositories:

    mkdir -p /home/francois/git-mirrors/libravatar-hg
    cd /home/francois/git-mirrors/
    git clone https://git.launchpad.net/~libravatar/libravatar git
    cd git
    git remote add github git@github.com:libravatar/libravatar.git
    
  • Do a test run for each of them (as francois):

    /usr/local/bin/git-mirror && echo success
    /usr/local/bin/hg-mirror && echo success
    
  • Create the following cron jobs in /etc/cron.d/libravatar_git-mirrors:

    50 * * * *      francois        /usr/local/bin/git-mirror
    55 * * * *      francois        /usr/local/bin/hg-mirror
    

From the old server

If migrating from one server to another:

  • Setup the database

    • Get a DB dump from the old server:

      pg_dump -Fc libravatar > libravatar.pg
      
    • Create a new database:

      createdb -O djangouser -E utf8 libravatar
      
    • Restore the database:

      pg_restore -d libravatar < libravatar.pg
      
  • Copy the contents of /var/lib/libravatar/ from the old server and fix the file permissions:

    chown -R root:root /var/lib/libravatar/avatar/* /var/lib/libravatar/user/*