Here are some notes on how to setup the main libravatar.org server after you've installed Debian and the usual server packages.

Basic setup

  • Install Debian and tweak a few things
  • Install rssh and uncomment allowrsync in /etc/rssh.conf
  • Setup appropriate mail aliases

    • Add these to /etc/aliases

      root: francois
      francois: francois@libravatar.org
      www-data: root
      libravatar-master: root
      
    • Run newaliases to update /etc/aliases.db

    • Test the aliases by running mail root
  • Ensure that the only entry for localhost in /etc/hosts is for 127.0.0.1 (if you leave the one for ::1, it will confuse gearman)

  • Copy the local logcheck rules from the old server and add the following to /etc/logcheck/logcheck.logfiles:

    /var/log/libravatar/error-cdn.log
    /var/log/libravatar/error-seccdn.log
    /var/log/libravatar/error-www.log
    /var/log/libravatar/workers.log
    /var/log/postgresql/postgresql-8.4-main.log
    

Firewall

  • In /etc/network/iptables.up.rules, put the following:

    # Set default policies
    *filter
    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    
    # Allow unlimited outbound traffic
    -A OUTPUT -j ACCEPT
    
    # Annoying attack bots
    -A INPUT -s xxx.xxx.xxx.xxx/xx -j DROP
    
    # Accept all other traffic
    -A INPUT -j ACCEPT
    
    COMMIT
    
  • Enable the rules using iptables-apply

  • In /etc/network/ip6tables.up.rules, put the following:

    # Set default policies
    *filter
    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    
    # Allow unlimited outbound traffic
    -A OUTPUT -j ACCEPT
    
    # Allow unlimited traffic on the loopback interface
    -A INPUT -i lo -j ACCEPT
    
    # TODO: ban attackers!
    
    # Accept all other traffic
    -A INPUT -j ACCEPT
    
    COMMIT
    
  • Enable the rules using ip6tables-apply

  • Set something like this in /etc/network/interfaces:

    auto eth0
    iface eth0 inet static
        address x.x.x.x
        netmask 255.255.255.0
        gateway x.x.x.x
        pre-up iptables-restore /etc/network/iptables.up.rules
    iface eth0 inet6 static
        address x::x
        netmask 64
        gateway x::x
        pre-up ip6tables-restore /etc/network/ip6tables.up.rules
    

Database

  • Install Postgres:

    • In /etc/postgresql/*/main/pg_hba.conf, change all local connections to trust
    • In /etc/postgresql/*/main/postgresql.conf, set these:

      log_min_duration_statement =  1000
      ssl = off
      
    • Create postgres users:

      sudo -u postgres createuser -s francois
      createuser -s root
      createuser -S -R -D djangouser
      
  • Install pgbouncer

    apt install pgbouncer
    
    • Turn it on in /etc/default/pgbouncer

      START=1
      
    • Put the following in /etc/pgbouncer/pgbouncer.ini:

      [databases]
      libravatar = dbname=libravatar
      
      [pgbouncer]
      admin_users = postgres
      
    • Put the following in /etc/pgbouncer/userlist.txt:

      "djangouser" ""
      "postgres" ""
      
    • Restart the daemon:

      systemctl restart pgbouncer
      

Web server

  • Create a global TLS configuration in /etc/apache2/conf-available/tls.conf:

    SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite          ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    SSLHonorCipherOrder     on
    SSLCompression          off
    
    SSLUseStapling          on
    SSLStaplingResponderTimeout 5
    SSLStaplingReturnResponderErrors off
    SSLStaplingCache        shmcb:/var/run/ocsp(128000)
    
  • Enable TLS and mod_rewrite:

    a2enconf tls
    a2enmod ssl
    a2enmod rewrite
    
  • Setup some Apache vhosts:

    • /etc/apache2/sites-enabled/000-default:

      <VirtualHost *:80>
          RewriteEngine On
          RewriteRule ^ https://www.libravatar.org [redirect=301,last]
      </VirtualHost>
      
    • /etc/apache2/sites-enabled/default-ssl:

      <VirtualHost *:443>
          SSLEngine on
          SSLCertificateFile /etc/libravatar/www.crt
          SSLCertificateKeyFile /etc/libravatar/www.pem
          SSLCertificateChainFile /etc/libravatar/www-chain.pem
      
          RewriteEngine On
          RewriteRule ^ https://www.libravatar.org [redirect=301,last]
      </VirtualHost>
      
    • /etc/apache2/sites-available/stats:

      <VirtualHost *:443>
          ServerName stats.libravatar.org
          ServerAdmin webmaster@libravatar.org
          DocumentRoot /var/cache/awstats
          Alias /awstats-icon/ /usr/share/awstats/icon/
          Alias /favicon.ico /usr/share/libravatar/libravatar/favicon.ico
      
          SSLEngine on
          SSLCertificateFile /etc/libravatar/stats.crt
          SSLCertificateKeyFile /etc/libravatar/stats.pem
          SSLCertificateChainFile /etc/libravatar/stats-chain.pem
      
          Header add Strict-Transport-Security: "max-age=15768000"
      
          <Location />
              AuthType Basic
              AuthName "Libravatar Stats"
              AuthUserFile /etc/apache2/stats.passwd
              Require valid-user
              Order allow,deny
              allow from all
              Options +indexes
          </Location>
      </VirtualHost>
      
  • Copy the stats password file (/etc/apache2/stats.passwd).

  • Enable the new vhost:

    a2ensite default-ssl
    a2ensite stats
    
  • Copy seccdn SSL certs in /etc/libravatar/ from the old server.

  • Install certbot and then create two separate certs:

    systemctl stop apache2
    certbot certonly -d www.libravatar.org -d libravatar.org -d selfoss.libravatar.org
    certbot certonly -d stats.libravatar.org
    systemctl start apache2
    
  • Symlink the letsencrypt certs in the right place:

    cd /etc/libravatar/
    ln -s ../letsencrypt/stats.libravatar.org/privkey.pem stats.pem
    ln -s ../letsencrypt/stats.libravatar.org/cert.pem stats.crt
    ln -s ../letsencrypt/stats.libravatar.org/chain.pem stats-chain.pem
    ln -s ../letsencrypt/www.libravatar.org/privkey.pem www.pem
    ln -s ../letsencrypt/www.libravatar.org/cert.pem www.crt
    ln -s ../letsencrypt/www.libravatar.org/chain.pem www-chain.pem
    
  • Install a cronjob to automatically renew these certs in /etc/cron.daily/certbot-renew-libravatar:

    #!/bin/bash
    
    /usr/bin/certbot renew --quiet --pre-hook "/bin/systemctl stop apache2.service" --post-hook "/bin/systemctl start apache2.service"
    
    pushd /etc/ > /dev/null
    /usr/bin/git add letsencrypt
    DIFFSTAT="$(/usr/bin/git diff --cached --stat)"
    if [ -n "$DIFFSTAT" ] ; then
        /usr/bin/git commit --quiet -m "Renewed letsencrypt certs"
        echo "$DIFFSTAT"
    fi
    popd > /dev/null
    
  • Install awstats and add this to /etc/awstats/awstats.conf.local:

    SiteDomain="libravatar.org"
    LogType=W
    LogFormat=1
    

Backups

  • Add this script to /usr/local/sbin/libravatar_backups:

    #!/bin/sh
    
    DUMP_DIR="/var/backups/libravatar"
    
    # Perform fresh backup
    DUMP_FILE="$DUMP_DIR/`date +%Y%m%dT%H%M%S`.pg"
    pg_dump -Fc libravatar > $DUMP_FILE
    chmod 600 $DUMP_FILE
    
    # Encrypt backup
    gpg --recipient 007c98d1 --encrypt $DUMP_FILE
    rm $DUMP_FILE
    
    # Purge old backups
    find $DUMP_DIR -ctime +7 -delete
    
  • Create an empty /var/backups/libravatar directory

  • Install duplicity (apt install duplicity)
  • Add my duplicity backup script to /home/francois/.backup/ and update:
    • GPG passhrase
    • Destination directory (which includes the hostname)
  • Create this cronjob in /etc/cron.d/libravatar_backups:

    # Local DB backups
    4 1,13 * * *       root    ionice -c3 nice -n10 /usr/local/sbin/libravatar_backups
    
    # Full backups to Amazon S3
    6 14 * * *      root    /home/francois/.backup/backup-selfoss
    6 23 * * 3      root    /home/francois/.backup/backup-selfoss --full
    
  • Run an initial backup:

    sudo /home/francois/.backup/backup-selfoss
    

Install application

  • Install Libravatar packages

    • Start by installing the apt-transport-https package
    • Add the Libravatar repository to /etc/apt/sources.list:

      deb https://apt.libravatar.org/ jessie main
      
    • Then install server packages:

      apt update
      apt install libravatar{,-common,-cdn,-cdn-common,-deployment,-www,-master}
      
    • Use 6432 for the database port number, otherwise accept all of the defaults

  • Install gearman-job-server:

    apt install --no-install-recommends gearman-job-server
    

Git mirrors

  • Install hg:

    apt install mercurial
    
  • Enable the convert mercurial extension by putting this in /etc/mercurial/hgrc:

    [extensions]
    convert =
    
  • Add this script in /usr/local/bin/git-mirror:

    #!/bin/bash
    
    DIR=/home/francois/git-mirrors
    cd $DIR/git
    git fetch --quiet origin 2> /dev/null
    git reset --quiet --hard origin/master
    git push --quiet github master 2> /dev/null
    exit 0
    
  • Add this script in /usr/local/bin/hg-mirror:

    #!/bin/bash
    DIR=/home/francois/git-mirrors
    cd $DIR/git
    git fetch --quiet origin 2> /dev/null
    git reset --quiet --hard origin/master
    cd $DIR
    hg --quiet convert git libravatar-hg
    cd $DIR/libravatar-hg
    hg --quiet update
    hg --quiet push ssh://bitbucket.org/libravatar/libravatar 2> /dev/null
    exit 0
    
  • Add this to /home/francois/.ssh/config:

    Host bitbucket.org
      User hg
      IdentityFile ~/.ssh/repository_mirrors
    Host github.com
      User git
      IdentityFile ~/.ssh/repository_mirrors
      AddressFamily inet
    
  • Copy repository_mirrors ssh key from old server into /home/francois/.ssh/

  • Create directory for mercurial mirror: mkdir -p /home/francois/git-mirrors/libravatar-hg
  • Prepare directory for git repository:

    cd /home/francois/git-mirrors/
    git clone https://git.launchpad.net/~libravatar/libravatar git
    cd git
    git remote add github git@github.com:libravatar/libravatar.git
    
  • Create the following cron jobs in /etc/cron.d/libravatar_git-mirrors:

    50 * * * *      francois        /usr/local/bin/git-mirror
    55 * * * *      francois        /usr/local/bin/hg-mirror
    

From the old server

If migrating from one server to another:

  • Setup the database

    • Get a DB dump from the old server:

      pg_dump -Fc libravatar > libravatar.pg
      
    • Create a new database:

      createdb -O djangouser -E utf8 libravatar
      
    • Restore the database:

      pg_restore -d libravatar < libravatar.pg
      
  • Copy the contents of /var/lib/libravatar/ from the old server and fix the file permissions:

    find /var/lib/libravatar/avatar/ -type f -exec chown libravatar-img:nogroup {} \;
    find /var/lib/libravatar/user/ -type f -exec chown libravatar-img:nogroup {} \;