Transition Technology: Ticket #763: Server Backups

ed — Tue, 22 Jul 2014 09:04:35 GMT

I am happy that you set this up to enable others access to backups without extra charge
I am concerned that the off-site backups aren't working; isn't it best to have off-site (rather than same-site) backups?

chris — Tue, 22 Jul 2014 09:47:44 GMT

Replying to ed:

I am happy that you set this up to enable others access to backups without extra charge

Great, I'll try to get this sorted today or tomorrow.

I am concerned that the off-site backups aren't working; isn't it best to have off-site (rather than same-site) backups?

Yes, we should have both, the disk in the Manchester server has now been replaced and next we are going to rebuild it as a BSD/ZFS backup server and then start syncing data to it again.

hours, totalhours changed

chris — Wed, 23 Jul 2014 14:50:08 GMT

hours changed from 0.0 to 1.75
totalhours changed from 0.0 to 1.75

Creating 3 accounts for backups, one per server, on store1.webarch.net:

sudo -i
bash
export USERNAME="tn-parrot"
zfs create tank/backupclients/$USERNAME
zfs set quota=100G tank/backupclients/$USERNAME
pw addgroup $USERNAME
pw useradd -n $USERNAME -s /usr/local/bin/rbash -d /tank/backupclients/$USERNAME -g $USERNAME
chmod 750 /tank/backupclients/$USERNAME
chown $USERNAME:$USERNAME /tank/backupclients/$USERNAME
vim /root/clients/backupstore/backupclients.yaml
#   - client: transitionnetwork.org
#     uid: tn-parrot
#     storage_path: tank/backupclients/tn-parrot
#     email: webproject@transitionnetwork.org
#     quota: 100G
vim /usr/local/etc/zfs-snapshot-mgmt.conf
#   tank/backupclients/tn-parrot:
#     recursive: false
#     # Create snapshots every 24 hours, starting at 06:00 UTC
#     creation_rule:
#       at_multiple: 1440
#       offset: 360
#     # Keep snapshots for 60 days
#     preservation_rules:
#       - { for_minutes: 86400, at_multiple: 1440, offset:  360 }
mkdir /tank/backupclients/$USERNAME/.ssh
chmod 700 /tank/backupclients/$USERNAME/.ssh
touch /tank/backupclients/$USERNAME/.ssh/authorized_keys
chmod 600 /tank/backupclients/$USERNAME/.ssh/authorized_keys
chown -R $USERNAME:$USERNAME /tank/backupclients/$USERNAME/.ssh
vim /tank/backupclients/$USERNAME/.ssh/authorized_keys

I added everybodies ssh public keys and the servers root ssh public keys and then repeated the above for:

export USERNAME="tn-penguin"
export USERNAME="tn-puffin"

But gave the tn-puffin account a 300GB quota.

On each server the /root/.ssh/config files were edited and sshing to the backup server was tested:

vim /root/.ssh/config
#Host *
#  CompressionLevel 9
#  CheckHostIP yes
#  StrictHostKeyChecking ask
#
#Host backup
#  Hostname store1.webarch.net
#  User tn-parrot
cat /root/.ssh/id_*.pub
ssh backup

I have put a copy of the backup script on wiki:AgileBackup and it was deployed like this:

wget https://svn.webarch.net/scripts/usr/local/bin/agile-backup -O /usr/local/bin/agile-backup --no-check-certificate
chmod 700 /usr/local/bin/agile-backup
mkdir /etc/agile/
chmod 700 /etc/agile/
vim /etc/agile/backup
chmod 600 /etc/agile/backup

The things being backed up are listed in /etc/agile/backup on each server, for wiki:ParrotServer:

/root
/home
/etc
/usr/local
/var/backups/mysql/sqldump
/var/spool/cron/crontabs

wiki:PenguinServer:

/root
/home
/etc
/web
/var/backups/mysql/sqldump
/var/spool/cron/crontabs

and wiki:PuffinServer:

/root
/etc
/home
/data
/var/backups/mysql/sqldump
/var/spool/cron/crontabs

procmail was installed on all 3 servers as it's used for the lockfile and the backups were run once manually to check everything is working.

The backup tasks were have been setup to run each night via cron, on wiki:PuffinServer this has been done using my account as BOA clobbers the root crontab on upgrades.

People should now be able to use SFTP to access the backups:

sftp tn-puffin@store1.webarch.net

The latest backups are in parrot.webarch.net and 60 days worth of snapshots are in ~/.zfs/, you can mount these backups locally, for example on Debian:

aptitude install sshfs
mkdir -p /media/tn-puffin
chmod 700 /media/tn-puffin/
mkdir /media/tn-puffin/latest
mkdir /media/tn-puffin/archive
echo "sshfs#tn-puffin@store1.webarch.net:puffin.webarch.net /media/tn-puffin/latest fuse ro,nobootwait 0 0" >> /etc/fstab
echo "sshfs#tn-puffin@store1.webarch.net::.zfs/snapshot /media/tn-puffin/archive fuse ro,nobootwait 0 0" >> /etc/fstab
mount -a

I have documented the backups:

hours, totalhours changed

chris — Tue, 02 Sep 2014 10:43:25 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.75 to 2.0

It looks like the way the space was devided wasn't ideal, the PenguinServer space is 95% used up:

NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        40.3G  59.7G  10.2G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin       99.6G   416M  10.2G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         111G   189G  23.1G  /tank/backupclients/tn-puffin

So adding 100GB to the space for penguin and removing 100GB from puffin:

zfs set quota=200G tank/backupclients/tn-penguin
zfs set quota=200G tank/backupclients/tn-puffin

And /root/clients/backupstore/backupclients.yaml was edited:

   - client: transitionnetwork.org
     uid: tn-parrot
     storage_path: tank/backupclients/tn-parrot
     email: webproject@transitionnetwork.org
     quota: 100G
   - client: transitionnetwork.org
     uid: tn-penguin
     storage_path: tank/backupclients/tn-penguin
     email: webproject@transitionnetwork.org
     quota: 200G
   - client: transitionnetwork.org
     uid: tn-puffin
     storage_path: tank/backupclients/tn-puffin
     email: webproject@transitionnetwork.org
     quota: 200G

And now we have:

NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        40.3G  59.7G  10.2G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin       99.6G   100G  10.2G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         111G  89.4G  23.1G  /tank/backupclients/tn-puffin

hours, totalhours changed

chris — Thu, 11 Dec 2014 10:00:58 GMT

hours changed from 0.0 to 0.25
totalhours changed from 2.0 to 2.25

The PenguinServer backup space is getting low:

zfs list | grep 'tn\|NAME'
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        56.8G  43.2G  10.6G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin        196G  3.64G  10.9G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         131G  69.3G  25.1G  /tank/backupclients/tn-puffin

This is what was set in /root/clients/backupstore/backupclients.yaml:

  - client: transitionnetwork.org
    uid: tn-parrot
    storage_path: tank/backupclients/tn-parrot
    email: webproject@transitionnetwork.org
    quota: 100G
  - client: transitionnetwork.org
    uid: tn-penguin
    storage_path: tank/backupclients/tn-penguin
    email: webproject@transitionnetwork.org
    quota: 200G
  - client: transitionnetwork.org
    uid: tn-puffin
    storage_path: tank/backupclients/tn-puffin
    email: webproject@transitionnetwork.org
    quota: 200G

So adding more space for PenguinServer and reducing the others:

zfs set quota=180G tank/backupclients/tn-puffin
zfs set quota=80G tank/backupclients/tn-parrot
zfs set quota=240G tank/backupclients/tn-penguin

So we now have the following in /root/clients/backupstore/backupclients.yaml:

  - client: transitionnetwork.org
    uid: tn-parrot
    storage_path: tank/backupclients/tn-parrot
    email: webproject@transitionnetwork.org
    quota: 80G
  - client: transitionnetwork.org
    uid: tn-penguin
    storage_path: tank/backupclients/tn-penguin
    email: webproject@transitionnetwork.org
    quota: 240G
  - client: transitionnetwork.org
    uid: tn-puffin
    storage_path: tank/backupclients/tn-puffin
    email: webproject@transitionnetwork.org
    quota: 180G

So this is now the state of the quotas:

zfs list | grep 'tn\|NAME'
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        56.8G  23.2G  10.6G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin        196G  43.6G  10.9G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         131G  49.3G  25.1G  /tank/backupclients/tn-puffin

cc changed

chris — Sat, 02 May 2015 09:21:38 GMT

cc ade added; ed removed

Just an update to report that the nightly disk images of all three servers are once again (and have been for the last couple of months or so) being synced to a backup server in Manchester every night.