Server Backups

[chris]

Two weeks ago annesley asked:

what off-site data storage, file backup and quick setup do we have?

I answered:

The 3 virtual servers have their file system mounted off a BSD/NFS/ZFS file server and the whole file system is backed up and stored onto another BSD/ZFS server in the same data centre. We did have backups also being copied to a server in Manchester but this is currently off-line as the Manchester server needs a disk swapping and rebuilding as a BSD/ZFS server.

A problem with this is that it's only me and Alan that have access to these backups, so I'd like to suggest I set up a new account for backups on our backup server and sort out cron jobs to rsync data to this account and document how people can access these backups.

The result would be that everybody would have SFTP access to 60 days worth of snapshots of backups from all three servers whenever needed without any need for my or Alan's intervention.

I expect this would take abount an hour to set up and another hour to document and help people understand it.

There would be no additional cost to the TN because backup space is already paid for.

[ed]

1. I am happy that you set this up to enable others access to backups without extra charge
2. I am concerned that the off-site backups aren't working; isn't it best to have off-site (rather than same-site) backups?

[chris]

Replying to ed:

1. I am happy that you set this up to enable others access to backups without extra charge

Great, I'll try to get this sorted today or tomorrow.

2. I am concerned that the off-site backups aren't working; isn't it best to have off-site (rather than same-site) backups?

Yes, we should have both, the disk in the Manchester server has now been replaced and next we are going to rebuild it as a BSD/ZFS backup server and then start syncing data to it again.

[chris]

Creating 3 accounts for backups, one per server, on store1.webarch.net:

sudo -i
bash
export USERNAME="tn-parrot"
zfs create tank/backupclients/$USERNAME
zfs set quota=100G tank/backupclients/$USERNAME
pw addgroup $USERNAME
pw useradd -n $USERNAME -s /usr/local/bin/rbash -d /tank/backupclients/$USERNAME -g $USERNAME
chmod 750 /tank/backupclients/$USERNAME
chown $USERNAME:$USERNAME /tank/backupclients/$USERNAME
vim /root/clients/backupstore/backupclients.yaml
#   - client: transitionnetwork.org
#     uid: tn-parrot
#     storage_path: tank/backupclients/tn-parrot
#     email: webproject@transitionnetwork.org 
#     quota: 100G
vim /usr/local/etc/zfs-snapshot-mgmt.conf
#   tank/backupclients/tn-parrot:
#     recursive: false
#     # Create snapshots every 24 hours, starting at 06:00 UTC
#     creation_rule:
#       at_multiple: 1440
#       offset: 360
#     # Keep snapshots for 60 days
#     preservation_rules:
#       - { for_minutes: 86400, at_multiple: 1440, offset:  360 }
mkdir /tank/backupclients/$USERNAME/.ssh
chmod 700 /tank/backupclients/$USERNAME/.ssh
touch /tank/backupclients/$USERNAME/.ssh/authorized_keys
chmod 600 /tank/backupclients/$USERNAME/.ssh/authorized_keys
chown -R $USERNAME:$USERNAME /tank/backupclients/$USERNAME/.ssh
vim /tank/backupclients/$USERNAME/.ssh/authorized_keys

I added everybodies ssh public keys and the servers root ssh public keys and then repeated the above for:

export USERNAME="tn-penguin"
export USERNAME="tn-puffin"

But gave the tn-puffin account a 300GB quota.

On each server the /root/.ssh/config files were edited and sshing to the backup server was tested:

vim /root/.ssh/config
#Host *
#  CompressionLevel 9
#  CheckHostIP yes
#  StrictHostKeyChecking ask
#
#Host backup
#  Hostname store1.webarch.net
#  User tn-parrot
cat /root/.ssh/id_*.pub
ssh backup

I have put a copy of the backup script on wiki:AgileBackup and it was deployed like this:

wget https://svn.webarch.net/scripts/usr/local/bin/agile-backup -O /usr/local/bin/agile-backup --no-check-certificate
chmod 700 /usr/local/bin/agile-backup
mkdir /etc/agile/
chmod 700 /etc/agile/
vim /etc/agile/backup
chmod 600 /etc/agile/backup

The things being backed up are listed in /etc/agile/backup on each server, for wiki:ParrotServer:

/root
/home
/etc
/usr/local
/var/backups/mysql/sqldump
/var/spool/cron/crontabs

wiki:PenguinServer:

/root
/home
/etc
/web
/var/backups/mysql/sqldump
/var/spool/cron/crontabs

and wiki:PuffinServer:

/root
/etc
/home
/data
/var/backups/mysql/sqldump
/var/spool/cron/crontabs

procmail was installed on all 3 servers as it's used for the lockfile and the backups were run once manually to check everything is working.

The backup tasks were have been setup to run each night via cron, on wiki:PuffinServer this has been done using my account as BOA clobbers the root crontab on upgrades.

People should now be able to use SFTP to access the backups:

sftp tn-puffin@store1.webarch.net

The latest backups are in parrot.webarch.net and 60 days worth of snapshots are in ~/.zfs/, you can mount these backups locally, for example on Debian:

aptitude install sshfs
mkdir -p /media/tn-puffin
chmod 700 /media/tn-puffin/
mkdir /media/tn-puffin/latest
mkdir /media/tn-puffin/archive
echo "sshfs#tn-puffin@store1.webarch.net:puffin.webarch.net /media/tn-puffin/latest fuse ro,nobootwait 0 0" >> /etc/fstab
echo "sshfs#tn-puffin@store1.webarch.net::.zfs/snapshot /media/tn-puffin/archive fuse ro,nobootwait 0 0" >> /etc/fstab
mount -a

I have documented the backups:

• PenguinServer#Backups
• PuffinServer#Backups
• ParrotServer#Backups

[chris]

It looks like the way the space was devided wasn't ideal, the PenguinServer space is 95% used up:

NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        40.3G  59.7G  10.2G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin       99.6G   416M  10.2G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         111G   189G  23.1G  /tank/backupclients/tn-puffin

So adding 100GB to the space for penguin and removing 100GB from puffin:

zfs set quota=200G tank/backupclients/tn-penguin
zfs set quota=200G tank/backupclients/tn-puffin 

And /root/clients/backupstore/backupclients.yaml was edited:

   - client: transitionnetwork.org
     uid: tn-parrot
     storage_path: tank/backupclients/tn-parrot
     email: webproject@transitionnetwork.org
     quota: 100G
 
   - client: transitionnetwork.org
     uid: tn-penguin
     storage_path: tank/backupclients/tn-penguin
     email: webproject@transitionnetwork.org
     quota: 200G
 
   - client: transitionnetwork.org
     uid: tn-puffin
     storage_path: tank/backupclients/tn-puffin
     email: webproject@transitionnetwork.org
     quota: 200G

And now we have:

NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        40.3G  59.7G  10.2G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin       99.6G   100G  10.2G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         111G  89.4G  23.1G  /tank/backupclients/tn-puffin

[chris]

The PenguinServer backup space is getting low:

zfs list | grep 'tn\|NAME'
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        56.8G  43.2G  10.6G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin        196G  3.64G  10.9G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         131G  69.3G  25.1G  /tank/backupclients/tn-puffin

This is what was set in /root/clients/backupstore/backupclients.yaml:

  - client: transitionnetwork.org
    uid: tn-parrot
    storage_path: tank/backupclients/tn-parrot
    email: webproject@transitionnetwork.org
    quota: 100G

  - client: transitionnetwork.org
    uid: tn-penguin
    storage_path: tank/backupclients/tn-penguin
    email: webproject@transitionnetwork.org
    quota: 200G

  - client: transitionnetwork.org
    uid: tn-puffin
    storage_path: tank/backupclients/tn-puffin
    email: webproject@transitionnetwork.org
    quota: 200G

So adding more space for PenguinServer and reducing the others:

zfs set quota=180G tank/backupclients/tn-puffin 
zfs set quota=80G tank/backupclients/tn-parrot
zfs set quota=240G tank/backupclients/tn-penguin

So we now have the following in /root/clients/backupstore/backupclients.yaml:

  - client: transitionnetwork.org
    uid: tn-parrot
    storage_path: tank/backupclients/tn-parrot
    email: webproject@transitionnetwork.org
    quota: 80G

  - client: transitionnetwork.org
    uid: tn-penguin
    storage_path: tank/backupclients/tn-penguin
    email: webproject@transitionnetwork.org
    quota: 240G

  - client: transitionnetwork.org
    uid: tn-puffin
    storage_path: tank/backupclients/tn-puffin
    email: webproject@transitionnetwork.org
    quota: 180G

So this is now the state of the quotas:

zfs list | grep 'tn\|NAME'
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
tank/backupclients/tn-parrot        56.8G  23.2G  10.6G  /tank/backupclients/tn-parrot
tank/backupclients/tn-penguin        196G  43.6G  10.9G  /tank/backupclients/tn-penguin
tank/backupclients/tn-puffin         131G  49.3G  25.1G  /tank/backupclients/tn-puffin

[chris]

Just an update to report that the nightly disk images of all three servers are once again (and have been for the last couple of months or so) being synced to a backup server in Manchester every night.