Quince to Puffin rsync script

[chris]

Request from Jim:

"can you set up an rsync script on the new box that can:

• Grab all the files from sites/default/files on the old site recursively
• Exclude the directories 'ctools', 'js', 'css' and 'tmp' folders
• Put them in the /data/disk/[o1 OR tn depending on setup]/static/sites/transitionnetwork.org/files (make the folder)
• Be able to be run any time to bring this up to date, including deleting files that are gone and replacing all changed (except those in the ignored folders)
• Set the permissions and owners afterwards (I'll get the correct values for this afterwards, just leave a space in the script at the end)

If you can put this in /data/disk/[o1 OR tn depending on setup]/scripts/pull-files-from-quince.sh or similar then that would be awesome. This will allow us to get all the files from the other site whenever and will be useful many times, including when we finally do the go-live."

[chris]

This script has been created and tested, details follow.

Create a directory for the script and the script:

mkdir /data/disk/tn/scripts
chown tn:users /data/disk/tn/scripts
chmod 700 /data/disk/tn/scripts
touch /data/disk/tn/scripts/pull-files-from-quince.sh
chown tn:users /data/disk/tn/scripts/pull-files-from-quince.sh
chmod 700 /data/disk/tn/scripts/pull-files-from-quince.sh

The directory /data/disk/tn/static currently has one file in it, EMPTY.txt with the content "empty" -- Jim should we delete this file when we create the /data/disk/tn/static/sites/transitionnetwork.org/files directory?

For the moment I have set the script up so that it writes the files into /tmp/ but of course this can be changed by editing the DEST_DIR variable.

Since there is 2.4G of files in /web/transitionnetwork.org/www/sites/default/files on quince I'd suggest moving the contents of /tmp/transitionnetwork.org/files to the place it's wanted to save time / resources compared with rsyncing the data from scratch.

Following is the content of the script:

#!/bin/bash

# See /trac/ticket/472

# The server we are copying the files from, see ~/.ssh/config
SOURCE_SERVER="quince"

# Directory on the source server where the files are
# no / at the end please
SOURCE_DIR="/web/transitionnetwork.org/www/sites/default/files"

# The place we want the files copied to
# no / at the end please
DEST_DIR="/tmp/transitionnetwork.org/files"

# The things we want to exclude
EXCLUDE="--exclude=ctools --exclude=js --exclude=css --exclude=tmp"

# The rsync command
RSYNC="rsync -aq --delete"

# Group and user owner to set on transfered files
CHOWN="tn:users"

# Directory permissions to set
CHMOD_DIR="755"

# File permissions to set
CHMOD_FILE="644"

# Check that the $DEST_DIR exists and if it doesn't then create it
if [[ ! -d "$DEST_DIR" ]]; then
  mkdir -p $DEST_DIR
fi

# The rsync command
echo "rsyncing from $SOURCE_SERVER:$SOURCE_DIR/ to $DEST_DIR/ excluding $EXCLUDE"
$RSYNC $EXCLUDE $SOURCE_SERVER:$SOURCE_DIR/ $DEST_DIR/

# The chown command
echo "chowning $DEST_DIR to $CHOWN"
chown -R $CHOWN $DEST_DIR

#  The chmod commands
cd $DEST_DIR
echo "chmodding directories in $DEST_DIR to $CHMOD_DIR" 
find -type d -exec chmod $CHMOD_DIR {} \;
echo "chmodding files in $DEST_DIR to $CHMOD_FILE" 
find -type f -exec chmod $CHMOD_FILE {} \;

[jim]

Thanks Chris.

FYI a couple of tweaks:

• DEST_DIR now is "/data/disk/tn/static/sites/transitionnetwork.org-PROD/files".
• RSYNC="rsync -az --delete" (use compression, don't be quiet [I like to see things happen!])

For some reason it still said nothing while it worked, but it did work! Thanks again.

[chris]

Replying to jim:

• RSYNC="rsync -az --delete" (use compression, don't be quiet [I like to see things happen!])

For some reason it still said nothing while it worked, but it did work!

To see the output you need -va (v for verbose), compression isn't going to speed things up as both virtual servers are on the same switch, glad it worked :-)