Transition Technology: Ticket #466: Puffin install and configuration

hours, totalhours changed

chris — Wed, 28 Nov 2012 13:42:56 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.0 to 0.1

hours, totalhours changed

chris — Fri, 30 Nov 2012 21:38:02 GMT

hours changed from 0.0 to 2.8
totalhours changed from 0.1 to 2.9

The new server is up and running, I have done the following:

Set up the domain names puffin.webarch.net, puffin.transitionnetwork.org newdev.transitionnetwork.org and *.newdev.transitionnetwork.org to point to it's IP address, 81.95.52.103 and also asked for reverse dns to be set to puffin.transitionnetwork.org and edited /etc/hosts and /etc/hostname

Installed the follwing:

aptitude install vim screen sudo wget lynx pwgen rsync

Added the following to /root/.bashrc:

export EDITOR=vim

Created /root/.vimrc with the following in it:

syntax on

Generated a passphrase less root ssh key pair for rsyncing data off the quince and kiwi:

ssh-keygen -t rsa -b 2048

Added this public key to /root/.ssh/authorized_keys on kiwi and quince with this at the start of it to limit it's use to the puffin IP address:

from="81.95.52.103" ssh-rsa AAAA...

Created /root/.ssh/config with the following in it:

Host quince
  Hostname quince.transitionnetwork.org
  User root
Host kiwi
  Hostname kiwi.transitionnetwork.org
  User root

Edited /etc/ssh/sshd_config and restarted ssh:

PermitRootLogin no
AllowGroups sudo sshaccess

Edited /etc/sudoers:

#%sudo ALL=(ALL) ALL
%sudo ALL=(ALL) NOPASSWD: ALL

Created user accounts for all the existing people who need it, matching the user and group ID's to the ones on quince, and copied across their ssh public keys and sent them email with their passwords, using the following:

export NEWUSER=username
ssh quince grep $NEWUSER /etc/passwd # get the ID
export NEWUSERID=123
addgroup --gid $NEWUSERID $NEWUSER
adduser --gid $NEWUSERID --uid $NEWUSERID $NEWUSER
adduser $NEWUSER sudo
mkdir /home/$NEWUSER/.ssh
chmod 700 /home/$NEWUSER/.ssh
rsync -av quince:/home/$NEWUSER/.ssh/authorized_keys /home/$NEWUSER/.ssh/
chmod 600 /home/$NEWUSER/.ssh/authorized_keys
chown -R $NEWUSER:$NEWUSER /home/$NEWUSER/.ssh

Installed a MTA and MUA:

aptitude install postfix mutt

Copied across the wild card ssl cert for *.transitionnetwork.org:

mkdir /etc/ssl/transitionnetwork.org/
chmod 700 /etc/ssl/transitionnetwork.org/
rsync -av quince:/etc/ssl/transitionnetwork.org/ /etc/ssl/transitionnetwork.org/

Edited the TLS settings in /etc/postfix/main.cf and restarted postfix:

# TLS parameters
smtpd_tls_cert_file = /etc/ssl/transitionnetwork.org/transitionnetwork.org.crt
smtpd_tls_key_file = /etc/ssl/transitionnetwork.org/transitionnetwork.org.key
smtpd_tls_CAfile = /etc/ssl/transitionnetwork.org/gandi.pem
smtpd_use_tls=yes
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_session_cache_timeout = 3600s
smtpd_tls_cipherlist = EDH:!EXP:!LOW
smtpd_tls_ask_ccert = yes
smtpd_tls_req_ccert = no
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtp_enforce_tls = no
smtp_tls_cert_file = /etc/ssl/transitionnetwork.org/transitionnetwork.org.crt
smtp_tls_key_file = /etc/ssl/transitionnetwork.org/transitionnetwork.org.key
smtp_tls_CAfile = /etc/ssl/transitionnetwork.org/gandi.pem
smtp_tls_loglevel = 1
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_session_cache_timeout = 3600s
myhostname = puffin.transitionnetwork.org

Edited /etc/aliases and added a destination for root emails and ran newaliases and sent a test email to root using mutt and then checked that TLS was used by looking at the mail headers.

Installed metche which will keep an eye on /root/Changelog and send system emails out.

Installed mysql and added the root password to /root/.my.cnf:

 aptitude install mysql-server-5.1

Installed backupninja and created a user on the remote backup server and set up ssh:

adduser puffin
mkdir /home/puffin/.ssh
vi /home/puffin/.ssh/authorized_keys
chmod 600 /home/puffin/.ssh/authorized_keys
chmod 700 /home/puffin/.ssh
chown -R puffin:puffin /home/puffin/.ssh

Ran ninjahelper and set up system, mysql and rdiff backups and tested that it was all working.

Installed git.

Installed https://drupal.org/project/barracuda:

cd
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
boa in-stable public newdev.transitionnetwork.org chris@webarchitects.co.uk o1
Barracuda [Fri Nov 30 21:25:25 GMT 2012] ==> BOA Skynet welcomes you aboard!
Barracuda [Fri Nov 30 21:25:29 GMT 2012] ==> INFO: NORMAL INIT
Barracuda [Fri Nov 30 21:25:30 GMT 2012] ==> INFO: Easy Public Setup Mode Active
Barracuda [Fri Nov 30 21:25:30 GMT 2012] ==> INFO: Creating your /root/.barracuda.cnf config file
Barracuda [Fri Nov 30 21:25:31 GMT 2012] ==> INFO: We need to install wget, axel, aptitude, netcat & git first, please wait...
Barracuda [Fri Nov 30 21:25:46 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Fri Nov 30 21:25:47 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Fri Nov 30 21:25:47 GMT 2012] ==> INFO: Downloading little helpers, please wait...
Barracuda [Fri Nov 30 21:25:50 GMT 2012] ==> INFO: Checking BARRACUDA version...
Barracuda [Fri Nov 30 21:25:50 GMT 2012] ==> INFO: Version test result: OK
Barracuda [Fri Nov 30 21:25:50 GMT 2012] ==> INFO: Installing lsb-release and dnsutils now, please wait...
Barracuda [Fri Nov 30 21:26:06 GMT 2012] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Fri Nov 30 21:26:09 GMT 2012] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.4
Barracuda [Fri Nov 30 21:26:11 GMT 2012] ==> EXIT on error due to invalid DNS setup
  * Your custom _MY_OWNIP is set to "81.95.52.103"
  * Your custom _MY_HOSTN is set to "newdev.transitionnetwork.org"
  * Your custom _MY_FRONT is set to "master.newdev.transitionnetwork.org"
  * Your _MY_HOSTN and/or _MY_FRONT doesn't match your _MY_OWNIP,
    or your hostname is not set properly yet.
  * Please make sure that below command returns your FQDN hostname "newdev.transitionnetwork.org":
  $ uname -n
  * Your server has to have already working FQDN hostname matching your IP address.
    This means you _have to_ configure the dns for your server IP/hostname _before_
    trying to use this install script. Reverse dns is _not_ required.
  * Your hostname appears to be newdev.transitionnetwork.org - are you sure it's a valid FQDN hostname?
  * Are you sure your FQDN hostname matches your IP address: 81.95.52.103?
Barracuda [Fri Nov 30 21:26:11 GMT 2012] ==> EXIT on error due to invalid DNS setup
Octopus [Fri Nov 30 21:26:11 GMT 2012] ==> BOA Skynet welcomes you aboard!
Octopus [Fri Nov 30 21:26:14 GMT 2012] ==> INFO: Creating your /root/.o1.octopus.cnf config file
Octopus [Fri Nov 30 21:26:15 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Octopus [Fri Nov 30 21:26:17 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Octopus [Fri Nov 30 21:26:17 GMT 2012] ==> ERROR: This script should be used only when the same version of BARRACUDA was used before
Octopus [Fri Nov 30 21:26:17 GMT 2012] ==> Your system has to be configured/upgraded by BARRACUDA version BOA-2.0.4 first
Octopus [Fri Nov 30 21:26:17 GMT 2012] ==> Bye
BOA in-stable completed
Bye

So the problems above were because the DNS for newdev.transitionnetwork.org hadn't yet updated, oops, I'll get some tea and carry on later...

Next steps:

rerun the barracuda installer once the dns has updated
Get the barracuda installed drupal to serve the transition network site.
Copy across the live data and database, perhaps splitting it into seperate inodedb files in the process and write a script so that this can be easilly done again.
Migrate all sites / applications we are keeping running off kiwi and quince.

hours, totalhours changed

chris — Fri, 30 Nov 2012 22:01:39 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.9 to 3.0

denyhosts was installed and it was set in /etc/denyhosts.conf to not send email, in the short time the server has been up two IP's got added to /etc/hosts.deny

/root/Changelog was created and a basic list of packages installed added to it:

2012-11-30      chris
        *       metche vim screen sudo wget lynx pwgen rsync : installed
        *       mysql-server-5.1 : installed
        *       backupninja : installed
        *       git : installed
        *       denyhosts : installed
        *       for further info see : https://tech.transitionnetwork.org/trac/ticket/466

jim — Sat, 01 Dec 2012 06:51:50 GMT

Hi Chris, the main thing with Barracuda (BOA = Barracuda + Octopus) is that you need the system as 'out of the box' as possible. Don't install anything at this point that isn't needed, there's a chance BOA will remove it, or fail... This is true of things going forward too: Things might get removed during updates (you can put them back, obviously) if they relate to PHP, MySQL, Redis or NginX.

Also, don't just install the site under the base Barracuda package -- add Octopus too as it comes with a load of goodies we'll need and has loads of anti-bot, high performance Drupal tweaks. http://drupal.org/project/octopus

Really good info and links here on the BOA wiki: http://groups.drupal.org/node/163784

Finally you'll need to understand the Aegir Drupal hosting system which manages sites etc: http://community.aegirproject.org/notebook

I'm off to CA now, but will be on emails etc much of the time.

hours, totalhours changed

chris — Sat, 01 Dec 2012 10:23:26 GMT

hours changed from 0.0 to 0.15
totalhours changed from 3.0 to 3.15

Replying to jim:

Hi Chris, the main thing with Barracuda (BOA = Barracuda + Octopus) is that you need the system as 'out of the box' as possible. Don't install anything at this point that isn't needed

I know, I did read the README.txt, INSTALL.txt and SSL.txt and UPGRADE.txt before starting and decided that it should be OK to do the following before installing it:

Configuring a MTA so that the server could securly send email out.
Configuring backups, this necessitated the installation of mysql.
Adding user accounts and configuring SSH.

Thanks for the links, I see the Aegir Open Atrium site is hosting spam, hope the trip goes well!

hours, totalhours changed

chris — Sat, 01 Dec 2012 10:49:58 GMT

hours changed from 0.0 to 0.43
totalhours changed from 3.15 to 3.58

Re-running the installer:

boa in-stable public newdev.transitionnetwork.org chris@webarchitects.co.uk o1
  Please update installers on your system
  using BOA Meta Installer and try again.
  $ wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
  $ bash BOA.sh.txt

So starting again:

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt ; bash BOA.sh.txt
BOA Meta Installer setup completed
Please check docs/INSTALL.txt and docs/UPGRADE.txt for how-to
Bye

boa in-stable public newdev.transitionnetwork.org chris@webarchitects.co.uk o1
Barracuda [Sat Dec  1 10:26:01 GMT 2012] ==> BOA Skynet welcomes you aboard!
Barracuda [Sat Dec  1 10:26:05 GMT 2012] ==> INFO: NORMAL INIT
Barracuda [Sat Dec  1 10:26:05 GMT 2012] ==> INFO: Easy Public Setup Mode Active
Barracuda [Sat Dec  1 10:26:05 GMT 2012] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Sat Dec  1 10:26:06 GMT 2012] ==> NOTE! Please review all config options displayed below
Barracuda [Sat Dec  1 10:26:06 GMT 2012] ==> NOTE! It will *override* all settings in the Barracuda script
###
### Configuration created on 121130-2125
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
_MY_HOSTN="newdev.transitionnetwork.org"
_MY_FRONT="master.newdev.transitionnetwork.org"
_THIS_DB_HOST=localhost
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV FTP"
_AUTOPILOT=YES
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
_LOAD_LIMIT_ONE=1444
_LOAD_LIMIT_TWO=888
_CUSTOM_CONFIG_CSF=NO
_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
_CUSTOM_CONFIG_PHP_5_3=NO
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121130-2125
### with Barracuda version BOA-2.0.4
###
Barracuda [Sat Dec  1 10:26:08 GMT 2012] ==> INFO: We need to install wget, axel, aptitude, netcat & git first, please wait...
Barracuda [Sat Dec  1 10:26:14 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Sat Dec  1 10:26:16 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Sat Dec  1 10:26:16 GMT 2012] ==> INFO: Downloading little helpers, please wait...
Barracuda [Sat Dec  1 10:26:19 GMT 2012] ==> INFO: Checking BARRACUDA version...
Barracuda [Sat Dec  1 10:26:19 GMT 2012] ==> INFO: Version test result: OK
Barracuda [Sat Dec  1 10:26:19 GMT 2012] ==> INFO: Installing lsb-release and dnsutils now, please wait...
Barracuda [Sat Dec  1 10:26:22 GMT 2012] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Sat Dec  1 10:26:24 GMT 2012] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.4
Barracuda [Sat Dec  1 10:26:27 GMT 2012] ==> DNS test OK - your FQDN hostname is newdev.transitionnetwork.org and it matches your IP 81.95.52.103
Barracuda [Sat Dec  1 10:26:27 GMT 2012] ==> Please double check whether this is a valid result of my test
Barracuda [Sat Dec  1 10:26:29 GMT 2012] ==> INSTALL START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your IP address appears to be 81.95.52.103 - are you sure it's a valid IP?
  * Your hostname appears to be newdev.transitionnetwork.org - are you sure it's a valid FQDN hostname?
Barracuda [Sat Dec  1 10:26:29 GMT 2012] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Sat Dec  1 10:26:31 GMT 2012] ==> INFO: Installing DNS cache pdnsd server, please wait...
Barracuda [Sat Dec  1 10:26:51 GMT 2012] ==> INFO: DNS cache pdnsd server installation completed
Barracuda [Sat Dec  1 10:26:53 GMT 2012] ==> INFO: Updating apt sources
Barracuda [Sat Dec  1 10:26:56 GMT 2012] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Sat Dec  1 10:27:00 GMT 2012] ==> INFO: Running aptitude update, please wait...
Barracuda [Sat Dec  1 10:27:33 GMT 2012] ==> INFO: Installing required libraries and tools
Barracuda [Sat Dec  1 10:27:33 GMT 2012] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Sat Dec  1 10:36:04 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec  1 10:36:06 GMT 2012] ==> INFO: Installed Nginx version nginx/1.2.5, upgrade required
Barracuda [Sat Dec  1 10:36:08 GMT 2012] ==> INFO: Installing Nginx, please wait...
Barracuda [Sat Dec  1 10:37:21 GMT 2012] ==> INFO: Installing MariaDB, please wait...
Barracuda [Sat Dec  1 10:38:49 GMT 2012] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Sat Dec  1 10:39:05 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec  1 10:39:07 GMT 2012] ==> INFO: Installed Nginx version nginx/1.3.8, no upgrade required
Barracuda [Sat Dec  1 10:39:09 GMT 2012] ==> INFO: Installing /usr/bin/wkhtmltopdf x86_64 version, please wait...
Barracuda [Sat Dec  1 10:39:14 GMT 2012] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Sat Dec  1 10:39:16 GMT 2012] ==> INFO: Installing VnStat monitor, please wait...
Barracuda [Sat Dec  1 10:39:26 GMT 2012] ==> INFO: Installing a few more tools, please wait...
Barracuda [Sat Dec  1 10:39:28 GMT 2012] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
Barracuda [Sat Dec  1 10:39:32 GMT 2012] ==> INFO: Installing PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 10:39:55 GMT 2012] ==> INFO: Installing PhpRedis for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 10:40:26 GMT 2012] ==> INFO: Installing UploadProgress for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 10:40:48 GMT 2012] ==> INFO: Installing JSMin for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 10:41:16 GMT 2012] ==> INFO: Installing bzr, please wait...
Barracuda [Sat Dec  1 10:41:48 GMT 2012] ==> INFO: Installing lshell, please wait...
Barracuda [Sat Dec  1 10:41:56 GMT 2012] ==> INFO: Installing latest Pure-FTPd server, please wait...
Barracuda [Sat Dec  1 10:43:24 GMT 2012] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
Barracuda [Sat Dec  1 10:44:22 GMT 2012] ==> INFO: Generating random password for Redis server
Barracuda [Sat Dec  1 10:44:22 GMT 2012] ==> INFO: Updating init scripts
Barracuda [Sat Dec  1 10:44:28 GMT 2012] ==> INFO: Restarting MariaDB
Barracuda [Sat Dec  1 10:44:38 GMT 2012] ==> INFO: Starting Redis, PHP-FPM and Nginx
Barracuda [Sat Dec  1 10:44:49 GMT 2012] ==> INFO: Generating random password for MariaDB
Barracuda [Sat Dec  1 10:44:52 GMT 2012] ==> INFO: OS and services installation completed
Barracuda [Sat Dec  1 10:44:54 GMT 2012] ==> INFO: Installing Aegir Master Instance, please wait...
Barracuda [Sat Dec  1 10:45:08 GMT 2012] ==> INFO: Running hosting-dispatch (1/3), please wait...
Barracuda [Sat Dec  1 10:45:14 GMT 2012] ==> INFO: Running hosting-dispatch (2/3), please wait...
Barracuda [Sat Dec  1 10:45:20 GMT 2012] ==> INFO: Running hosting-dispatch (3/3), please wait...
cat: /var/aegir/.drush/hostmaster.alias.drushrc.php: No such file or directory
Barracuda [Sat Dec  1 10:45:25 GMT 2012] ==> INFO: Aegir Master Instance installation completed
Barracuda [Sat Dec  1 10:45:28 GMT 2012] ==> INFO: Installing default SSL Wildcard Nginx Proxy, please wait...
Barracuda [Sat Dec  1 10:45:33 GMT 2012] ==> INFO: Installing Chive MariaDB Manager, please wait...
Barracuda [Sat Dec  1 10:45:37 GMT 2012] ==> INFO: Chive MariaDB Manager installation completed
Barracuda [Sat Dec  1 10:45:39 GMT 2012] ==> INFO: MariaDB final setup
Barracuda [Sat Dec  1 10:45:55 GMT 2012] ==> INFO: MariaDB setup completed
Barracuda [Sat Dec  1 10:45:55 GMT 2012] ==> INFO: You can now log in as root by typing only 'mysql' on the command line
Barracuda [Sat Dec  1 10:45:58 GMT 2012] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Sat Dec  1 10:46:00 GMT 2012] ==> ALRT: Something went wrong
Barracuda [Sat Dec  1 10:46:00 GMT 2012] ==> ALRT: Please check the install log for details:
Barracuda [Sat Dec  1 10:46:00 GMT 2012] ==> ALRT: /var/aegir/install.log
Barracuda [Sat Dec  1 10:46:12 GMT 2012] ==> INFO: Installing csf/lfd firewall, please wait...
Barracuda [Sat Dec  1 10:46:18 GMT 2012] ==> INFO: csf/lfd firewall installation completed
Barracuda [Sat Dec  1 10:46:20 GMT 2012] ==> CARD: Now charging your credit card for this automated install service...
Barracuda [Sat Dec  1 10:46:26 GMT 2012] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Sat Dec  1 10:46:30 GMT 2012] ==> Final post-install cleaning, please wait a moment...
Timeout, server not responding.

This is the point at which I regret not running the install in screen, do'h!

hours, totalhours changed

chris — Sat, 01 Dec 2012 11:23:29 GMT

hours changed from 0.0 to 0.56
totalhours changed from 3.58 to 4.14

So, let's try running the installer, in screen, again:

sudo -i
screen
cd
boa in-stable public newdev.transitionnetwork.org chris@webarchitects.co.uk o1
  Another BOA installer is running probably - /var/run/boa_run.pid exists

So I'll leave this a while and have a look around at what is installed and running...

There is a "Under Construction" page up at http://newdev.transitionnetwork.org/ and also at https://newdev.transitionnetwork.org/ with an Ageir generated wildcard SSL certificate.
A /home/redis/ directory and system accounts for messagebus, pdnsd, redis and aegir have been created with shells set to /bin/false
nginx is up and running, but there are no sites configured in the /etc/nginx/sites-available/ directory.
Other services running include, boa, pure-ftpd, php-fpm, mysqld, redis-server, nginx, pdnsd.

The PID file still exists and it hasn't been updated for 25 mins:

-rw-r--r-- 1 root root 0 Dec  1 10:46 /var/run/boa_run.pid

I haven't been sent a email, which I was expecting, from the installer, but I have got a SSH login alert email sent to root by lfd, which is a Perl script which has been installed at /usr/sbin/lfd

Lots of iptables rules have been set up:

root@newdev:/etc/nginx# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  dns1.webarchitects.co.uk  anywhere            tcp dpt:domain
ACCEPT     udp  --  dns1.webarchitects.co.uk  anywhere            udp dpt:domain
ACCEPT     tcp  --  dns1.webarchitects.co.uk  anywhere            tcp spt:domain
ACCEPT     udp  --  dns1.webarchitects.co.uk  anywhere            udp spt:domain
LOCALINPUT  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
INVALID    tcp  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp-data
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:www
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:cvspserver
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:5280
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:git
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpts:30000:50000
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:20
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:fsp
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:ntp
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:snmp
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpts:33434:33523
ACCEPT     icmp --  anywhere             anywhere            icmp echo-request limit: avg 1/sec burst 5
ACCEPT     icmp --  anywhere             anywhere            icmp echo-reply limit: avg 1/sec burst 5
ACCEPT     icmp --  anywhere             anywhere            icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere            icmp destination-unreachable
LOGDROPIN  all  --  anywhere             anywhere
Chain FORWARD (policy DROP)
target     prot opt source               destination
Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             dns1.webarchitects.co.uk tcp dpt:domain
ACCEPT     udp  --  anywhere             dns1.webarchitects.co.uk udp dpt:domain
ACCEPT     tcp  --  anywhere             dns1.webarchitects.co.uk tcp spt:domain
ACCEPT     udp  --  anywhere             dns1.webarchitects.co.uk udp spt:domain
LOCALOUTPUT  all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:domain
ACCEPT     udp  --  anywhere             anywhere            udp spt:domain
ACCEPT     all  --  anywhere             anywhere
INVALID    tcp  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp-data
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:www
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:pop3
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:imap2
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssmtp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:submission
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:rsync
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:imaps
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:pop3s
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:cvspserver
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:mysql
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:5280
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:git
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:hkp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpts:30000:50000
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:20
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:fsp
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:113
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:ntp
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:snmp
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpts:33434:33523
ACCEPT     icmp --  anywhere             anywhere            icmp echo-reply
ACCEPT     icmp --  anywhere             anywhere            icmp echo-request
ACCEPT     icmp --  anywhere             anywhere            icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere            icmp destination-unreachable
LOGDROPOUT  all  --  anywhere             anywhere
Chain INVALID (2 references)
target     prot opt source               destination
INVDROP    all  --  anywhere             anywhere            state INVALID
INVDROP    tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
INVDROP    tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG
INVDROP    tcp  --  anywhere             anywhere            tcp flags:FIN,SYN/FIN,SYN
INVDROP    tcp  --  anywhere             anywhere            tcp flags:SYN,RST/SYN,RST
INVDROP    tcp  --  anywhere             anywhere            tcp flags:FIN,RST/FIN,RST
INVDROP    tcp  --  anywhere             anywhere            tcp flags:FIN,ACK/FIN
INVDROP    tcp  --  anywhere             anywhere            tcp flags:PSH,ACK/PSH
INVDROP    tcp  --  anywhere             anywhere            tcp flags:ACK,URG/URG
INVDROP    tcp  --  anywhere             anywhere            tcp flags:!FIN,SYN,RST,ACK/SYN state NEW
Chain INVDROP (10 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
Chain LOCALINPUT (1 references)
target     prot opt source               destination
DROP       tcp  --  222.37.135.2         anywhere            tcp dpt:ssh
DROP       tcp  --  80-252-241-37.hosting.wildpark.net  anywhere            tcp dpt:ssh
Chain LOCALOUTPUT (1 references)
target     prot opt source               destination
Chain LOGDROPIN (1 references)
target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere            tcp dpt:bootps
DROP       udp  --  anywhere             anywhere            udp dpt:bootps
DROP       tcp  --  anywhere             anywhere            tcp dpt:bootpc
DROP       udp  --  anywhere             anywhere            udp dpt:bootpc
DROP       tcp  --  anywhere             anywhere            tcp dpt:sunrpc
DROP       udp  --  anywhere             anywhere            udp dpt:sunrpc
DROP       tcp  --  anywhere             anywhere            tcp dpt:auth
DROP       udp  --  anywhere             anywhere            udp dpt:113
DROP       tcp  --  anywhere             anywhere            tcp dpts:loc-srv:netbios-ssn
DROP       udp  --  anywhere             anywhere            udp dpts:loc-srv:netbios-ssn
DROP       tcp  --  anywhere             anywhere            tcp dpt:microsoft-ds
DROP       udp  --  anywhere             anywhere            udp dpt:microsoft-ds
DROP       tcp  --  anywhere             anywhere            tcp dpt:isakmp
DROP       udp  --  anywhere             anywhere            udp dpt:isakmp
DROP       tcp  --  anywhere             anywhere            tcp dpt:login
DROP       udp  --  anywhere             anywhere            udp dpt:who
DROP       tcp  --  anywhere             anywhere            tcp dpt:520
DROP       udp  --  anywhere             anywhere            udp dpt:route
LOG        tcp  --  anywhere             anywhere            limit: avg 30/min burst 5 LOG level warning prefix `Firewall: *TCP_IN Blocked* '
LOG        udp  --  anywhere             anywhere            limit: avg 30/min burst 5 LOG level warning prefix `Firewall: *UDP_IN Blocked* '
LOG        icmp --  anywhere             anywhere            limit: avg 30/min burst 5 LOG level warning prefix `Firewall: *ICMP_IN Blocked* '
DROP       all  --  anywhere             anywhere
Chain LOGDROPOUT (1 references)
target     prot opt source               destination
LOG        tcp  --  anywhere             anywhere            limit: avg 30/min burst 5 LOG level warning prefix `Firewall: *TCP_OUT Blocked* '
LOG        udp  --  anywhere             anywhere            limit: avg 30/min burst 5 LOG level warning prefix `Firewall: *UDP_OUT Blocked* '
LOG        icmp --  anywhere             anywhere            limit: avg 30/min burst 5 LOG level warning prefix `Firewall: *ICMP_OUT Blocked* '
DROP       all  --  anywhere             anywhere

I'll get another cuppa and if the lock file still exists I think it'll time to delete it and re-run the installer...

hours, totalhours changed

chris — Sat, 01 Dec 2012 12:14:05 GMT

hours changed from 0.0 to 0.36
totalhours changed from 4.14 to 4.5

OK, the lock file is over an hour old, deleting it and re-running the installer:

rm /var/run/boa_run.pid
boa in-stable public newdev.transitionnetwork.org chris@webarchitects.co.uk o1
Barracuda [Sat Dec  1 11:54:34 GMT 2012] ==> BOA Skynet welcomes you aboard!
Barracuda [Sat Dec  1 11:54:38 GMT 2012] ==> INFO: FORCED INIT
Barracuda [Sat Dec  1 11:54:38 GMT 2012] ==> INFO: Easy Public Setup Mode Active
Barracuda [Sat Dec  1 11:54:38 GMT 2012] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Sat Dec  1 11:54:39 GMT 2012] ==> NOTE! Please review all config options displayed below
Barracuda [Sat Dec  1 11:54:39 GMT 2012] ==> NOTE! It will *override* all settings in the Barracuda script
###
### Configuration created on 121130-2125
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
_MY_HOSTN="newdev.transitionnetwork.org"
_MY_FRONT="master.newdev.transitionnetwork.org"
_THIS_DB_HOST=localhost
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV FTP"
_AUTOPILOT=YES
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
_LOAD_LIMIT_ONE=1444
_LOAD_LIMIT_TWO=888
_CUSTOM_CONFIG_CSF=NO
_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
_CUSTOM_CONFIG_PHP_5_3=NO
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121130-2125
### with Barracuda version BOA-2.0.4
###
Barracuda [Sat Dec  1 11:54:41 GMT 2012] ==> INFO: We need to install wget, axel, aptitude, netcat & git first, please wait...
Barracuda [Sat Dec  1 11:54:48 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Sat Dec  1 11:54:50 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Sat Dec  1 11:54:50 GMT 2012] ==> INFO: Downloading little helpers, please wait...
Barracuda [Sat Dec  1 11:54:52 GMT 2012] ==> INFO: Checking BARRACUDA version...
Barracuda [Sat Dec  1 11:54:52 GMT 2012] ==> INFO: Version test result: OK
Barracuda [Sat Dec  1 11:54:52 GMT 2012] ==> INFO: Installing lsb-release and dnsutils now, please wait...
Barracuda [Sat Dec  1 11:54:55 GMT 2012] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Sat Dec  1 11:54:57 GMT 2012] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.4
Barracuda [Sat Dec  1 11:55:00 GMT 2012] ==> DNS test OK - your FQDN hostname is newdev.transitionnetwork.org and it matches your IP 81.95.52.103
Barracuda [Sat Dec  1 11:55:00 GMT 2012] ==> Please double check whether this is a valid result of my test
Barracuda [Sat Dec  1 11:55:02 GMT 2012] ==> INSTALL START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your IP address appears to be 81.95.52.103 - are you sure it's a valid IP?
  * Your hostname appears to be newdev.transitionnetwork.org - are you sure it's a valid FQDN hostname?
Barracuda [Sat Dec  1 11:55:02 GMT 2012] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Sat Dec  1 11:55:04 GMT 2012] ==> INFO: Updating apt sources
Barracuda [Sat Dec  1 11:55:06 GMT 2012] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Sat Dec  1 11:55:10 GMT 2012] ==> INFO: Running aptitude update, please wait...
Barracuda [Sat Dec  1 11:55:28 GMT 2012] ==> INFO: Installing required libraries and tools
Barracuda [Sat Dec  1 11:55:28 GMT 2012] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Sat Dec  1 11:55:41 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec  1 11:55:43 GMT 2012] ==> INFO: Installed Nginx version nginx/1.3.8, no upgrade required
Barracuda [Sat Dec  1 11:55:45 GMT 2012] ==> INFO: Installing Nginx, please wait...
Barracuda [Sat Dec  1 11:56:57 GMT 2012] ==> INFO: Installing MariaDB, please wait...
Barracuda [Sat Dec  1 11:58:26 GMT 2012] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Sat Dec  1 11:59:11 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec  1 11:59:13 GMT 2012] ==> INFO: Installed Nginx version nginx/1.3.8, no upgrade required
Barracuda [Sat Dec  1 11:59:16 GMT 2012] ==> INFO: Installing Nginx, please wait...
Barracuda [Sat Dec  1 12:00:30 GMT 2012] ==> INFO: Installing /usr/bin/wkhtmltopdf x86_64 version, please wait...
Barracuda [Sat Dec  1 12:00:35 GMT 2012] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Sat Dec  1 12:00:37 GMT 2012] ==> INFO: Installing VnStat monitor, please wait...
Barracuda [Sat Dec  1 12:00:47 GMT 2012] ==> INFO: Installing a few more tools, please wait...
Barracuda [Sat Dec  1 12:00:49 GMT 2012] ==> INFO: Installed PHP version 5.3.19-1~dotdeb.0, no upgrade required
Barracuda [Sat Dec  1 12:00:49 GMT 2012] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
Barracuda [Sat Dec  1 12:00:53 GMT 2012] ==> INFO: Installing PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 12:00:57 GMT 2012] ==> INFO: Installing PhpRedis for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 12:01:30 GMT 2012] ==> INFO: Installing UploadProgress for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 12:01:52 GMT 2012] ==> INFO: Installing JSMin for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 12:02:20 GMT 2012] ==> INFO: Installing bzr, please wait...
Barracuda [Sat Dec  1 12:02:50 GMT 2012] ==> INFO: Installing lshell, please wait...
Barracuda [Sat Dec  1 12:02:55 GMT 2012] ==> INFO: Installing latest Pure-FTPd server, please wait...
Barracuda [Sat Dec  1 12:04:23 GMT 2012] ==> INFO: Installed Redis version 2.6.4, no rebuild required
Barracuda [Sat Dec  1 12:04:25 GMT 2012] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
Barracuda [Sat Dec  1 12:05:32 GMT 2012] ==> INFO: Updating init scripts
Barracuda [Sat Dec  1 12:05:37 GMT 2012] ==> INFO: Restarting MariaDB
Barracuda [Sat Dec  1 12:05:46 GMT 2012] ==> INFO: Starting Redis, PHP-FPM and Nginx
Barracuda [Sat Dec  1 12:05:57 GMT 2012] ==> INFO: Generating random password for MariaDB
Barracuda [Sat Dec  1 12:06:00 GMT 2012] ==> INFO: OS and services installation completed
Barracuda [Sat Dec  1 12:06:02 GMT 2012] ==> INFO: Installing Aegir Master Instance, please wait...
Barracuda [Sat Dec  1 12:06:16 GMT 2012] ==> INFO: Running hosting-dispatch (1/3), please wait...
Barracuda [Sat Dec  1 12:06:22 GMT 2012] ==> INFO: Running hosting-dispatch (2/3), please wait...
Barracuda [Sat Dec  1 12:06:28 GMT 2012] ==> INFO: Running hosting-dispatch (3/3), please wait...
cat: /var/aegir/.drush/hostmaster.alias.drushrc.php: No such file or directory
Barracuda [Sat Dec  1 12:06:33 GMT 2012] ==> INFO: Aegir Master Instance installation completed
Barracuda [Sat Dec  1 12:06:36 GMT 2012] ==> INFO: Installing default SSL Wildcard Nginx Proxy, please wait...
Barracuda [Sat Dec  1 12:06:41 GMT 2012] ==> INFO: MariaDB final setup
Barracuda [Sat Dec  1 12:06:58 GMT 2012] ==> INFO: MariaDB setup completed
Barracuda [Sat Dec  1 12:06:58 GMT 2012] ==> INFO: You can now log in as root by typing only 'mysql' on the command line
Barracuda [Sat Dec  1 12:07:00 GMT 2012] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Sat Dec  1 12:07:03 GMT 2012] ==> ALRT: Something went wrong
Barracuda [Sat Dec  1 12:07:03 GMT 2012] ==> ALRT: Please check the install log for details:
Barracuda [Sat Dec  1 12:07:03 GMT 2012] ==> ALRT: /var/aegir/install.log
Barracuda [Sat Dec  1 12:07:13 GMT 2012] ==> CARD: Now charging your credit card for this automated install service...
Barracuda [Sat Dec  1 12:07:19 GMT 2012] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Sat Dec  1 12:07:23 GMT 2012] ==> Final post-install cleaning, please wait a moment...
Barracuda [Sat Dec  1 12:07:30 GMT 2012] ==> BYE!
Octopus [Sat Dec  1 12:07:34 GMT 2012] ==> BOA Skynet welcomes you aboard!
Octopus [Sat Dec  1 12:07:37 GMT 2012] ==> INFO: Reading your /root/.o1.octopus.cnf config file
/var/backups/OCTOPUS.sh.txt: line 713: /data/disk/o1/log/fpm.txt: No such file or directory
/var/backups/OCTOPUS.sh.txt: line 722: /data/disk/o1/log/cli.txt: No such file or directory
Octopus [Sat Dec  1 12:07:38 GMT 2012] ==> NOTE! Please review all config options displayed below
###
### Configuration created on 121130-2126 with
### Octopus version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Octopus script.
###
_USER="o1"
_MY_EMAIL="chris@webarchitects.co.uk"
_PLATFORMS_LIST="ALL"
_ALLOW_UNSUPPORTED=NO
_AUTOPILOT=NO
_HM_ONLY=NO
_O_CONTRIB_UP=NO
_DEBUG_MODE=NO
_MY_OWNIP=
_FORCE_GIT_MIRROR=""
_THIS_DB_HOST=localhost
_DNS_SETUP_TEST=NO
_HOT_SAUCE=NO
_USE_CURRENT=YES
_REMOTE_CACHE_IP=127.0.0.1
_LOCAL_NETWORK_IP=
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
_USE_STOCK=NO
###
### NOTE: the group of settings displayed bellow will be *overriden*
### by config files stored in the /data/disk/o1/log/ directory,
### but only on upgrade.
###
_DOMAIN="o1.newdev.transitionnetwork.org"
_CLIENT_EMAIL="chris@webarchitects.co.uk"
_CLIENT_OPTION="SSD"
_CLIENT_SUBSCR="Y"
_CLIENT_CORES="8"
###
### Configuration created on 121130-2126 with
### Octopus version BOA-2.0.4
###
Do you want to proceed with the install? [Y/n] Y
Octopus [Sat Dec  1 12:10:08 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Octopus [Sat Dec  1 12:10:09 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Octopus [Sat Dec  1 12:10:10 GMT 2012] ==> NEW Aegir Satellite Instance setup in progress...
Octopus [Sat Dec  1 12:10:10 GMT 2012] ==> START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your client e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your Aegir control panel for this instance will be available at https://o1.newdev.transitionnetwork.org
  * Your Aegir system user for this instance will be o1
  * This Octopus will use PHP-CLI 5.3 for all sites
  * This Octopus will use PHP-FPM 5.3 both for D6 and D7 sites
  * This Octopus includes platforms: ALL / Unsupported: NO
  * This Octopus options are listed as SSD / Y / 8 C
Do you want to proceed? [Y/n] Y
Octopus [Sat Dec  1 12:10:48 GMT 2012] ==> 8s before we will continue...
Octopus [Sat Dec  1 12:10:48 GMT 2012] ==> 8s before we will continue...
Octopus [Sat Dec  1 12:11:07 GMT 2012] ==> INIT A: Aegir automated install script part A
Octopus [Sat Dec  1 12:11:07 GMT 2012] ==> INFO A: Checking OCTOPUS version, please wait...
Octopus [Sat Dec  1 12:11:07 GMT 2012] ==> INFO: Version test result: OK
Octopus [Sat Dec  1 12:11:07 GMT 2012] ==> INIT A: INIT
Octopus [Sat Dec  1 12:11:07 GMT 2012] ==> INIT A: Shared platforms code v.001 (hot new) will be created
Octopus [Sat Dec  1 12:11:07 GMT 2012] ==> INIT A: Creating directories with correct permissions, please wait...
Octopus [Sat Dec  1 12:11:09 GMT 2012] ==> INIT A: Adding user, please wait...
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
Octopus [Sat Dec  1 12:11:16 GMT 2012] ==> INIT A: Switching user and running AegirSetupB, please wait...
Octopus [Sat Dec  1 12:11:18 GMT 2012] ==> INIT B: Aegir automated install script part B
Octopus [Sat Dec  1 12:11:18 GMT 2012] ==> INIT B: Creating directories with correct permissions
Octopus [Sat Dec  1 12:11:20 GMT 2012] ==> INIT B: Running standard installer
Octopus [Sat Dec  1 12:11:22 GMT 2012] ==> INIT B: Downloading drush, please wait...
Octopus [Sat Dec  1 12:11:25 GMT 2012] ==> INIT B: Drush seems to be functioning properly
Octopus [Sat Dec  1 12:11:25 GMT 2012] ==> INIT B: Installing provision backend in /data/disk/o1/.drush
Octopus [Sat Dec  1 12:11:26 GMT 2012] ==> INIT B: Downloading Drush and Provision extensions, please wait...
Octopus [Sat Dec  1 12:11:32 GMT 2012] ==> INIT B: Running hostmaster-install, please wait...
AegirSetupB.sh.txt: line 456: cd: /data/disk/o1/aegir/distro/001: No such file or directory
Octopus [Sat Dec  1 12:11:36 GMT 2012] ==> INIT B: Running hosting-dispatch (1/3), please wait...
Octopus [Sat Dec  1 12:11:42 GMT 2012] ==> INIT B: Running hosting-dispatch (2/3), please wait...
Octopus [Sat Dec  1 12:11:48 GMT 2012] ==> INIT B: Running hosting-dispatch (3/3), please wait...
Octopus [Sat Dec  1 12:11:49 GMT 2012] ==> INIT B: Simple check if Aegir install is successful
Octopus [Sat Dec  1 12:11:51 GMT 2012] ==> INIT B: FATAL ERROR: Required file /data/disk/o1/aegir/distro/001/sites/o1.newdev.transitionnetwork.org/settings.php does not exist
Octopus [Sat Dec  1 12:11:51 GMT 2012] ==> INIT B: FATAL ERROR: Aborting AegirSetupB installer NOW!
Octopus [Sat Dec  1 12:11:51 GMT 2012] ==> INIT A: FATAL ERROR: AegirSetupB installer failed
Octopus [Sat Dec  1 12:11:51 GMT 2012] ==> INIT A: FATAL ERROR: Aborting AegirSetupA installer NOW!
Octopus [Sat Dec  1 12:11:51 GMT 2012] ==> FATAL ERROR: AegirSetupA installer failed
Octopus [Sat Dec  1 12:11:51 GMT 2012] ==> FATAL ERROR: Aborting Octopus installer NOW!
BOA in-stable completed
Bye

So, Jim was right, it looks like the fact that Mysql was installed before the installer was run has caused it to fail. I'll come back to this on Monday, it might be that it's easier to start from scratch with a fresh debian install rather than sort out this failed install. Sorry :-(

hours, totalhours changed

chris — Sat, 01 Dec 2012 16:41:45 GMT

hours changed from 0.0 to 0.7
totalhours changed from 4.5 to 5.2

I have just rebuilt puffin from scratch, it took an hour and a half as I did some debugging on our virtual server build scripts, but I'm not putting the time for this down here.

So, starting again...

ssh root@puffin.webarch.net
aptitude install screen
screen
cd
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
  BOA Meta Installer setup completed
  Please check docs/INSTALL.txt and docs/UPGRADE.txt for how-to
  Bye
boa in-stable public newdev.transitionnetwork.org chris@webarchitects.co.uk o1
Barracuda [Sat Dec  1 16:03:27 GMT 2012] ==> BOA Skynet welcomes you aboard!
Barracuda [Sat Dec  1 16:03:31 GMT 2012] ==> INFO: NORMAL INIT
Barracuda [Sat Dec  1 16:03:31 GMT 2012] ==> INFO: Easy Public Setup Mode Active
Barracuda [Sat Dec  1 16:03:31 GMT 2012] ==> INFO: Creating your /root/.barracuda.cnf config file
Barracuda [Sat Dec  1 16:03:32 GMT 2012] ==> INFO: We need to install wget, axel, aptitude, netcat & git first, please wait...
Barracuda [Sat Dec  1 16:04:07 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Sat Dec  1 16:04:11 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Sat Dec  1 16:04:11 GMT 2012] ==> INFO: Downloading little helpers, please wait...
Barracuda [Sat Dec  1 16:04:17 GMT 2012] ==> INFO: Checking BARRACUDA version...
Barracuda [Sat Dec  1 16:04:17 GMT 2012] ==> INFO: Version test result: OK
Barracuda [Sat Dec  1 16:04:17 GMT 2012] ==> INFO: Installing lsb-release and dnsutils now, please wait...
Barracuda [Sat Dec  1 16:04:41 GMT 2012] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Sat Dec  1 16:04:44 GMT 2012] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.4
Barracuda [Sat Dec  1 16:04:46 GMT 2012] ==> DNS test OK - your FQDN hostname is newdev.transitionnetwork.org and it matches your IP 81.95.52.103
Barracuda [Sat Dec  1 16:04:46 GMT 2012] ==> Please double check whether this is a valid result of my test
Barracuda [Sat Dec  1 16:04:48 GMT 2012] ==> INSTALL START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your IP address appears to be 81.95.52.103 - are you sure it's a valid IP?
  * Your hostname appears to be newdev.transitionnetwork.org - are you sure it's a valid FQDN hostname?
Barracuda [Sat Dec  1 16:04:48 GMT 2012] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Sat Dec  1 16:04:50 GMT 2012] ==> INFO: Installing DNS cache pdnsd server, please wait...
Barracuda [Sat Dec  1 16:05:09 GMT 2012] ==> INFO: DNS cache pdnsd server installation completed
Barracuda [Sat Dec  1 16:05:11 GMT 2012] ==> INFO: Updating apt sources
Barracuda [Sat Dec  1 16:05:13 GMT 2012] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Sat Dec  1 16:05:17 GMT 2012] ==> INFO: Running aptitude update, please wait...
Barracuda [Sat Dec  1 16:05:52 GMT 2012] ==> INFO: Installing required libraries and tools
Barracuda [Sat Dec  1 16:05:52 GMT 2012] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Sat Dec  1 16:13:32 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec  1 16:13:34 GMT 2012] ==> INFO: Installed Nginx version nginx/1.2.5, upgrade required
Barracuda [Sat Dec  1 16:13:36 GMT 2012] ==> INFO: Installing Nginx, please wait...
Barracuda [Sat Dec  1 16:14:36 GMT 2012] ==> INFO: Installing MariaDB, please wait...
Barracuda [Sat Dec  1 16:16:01 GMT 2012] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Sat Dec  1 16:16:15 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec  1 16:16:18 GMT 2012] ==> INFO: Installed Nginx version nginx/1.3.8, no upgrade required
Barracuda [Sat Dec  1 16:16:20 GMT 2012] ==> INFO: Installing /usr/bin/wkhtmltopdf x86_64 version, please wait...
Barracuda [Sat Dec  1 16:16:25 GMT 2012] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Sat Dec  1 16:16:28 GMT 2012] ==> INFO: Installing VnStat monitor, please wait...
Barracuda [Sat Dec  1 16:16:37 GMT 2012] ==> INFO: Installing a few more tools, please wait...
Barracuda [Sat Dec  1 16:16:39 GMT 2012] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
Barracuda [Sat Dec  1 16:16:44 GMT 2012] ==> INFO: Installing PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 16:17:04 GMT 2012] ==> INFO: Installing PhpRedis for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 16:17:27 GMT 2012] ==> INFO: Installing UploadProgress for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 16:17:41 GMT 2012] ==> INFO: Installing JSMin for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec  1 16:18:01 GMT 2012] ==> INFO: Installing bzr, please wait...
Barracuda [Sat Dec  1 16:18:33 GMT 2012] ==> INFO: Installing lshell, please wait...
Barracuda [Sat Dec  1 16:18:39 GMT 2012] ==> INFO: Installing latest Pure-FTPd server, please wait...
Barracuda [Sat Dec  1 16:19:35 GMT 2012] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
Barracuda [Sat Dec  1 16:20:25 GMT 2012] ==> INFO: Generating random password for Redis server
Barracuda [Sat Dec  1 16:20:25 GMT 2012] ==> INFO: Updating init scripts
Barracuda [Sat Dec  1 16:20:30 GMT 2012] ==> INFO: Restarting MariaDB
Barracuda [Sat Dec  1 16:20:38 GMT 2012] ==> INFO: Starting Redis, PHP-FPM and Nginx
Barracuda [Sat Dec  1 16:20:48 GMT 2012] ==> INFO: Generating random password for MariaDB
Barracuda [Sat Dec  1 16:20:51 GMT 2012] ==> INFO: OS and services installation completed
Barracuda [Sat Dec  1 16:20:53 GMT 2012] ==> INFO: Installing Aegir Master Instance, please wait...
Barracuda [Sat Dec  1 16:21:40 GMT 2012] ==> INFO: Running hosting-dispatch (1/3), please wait...
Barracuda [Sat Dec  1 16:21:46 GMT 2012] ==> INFO: Running hosting-dispatch (2/3), please wait...
Barracuda [Sat Dec  1 16:21:52 GMT 2012] ==> INFO: Running hosting-dispatch (3/3), please wait...
Barracuda [Sat Dec  1 16:21:57 GMT 2012] ==> INFO: Aegir Master Instance installation completed
Barracuda [Sat Dec  1 16:22:00 GMT 2012] ==> INFO: Installing default SSL Wildcard Nginx Proxy, please wait...
Barracuda [Sat Dec  1 16:22:05 GMT 2012] ==> INFO: Installing Chive MariaDB Manager, please wait...
Barracuda [Sat Dec  1 16:22:09 GMT 2012] ==> INFO: Chive MariaDB Manager installation completed
Barracuda [Sat Dec  1 16:22:11 GMT 2012] ==> INFO: MariaDB final setup
Barracuda [Sat Dec  1 16:22:25 GMT 2012] ==> INFO: MariaDB setup completed
Barracuda [Sat Dec  1 16:22:25 GMT 2012] ==> INFO: You can now log in as root by typing only 'mysql' on the command line
Barracuda [Sat Dec  1 16:22:27 GMT 2012] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Sat Dec  1 16:22:29 GMT 2012] ==> INFO: Congratulations, Aegir backend and frontend have been installed successfully
Barracuda [Sat Dec  1 16:22:29 GMT 2012] ==> NOTE! Please wait 2 minutes before opening the following link in your web browser:
Barracuda [Sat Dec  1 16:22:29 GMT 2012] ==> LINK: http://master.newdev.transitionnetwork.org/user/reset/1/XXXXX
Barracuda [Sat Dec  1 16:22:41 GMT 2012] ==> INFO: Installing csf/lfd firewall, please wait...
Barracuda [Sat Dec  1 16:22:47 GMT 2012] ==> INFO: csf/lfd firewall installation completed
Barracuda [Sat Dec  1 16:22:49 GMT 2012] ==> CARD: Now charging your credit card for this automated install service...
Barracuda [Sat Dec  1 16:22:55 GMT 2012] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Sat Dec  1 16:22:59 GMT 2012] ==> Final post-install cleaning, please wait a moment...
Barracuda [Sat Dec  1 16:23:09 GMT 2012] ==> BYE!
Octopus [Sat Dec  1 16:23:13 GMT 2012] ==> BOA Skynet welcomes you aboard!
Octopus [Sat Dec  1 16:23:16 GMT 2012] ==> INFO: Creating your /root/.o1.octopus.cnf config file
Octopus [Sat Dec  1 16:23:17 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Octopus [Sat Dec  1 16:23:19 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Octopus [Sat Dec  1 16:23:20 GMT 2012] ==> NEW Aegir Satellite Instance setup in progress...
Octopus [Sat Dec  1 16:23:20 GMT 2012] ==> START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your client e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your Aegir control panel for this instance will be available at https://o1.newdev.transitionnetwork.org
  * Your Aegir system user for this instance will be o1
  * This Octopus will use PHP-CLI 5.3 for all sites
  * This Octopus will use PHP-FPM 5.3 both for D6 and D7 sites
  * This Octopus includes platforms: ALL / Unsupported: NO
  * This Octopus options are listed as SSD / Y / 8 C
Do you want to proceed? [Y/n]  Y
Octopus [Sat Dec  1 16:23:53 GMT 2012] ==> 8s before we will continue...
Octopus [Sat Dec  1 16:24:07 GMT 2012] ==> INIT A: Aegir automated install script part A
Octopus [Sat Dec  1 16:24:07 GMT 2012] ==> INFO A: Checking OCTOPUS version, please wait...
Octopus [Sat Dec  1 16:24:07 GMT 2012] ==> INFO: Version test result: OK
Octopus [Sat Dec  1 16:24:07 GMT 2012] ==> INIT A: INIT
Octopus [Sat Dec  1 16:24:07 GMT 2012] ==> INIT A: Shared platforms code v.001 (hot new) will be created
Octopus [Sat Dec  1 16:24:08 GMT 2012] ==> INIT A: Creating directories with correct permissions, please wait...
Octopus [Sat Dec  1 16:24:10 GMT 2012] ==> INIT A: Adding user, please wait...
Octopus [Sat Dec  1 16:24:16 GMT 2012] ==> INIT A: Switching user and running AegirSetupB, please wait...
Octopus [Sat Dec  1 16:24:18 GMT 2012] ==> INIT B: Aegir automated install script part B
Octopus [Sat Dec  1 16:24:18 GMT 2012] ==> INIT B: Creating directories with correct permissions
Octopus [Sat Dec  1 16:24:21 GMT 2012] ==> INIT B: Running standard installer
Octopus [Sat Dec  1 16:24:23 GMT 2012] ==> INIT B: Downloading drush, please wait...
Octopus [Sat Dec  1 16:24:25 GMT 2012] ==> INIT B: Drush seems to be functioning properly
Octopus [Sat Dec  1 16:24:25 GMT 2012] ==> INIT B: Installing provision backend in /data/disk/o1/.drush
Octopus [Sat Dec  1 16:24:26 GMT 2012] ==> INIT B: Downloading Drush and Provision extensions, please wait...
Octopus [Sat Dec  1 16:24:32 GMT 2012] ==> INIT B: Running hostmaster-install, please wait...
Octopus [Sat Dec  1 16:25:14 GMT 2012] ==> INIT B: Running hosting-dispatch (1/3), please wait...
Octopus [Sat Dec  1 16:25:21 GMT 2012] ==> INIT B: Running hosting-dispatch (2/3), please wait...
Octopus [Sat Dec  1 16:25:28 GMT 2012] ==> INIT B: Running hosting-dispatch (3/3), please wait...
Octopus [Sat Dec  1 16:25:30 GMT 2012] ==> INIT B: Simple check if Aegir install is successful
Octopus [Sat Dec  1 16:25:32 GMT 2012] ==> INIT B: Aegir install test result: OK
Octopus [Sat Dec  1 16:25:32 GMT 2012] ==> INIT B: Enhancing Aegir UI, please wait...
Octopus [Sat Dec  1 16:26:16 GMT 2012] ==> INIT A: Aegir Satellite Instance installation completed
Octopus [Sat Dec  1 16:26:20 GMT 2012] ==> INIT A: Creating shared directories, please wait...
Octopus [Sat Dec  1 16:27:15 GMT 2012] ==> INIT A: Switching user and running Platforms build
Octopus [Sat Dec  1 16:27:18 GMT 2012] ==> INIT C: Aegir automated install script part C
Octopus [Sat Dec  1 16:27:18 GMT 2012] ==> INIT C: Shared platforms code v.001 (hot new) will be created
Octopus [Sat Dec  1 16:27:18 GMT 2012] ==> INIT C: Preparing Pressflow +Extra 6.26.2 core, please wait...
Acquia 6.26.2 P.001 - http://bit.ly/acquiadrupal [Y/n] Y
Octopus [Sat Dec  1 16:27:52 GMT 2012] ==> DISTRO: Acquia 6.26.2 P.001 installation in progress...
Octopus [Sat Dec  1 16:28:08 GMT 2012] ==> DISTRO: Acquia 6.26.2 P.001 installation completed
CiviCRM 4.1.6 6.26.2 P.001 - http://civicrm.org [Y/n] n
Octopus [Sat Dec  1 16:28:23 GMT 2012] ==> DISTRO: CiviCRM 4.1.6 6.26.2 P.001 installation skipped
CiviCRM 4.2.6 7.17.1 P.001 - http://civicrm.org [Y/n] n
Octopus [Sat Dec  1 16:28:42 GMT 2012] ==> DISTRO: CiviCRM 4.2.6 7.17.1 P.001 installation skipped
CiviCRM 3.4.8 6.26.2 P.001 - http://civicrm.org [Y/n] n
Octopus [Sat Dec  1 16:29:00 GMT 2012] ==> DISTRO: CiviCRM 3.4.8 6.26.2 P.001 installation skipped
Conference 1.0-rc2 6.26.2 P.001 - http://usecod.com [Y/n] n
Octopus [Sat Dec  1 16:29:06 GMT 2012] ==> DISTRO: Conference 1.0-rc2 6.26.2 P.001 installation skipped
Commerce 1.11.1 7.17.1 P.001 - http://drupalcommerce.org [Y/n] n
Octopus [Sat Dec  1 16:29:11 GMT 2012] ==> DISTRO: Commerce 1.11.1 7.17.1 P.001 installation skipped
Commerce 2.0-rc4 7.17.1 P.001 - http://drupalcommerce.org [Y/n] n
Octopus [Sat Dec  1 16:29:15 GMT 2012] ==> DISTRO: Commerce 2.0-rc4 7.17.1 P.001 installation skipped
Commons 2.10 6.26.2 P.001 - http://acquia.com/drupalcommons [Y/n] n
Octopus [Sat Dec  1 16:29:58 GMT 2012] ==> DISTRO: Commons 2.10 6.26.2 P.001 installation skipped
Drupal 6.26.2 P.001 - http://pressflow.org [Y/n] y
Octopus [Sat Dec  1 16:30:11 GMT 2012] ==> DISTRO: Drupal 6.26.2 D.001 installation in progress...
Octopus [Sat Dec  1 16:30:13 GMT 2012] ==> DISTRO: Drupal 6.26.2 D.001 installation completed
Octopus [Sat Dec  1 16:30:15 GMT 2012] ==> DISTRO: Drupal 6.26.2 S.001 installation in progress...
Octopus [Sat Dec  1 16:30:17 GMT 2012] ==> DISTRO: Drupal 6.26.2 S.001 installation completed
Octopus [Sat Dec  1 16:30:19 GMT 2012] ==> DISTRO: Drupal 6.26.2 P.001 installation in progress...
Octopus [Sat Dec  1 16:30:21 GMT 2012] ==> DISTRO: Drupal 6.26.2 P.001 installation completed
Drupal 7.17.1 P.001 - http://drupal.org/drupal-7.17 [Y/n] n
7.17.1 P.001 installation skipped
Drupal 8.0-dev-120915 P.001 - http://drupal.org [Y/n] n
Octopus [Sat Dec  1 16:31:04 GMT 2012] ==> DISTRO: Drupal 8.0-dev-120915 P.001 installation skipped
ELMS 1.0-b1 6.26.2 P.001 - http://elms.psu.edu [Y/n] n
Octopus [Sat Dec  1 16:31:25 GMT 2012] ==> DISTRO: ELMS 1.0-b1 6.26.2 P.001 installation skipped
Feature Server 1.1 6.26.2 P.001 - http://bit.ly/fservermore [Y/n] n
Octopus [Sat Dec  1 16:31:32 GMT 2012] ==> DISTRO: Feature Server 1.1 6.26.2 P.001 installation skipped
MNews 1.2.2 6.26.2 P.001 - http://managingnews.com [Y/n] n
Octopus [Sat Dec  1 16:31:43 GMT 2012] ==> DISTRO: MNews 1.2.2 6.26.2 P.001 installation skipped
NodeStream 1.5.1 6.26.2 P.001 - http://nodestream.org [Y/n] n
Octopus [Sat Dec  1 16:31:48 GMT 2012] ==> DISTRO: NodeStream 1.5.1 6.26.2 P.001 installation skipped
NodeStream 2.0-b8 7.17.1 P.001 - http://nodestream.org [Y/n] n
Octopus [Sat Dec  1 16:31:53 GMT 2012] ==> DISTRO: NodeStream 2.0-b8 7.17.1 P.001 installation skipped
Open Atrium 1.6.1 6.26.2 P.001 - http://openatrium.com [Y/n] n
Octopus [Sat Dec  1 16:31:58 GMT 2012] ==> DISTRO: Open Atrium 1.6.1 6.26.2 P.001 installation skipped
OpenChurch 1.11-b5 7.17.1 P.001 - http://openchurchsite.com [Y/n] n
Octopus [Sat Dec  1 16:32:02 GMT 2012] ==> DISTRO: OpenChurch 1.11-b5 7.17.1 P.001 installation skipped
Open Deals 1.11 7.17.1 P.001 - http://opendealsapp.com [Y/n] n
Octopus [Sat Dec  1 16:32:05 GMT 2012] ==> DISTRO: Open Deals 1.11 7.17.1 P.001 installation skipped
Open Outreach 1.0-rc6 7.17.1 P.001 - http://openoutreach.org [Y/n] n
Octopus [Sat Dec  1 16:32:09 GMT 2012] ==> DISTRO: Open Outreach 1.0-rc6 7.17.1 P.001 installation skipped
OpenPublish 3.0-b7 7.17.1 P.001 - http://openpublishapp.com [Y/n] n
Octopus [Sat Dec  1 16:32:13 GMT 2012] ==> DISTRO: OpenPublish 3.0-b7 7.17.1 P.001 installation skipped
OpenScholar 2.0-rc1 6.26.2 P.001 - http://openscholar.harvard.edu [Y/n] n
Octopus [Sat Dec  1 16:32:17 GMT 2012] ==> DISTRO: OpenScholar 2.0-rc1 6.26.2 P.001 installation skipped
Panopoly 1.0-rc2 7.17.1 P.001 - http://drupal.org/project/panopoly [Y/n] n
Octopus [Sat Dec  1 16:32:22 GMT 2012] ==> DISTRO: Panopoly 1.0-rc2 7.17.1 P.001 installation skipped
Prosepoint 0.43 6.26.2 P.001 - http://prosepoint.org [Y/n] n
Octopus [Sat Dec  1 16:32:26 GMT 2012] ==> DISTRO: Prosepoint 0.43 6.26.2 P.001 installation skipped
Ubercart 2.10.1 6.26.2 P.001 - http://ubercart.org [Y/n] n
Octopus [Sat Dec  1 16:32:30 GMT 2012] ==> DISTRO: Ubercart 2.10.1 6.26.2 P.001 installation skipped
Ubercart 3.2.1 7.17.1 P.001 - http://ubercart.org [Y/n] n
Octopus [Sat Dec  1 16:32:34 GMT 2012] ==> DISTRO: Ubercart 3.2.1 7.17.1 P.001 installation skipped
Octopus [Sat Dec  1 16:32:36 GMT 2012] ==> INIT C: Removing some old core themes, please wait...
Octopus [Sat Dec  1 16:32:36 GMT 2012] ==> INIT C: Running Platforms Save & Verify tasks, please wait...
Octopus [Sat Dec  1 16:33:02 GMT 2012] ==> INIT A: Platforms installation completed
Octopus [Sat Dec  1 16:33:02 GMT 2012] ==> INIT A: Cleaning up various dot files, please wait...
Octopus [Sat Dec  1 16:33:06 GMT 2012] ==> INIT A: Adding ftps/lshell user
Octopus [Sat Dec  1 16:33:09 GMT 2012] ==> INIT A: Adding symlink to the sites backups
Octopus [Sat Dec  1 16:33:11 GMT 2012] ==> INIT A: Adding symlink to the system drush_make
Octopus [Sat Dec  1 16:33:13 GMT 2012] ==> INIT A: Adding symlink to the system registry_rebuild
Octopus [Sat Dec  1 16:33:15 GMT 2012] ==> INIT A: Adding symlink to the clients directory
Octopus [Sat Dec  1 16:33:17 GMT 2012] ==> INIT A: Creating ftp symlinks
Octopus [Sat Dec  1 16:33:19 GMT 2012] ==> INIT A: Preparing setupmail.txt
Octopus [Sat Dec  1 16:33:21 GMT 2012] ==> INIT A: Sending setup e-mail on init, please wait...
Octopus [Sat Dec  1 16:33:24 GMT 2012] ==> INIT A: New entry added to /data/disk/o1/log/octopus_log.txt
Octopus [Sat Dec  1 16:33:24 GMT 2012] ==> INIT A: Final cleaning, please wait a moment...
Octopus [Sat Dec  1 16:33:26 GMT 2012] ==> INFO: Congratulations, Aegir backend and frontend have been installed successfully
Octopus [Sat Dec  1 16:33:26 GMT 2012] ==> NOTE! Please wait 1 minute before opening the following link in your web browser:
Octopus [Sat Dec  1 16:33:26 GMT 2012] ==> LINK: http://o1.newdev.transitionnetwork.org/user/reset/1/XXX
Octopus [Sat Dec  1 16:33:26 GMT 2012] ==> INIT A: Starting the cron now
Octopus [Sat Dec  1 16:33:26 GMT 2012] ==> INIT A: All done!
Octopus [Sat Dec  1 16:33:26 GMT 2012] ==> BYE!
BOA in-stable completed
Bye

The only mistake I think I made above was to say Y to installing http://bit.ly/acquiadrupal when all we need is the D6 Pressflow AFAIK.

So, all good so far, TODO:

Have a play with Ageir and work out what it can do with it.
Make Ageir use HTTPS only (currently it works with both but allows HTTP),
Redo the configuration done in ticket:466#comment:2 (user accounts, backups etc etc)
Copy across the live data and database, perhaps splitting it into seperate inodedb files in the process and write a script so that this can be easilly done again.
Migrate all sites / applications we are keeping running off kiwi and quince.

hours, totalhours changed

chris — Sat, 01 Dec 2012 17:04:03 GMT

hours changed from 0.0 to 0.35
totalhours changed from 5.2 to 5.55

So, taking items from the the TODO at the end of ticket:466#comment:9, starting with:

Have a play with Ageir and work out what it can do with it.

I found on this page https://o1.newdev.transitionnetwork.org/admin/hosting/features the following option enabled and I have disabled it:

 Sign up form
   Provides a simpler signup form that can be opened to anonymous users.

However it wouldn't accept that:

You cannot disable hosting_signup because aegir_custom_settings depends on it

More looking around is needed, but moving onto the next item in the TODO list:

Make Ageir use HTTPS only (currently it works with both but allows HTTP)

On https://o1.newdev.transitionnetwork.org/hosting/platforms there are these options which are not enabled by default:

  Nginx +SSL servers
    Allow hostmaster to configure NGINX web servers with SSL support.
    Depends on: Web Server (enabled), SSL support (disabled), Nginx servers (enabled), Hosting (enabled), Modal Frame API (enabled), jQuery UI (enabled)
  SSL support
    Allow hostmaster to configure web servers with SSL support
    Depends on: Hosting (enabled), Web Server (enabled), Modal Frame API (enabled), jQuery UI (enabled)
    Required by: Nginx +SSL servers (disabled)

So I have enabled them.

I have noted that the emails sent from the server didn't use TLS so, as expected, that needs sorting out.

hours, totalhours changed

chris — Sat, 01 Dec 2012 18:58:35 GMT

hours changed from 0.0 to 0.2
totalhours changed from 5.55 to 5.75

Testing the install of a site, calling it dev and once it's set up I'll see if I can copy dev data to it.

Clicked "Administration" -> "Add Site and Client" and filled in these values:

Domain name:           dev.newdev.transitionnetwork.org
Installation profile:  Pressflow
Platform:              Drupal 6.26.2 P.001
Language:              English
Database server:       localhost
Domain aliases:        dev.transitionnetwork.org
                       www.dev.transitionnetwork.org
Internal name:         dev
Email address:         chris@webarch.net
Client name:           Transition Network Development Server

This generated:

User warning: Duplicate entry &#039;3-user&#039; for key &#039;uid_up_type&#039; query: INSERT INTO userprotect (uid, up_roles, up_name, up_mail, up_pass, up_status, up_openid, up_delete, up_edit, up_type) VALUES (3, 1, 0, 0, 0, 0, 0, 0, 0, &#039;user&#039;) in _db_query() (line 169 of /data/disk/o1/aegir/distro/001/includes/database.mysqli.inc).
transitionnetwor has been protected from the following editing operations: roles
Task install was added to the queue. Next queue run is 18:53:06+0000, server time is 18:53:05+0000.
Your site (dev.newdev.transitionnetwork.org) has been requested, and the moment it is ready you will receive a mail at chris@webarch.net with instructions on how to log into it.

So, that worked, there is a site here https://dev.newdev.transitionnetwork.org/ and I'll set up SSH so I can copy across the dev site from the dev server and while I'm at it set up some other system stuff.

hours, totalhours changed

chris — Sat, 01 Dec 2012 19:22:42 GMT

hours changed from 0.0 to 0.36
totalhours changed from 5.75 to 6.11

Some things to note if we start again with this process...

The "o1" used for the install results in system users being created like this:

ls -lah /home/
drwx------  6 o1.ftp              users 4.0K Dec  1 18:55 o1.ftp
drwx------  4 o1.transitionnetwor users 4.0K Dec  1 19:00 o1.transitionnetwor

With hindsight perhaps I should have used "tn" as a prefix.

Also note that the username, "o1.transitionnetwor" has been generated from the "Client name" in the add site stage, for this I should have used "Dev" and then we would have a "tn.dev" user.

In the /home/o1.transitionnetwor/ directory we have:

drwx------ 2 o1.transitionnetwor users 4.0K Dec  1 18:55 .drush
drwx------ 2 o1.transitionnetwor users 4.0K Dec  1 18:55 .ssh
lrwxrwxrwx 1 root                root    38 Dec  1 19:05 sites -> /data/disk/o1/clients/transitionnetwor

And in the /data/disk/o1/clients/transitionnetwor directory we have another symlink:

lrwxrwxrwx 1 o1 users   85 Dec  1 18:53 dev.newdev.transitionnetwork.org -> /data/disk/o1/distro/001/pressflow-6.26.2-prod/sites/dev.newdev.transitionnetwork.org

And in /data/disk/o1/distro/001/pressflow-6.26.2-prod/sites/dev.newdev.transitionnetwork.org we have the actual site:

-r--r-----  1 o1 users     61K Dec  1 18:53 drushrc.php
drwxrws--- 15 o1 www-data 4.0K Dec  1 18:53 files
drwxrwsr-x  2 o1 users    4.0K Dec  1 18:53 libraries
-r--r-----  1 o1 www-data   28 Dec  1 18:53 local.settings.php
drwxrwsr-x  2 o1 users    4.0K Dec  1 18:53 modules
drwxrws---  5 o1 www-data 4.0K Dec  1 18:53 private
-r--r-----  1 o1 www-data 3.2K Dec  1 18:53 settings.php
drwxrwsr-x  2 o1 users    4.0K Dec  1 18:53 themes

So, I'm happy installing the mysql data and the files from the live dev site but what isn't clear to me is how we will install the modules and theme while still using Ageir, at the top of settings.php we have:

/**
 * @file Drupal's settings.php file
 *
 * This file was automatically generated by Aegir 6.x-2.x
 * on Sat, 01 Dec 2012 18:53:10 +0000.
 *
 * If it is still managed by Aegir, changes to this file may be
 * lost. If it is not managed by aegir, you should remove this header
 * to avoid further confusion.
 */

hours, totalhours changed

chris — Sun, 02 Dec 2012 15:09:43 GMT

hours changed from 0.0 to 0.1
totalhours changed from 6.11 to 6.21

This looks like the way to import the TN sites to Ageir:

http://community.aegirproject.org/content/importing-complete-drupal-platform

Jim - does that make sense to you?

hours, totalhours changed

jim — Sun, 02 Dec 2012 16:48:19 GMT

hours changed from 0.0 to 0.25
totalhours changed from 6.21 to 6.46

That link is good BUT do not add any sites under the 'base' Aegir install -- all sites should be done via the o1 account and placed in /data/disk/o1/static... See the BOA specific from Omega8cc (the people who maintain it and run it commercially for their hosting):

http://omega8.cc/ <-- see the various links in the 'library' section on the left, lots of handy stuff here.

All interaction with the system for web stuff needs to be via the o1 account (I think its su /bin/bash - o1 to switch user accounts from memory).

Putting stuff in /var/www or /var/aegir will lose some of the good stuff Octopus brings, plus mean we need root or high access accounts for developers when the ability to switch to o1 from our own accounts should be all that's needed on a regular basis.

Finally all that o1.* stuff is also quite neat as it creates limited user accounts for each 'client' in the system... Each with SFTP access, LShell SSH and quotas etc. Less useful for us, but again if we want to add a new site, or run a development environment on the server etc, we can with no risk to the other sites.

hours, totalhours changed

chris — Mon, 03 Dec 2012 14:32:09 GMT

hours changed from 0.0 to 1.0
totalhours changed from 6.46 to 7.46

Thanks Jim.

Following How to add custom platform properly? I have copied the files from Kiwi:

cd /data/disk/o1/static
mkdir dev.transitionnetwork.org
rsync -av rsync -av kiwi:/web/dev.transitionnetwork.org.webarch.net/www/ dev.transitionnetwork.org/
chown -R o1:users dev.transitionnetwork.org/
chmod 775 dev.transitionnetwork.org/

I then added it as a platform using the form here with these values:

Platform Name:           Dev - Kiwi Import
Platform Path:           /data/disk/o1/static/dev.transitionnetwork.org
Drush make option:       Working copy - preserves SCM files
Platform access control: Transition Network Development Server

This created a platform at https://o1.newdev.transitionnetwork.org/hosting/c/platform_DevKiwiImport

I couldn't see how to switch the site I set up at http://dev.newdev.transitionnetwork.org/ to use the new platform so I created a new site at http://newdev.newdev.transitionnetwork.org/ But it doesn't appear to be using the Transition Network templates and I'm not sure what the best way to import the database is, manually, using drush or using a web interface?

I have also created a Octopus admin account for Jim and emailed him the details.

I haven't set up backups and other things for puffin yet as I'm not convinced we won't be starting from scratch again...

hours, totalhours changed

chris — Tue, 04 Dec 2012 20:43:04 GMT

hours changed from 0.0 to 0.22
totalhours changed from 7.46 to 7.68

I have started to read around and document what the BOA stack is and how it works, see wiki:PuffinServer#BarracudaOctopusAgeir.

cc, hours, totalhours changed

chris — Tue, 04 Dec 2012 22:53:44 GMT

cc mark added
hours changed from 0.0 to 1.76
totalhours changed from 7.68 to 9.44

I have just watched four "Drupal Tutorial - Aegir on Omega 8" videos, Introduction, Site Migration, Making a Platform and Configuring, I suggest that Laura and Mark watch these to get an idea about what we are getting into.

I have come to the conclusion that it's going to be Jim, Laura and Mark who are using the BOA web interface to backup, clone and migrate the TN drupal site after building platforms using drush and git.

I think we need to seriously consider having a seperate virtual server for BOA and one for everything else, I really think that mixing the BOA stack with a load of other things on the same virtual server is going to be asking for trouble, so I suggest:

puffin.webarch.net 4GB RAM for BOA
penguin.webarch.net 2GB RAM for Mediawiki, Piwik, Trac, static, archives and everything else

Done this sound like a plan?

chris — Sat, 08 Dec 2012 13:24:22 GMT

Just to remind peole that I think this ticket is stalled at this point:

https://tech.transitionnetwork.org/trac/ticket/466#comment:17

ed — Mon, 10 Dec 2012 22:51:00 GMT

As discussed today - Jim needs to contact Chris and set up/document how it works - also to discuss the puffin/penguin

jim — Fri, 14 Dec 2012 00:34:37 GMT

Not forgotten about this, just not had a moment... Will get my thinking trousers on and reply properly, but the idea for the partition is probably be a good one.

chris — Fri, 14 Dec 2012 12:43:29 GMT

Replying to jim:

the idea for the partition is probably be a good one.

I have just discussed this with Ed and in order that we can get on with things I'll split it into two seperate servers now.

hours, totalhours changed

chris — Sat, 15 Dec 2012 16:34:27 GMT

hours changed from 0.0 to 0.9
totalhours changed from 9.44 to 10.34

So, on a brand new 4GB RAM puffin:

ssh root@puffin.webarch.net
echo "puffin.webarch.net" > /etc/hostname # fqdn for outgoing email
aptitude install screen
screen
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
  BOA Meta Installer setup completed
  Please check docs/INSTALL.txt and docs/UPGRADE.txt for how-to
  Bye
boa in-stable public puffin.webarch.net chris@webarchitects.co.uk tn
Barracuda [Sat Dec 15 15:45:57 GMT 2012] ==> BOA Skynet welcomes you aboard!
Barracuda [Sat Dec 15 15:46:01 GMT 2012] ==> INFO: NORMAL INIT
Barracuda [Sat Dec 15 15:46:02 GMT 2012] ==> INFO: Easy Public Setup Mode Active
Barracuda [Sat Dec 15 15:46:02 GMT 2012] ==> INFO: Creating your /root/.barracuda.cnf config file
Barracuda [Sat Dec 15 15:46:03 GMT 2012] ==> INFO: We need to install wget, axel, aptitude, netcat & git first, please wait...
Barracuda [Sat Dec 15 15:46:48 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Sat Dec 15 15:46:50 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Sat Dec 15 15:46:50 GMT 2012] ==> INFO: Downloading little helpers, please wait...
Barracuda [Sat Dec 15 15:46:53 GMT 2012] ==> INFO: Checking BARRACUDA version...
Barracuda [Sat Dec 15 15:46:53 GMT 2012] ==> INFO: Version test result: OK
Barracuda [Sat Dec 15 15:46:53 GMT 2012] ==> INFO: Installing lsb-release and dnsutils now, please wait...
Barracuda [Sat Dec 15 15:47:19 GMT 2012] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Sat Dec 15 15:47:21 GMT 2012] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.4
Barracuda [Sat Dec 15 15:47:23 GMT 2012] ==> EXIT on error due to invalid DNS setup
  * Your custom _MY_OWNIP is set to "81.95.52.103"
  * Your custom _MY_HOSTN is set to "puffin.webarch.net"
  * Your custom _MY_FRONT is set to "master.puffin.webarch.net"
  * Your _MY_HOSTN and/or _MY_FRONT doesn't match your _MY_OWNIP,
    or your hostname is not set properly yet.
  * Please make sure that below command returns your FQDN hostname "puffin.webarch.net":
  $ uname -n
  * Your server has to have already working FQDN hostname matching your IP address.
    This means you _have to_ configure the dns for your server IP/hostname _before_
    trying to use this install script. Reverse dns is _not_ required.
  * Your hostname appears to be puffin.webarch.net - are you sure it's a valid FQDN hostname?
  * Are you sure your FQDN hostname matches your IP address: 81.95.52.103?
Barracuda [Sat Dec 15 15:47:24 GMT 2012] ==> EXIT on error due to invalid DNS setup
Octopus [Sat Dec 15 15:47:24 GMT 2012] ==> ALRT: Percona server not running!
Octopus [Sat Dec 15 15:47:24 GMT 2012] ==> EXIT: We can't proceed and will exit now
Octopus [Sat Dec 15 15:47:24 GMT 2012] ==> HINT: Please (re)start Percona server and then run Octopus installer again
Octopus [Sat Dec 15 15:47:24 GMT 2012] ==> Bye
BOA in-stable completed
Bye

The problem here was that I hadn't set up a wild card DNS entry for *.puffin.webarch.net, so that was set up and tested:

dig @dns0.webarchitects.co.uk transition.puffin.webarch.net
;; ANSWER SECTION:
transition.puffin.webarch.net. 3600 IN  A       81.95.52.103

And the script was run again:

boa in-stable public puffin.webarch.net chris@webarchitects.co.uk tn
Barracuda [Sat Dec 15 15:55:15 GMT 2012] ==> BOA Skynet welcomes you aboard!
Barracuda [Sat Dec 15 15:55:20 GMT 2012] ==> INFO: NORMAL INIT
Barracuda [Sat Dec 15 15:55:20 GMT 2012] ==> INFO: Easy Public Setup Mode Active
Barracuda [Sat Dec 15 15:55:20 GMT 2012] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Sat Dec 15 15:55:21 GMT 2012] ==> NOTE! Please review all config options displayed below
Barracuda [Sat Dec 15 15:55:21 GMT 2012] ==> NOTE! It will *override* all settings in the Barracuda script
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
_MY_HOSTN="puffin.webarch.net"
_MY_FRONT="master.puffin.webarch.net"
_THIS_DB_HOST=localhost
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV FTP"
_AUTOPILOT=YES
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
_LOAD_LIMIT_ONE=1444
_LOAD_LIMIT_TWO=888
_CUSTOM_CONFIG_CSF=NO
_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
_CUSTOM_CONFIG_PHP_5_3=NO
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
Barracuda [Sat Dec 15 15:55:22 GMT 2012] ==> INFO: We need to install wget, axel, aptitude, netcat & git first, please wait...
Barracuda [Sat Dec 15 15:55:29 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Sat Dec 15 15:55:31 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Sat Dec 15 15:55:31 GMT 2012] ==> INFO: Downloading little helpers, please wait...
Barracuda [Sat Dec 15 15:55:33 GMT 2012] ==> INFO: Checking BARRACUDA version...
Barracuda [Sat Dec 15 15:55:33 GMT 2012] ==> INFO: Version test result: OK
Barracuda [Sat Dec 15 15:55:33 GMT 2012] ==> INFO: Installing lsb-release and dnsutils now, please wait...
Barracuda [Sat Dec 15 15:55:36 GMT 2012] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Sat Dec 15 15:55:39 GMT 2012] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.4
Barracuda [Sat Dec 15 15:55:41 GMT 2012] ==> DNS test OK - your FQDN hostname is puffin.webarch.net and it matches your IP 81.95.52.103
Barracuda [Sat Dec 15 15:55:41 GMT 2012] ==> Please double check whether this is a valid result of my test
Barracuda [Sat Dec 15 15:55:43 GMT 2012] ==> INSTALL START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your IP address appears to be 81.95.52.103 - are you sure it's a valid IP?
  * Your hostname appears to be puffin.webarch.net - are you sure it's a valid FQDN hostname?
Barracuda [Sat Dec 15 15:55:43 GMT 2012] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Sat Dec 15 15:55:45 GMT 2012] ==> INFO: Installing DNS cache pdnsd server, please wait...
Barracuda [Sat Dec 15 15:56:05 GMT 2012] ==> INFO: DNS cache pdnsd server installation completed
Barracuda [Sat Dec 15 15:56:07 GMT 2012] ==> INFO: Updating apt sources
Barracuda [Sat Dec 15 15:56:09 GMT 2012] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Sat Dec 15 15:56:13 GMT 2012] ==> INFO: Running aptitude update, please wait...
Barracuda [Sat Dec 15 15:56:44 GMT 2012] ==> INFO: Installing required libraries and tools
Barracuda [Sat Dec 15 15:56:44 GMT 2012] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Sat Dec 15 16:05:55 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec 15 16:05:57 GMT 2012] ==> INFO: Installed Nginx version nginx/1.2.6, upgrade required
Barracuda [Sat Dec 15 16:05:59 GMT 2012] ==> INFO: Installing Nginx, please wait...
Barracuda [Sat Dec 15 16:07:11 GMT 2012] ==> INFO: Installing MariaDB, please wait...
Barracuda [Sat Dec 15 16:08:31 GMT 2012] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Sat Dec 15 16:08:46 GMT 2012] ==> INFO: Testing Nginx version...
Barracuda [Sat Dec 15 16:08:48 GMT 2012] ==> INFO: Installed Nginx version nginx/1.3.8, no upgrade required
Barracuda [Sat Dec 15 16:08:50 GMT 2012] ==> INFO: Installing /usr/bin/wkhtmltopdf x86_64 version, please wait...
Barracuda [Sat Dec 15 16:08:59 GMT 2012] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Sat Dec 15 16:09:02 GMT 2012] ==> INFO: Installing VnStat monitor, please wait...
Barracuda [Sat Dec 15 16:09:11 GMT 2012] ==> INFO: Installing a few more tools, please wait...
Barracuda [Sat Dec 15 16:09:13 GMT 2012] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
Barracuda [Sat Dec 15 16:09:27 GMT 2012] ==> INFO: Installing PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec 15 16:09:50 GMT 2012] ==> INFO: Installing PhpRedis for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec 15 16:10:21 GMT 2012] ==> INFO: Installing UploadProgress for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec 15 16:10:42 GMT 2012] ==> INFO: Installing JSMin for PHP-FPM 5.3.18, please wait...
Barracuda [Sat Dec 15 16:11:10 GMT 2012] ==> INFO: Installing bzr, please wait...
Barracuda [Sat Dec 15 16:11:54 GMT 2012] ==> INFO: Installing lshell, please wait...
Barracuda [Sat Dec 15 16:12:01 GMT 2012] ==> INFO: Installing latest Pure-FTPd server, please wait...
Barracuda [Sat Dec 15 16:13:29 GMT 2012] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
Barracuda [Sat Dec 15 16:14:28 GMT 2012] ==> INFO: Generating random password for Redis server
Barracuda [Sat Dec 15 16:14:29 GMT 2012] ==> INFO: Updating init scripts
Barracuda [Sat Dec 15 16:14:34 GMT 2012] ==> INFO: Restarting MariaDB
Barracuda [Sat Dec 15 16:14:44 GMT 2012] ==> INFO: Starting Redis, PHP-FPM and Nginx
Barracuda [Sat Dec 15 16:14:55 GMT 2012] ==> INFO: Generating random password for MariaDB
Barracuda [Sat Dec 15 16:14:58 GMT 2012] ==> INFO: OS and services installation completed
Barracuda [Sat Dec 15 16:15:00 GMT 2012] ==> INFO: Installing Aegir Master Instance, please wait...
Barracuda [Sat Dec 15 16:15:58 GMT 2012] ==> INFO: Running hosting-dispatch (1/3), please wait...
Barracuda [Sat Dec 15 16:16:04 GMT 2012] ==> INFO: Running hosting-dispatch (2/3), please wait...
Barracuda [Sat Dec 15 16:16:10 GMT 2012] ==> INFO: Running hosting-dispatch (3/3), please wait...
Barracuda [Sat Dec 15 16:16:19 GMT 2012] ==> INFO: Aegir Master Instance installation completed
Barracuda [Sat Dec 15 16:16:23 GMT 2012] ==> INFO: Installing default SSL Wildcard Nginx Proxy, please wait...
Barracuda [Sat Dec 15 16:16:27 GMT 2012] ==> INFO: Installing Chive MariaDB Manager, please wait...
Barracuda [Sat Dec 15 16:16:34 GMT 2012] ==> INFO: Chive MariaDB Manager installation completed
Barracuda [Sat Dec 15 16:16:36 GMT 2012] ==> INFO: MariaDB final setup
Barracuda [Sat Dec 15 16:16:53 GMT 2012] ==> INFO: MariaDB setup completed
Barracuda [Sat Dec 15 16:16:53 GMT 2012] ==> INFO: You can now log in as root by typing only 'mysql' on the command line
Barracuda [Sat Dec 15 16:16:56 GMT 2012] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Sat Dec 15 16:16:58 GMT 2012] ==> INFO: Congratulations, Aegir backend and frontend have been installed successfully
Barracuda [Sat Dec 15 16:16:58 GMT 2012] ==> NOTE! Please wait 2 minutes before opening the following link in your web browser:
Barracuda [Sat Dec 15 16:16:58 GMT 2012] ==> LINK: http://master.puffin.webarch.net/user/reset/1/1355588148/db2776a99686789d1865b0225c659162
Barracuda [Sat Dec 15 16:17:10 GMT 2012] ==> INFO: Installing csf/lfd firewall, please wait...
Barracuda [Sat Dec 15 16:17:16 GMT 2012] ==> INFO: csf/lfd firewall installation completed
Barracuda [Sat Dec 15 16:17:18 GMT 2012] ==> CARD: Now charging your credit card for this automated install service...
Barracuda [Sat Dec 15 16:17:25 GMT 2012] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Sat Dec 15 16:17:29 GMT 2012] ==> Final post-install cleaning, please wait a moment...
Barracuda [Sat Dec 15 16:17:35 GMT 2012] ==> BYE!
Octopus [Sat Dec 15 16:17:39 GMT 2012] ==> BOA Skynet welcomes you aboard!
Octopus [Sat Dec 15 16:17:42 GMT 2012] ==> INFO: Creating your /root/.tn.octopus.cnf config file
Octopus [Sat Dec 15 16:17:43 GMT 2012] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Octopus [Sat Dec 15 16:17:45 GMT 2012] ==> INFO: GitHub mirror repository will be used for this install
Octopus [Sat Dec 15 16:17:46 GMT 2012] ==> NEW Aegir Satellite Instance setup in progress...
Octopus [Sat Dec 15 16:17:46 GMT 2012] ==> START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your client e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your Aegir control panel for this instance will be available at https://tn.puffin.webarch.net
  * Your Aegir system user for this instance will be tn
  * This Octopus will use PHP-CLI 5.3 for all sites
  * This Octopus will use PHP-FPM 5.3 both for D6 and D7 sites
  * This Octopus includes platforms: ALL / Unsupported: NO
  * This Octopus options are listed as SSD / Y / 8 C
Do you want to proceed? [Y/n] Y
Octopus [Sat Dec 15 16:19:32 GMT 2012] ==> 8s before we will continue...
Octopus [Sat Dec 15 16:19:32 GMT 2012] ==> 8s before we will continue...
Octopus [Sat Dec 15 16:19:51 GMT 2012] ==> INIT A: Aegir automated install script part A
Octopus [Sat Dec 15 16:19:51 GMT 2012] ==> INFO A: Checking OCTOPUS version, please wait...
Octopus [Sat Dec 15 16:19:51 GMT 2012] ==> INFO: Version test result: OK
Octopus [Sat Dec 15 16:19:51 GMT 2012] ==> INIT A: INIT
Octopus [Sat Dec 15 16:19:51 GMT 2012] ==> INIT A: Shared platforms code v.001 (hot new) will be created
Octopus [Sat Dec 15 16:19:51 GMT 2012] ==> INIT A: Creating directories with correct permissions, please wait...
Octopus [Sat Dec 15 16:19:53 GMT 2012] ==> INIT A: Adding user, please wait...
Octopus [Sat Dec 15 16:20:00 GMT 2012] ==> INIT A: Switching user and running AegirSetupB, please wait...
Octopus [Sat Dec 15 16:20:02 GMT 2012] ==> INIT B: Aegir automated install script part B
Octopus [Sat Dec 15 16:20:02 GMT 2012] ==> INIT B: Creating directories with correct permissions
Octopus [Sat Dec 15 16:20:04 GMT 2012] ==> INIT B: Running standard installer
Octopus [Sat Dec 15 16:20:06 GMT 2012] ==> INIT B: Downloading drush, please wait...
Octopus [Sat Dec 15 16:20:09 GMT 2012] ==> INIT B: Drush seems to be functioning properly
Octopus [Sat Dec 15 16:20:09 GMT 2012] ==> INIT B: Installing provision backend in /data/disk/tn/.drush
Octopus [Sat Dec 15 16:20:10 GMT 2012] ==> INIT B: Downloading Drush and Provision extensions, please wait...
Octopus [Sat Dec 15 16:20:16 GMT 2012] ==> INIT B: Running hostmaster-install, please wait...
Octopus [Sat Dec 15 16:21:02 GMT 2012] ==> INIT B: Running hosting-dispatch (1/3), please wait...
Octopus [Sat Dec 15 16:21:09 GMT 2012] ==> INIT B: Running hosting-dispatch (2/3), please wait...
Octopus [Sat Dec 15 16:21:16 GMT 2012] ==> INIT B: Running hosting-dispatch (3/3), please wait...
Octopus [Sat Dec 15 16:21:17 GMT 2012] ==> INIT B: Simple check if Aegir install is successful
Octopus [Sat Dec 15 16:21:19 GMT 2012] ==> INIT B: Aegir install test result: OK
Octopus [Sat Dec 15 16:21:19 GMT 2012] ==> INIT B: Enhancing Aegir UI, please wait...
Octopus [Sat Dec 15 16:22:04 GMT 2012] ==> INIT A: Aegir Satellite Instance installation completed
Octopus [Sat Dec 15 16:22:14 GMT 2012] ==> INIT A: Creating shared directories, please wait...
Octopus [Sat Dec 15 16:24:13 GMT 2012] ==> INIT A: Switching user and running Platforms build
Octopus [Sat Dec 15 16:24:15 GMT 2012] ==> INIT C: Aegir automated install script part C
Octopus [Sat Dec 15 16:24:15 GMT 2012] ==> INIT C: Shared platforms code v.001 (hot new) will be created
Octopus [Sat Dec 15 16:24:15 GMT 2012] ==> INIT C: Preparing Pressflow +Extra 6.26.2 core, please wait...
Acquia 6.26.2 P.001 - http://bit.ly/acquiadrupal [Y/n] n
Octopus [Sat Dec 15 16:24:50 GMT 2012] ==> DISTRO: Acquia 6.26.2 P.001 installation skipped
CiviCRM 4.1.6 6.26.2 P.001 - http://civicrm.org [Y/n] n
Octopus [Sat Dec 15 16:24:55 GMT 2012] ==> DISTRO: CiviCRM 4.1.6 6.26.2 P.001 installation skipped
CiviCRM 4.2.6 7.17.1 P.001 - http://civicrm.org [Y/n] n
Octopus [Sat Dec 15 16:24:59 GMT 2012] ==> DISTRO: CiviCRM 4.2.6 7.17.1 P.001 installation skipped
CiviCRM 3.4.8 6.26.2 P.001 - http://civicrm.org [Y/n] n
Octopus [Sat Dec 15 16:25:02 GMT 2012] ==> DISTRO: CiviCRM 3.4.8 6.26.2 P.001 installation skipped
Conference 1.0-rc2 6.26.2 P.001 - http://usecod.com [Y/n] n
Octopus [Sat Dec 15 16:25:07 GMT 2012] ==> DISTRO: Conference 1.0-rc2 6.26.2 P.001 installation skipped
Commerce 1.11.1 7.17.1 P.001 - http://drupalcommerce.org [Y/n] n
Octopus [Sat Dec 15 16:25:10 GMT 2012] ==> DISTRO: Commerce 1.11.1 7.17.1 P.001 installation skipped
Commerce 2.0-rc4 7.17.1 P.001 - http://drupalcommerce.org [Y/n] n
Octopus [Sat Dec 15 16:25:14 GMT 2012] ==> DISTRO: Commerce 2.0-rc4 7.17.1 P.001 installation skipped
Commons 2.10 6.26.2 P.001 - http://acquia.com/drupalcommons [Y/n] n
Octopus [Sat Dec 15 16:25:18 GMT 2012] ==> DISTRO: Commons 2.10 6.26.2 P.001 installation skipped
Drupal 6.26.2 P.001 - http://pressflow.org [Y/n] y
Octopus [Sat Dec 15 16:25:25 GMT 2012] ==> DISTRO: Drupal 6.26.2 D.001 installation in progress...
Octopus [Sat Dec 15 16:25:27 GMT 2012] ==> DISTRO: Drupal 6.26.2 D.001 installation completed
Octopus [Sat Dec 15 16:25:29 GMT 2012] ==> DISTRO: Drupal 6.26.2 S.001 installation in progress...
Octopus [Sat Dec 15 16:25:32 GMT 2012] ==> DISTRO: Drupal 6.26.2 S.001 installation completed
Octopus [Sat Dec 15 16:25:34 GMT 2012] ==> DISTRO: Drupal 6.26.2 P.001 installation in progress...
Octopus [Sat Dec 15 16:25:36 GMT 2012] ==> DISTRO: Drupal 6.26.2 P.001 installation completed
Drupal 7.17.1 P.001 - http://drupal.org/drupal-7.17 [Y/n] n
Octopus [Sat Dec 15 16:25:43 GMT 2012] ==> DISTRO: Drupal 7.17.1 P.001 installation skipped
Drupal 8.0-dev-120915 P.001 - http://drupal.org [Y/n] n
Octopus [Sat Dec 15 16:25:48 GMT 2012] ==> DISTRO: Drupal 8.0-dev-120915 P.001 installation skipped
ELMS 1.0-b1 6.26.2 P.001 - http://elms.psu.edu [Y/n] n
Octopus [Sat Dec 15 16:25:52 GMT 2012] ==> DISTRO: ELMS 1.0-b1 6.26.2 P.001 installation skipped
Feature Server 1.1 6.26.2 P.001 - http://bit.ly/fservermore [Y/n] n
Octopus [Sat Dec 15 16:25:57 GMT 2012] ==> DISTRO: Feature Server 1.1 6.26.2 P.001 installation skipped
MNews 1.2.2 6.26.2 P.001 - http://managingnews.com [Y/n] n
Octopus [Sat Dec 15 16:26:01 GMT 2012] ==> DISTRO: MNews 1.2.2 6.26.2 P.001 installation skipped
NodeStream 1.5.1 6.26.2 P.001 - http://nodestream.org [Y/n] n
Octopus [Sat Dec 15 16:26:05 GMT 2012] ==> DISTRO: NodeStream 1.5.1 6.26.2 P.001 installation skipped
NodeStream 2.0-b8 7.17.1 P.001 - http://nodestream.org [Y/n] n
Octopus [Sat Dec 15 16:26:09 GMT 2012] ==> DISTRO: NodeStream 2.0-b8 7.17.1 P.001 installation skipped
Open Atrium 1.6.1 6.26.2 P.001 - http://openatrium.com [Y/n] n
Octopus [Sat Dec 15 16:26:17 GMT 2012] ==> DISTRO: Open Atrium 1.6.1 6.26.2 P.001 installation skipped
OpenChurch 1.11-b5 7.17.1 P.001 - http://openchurchsite.com [Y/n] n
Octopus [Sat Dec 15 16:26:43 GMT 2012] ==> DISTRO: OpenChurch 1.11-b5 7.17.1 P.001 installation skipped
Open Deals 1.11 7.17.1 P.001 - http://opendealsapp.com [Y/n] n
Octopus [Sat Dec 15 16:26:47 GMT 2012] ==> DISTRO: Open Deals 1.11 7.17.1 P.001 installation skipped
Open Outreach 1.0-rc6 7.17.1 P.001 - http://openoutreach.org [Y/n] n
Octopus [Sat Dec 15 16:26:50 GMT 2012] ==> DISTRO: Open Outreach 1.0-rc6 7.17.1 P.001 installation skipped
OpenPublish 3.0-b7 7.17.1 P.001 - http://openpublishapp.com [Y/n] n
Octopus [Sat Dec 15 16:26:54 GMT 2012] ==> DISTRO: OpenPublish 3.0-b7 7.17.1 P.001 installation skipped
OpenScholar 2.0-rc1 6.26.2 P.001 - http://openscholar.harvard.edu [Y/n] n
Octopus [Sat Dec 15 16:26:57 GMT 2012] ==> DISTRO: OpenScholar 2.0-rc1 6.26.2 P.001 installation skipped
Panopoly 1.0-rc2 7.17.1 P.001 - http://drupal.org/project/panopoly [Y/n] n
Octopus [Sat Dec 15 16:27:01 GMT 2012] ==> DISTRO: Panopoly 1.0-rc2 7.17.1 P.001 installation skipped
Prosepoint 0.43 6.26.2 P.001 - http://prosepoint.org [Y/n] n
Octopus [Sat Dec 15 16:27:04 GMT 2012] ==> DISTRO: Prosepoint 0.43 6.26.2 P.001 installation skipped
Ubercart 2.10.1 6.26.2 P.001 - http://ubercart.org [Y/n] n
Octopus [Sat Dec 15 16:27:08 GMT 2012] ==> DISTRO: Ubercart 2.10.1 6.26.2 P.001 installation skipped
Ubercart 3.2.1 7.17.1 P.001 - http://ubercart.org [Y/n] n
Octopus [Sat Dec 15 16:27:11 GMT 2012] ==> DISTRO: Ubercart 3.2.1 7.17.1 P.001 installation skipped
Octopus [Sat Dec 15 16:27:13 GMT 2012] ==> INIT C: Removing some old core themes, please wait...
Octopus [Sat Dec 15 16:27:13 GMT 2012] ==> INIT C: Running Platforms Save & Verify tasks, please wait...
Octopus [Sat Dec 15 16:27:33 GMT 2012] ==> INIT A: Platforms installation completed
Octopus [Sat Dec 15 16:27:33 GMT 2012] ==> INIT A: Cleaning up various dot files, please wait...
Octopus [Sat Dec 15 16:27:38 GMT 2012] ==> INIT A: Adding ftps/lshell user
Octopus [Sat Dec 15 16:27:40 GMT 2012] ==> INIT A: Adding symlink to the sites backups
Octopus [Sat Dec 15 16:27:42 GMT 2012] ==> INIT A: Adding symlink to the system drush_make
Octopus [Sat Dec 15 16:27:44 GMT 2012] ==> INIT A: Adding symlink to the system registry_rebuild
Octopus [Sat Dec 15 16:27:47 GMT 2012] ==> INIT A: Adding symlink to the clients directory
Octopus [Sat Dec 15 16:27:49 GMT 2012] ==> INIT A: Creating ftp symlinks
Octopus [Sat Dec 15 16:27:51 GMT 2012] ==> INIT A: Preparing setupmail.txt
Octopus [Sat Dec 15 16:27:53 GMT 2012] ==> INIT A: Sending setup e-mail on init, please wait...
Octopus [Sat Dec 15 16:27:55 GMT 2012] ==> INIT A: New entry added to /data/disk/tn/log/octopus_log.txt
Octopus [Sat Dec 15 16:27:55 GMT 2012] ==> INIT A: Final cleaning, please wait a moment...
Octopus [Sat Dec 15 16:27:58 GMT 2012] ==> INFO: Congratulations, Aegir backend and frontend have been installed successfully
Octopus [Sat Dec 15 16:27:58 GMT 2012] ==> NOTE! Please wait 1 minute before opening the following link in your web browser:
Octopus [Sat Dec 15 16:27:58 GMT 2012] ==> LINK: http://tn.puffin.webarch.net/user/reset/1/XXX
Octopus [Sat Dec 15 16:27:58 GMT 2012] ==> INIT A: Starting the cron now
Octopus [Sat Dec 15 16:27:58 GMT 2012] ==> INIT A: All done!
Octopus [Sat Dec 15 16:27:58 GMT 2012] ==> BYE!
BOA in-stable completed
Bye

I have forwarded the one time login links for https://tn.puffin.webarch.net/ and https://master.puffin.webarch.net/ to Jim so he can sort out the BOA side of things.

Next I'll sort out the SSL certificate for the server, add ssh public keys to the root account for Jim and myself and also sort out backups.

Than I'll concentrate on PenguinServer ticket:470 for a while.

hours, totalhours changed

chris — Sat, 15 Dec 2012 16:59:40 GMT

hours changed from 0.0 to 0.5
totalhours changed from 10.34 to 10.84

Generate a ssh key and then add it to kiwi and quince:

cd /root/
ssh-keygen -t rsa -n 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

The key has been added to /root/authorized_keys on quince and kiwi with the IP address for extra security:

from="81.95.52.103" ssh-rsa AAAA...

And ~/.ssh/config was created with this in to make ssh'ing simpler:

Host kiwi
  Hostname kiwi.webarch.net
  User root
Host quince
  Hostname quince.webarch.net
  User root

Fix vim syntax highlighting:

echo "syntax on" >> ~/.vimrc

Fix default editor:

echo "export EDITOR='vim'" >> ~/.bashrc

Install backupninja:

aptitude install backupninja

Created a /root/Changelog to document system updates:

2012-12-15      chris
        *       backupninja : installed

Install metche to keep track of the Changelog:

aptitude install metche

Installed a couple of scripts to /usr/local/bin for updating Changelog and running aptitude, a-up and logchanges.

Sorted out root email:

echo "root: chris@webarchitects.co.uk" >> /etc/aliases
newaliases
/etc/init.d/postfix restart

hours, totalhours changed

chris — Sat, 15 Dec 2012 17:22:21 GMT

hours changed from 0.0 to 0.15
totalhours changed from 10.84 to 10.99

I have set up backupninja and done a test run, the scripts are in /etc/backup.d/.

Next I'll sort the certificate and postfix TLS out.

hours, totalhours changed

chris — Sat, 15 Dec 2012 20:25:50 GMT

hours changed from 0.0 to 1.45
totalhours changed from 10.99 to 12.44

SSL cert copied across:

mkdir /etc/ssl/transitionnetwork.org
rsync -av quince:/etc/ssl/transitionnetwork.org/ /etc/ssl/transitionnetwork.org/

Then the existing TLS configuration section in /etc/postfix/main.cf was replaced with:

smtpd_tls_cert_file = /etc/ssl/transitionnetwork.org/transitionnetwork.org.crt
smtpd_tls_key_file =  /etc/ssl/transitionnetwork.org/transitionnetwork.org.key
smtpd_tls_CAfile =    /etc/ssl/transitionnetwork.org/gandi.pem
smtpd_use_tls = yes
smtpd_enforce_tls = no
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_session_cache_timeout = 3600s
smtpd_tls_cipherlist = EDH:!EXP:!LOW
smtpd_tls_ask_ccert = yes
smtpd_tls_req_ccert = no
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtp_enforce_tls = no
smtp_tls_cert_file = /etc/ssl/transitionnetwork.org/transitionnetwork.org.crt
smtp_tls_key_file =  /etc/ssl/transitionnetwork.org/transitionnetwork.org.key
smtp_tls_CAfile =    /etc/ssl/transitionnetwork.org/gandi.pem
smtp_tls_loglevel = 1
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_session_cache_timeout = 3600s

And mutt was used to test TLS.

The gandi.pem file was created like this:

wget http://crt.gandi.net/GandiStandardSSLCA.crt -O GandiStandardSSLCA.crt
wget http://crt.usertrust.com/UTNAddTrustServer_CA.crt -O UTNAddTrustServer_CA.crt
wget http://crt.usertrust.com/AddTrustExternalCARoot.crt -O AddTrustExternalCARoot.crt
openssl x509 -inform DER -in GandiStandardSSLCA.crt -out GandiStandardSSLCA.pem
openssl x509 -inform DER -in AddTrustExternalCARoot.crt -out AddTrustExternalCARoot.pem
openssl x509 -inform DER -in UTNAddTrustServer_CA.crt -out UTNAddTrustServer_CA.pem
cat GandiStandardSSLCA.pem > gandi.pem
cat UTNAddTrustServer_CA.pem >> gandi.pem
cat AddTrustExternalCARoot.pem >> gandi.pem

And for Aegir and nginx / ftp the answer was found here:

https://github.com/omega8cc/nginx-for-drupal/issues/8#issuecomment-422955

cd /etc/ssl/private/
mv nginx-wild-ssl.crt nginx-wild-ssl.crt.old
mv nginx-wild-ssl.key nginx-wild-ssl.key.old
mv pure-ftpd.pem pure-ftpd.pem.old
ln -s ../transitionnetwork.org/transitionnetwork.org.key nginx-wild-ssl.key
ln -s ../transitionnetwork.org/transitionnetwork.org.crt nginx-wild-ssl.crt
ln -s ../transitionnetwork.org/transitionnetwork.org.pem pure-ftpd.pem

After nginx was restarted the gandi wild card cert was being used.

backupninja has been documented wiki:PuffinServer#Backupninja

postfix documentation: wiki:PuffinServer#Postfix

nginx documentation: wiki:PuffinServer#Nginx

I think that is me done on puffin, unless I have forgotten anything, as far as I'm aware the rest of the configuration and management of the sites on the server can be done via the web interface. So I'll next do some work on penguin, ticket:470

hours, totalhours changed

chris — Tue, 08 Jan 2013 12:57:05 GMT

hours changed from 0.0 to 0.6
totalhours changed from 12.44 to 13.04

Updating BOA, following http://drupalcode.org/project/barracuda.git/blob/HEAD:/docs/UPGRADE.txt

This has also been documented here wiki:PuffinServer#UpgradingBOA

screen
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
  BOA Meta Installer setup completed
  Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
  Bye

Updating barracuda (output trimmed):

barracuda up-stable
  Barracuda [Tue Jan  8 12:32:42 GMT 2013] ==> BOA Skynet welcomes you aboard!
  Barracuda [Tue Jan  8 12:32:46 GMT 2013] ==> INFO: UPGRADE
  Barracuda [Tue Jan  8 12:32:46 GMT 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
  Barracuda [Tue Jan  8 12:32:47 GMT 2013] ==> NOTE! Please review all config options displayed below
  Barracuda [Tue Jan  8 12:32:47 GMT 2013] ==> NOTE! It will *override* all settings in the Barracuda script
  Barracuda [Tue Jan  8 12:32:49 GMT 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
  Barracuda [Tue Jan  8 12:32:51 GMT 2013] ==> INFO: GitHub mirror repository will be used for this install
  Barracuda [Tue Jan  8 12:32:51 GMT 2013] ==> INFO: Downloading little helpers, please wait...
  Barracuda [Tue Jan  8 12:32:53 GMT 2013] ==> INFO: Checking BARRACUDA version...
  Barracuda [Tue Jan  8 12:32:53 GMT 2013] ==> INFO: Version test result: OK
  Barracuda [Tue Jan  8 12:32:53 GMT 2013] ==> INFO: Checking your Debian or Ubuntu version...
  Barracuda [Tue Jan  8 12:32:56 GMT 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.5
  Barracuda [Tue Jan  8 12:33:00 GMT 2013] ==> INFO: Cleaning up temp files in /var/opt/
  Barracuda [Tue Jan  8 12:33:02 GMT 2013] ==> INFO: Updating apt sources
  Barracuda [Tue Jan  8 12:33:04 GMT 2013] ==> INFO: We will use Debian mirror ftp.debian.org
  Barracuda [Tue Jan  8 12:33:08 GMT 2013] ==> INFO: Running aptitude update, please wait...
  Barracuda [Tue Jan  8 12:33:17 GMT 2013] ==> INFO: Upgrading required libraries and tools
  Barracuda [Tue Jan  8 12:33:17 GMT 2013] ==> INFO: Now waiting 60 seconds for php-fpm pre-upgrade shutdown...
  Barracuda [Tue Jan  8 12:34:20 GMT 2013] ==> NOTE! This step may take a few minutes, please wait...
  Barracuda [Tue Jan  8 12:35:09 GMT 2013] ==> INFO: Testing Nginx version...
  Barracuda [Tue Jan  8 12:35:11 GMT 2013] ==> INFO: Upgrading Nginx, please wait...
  Barracuda [Tue Jan  8 12:36:31 GMT 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
  Barracuda [Tue Jan  8 12:37:15 GMT 2013] ==> INFO: Testing Nginx version...
  Barracuda [Tue Jan  8 12:37:17 GMT 2013] ==> INFO: Installed Nginx version nginx/1.3.9, no upgrade required
  Barracuda [Tue Jan  8 12:37:19 GMT 2013] ==> INFO: Checking SMTP connections, please wait...
  Barracuda [Tue Jan  8 12:37:21 GMT 2013] ==> INFO: Upgrading a few more tools, please wait...
  Barracuda [Tue Jan  8 12:37:24 GMT 2013] ==> INFO: Checking if PHP upgrade is available
  Barracuda [Tue Jan  8 12:37:26 GMT 2013] ==> INFO: Installed PHP version 5.3.20-1~dotdeb.0, no upgrade required
  Barracuda [Tue Jan  8 12:37:26 GMT 2013] ==> INFO: Installing PhpRedis upgrade for PHP-FPM 5.3.20, please wait...
  Barracuda [Tue Jan  8 12:37:59 GMT 2013] ==> INFO: Installing JSMin upgrade for PHP-FPM 5.3.20, please wait...
  Barracuda [Tue Jan  8 12:38:26 GMT 2013] ==> INFO: Installing latest Pure-FTPd server, please wait...
  Barracuda [Tue Jan  8 12:39:58 GMT 2013] ==> INFO: Installed Redis version 2.6.4, upgrade/rebuild required
  Barracuda [Tue Jan  8 12:40:00 GMT 2013] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
  Barracuda [Tue Jan  8 12:41:08 GMT 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
  Barracuda [Tue Jan  8 12:41:18 GMT 2013] ==> INFO: OS and services upgrade completed
  Barracuda [Tue Jan  8 12:41:20 GMT 2013] ==> INFO: Restarting MariaDB server, please wait...
  Barracuda [Tue Jan  8 12:42:35 GMT 2013] ==> INFO: Running Aegir Master Instance upgrade
  Barracuda [Tue Jan  8 12:42:37 GMT 2013] ==> INFO: Test OK, we can proceed with Hostmaster upgrade
  Barracuda [Tue Jan  8 12:42:37 GMT 2013] ==> INFO: Downloading drush, please wait...
  Barracuda [Tue Jan  8 12:42:41 GMT 2013] ==> INFO: Drush seems to be functioning properly
  Barracuda [Tue Jan  8 12:42:41 GMT 2013] ==> INFO: Moving old directories
  Barracuda [Tue Jan  8 12:42:41 GMT 2013] ==> INFO: Installing provision backend in /var/aegir/.drush
  Barracuda [Tue Jan  8 12:42:42 GMT 2013] ==> INFO: Downloading Drush and Provision extensions, please wait...
  Barracuda [Tue Jan  8 12:42:44 GMT 2013] ==> INFO: Running hostmaster-migrate, please wait...
  Barracuda [Tue Jan  8 12:43:47 GMT 2013] ==> INFO: Aegir Master Instance upgrade completed
  Barracuda [Tue Jan  8 12:43:48 GMT 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
  Barracuda [Tue Jan  8 12:43:48 GMT 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
  Barracuda [Tue Jan  8 12:43:54 GMT 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
  Barracuda [Tue Jan  8 12:43:58 GMT 2013] ==> Final post-upgrade cleaning, please wait a moment...
  Barracuda [Tue Jan  8 12:44:10 GMT 2013] ==> BYE!
  BARRACUDA upgrade completed
  Bye

Updating octopus (output trimmed):

octopus up-stable all
  Octopus [Tue Jan  8 12:45:20 GMT 2013] ==> BOA Skynet welcomes you aboard!
  Octopus [Tue Jan  8 12:45:23 GMT 2013] ==> INFO: Reading your /root/.tn.octopus.cnf config file
  Octopus [Tue Jan  8 12:45:43 GMT 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
  Octopus [Tue Jan  8 12:45:45 GMT 2013] ==> INFO: GitHub mirror repository will be used for this install
  Octopus [Tue Jan  8 12:45:46 GMT 2013] ==> UPGRADE in progress...
  Octopus [Tue Jan  8 12:45:46 GMT 2013] ==> START -> checkpoint:
    * Your Aegir control panel for this instance is available at https://tn.puffin.webarch.net
    * Your Aegir system user for this instance is tn
    * This Octopus will use PHP-CLI 5.3 for all sites
    * This Octopus will use PHP-FPM 5.3 both for D6 and D7 sites
    * This Octopus includes platforms: ALL / Unsupported: NO
    * This Octopus options are listed as SSD / Y / 8 C
  Do you want to proceed? [Y/n] y
  Octopus [Tue Jan  8 12:46:03 GMT 2013] ==> 8s before we will continue...
  Octopus [Tue Jan  8 12:46:17 GMT 2013] ==> INIT A: Aegir automated install script part A
  Octopus [Tue Jan  8 12:46:17 GMT 2013] ==> INFO A: Checking OCTOPUS version, please wait...
  Octopus [Tue Jan  8 12:46:17 GMT 2013] ==> INFO: Version test result: OK
  Octopus [Tue Jan  8 12:46:17 GMT 2013] ==> UPGRADE A: UPGRADE
  Octopus [Tue Jan  8 12:46:20 GMT 2013] ==> UPGRADE A: Shared platforms code v.002 (hot new) will be created
  Octopus [Tue Jan  8 12:46:20 GMT 2013] ==> UPGRADE A: Creating directories with correct permissions, please wait...
  Do you want to upgrade this Aegir Satellite Instance? [Y/n] y
  Octopus [Tue Jan  8 12:46:35 GMT 2013] ==> UPGRADE A: Switching user and running AegirSetupB, please wait...
  Octopus [Tue Jan  8 12:46:38 GMT 2013] ==> INIT B: Aegir automated install script part B
  Octopus [Tue Jan  8 12:46:38 GMT 2013] ==> UPGRADE B: Creating directories with correct permissions
  Octopus [Tue Jan  8 12:46:40 GMT 2013] ==> UPGRADE B: Running standard installer
  Octopus [Tue Jan  8 12:46:42 GMT 2013] ==> UPGRADE B: Downloading drush, please wait...
  Octopus [Tue Jan  8 12:46:45 GMT 2013] ==> UPGRADE B: Drush seems to be functioning properly
  Octopus [Tue Jan  8 12:46:45 GMT 2013] ==> UPGRADE B: Installing provision backend in /data/disk/tn/.drush
  Octopus [Tue Jan  8 12:46:46 GMT 2013] ==> UPGRADE B: Downloading Drush and Provision extensions, please wait...
  Octopus [Tue Jan  8 12:46:50 GMT 2013] ==> UPGRADE B: Hostmaster STATUS: upgrade start
  Octopus [Tue Jan  8 12:46:52 GMT 2013] ==> UPGRADE B: Running hostmaster-migrate, please wait...
  Octopus [Tue Jan  8 12:47:52 GMT 2013] ==> UPGRADE B: Hostmaster STATUS: upgrade completed
  Octopus [Tue Jan  8 12:47:52 GMT 2013] ==> UPGRADE B: Simple check if Aegir upgrade is successful
  Octopus [Tue Jan  8 12:47:54 GMT 2013] ==> UPGRADE B: Aegir upgrade test result: OK
  Octopus [Tue Jan  8 12:47:54 GMT 2013] ==> UPGRADE B: Enhancing Aegir UI, please wait...
  Octopus [Tue Jan  8 12:48:38 GMT 2013] ==> UPGRADE A: Aegir Satellite Instance upgrade completed
  Do you want to install some latest, ready to use platforms? [Y/n] n
  Octopus [Tue Jan  8 12:49:40 GMT 2013] ==> UPGRADE A: No new platforms added this time
  Octopus [Tue Jan  8 12:49:42 GMT 2013] ==> UPGRADE A: Cleaning up various dot files, please wait...
  /opt/tmp/nginx-for-drupal/aegir/scripts/AegirSetupA.sh.txt: line 1122: cd: /data/all/002: No such file or directory
  touch: cannot touch `/data/all/002/dot-files-ctrl-BOA-2.0.5': No such file or directory
  ln: creating symbolic link `/home/tn.ftp/platforms/002/keys': No such file or directory
  touch: cannot touch `/data/all/002/javascript_aggregator.out.txt': No such file or directory
  Octopus [Tue Jan  8 12:49:46 GMT 2013] ==> UPGRADE A: Preparing setupmail.txt
  Octopus [Tue Jan  8 12:49:49 GMT 2013] ==> UPGRADE A: New entry added to /data/disk/tn/log/octopus_log.txt
  Octopus [Tue Jan  8 12:49:49 GMT 2013] ==> UPGRADE A: Final cleaning, please wait a moment...
  Octopus [Tue Jan  8 12:49:51 GMT 2013] ==> UPGRADE A: Adding missing strongarm module to old fserver platforms
  Octopus [Tue Jan  8 12:49:51 GMT 2013] ==> UPGRADE A: Removed /data/all/001/o_contrib/backup_migrate
  Octopus [Tue Jan  8 12:49:51 GMT 2013] ==> UPGRADE A: Removed /data/all/001/o_contrib/purge
  Octopus [Tue Jan  8 12:49:51 GMT 2013] ==> UPGRADE A: Removed /data/all/001/o_contrib/expire
  Octopus [Tue Jan  8 12:49:55 GMT 2013] ==> UPGRADE A: New modules in /data/all/001/o_contrib: purge expire backup_migrate
  Octopus [Tue Jan  8 12:49:55 GMT 2013] ==> UPGRADE A: Removed /data/all/001/o_contrib_seven/purge
  Octopus [Tue Jan  8 12:49:55 GMT 2013] ==> UPGRADE A: Removed /data/all/001/o_contrib_seven/expire
  Octopus [Tue Jan  8 12:49:55 GMT 2013] ==> UPGRADE A: Removed /data/all/001/o_contrib_seven/filefield_nginx_progress
  Octopus [Tue Jan  8 12:49:59 GMT 2013] ==> UPGRADE A: New modules in /data/all/001/o_contrib_seven: filefield_nginx_progress purge expire
  Octopus [Tue Jan  8 12:49:59 GMT 2013] ==> UPGRADE A: Starting the cron now
  Octopus [Tue Jan  8 12:49:59 GMT 2013] ==> UPGRADE A: All done!
  Octopus [Tue Jan  8 12:49:59 GMT 2013] ==> BYE!
  Done for /data/disk/tn
  OCTOPUS upgrade completed
  Bye

chris — Tue, 08 Jan 2013 14:40:50 GMT

A script has been created for rsyncing files from NewLiveServer to puffin, see ticket:472 and a link to this ticket has also been added to the wiki page at PuffinServer.

hours, totalhours changed

chris — Sat, 19 Jan 2013 13:56:33 GMT

hours changed from 0.0 to 0.25
totalhours changed from 13.04 to 13.29

Jim was accidentially blocked by the firewall on puffin, so I have unblocked him and documented how to do this here wiki:PuffinServer#CSFLDF

jim — Wed, 23 Jan 2013 22:00:54 GMT

The server is plagued by segfaults, so I'm going to run the update again to see if that brings a better/patched/more stable version of MySQL or PHP-FPM... Following usual process as documented in above ticket.

puffin:~# tail /var/log/syslog
Jan 23 21:41:01 puffin /USR/SBIN/CRON[1501]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jan 23 21:41:32 puffin mysqld: 130123 21:41:32 [Warning] Aborted connection 140990 to db: 'transitionnetwor' user: 'transitionnetwor' host: 'localhost' (Unknown error)
Jan 23 21:41:52 puffin mysqld: 130123 21:41:52 [Warning] Aborted connection 141009 to db: 'transitionnetwor' user: 'transitionnetwor' host: 'localhost' (Unknown error)
Jan 23 21:42:01 puffin /USR/SBIN/CRON[1935]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jan 23 21:42:01 puffin /USR/SBIN/CRON[1936]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jan 23 21:42:01 puffin /USR/SBIN/CRON[1938]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jan 23 21:42:01 puffin /USR/SBIN/CRON[1939]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jan 23 21:42:32 puffin postfix/pickup[28300]: 8AB4225C018: uid=0 from=<root>
Jan 23 21:42:32 puffin postfix/cleanup[2226]: 8AB4225C018: message-id=<20130123214232.8AB4225C018@puffin.webarch.net>
Jan 23 21:42:32 puffin postfix/qmgr[3817]: 8AB4225C018: from=<root@puffin.webarch.net>, size=622, nrcpt=1 (queue active)

Since the issue seems to be connection to MySQL, I've altered #~/.barracuda.cnf to remove the IP address (since this is the recommended setting), the hunch being it's trying to use the external IP for MySQL connections, rather than 127.0.0.1/localhost.

jim — Wed, 23 Jan 2013 22:05:03 GMT

Have also added CGP Collectd monitor to the install by changing .barracuda.cnf:

_XTRAS_LIST="PDS CSF CHV FTP CGP"

(added 'CGP').

Also Chris, do we want FTP server running if we have SSH? I've personally disabled it on mine and feel that, unless we had multiple clients, it's adding nothing other than a possible attack vector... The solution is to remove 'FTP' from above and re-run the usual install.

Doing upgrade now with new settings.

jim — Wed, 23 Jan 2013 22:28:48 GMT

Updated completed, Collectd installed, but no updates... relevant sections:

Testing Nginx version...
Barracuda [Wed Jan 23 22:06:51 GMT 2013] ==> INFO: Installed Nginx version nginx/1.3.9, no upgrade required
Barracuda [Wed Jan 23 22:06:53 GMT 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Wed Jan 23 22:07:38 GMT 2013] ==> INFO: Testing Nginx version...
Barracuda [Wed Jan 23 22:07:40 GMT 2013] ==> INFO: Installed Nginx version nginx/1.3.9, no upgrade required
Barracuda [Wed Jan 23 22:07:42 GMT 2013] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Wed Jan 23 22:07:45 GMT 2013] ==> INFO: Upgrading a few more tools, please wait...
Barracuda [Wed Jan 23 22:07:47 GMT 2013] ==> INFO: Checking if PHP upgrade is available
Barracuda [Wed Jan 23 22:07:50 GMT 2013] ==> INFO: Installed PHP version 5.3.20-1~dotdeb.0, no upgrade required
Barracuda [Wed Jan 23 22:07:54 GMT 2013] ==> INFO: Installed Redis version 2.6.7, no upgrade/rebuild required
Barracuda [Wed Jan 23 22:07:59 GMT 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
Barracuda [Wed Jan 23 22:08:10 GMT 2013] ==> INFO: OS and services upgrade completed
Barracuda [Wed Jan 23 22:08:12 GMT 2013] ==> INFO: Restarting MariaDB server, please wait...
Barracuda [Wed Jan 23 22:09:26 GMT 2013] ==> INFO: Running Aegir Master Instance upgrade
Barracuda [Wed Jan 23 22:09:29 GMT 2013] ==> INFO: Testing previous install...
Barracuda [Wed Jan 23 22:09:29 GMT 2013] ==> INFO: Test OK, we can proceed with Hostmaster upgrade
Barracuda [Wed Jan 23 22:09:29 GMT 2013] ==> INFO: Downloading drush, please wait...
Barracuda [Wed Jan 23 22:09:31 GMT 2013] ==> INFO: Drush seems to be functioning properly
Barracuda [Wed Jan 23 22:09:31 GMT 2013] ==> INFO: Moving old directories
Barracuda [Wed Jan 23 22:09:32 GMT 2013] ==> INFO: Installing provision backend in /var/aegir/.drush
Barracuda [Wed Jan 23 22:09:33 GMT 2013] ==> INFO: Downloading Drush and Provision extensions, please wait...
Barracuda [Wed Jan 23 22:09:37 GMT 2013] ==> INFO: Running hostmaster-migrate, please wait...
Barracuda [Wed Jan 23 22:11:01 GMT 2013] ==> INFO: Aegir Master Instance upgrade completed
Barracuda [Wed Jan 23 22:11:02 GMT 2013] ==> INFO: Installing Collectd Graph Panel, please wait...
Barracuda [Wed Jan 23 22:12:19 GMT 2013] ==> INFO: Collectd Graph Panel installation completed
Barracuda [Wed Jan 23 22:12:22 GMT 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt

NOTE: CGP/Collectd is now available at: http://cgp.master.puffin.webarch.net/ I've documented this here https://tech.transitionnetwork.org/trac/wiki/PuffinServer#CGPCollectdsystemmonitor

So there's no change, and I still get repeated 502 'Bad Gateway' errors when trying to use the server. Syslog shows:

Jan 23 22:14:38 puffin mysqld: 130123 22:14:38 [Warning] Aborted connection 134 to db: 'v01newdevtransit' user: 'v01newdevtransit' host: 'localhost' (Unknown error)

However, a restart of mysql seems to have fixed it -- as it did on Sunday and Monday, so the result it temporary... So we have a buggy MySQL or PHP-FPM setup somehow.

I'll trawl the logs, but it might be worth re-running the install with one of these options set to 'YES' in the .barracuda.cnf -- the latter being the 'nuclear' option. _NGX_FORCE_REINSTALL _PHP_FORCE_REINSTALL _SQL_FORCE_REINSTALL _FULL_FORCE_REINSTALL

For now I'll look at the logs and raise another ticket for this if this continues to happen.

jim — Wed, 23 Jan 2013 22:51:44 GMT

More syslog

Jan 23 22:30:49 puffin kernel: [3395287.943609] php-fpm[12862]: segfault at 4a9f2120 ip 00000000006d697d sp 00007fff1e8855b0 error 4 in php5-fpm[400000+826000]

And I'm seeing a lot of errors in tail -n10000 /var/syslog | grep drupal

Jan 23 06:59:10 puffin drupal: http://v01.newdev.transitionnetwork.org|1358924350|cron|127.0.0.1|http://v01.newdev.transitionnetwork.org/index.php||0||Cron has been running for more than an hour and is most likely stuck.
Jan 23 07:24:12 puffin drupal: http://transitionnetwork.org|1358925852|cron|127.0.0.1|http://transitionnetwork.org/index.php||0||Cron run completed.
Jan 23 07:59:18 puffin drupal: http://v01.newdev.transitionnetwork.org|1358927958|aggregator|127.0.0.1|http://v01.newdev.transitionnetwork.org/index.php||0||The feed from REconomy Project Feed seems to be broken, due to an error "Invalid document end" on line 1.
Jan 23 07:59:55 puffin drupal: http://v01.newdev.transitionnetwork.org|1358927995|php|66.249.76.194|http://v01.newdev.transitionnetwork.org/||0||Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 &&
Jan 23 07:59:55 puffin drupal: http://v01.newdev.transitionnetwork.org|1358927995|php|66.249.76.194|http://v01.newdev.transitionnetwork.org/||0||Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 &&
Jan 23 07:59:55 puffin drupal: http://v01.newdev.transitionnetwork.org|1358927995|php|66.249.76.194|http://v01.newdev.transitionnetwork.org/||0||Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 &&
...

So I now wonder if there's an issue with the TN website somehow, the above v01.newdev.transitionnetwork.org was my first (broken) attempt. It seems to be to do with some regexp compilation issue. Will look on Drupal.org for this.

It's possible that this is something that runs on cron, and somehow messes up things. In the distant past I remember seeing a similar issue, possibly even on TN.org...

I'll keep prodding, but in the mean time delete v01.newdev.transitionnetwork.org since it's dead anyway.

jim — Thu, 24 Jan 2013 00:09:54 GMT

Quick updates:

The preg_match() error is fixed (was a Drupal CTools bug) - ignore that!
I reckon the php-fpm segfault is a red-herring related to mysql restart - ignore that too!
I've patched other Drupal modules and the site is nearly working as expected... Lets see if these fixes stop the database connection issue...

Chris, I saw a bunch of emails being sent to you when I was watching syslog... Apologies for these, but I hope to be fixing them slowly but surely.

More tomorrow, sleep time.

hours, totalhours changed

chris — Thu, 24 Jan 2013 10:01:05 GMT

hours changed from 0.0 to 0.25
totalhours changed from 13.29 to 13.54

Replying to jim:

do we want FTP server running if we have SSH? I've personally disabled it on mine and feel that, unless we had multiple clients, it's adding nothing other than a possible attack vector... The solution is to remove 'FTP' from above and re-run the usual install.

No, we don't need a FTP server, I have removed FTP from /root/.barracuda.cnf and re-run the upgrade steps as documented on the wiki, wiki:PuffinServer#UpgradingBOA

I don't know why MariaDB has been causing problems, I've not used it before, given the way Oracle has been behaving I would expect most Linux distros will switch from MySQL to MariaDB in time.

hours, totalhours changed

chris — Thu, 24 Jan 2013 10:12:50 GMT

hours changed from 0.0 to 0.1
totalhours changed from 13.54 to 13.64

Replying to jim:

NOTE: CGP/Collectd is now available at: http://cgp.master.puffin.webarch.net/ I've documented this here https://tech.transitionnetwork.org/trac/wiki/PuffinServer#CGPCollectdsystemmonitor

Thanks, I had been assuming that we would run a collectd server on penguin and clients on penguin and puffin, but perhaps if it doesn't use the same system resources as munin we can run either run and client and server on all machines or use puffin as the main server?

jim — Sat, 26 Jan 2013 19:47:41 GMT

The crashes plaguing me a few days back seem to be gone -- looks like they were caused by one or more of the issues raised in the above 6 comments.

NOTE in syslog I just saw LOTS (thousands) of:

Jan 26 13:19:01 puffin pure-ftpd: (?@190.58.249.69) [INFO] New connection from 190.58.249.69
Jan 26 13:19:01 puffin pure-ftpd: (?@190.58.249.69) [DEBUG] Command [user] [Administrator]
Jan 26 13:19:01 puffin pure-ftpd: (?@190.58.249.69) [WARNING] Sorry, cleartext sessions are not accepted on this server.#012Please reconnect using SSL/TLS security mechanisms.

So I ran a killall pure-ftpd to ensure no instances of the FTP server Chris removed were running. Then I edited /etc/csf.conf and removed access to ports 20 and 21, then csf -r to restart the firewall and LFD.

The above change will be overwritten on next BOA update unless we set _CUSTOM_CSF in #~/.barracuda.cnf to 'YES'. I'll leave this for now but we should close all ports not being used, and consider moving SSH port elsewhere too if possible. Thoughts Chris?

Anyway, since the DB/PHP issue seems to be gone, I'll proceed with the import of the TN site on ticket #478 Import TN.org site from Quince to Puffin.

hours, totalhours changed

chris — Sun, 27 Jan 2013 17:03:27 GMT

hours changed from 0.0 to 0.1
totalhours changed from 13.64 to 13.74

Replying to jim:

I ran a killall pure-ftpd to ensure no instances of the FTP server Chris removed were running.

Thanks, sorry I forgot to check if it was still running.

Then I edited /etc/csf.conf and removed access to ports 20 and 21, then csf -r to restart the firewall and LFD.

The above change will be overwritten on next BOA update unless we set _CUSTOM_CSF in #~/.barracuda.cnf to 'YES'.

Are there any disadvantages to doing that?

I'll leave this for now but we should close all ports not being used, and consider moving SSH port elsewhere too if possible. Thoughts Chris?

We could move SSH to a different port if you want, LDF is blocking several IP's that are trying to get i each dayn, but this is not unusual. We could also disable passwords so it's SSH keys only.

jim — Mon, 28 Jan 2013 00:34:45 GMT

@Chris:

Are there any disadvantages to doing that? (set custom CSF flag: well we wouldn't get any BOA-provided CSF tweaks (rare), but we'd be able to run our own ports at will. I do that on my VPS.
SSH: On my box I moved the SSH port and saw a drop to almost 0 for drive-bys in the CSF/LFD logs. I also disabled password and logging in as root, forcing login to my account via SSH only, then sudo -i and a password to go further. Your call as to what level of security you want to set on SSH.

NOTE DB/502 issues have reappeared today (see https://tech.transitionnetwork.org/trac/ticket/466#comment:29) - I have therefore reinstalled Barracuda with _PHP_FORCE_REINSTALL=YES, and used barracuda up-head to bring latest updates and fixes.

Problems fixed for now but the last time this happened they were back again. Hopefully fixed for good, else we need to switch _DEBUG_MODE=YES to get some meatier logs.

SSL -- Chris, your comments please

SSL was enabled by turning on 'SSL' and 'NginX SSL' in the black admin menu -> Hosting -> Features at master.puffin.webarch.net
Edited the puffin.webarch.net server node via Content Management -> Content, set server to 'nginx_ssl' from 'nginx'.
Sites and server was auto-verified... Green lights on these tasks
Edited 'transitionnetwork.org' site and set Encryption to 'Enabled'. This leave question of encryption key, and per Aegir: Using SSL - Notes for Nginx users (foot of page), I left it set to transitionnetwork.org hopeing this will pick up the one Chris added, or allow us to replace this auto-generated one. I set the SSL domain to 'transitionnetwork.org' -- is this correct?
Site reverified itself, now says "This site is using the SSL certificate and SSL key located at config/ssl.d/transitionnetwork.org/openssl.crt and config/ssl.d/transitionnetwork.org/openssl.key." -- Chris, is this where you put the keys? Can you put them there?

hours, totalhours changed

chris — Mon, 28 Jan 2013 10:08:43 GMT

hours changed from 0.0 to 0.8
totalhours changed from 13.74 to 14.54

I have symlinked to the cert and key in /etc/ssl/:

cd /data/disk/tn/config/ssl.d/transitionnetwork.org
mkdir bak
mv *.* bak/
ln -s /etc/ssl/transitionnetwork.org/transitionnetwork.org.key openssl.key
ln -s /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem openssl.crt

Regarding the SSL domain, I guess it should be set to 'www.transitionnetwork.org'.

When we do the DNS switch I think we should also edit the server_name in the files in /var/aegir/config/server_master/nginx/vhost.d/, but this might be best done via the web interface?

When requesting the front page at http://newlive.puffin.webarch.net/ these files are 404's:

We either need to make them appear at their existing (old) address' or add a redirect from /sites/default/ to /sites/transitionnetwork.org/, I'm not sure where in the Nginx configuration to add this without it being overwritten, perhaps we should do it with a symlink?

Next I'll restrict SSH access as flagged up by Jim and then I want to revisit the wiki:LoadTimes page -- testing the new server vs the old one it doesn't currently seem quicker via pingdom:

http://tools.pingdom.com/fpt/#!/SqZPGMH7/http://www.transitionnetwork.org -- No, of reqs: 35, Total size: 1.1 MB, Total time: 656 ms
http://tools.pingdom.com/fpt/#!/cOgL7T7O9/http://newlive.puffin.webarch.net -- No, of reqs: 27, Total size: 1.6 MB, Total time: 4.49 s

hours, totalhours changed

chris — Mon, 28 Jan 2013 10:40:42 GMT

hours changed from 0.0 to 0.4
totalhours changed from 14.54 to 14.94

I have added new users accounts for myself and Jim on puffin, copied ssh public keys from quince, added us to the sudo group and I have changed these lines in /etc/ssh/sshd_config:

PermitRootLogin no
AllowGroups sudo

So now only users in the sudo group can login via SSH.

I have also edited /etc/sudoers and set it so that members of the sudo group don't need to type a password to use sudo:

%sudo   ALL = NOPASSWD: ALL

I haven't changed the SSH port (if Jim is keen on this it could be changed) and I haven't restarted ssh -- these changes will only take effect after a ssh restart -- Jim can you check that you can ssh in and sudo as user jim first?

chris — Mon, 28 Jan 2013 11:26:07 GMT

I have created a separate ticket for the 502 bad gateway errors, ticket:483 -- I think this issue is probably a blocker.

jim — Mon, 28 Jan 2013 11:42:50 GMT

Chris said: "When requesting the front page at http://newlive.puffin.webarch.net/ these files are 404's"

Some paths are absolute, so they will fail until the DNS switch happens... The actual path (not including the domain part) is correct on the new server, but not on the old one. e.g. Changing http://www.transitionnetwork.org/sites/default/files/uploaded/u7/rp.jpg to http://newlive.puffin.webarch.net/sites/default/files/uploaded/u7/rp.jpg works fine.

Hence this issue will not be fixed now, but will go away after the DNS switch.

jim — Mon, 28 Jan 2013 11:43:16 GMT

(to be clear: no redirects needed)

chris — Mon, 28 Jan 2013 11:57:10 GMT

Replying to jim:

his issue will not be fixed now, but will go away after the DNS switch.

Hopefully that's true for all the internal links but what about external ones -- if anyone had linked to a file on the site from another Transition site (plus search engines etc) -- I think a Nginx redirect will still be needed, I'm just not sure where to put it so that BOA doesn't overwrote it.

jim — Mon, 28 Jan 2013 12:21:02 GMT

It is true for internal links, honest!.

As for external, your and Ed's call... People shouldn't be linking directly to files IMHO, but the page that contains them, since they are likely to be updated at the drop of a hat anyway. I say: "tough titties to cross-linkers" (but that's because I'm acting all badass).

And if the redirect is needed, you should check documentation first -- I'm not sure about setting up a redirect in the 'BOA way' properly, though there are plenty of places it can be added safely -- just need to make sure it won't confuse Aegir.

chris — Mon, 28 Jan 2013 12:32:10 GMT

Replying to jim:

People shouldn't be linking directly to files

Why not?

jim — Mon, 28 Jan 2013 12:47:26 GMT

Why not?

Because! It makes this job harder ;-) Seriously though, if you/Ed feel the need go for it, just research the best way first to ensure Aegir isn't confused.

ed — Mon, 28 Jan 2013 13:04:08 GMT

What type of links are we talking about here?

Incoming links from hundreds of sites to TN.org
Specific file links (e.g. resources)
Internal links within TN.org

chris — Mon, 28 Jan 2013 13:25:32 GMT

Replying to ed:

What type of links are we talking about here?

Incoming links from hundreds of sites to TN.org
Specific file links (e.g. resources)
Internal links within TN.org
?

Links to any uploaded files from places other than the Transition Network site itself.

hours, totalhours changed

chris — Mon, 28 Jan 2013 18:32:02 GMT

hours changed from 0.0 to 0.25
totalhours changed from 14.94 to 15.19

Using apache bench to test download time of the front page BOA on wiki:PuffinServer is almost as fast as varnish on wiki:NewLiveServer, see wiki:LoadTimes#a2013-01-28

jim — Mon, 28 Jan 2013 19:06:41 GMT

Varnish vs Nginx speedcache comparison good!

However, Puffin appears to be slower for Drupal operations: My measly 1.5Gb Linode (Babylon) is actually faster in side-by-side comparison than Puffin too, which is strange given that Puffin is a quieter VPS with much more horsepower. Database operations (like postcode search on /nearby) are 1.5-2x as fast on Babylon compared to Puffin. Odd...

Perhaps this implies the issues on Puffin are deeper than thought, or that normal operation is slowed because of the SSL and other issue per #483? Compare these as needed:

http://tn.i-jk.co.uk/ (Babylon)
http://newlive.puffin.webarch.net/ (Puffin)

I realise Babylon is not transferring images, but there's not that many on the site and they'd be cached by the second request, so I doubt they're the cause...

I'll now fix SSL per ticket:478#comment:12 as this misconfiguration could be slowing things.

hours, totalhours changed

chris — Mon, 28 Jan 2013 19:41:14 GMT

hours changed from 0.0 to 0.1
totalhours changed from 15.19 to 15.29

Replying to jim:

Varnish vs Nginx speedcache comparison good!

It seems to show that Nginx on a VPS with 14 CPUs and 4GB of RAM is almost as fast as Varnish on a VPS with 4 CPUs and 3GB of RAM... I was hoping that Puffin/Nginx would be faster than Quince/Varnish...

However, Puffin appears to be slower for Drupal operations: My measly 1.5Gb Linode (Babylon) is actually faster in side-by-side comparison than Puffin too

How did you measure that?

Compare these as needed:

http://tn.i-jk.co.uk/ (Babylon)
http://newlive.puffin.webarch.net/ (Puffin)

I have added some results for you server, see wiki:LoadTimes#a2013-01-28 but it's not a fair comparison as the test I did were from another server in the same rack.

jim — Mon, 28 Jan 2013 20:24:52 GMT

Replying to chris:

I was hoping that Puffin/Nginx would be faster than Quince/Varnish...

It will be for 'normal' Drupal page flinging ops... Testing raw throughput isn't so important (as long as it's fast enough).

The Nginx speed cache is ~20% slower than Varnish, but takes no appreciable extra memory/CPU. It can be made faster by moving the cached pages from spinning disk to SSD, or even to Redis, though that's outside my knowledge by a _long_ way.

How did you measure that?

I was testing Drupal, not the server... so I went to both sites and went to same pages. My browser (on a v fast PC) showed the page generally faster on Babylon... The main 'DB' test was http://tn.i-jk.co.uk/nearby and http://newlive.puffin.webarch.net/nearby, where I entered a couple of postcodes then hit return quickly (within 1/2 second of eachother) on Puffin then Babylon's pages. Babylon returned before puffin by about a 1/2 to 1 second.

Not super scientific, but more 'Drupally' than a simple ab I reckon. Would be good to sort out some proper Drupal benchmarks, but that's for another day.

I have added some results for you server, see wiki:LoadTimes#a2013-01-28 but it's not a fair comparison as the test I did were from another server in the same rack.

Then my Linode holds up OK! Interesting, thanks. I reckon Puffin will kick ass when we're done fiddling.

SSL fixed now.

hours, totalhours changed

chris — Mon, 28 Jan 2013 20:42:26 GMT

hours changed from 0.0 to 0.1
totalhours changed from 15.29 to 15.39

Replying to jim:

I was testing Drupal, not the server... so I went to both sites and went to same pages. My browser (on a v fast PC) showed the page generally faster on Babylon... The main 'DB' test was http://tn.i-jk.co.uk/nearby and http://newlive.puffin.webarch.net/nearby, where I entered a couple of postcodes then hit return quickly (within 1/2 second of eachother) on Puffin then Babylon's pages. Babylon returned before puffin by about a 1/2 to 1 second.

Thanks, I'll write a script to test with lots of different post codes.

SSL fixed now.

Thanks.

hours, totalhours changed

chris — Thu, 31 Jan 2013 21:27:30 GMT

hours changed from 0.0 to 1.0
totalhours changed from 15.39 to 16.39

Replying to jim:

Database operations (like postcode search on /nearby) are 1.5-2x as fast on Babylon compared to Puffin. Odd...

There is a big difference between quince and puffin -- I have written the following script to test the nearby search facility I incremented the PO_NUMBER to ensure that no two identical requests were made each time the script was run (don't want cached pages), these are the results from the second run:

newlive.puffin.webarch.net start: Thu Jan 31 21:00:46 GMT 2013
newlive.puffin.webarch.net end: Thu Jan 31 21:02:48 GMT 2013
www.transitionnetwork.org start: Thu Jan 31 21:02:48 GMT 2013
www.transitionnetwork.org end: Thu Jan 31 21:04:08 GMT 2013

Puffin took 2 min 8 seconds and quince took 1 min 20 seconds.

I ran it for a third time and triggered a 502 on puffin, it took 2 min 3 seconds on puffin and 1 min 20 seconds on quince:

newlive.puffin.webarch.net start: Thu Jan 31 21:08:43 GMT 2013
Error code response from server
newlive.puffin.webarch.net end: Thu Jan 31 21:10:47 GMT 2013
www.transitionnetwork.org start: Thu Jan 31 21:10:47 GMT 2013
www.transitionnetwork.org end: Thu Jan 31 21:12:07 GMT 2013

I then ran it for a 4th time:

newlive.puffin.webarch.net start: Thu Jan 31 21:22:25 GMT 2013
newlive.puffin.webarch.net end: Thu Jan 31 21:24:28 GMT 2013
www.transitionnetwork.org start: Thu Jan 31 21:24:28 GMT 2013
www.transitionnetwork.org end: Thu Jan 31 21:25:52 GMT 2013

Puffin took 2 mins and 2 seconds and quince took 1 min 24 seconds.

From this test I conclude that the way puffin is currently configured (BOA / Nginx) it is slower than quince (Varnish / Apache).

This is the script if anyone else wants to try it (change the PO_NUMBER variable, I have used 1, 2, 3 and 4 so far):

#!/bin/bash
DOMAINS="newlive.puffin.webarch.net www.transitionnetwork.org"
for DOMAIN in $DOMAINS ; do
    echo "$DOMAIN start: $(date)"
    PO_LETTERS="AB AL B BA BB BD BH BL BN BR BS BT CA CB CF CH CM CO CR CT CV CW DA DD DE DG DH DL DN DT DY E EC EH EN EX FK FY G GL GU HA HD HG HP HR HS HU HX IG IP IV KA KT KW KY L LA LD LE LL LN LS LU M ME MK ML N NE NG NN NP NR NW OL OX PA PE PH PL PO PR RG RH RM S SA SE SG SK SL SM SN SO SP SR SS ST SW SY TA TD TF TN TQ TR TS TW UB W WA WC WD WF WN WR WS WV YO ZE"
    for PO_LETTER in $PO_LETTERS ; do
      PO_NUMBER="3"
      PO="$PO_LETTER$PO_NUMBER"
      wget -q --user-agent="Mozilla/5.0 (TN Speed Test)" "http://$DOMAIN/nearby?distance[postal_code]=$PO&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile" -O /dev/null
      # catch http errors
      STATUS=$?
      if [ $STATUS == "8" ] ; then
         echo "Error code response from server"
      fi
    done
    echo "$DOMAIN end: $(date)"
done

I'll follow up about the 502 on ticket:483

hours, totalhours changed

chris — Fri, 01 Feb 2013 09:58:51 GMT

hours changed from 0.0 to 0.25
totalhours changed from 16.39 to 16.64

Replying to chris:

From this test I conclude that the way puffin is currently configured (BOA / Nginx) it is slower than quince (Varnish / Apache).

I expect (this is a essentially a guess) that the reason for this is that on quince we have a lot more memory allocated to serving each request -- if there was a huge traffic spike quince would really struggle where as puffin would probably just take it in it stride.

If I'm correct then it suggests we should consider doing one of three things:

Keep BOA the way it's set up out of the box and reduce the resources puffin has (as it's not going to make good use of them), drop it from 4GB RAM to 2GB and use the 2GB for another virtual machine doing something else (Wordpress farm?).
Keep BOA and look at how settings on puffin can be tweaked to increase it's speed by making more use of the resources it has.
Consider keeping the Apache / Varnish setup -- we could do a more accurate comparison of the quince setup vs. the puffin setup by copying the quince virtual machine over to the same hardware that puffin is on, setting up both virtual machines to have the same amount of RAM and then testing them side by side.

I'd be happy with any of the three approaches above, Jim what do you think, do you have any other suggestions?

Some work has been done on the wiki page:

jim — Fri, 01 Feb 2013 10:40:27 GMT

Personally, I think 2GB of memory for the server would be plenty for now. However, I'd resist dropping Aegir because that give TN so much flexibility to run LOTS of things from Drupal land... And the talk of Community tools, an online store, event management and other things that can be installed in a minute or so with a Makefile, and managed through a life cycle. And the instant Dev/Stage/Test? as needed.

But I don't think we've quite got to the place where we know it's BOA causing every problem. It seems to me we have 2 things happening:

502s over on #483
Slow performance.

I think 2 is the biggest chin scratcher... and I think it might point to a misconigured VM... The reason is I ran your postcode script locally on Babylon, changing DOMAINS="tn.i-jk.co.uk" -- the results are shocking:

babylon:~# ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:08:35 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:09:59 GMT 2013

run 1 = 1m 24s

babylon:~# ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:12:09 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:12:13 GMT 2013
babylon:~# ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:13:29 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:13:32 GMT 2013

runs two and three: 3-4 seconds! Redis + MySQL + Nginx + PHP-FPM is FAST...

I couldn't believe it, so here's a section of the access log from Babylon as proof the requests were coming in.:

babylon:~# tail -n100 /var/log/nginx/access.log
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:30 +0000] "GET /nearby?distance[postal_code]=E3&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile HTTP/1.0" 200 48708 226 49415 "-" "Mozilla/5.0 (TN Speed Test)" 0.027 "-"
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:30 +0000] "GET /nearby?distance[postal_code]=EC3&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile HTTP/1.0" 200 47936 227 48644 "-" "Mozilla/5.0 (TN Speed Test)" 0.026 "-"
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:30 +0000] "GET /nearby?distance[postal_code]=EH3&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile HTTP/1.0" 200 45342 227 46050 "-" "Mozilla/5.0 (TN Speed Test)" 0.025 "-"
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:30 +0000] "GET /nearby?
  ... snip ...
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:32 +0000] "GET /nearby?distance[postal_code]=WV3&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile HTTP/1.0" 200 47199 227 47907 "-" "Mozilla/5.0 (TN Speed Test)" 0.031 "-"
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:32 +0000] "GET /nearby?distance[postal_code]=YO3&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile HTTP/1.0" 200 51965 227 52673 "-" "Mozilla/5.0 (TN Speed Test)" 0.026 "-"
"178.79.169.201" tn.i-jk.co.uk [01/Feb/2013:10:13:32 +0000] "GET /nearby?distance[postal_code]=ZE3&distance[country]=uk&distance[search_distance]=50&distance[search_units]=mile HTTP/1.0" 200 51965 227 52673 "-" "Mozilla/5.0 (TN Speed Test)" 0.026 "-"

I ran it again on my local machine to add in the internet latency:

jimk@monolith:~$ ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:20:28 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:21:01 GMT 2013
jimk@monolith:~$ ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:22:59 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:23:38 GMT 2013

33 seconds and 39 seconds, cache still warm from running on Babylon directly.

Conclusion:

Puffin is NOT configured properly, either at a software, hardware or VM level. What's the IO like on it? The memory throughput? Are there any good benchmarks we can run here?
We should it reboot to ensure all the things are bedded in and running as expected -- Redis and Speed Cache are what making my Babylon server so fast, and they should be running on Puffin. I've done little to the BOA side on mine, mostly mucking around with mail stuff.
After reboot, perhaps you fancy trying the BOA performance tuning script 'BOND.sh'?: http://drupalcode.org/project/barracuda.git/blob/HEAD:/aegir/tools/BOND.sh.txt -- It will tune the BOA system for available memory and CPU -- might be worth running to see what it recommends, and comparing that to what you've seen so far.

I just can't believe my Linode, whose parent hardware is running a bunch of busy VMs, plus the VM has a 15 of its own Drupal sites, and is on mediocre-to-good Xeon platform (see http://forum.linode.com/viewtopic.php?t=4326) that's a probably couple of years old at least is outperforming both Quince and Puffin.

There must be something not quite right on Puffin, don't you think Chris? I wouldn't have recommended BOA if I didn't think it was fast, but on Puffin, it's just not. Odd.

hours, totalhours changed

chris — Fri, 01 Feb 2013 17:59:10 GMT

hours changed from 0.0 to 0.5
totalhours changed from 16.64 to 17.14

Replying to jim:

Personally, I think 2GB of memory for the server would be plenty for now.

For a BOA server I agree.

However, I'd resist dropping Aegir because that give TN so much flexibility to run LOTS of things from Drupal land... And the talk of Community tools, an online store, event management and other things that can be installed in a minute or so with a Makefile, and managed through a life cycle. And the instant Dev/Stage/Test? as needed.

Fair enough.

But I don't think we've quite got to the place where we know it's BOA causing every problem. It seems to me we have 2 things happening:

502s over on #483
Slow performance.
I think 2 is the biggest chin scratcher... and I think it might point to a misconigured VM... The reason is I ran your postcode script locally on Babylon, changing DOMAINS="tn.i-jk.co.uk" -- the results are shocking:
babylon:~# ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:08:35 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:09:59 GMT 2013
run 1 = 1m 24s

That is the same speed as quince. (1m 20s and 1m 24s see ticket:466#comment:55).

babylon:~# ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:12:09 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:12:13 GMT 2013
babylon:~# ./postcode.sh
tn.i-jk.co.uk start: Fri Feb  1 10:13:29 GMT 2013
tn.i-jk.co.uk end: Fri Feb  1 10:13:32 GMT 2013

runs two and three: 3-4 seconds! Redis + MySQL + Nginx + PHP-FPM is FAST...

That is because you didn't increment the PO_NUMBER variable? If you didn't increment this number then it will have just been testing the cache rather than page generation.

Conclusion:

Puffin is NOT configured properly, either at a software, hardware or VM level. What's the IO like on it? The memory throughput? Are there any good benchmarks we can run here?

There are some munin stats from the server the VM is on:

https://ecodissident.net/munin/webarch.net/red.webarch.net/index.html

I agree that testing IO and memory throughput is a good idea.

This is the output of iostat -x:

Linux 2.6.32-5-xen-amd64 (puffin.webarch.net)   02/01/13        _x86_64_        (14 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.22    0.00    0.28    0.07    0.01   99.42
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap2            0.00     5.29    0.19    9.44     8.97   189.97    20.67     0.48   49.78   1.42   1.37
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     9.35     0.00    8.34   1.29   0.00

The output on quince:

Linux 2.6.32-5-xen-amd64 (quince.webarch.net)   01/02/13        _x86_64_        (4 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.87    0.04    1.94    4.95    6.37   79.82
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap2            0.01    57.52    5.36   51.59   171.43   872.98    18.34     0.10    4.33   0.20   1.12
xvdap1            0.00     0.02    0.10    0.05     0.82     0.55     9.31     0.01   63.39  13.10   0.19
xvdap3            0.00    21.12    2.65   19.16    62.08   322.28    17.62     0.06    2.79   4.13   9.00

And hdparm -t /dev/xvda2 on puffin, which has 8 SAS disks running in RAID 10:

/dev/xvda2:
 Timing buffered disk reads: 184 MB in  3.04 seconds =  60.56 MB/sec

Quince has a root partition on a pair of mirrored SATA disks:

/dev/xvda2:
 Timing buffered disk reads: 196 MB in  3.02 seconds =  65.00 MB/sec

And the database is on a pair of mirrored SCSI disks:

/dev/xvda3:
 Timing buffered disk reads: 312 MB in  3.01 seconds = 103.58 MB/sec

I'll look for some other tests we can run.

We should it reboot to ensure all the things are bedded in and running as expected -- Redis and Speed Cache are what making my Babylon server so fast, and they should be running on Puffin. I've done little to the BOA side on mine, mostly mucking around with mail stuff.

OK, it's been rebooted.

After reboot, perhaps you fancy trying the BOA performance tuning script 'BOND.sh'?: http://drupalcode.org/project/barracuda.git/blob/HEAD:/aegir/tools/BOND.sh.txt -- It will tune the BOA system for available memory and CPU -- might be worth running to see what it recommends, and comparing that to what you've seen so far.

OK.

I just can't believe my Linode, whose parent hardware is running a bunch of busy VMs, plus the VM has a 15 of its own Drupal sites, and is on mediocre-to-good Xeon platform (see http://forum.linode.com/viewtopic.php?t=4326) that's a probably couple of years old at least is outperforming both Quince and Puffin.

It's matching quince speed wise.

There must be something not quite right on Puffin, don't you think Chris? I wouldn't have recommended BOA if I didn't think it was fast, but on Puffin, it's just not. Odd.

Agreed. More testing needed.

jim — Fri, 01 Feb 2013 18:27:55 GMT

OK I've done some more research into this...

Running the postcode.sh Chris made locally on Quince returns in 3-4 seconds. So the performance is good. However I think the cause of slowness might be a networking issue... I did some pings to compare things:

On Puffin, Ping localhost

puffin:~# ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_req=1 ttl=64 time=0.107 ms
64 bytes from localhost (127.0.0.1): icmp_req=2 ttl=64 time=0.097 ms
64 bytes from localhost (127.0.0.1): icmp_req=3 ttl=64 time=0.100 ms
64 bytes from localhost (127.0.0.1): icmp_req=4 ttl=64 time=0.088 ms
64 bytes from localhost (127.0.0.1): icmp_req=5 ttl=64 time=0.102 ms
64 bytes from localhost (127.0.0.1): icmp_req=6 ttl=64 time=0.105 ms
64 bytes from localhost (127.0.0.1): icmp_req=7 ttl=64 time=0.106 ms
^C
--- localhost ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 5996ms
rtt min/avg/max/mdev = 0.088/0.100/0.107/0.013 ms

... 0.1ms for a localhost ping is slow. Comparison:

On Quince, ping localhost

jim@quince:~$ ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
[snip]
--- localhost ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 5998ms
rtt min/avg/max/mdev = 0.032/0.048/0.055/0.008 ms

... Only 0.048ms average. And on my machine: rtt min/avg/max/mdev = 0.037/0.050/0.054/0.007 ms = 0.05ms.

Indeed, pinging:

Quince ping puffin.webarch.net = 1.283 ms average
Puffin ping quince.webarch.net = 0.584 ms average

Now if MySQL or NginX are using localhost, 127.0.0.1 or puffin.webarch.net, all of which are over twice as slow on Puffin vs Quince, I wonder if they latencies are adding up? Haven't run a longer test, but maybe some packets are being lost?

Anyway this seems like a good place to investigate, doesn't it Chris? Could the VM or Hypervisor have a wonky network setup? Or maybe there's a network hardware issue with the host Puffin lives on?

Would be great to run some actual benchmarks on each subsystem to establish any slowness:

Network (I've kinda done a simple version of this)
Disk IO
Memory IO
MySQL throughput
anything else?

jim — Fri, 01 Feb 2013 18:32:30 GMT

Ignore the last bit! You posted at the same time as me...

Apols for missing the cache thing -- I was expecting caching to make a huge difference, but didn't realise the PO_NUMBER was being incremented... Now I re-read your post I see what I missed.

But for me the network latency on Puffin could explain a few things. I'm in for another hour tonight Chris, shout if you need me to investigate/test/do anything.

hours, totalhours changed

chris — Fri, 01 Feb 2013 22:34:25 GMT

hours changed from 0.0 to 1.0
totalhours changed from 17.14 to 18.14

Interesting, I have been using sockets rather than the loopback address (the change for php-fpm / nginx is recorded in ticket:357#comment:16), so on wiki:PenguinServer php-fpm listens for connections using a socket, we have this in /etc/php5/fpm/pool.d/www.conf:

;listen = 127.0.0.1:9000
listen = /var/run/php5-fpm/phpfpm.sock

And in the Penguin Nginx config:

fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;

Where as on wiki:PuffinServer we have this in /opt//local/etc/php53-fpm.conf:

listen = 127.0.0.1:9090

And I can't work out which Nginx config file the site on Puffin uses, but there are a couple of values for fastcgi_pass:

fastcgi_pass 127.0.0.1:9090;
fastcgi_pass phpcgi;

I theory sockets should be quicker as they don't have the TCP overhead but after searching for some good article on this tonight I haven't actually found any.

We could try switching Puffin to use a socket?

I don't understand why pinging the loopback address is slower on puffin than quince, especially given that there is far more processing power on puffin (14 AMD 2000Mhz cores on puffin vs 4 Intel 2000Mhz cores on quince).

hours, totalhours changed

chris — Sun, 03 Feb 2013 23:06:21 GMT

hours changed from 0.0 to 1.0
totalhours changed from 18.14 to 19.14

On puffin the results from iperf look like this:

iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 3165 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 7.8 sec  1.84 GBytes  2.02 Gbits/sec

Comparing puffin and quince with 100000 pings, first puffin:

ping -f localhost -c 100000
PING localhost (127.0.0.1) 56(84) bytes of data.
--- localhost ping statistics ---
100000 packets transmitted, 100000 received, 0% packet loss, time 8198ms
rtt min/avg/max/mdev = 0.012/0.013/0.680/0.006 ms, ipg/ewma 0.081/0.014 ms

And then quince:

ping -f localhost -c 100000
PING localhost (127.0.0.1) 56(84) bytes of data.
--- localhost ping statistics ---
100000 packets transmitted, 100000 received, 0% packet loss, time 8837ms
rtt min/avg/max/mdev = 0.009/0.011/26.778/0.120 ms, pipe 2, ipg/ewma 0.088/0.009 ms

Testing disk speed using iozone, quince:

iozone -s 2g -r 64 -i 0 -i 1 -t 1
        Iozone: Performance Test of File I/O
                Version $Revision: 3.308 $
                Compiled for 64 bit mode.
                Build: linux
        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
        Run began: Sun Feb  3 22:49:03 2013
        File size set to 2097152 KB
        Record Size 64 KB
        Command line used: iozone -s 2g -r 64 -i 0 -i 1 -t 1
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 1 process
        Each process writes a 2097152 Kbyte file in 64 Kbyte records
        Children see throughput for  1 initial writers  =   12439.18 KB/sec
        Parent sees throughput for  1 initial writers   =    8739.77 KB/sec
        Min throughput per process                      =   12439.18 KB/sec
        Max throughput per process                      =   12439.18 KB/sec
        Avg throughput per process                      =   12439.18 KB/sec
        Min xfer                                        = 2097152.00 KB
        Children see throughput for  1 rewriters        =    7506.82 KB/sec
        Parent sees throughput for  1 rewriters         =    6485.36 KB/sec
        Min throughput per process                      =    7506.82 KB/sec
        Max throughput per process                      =    7506.82 KB/sec
        Avg throughput per process                      =    7506.82 KB/sec
        Min xfer                                        = 2097152.00 KB
        Children see throughput for  1 readers          =   35895.11 KB/sec
        Parent sees throughput for  1 readers           =   35892.10 KB/sec
        Min throughput per process                      =   35895.11 KB/sec
        Max throughput per process                      =   35895.11 KB/sec
        Avg throughput per process                      =   35895.11 KB/sec
        Min xfer                                        = 2097152.00 KB
        Children see throughput for 1 re-readers        =   20054.65 KB/sec
        Parent sees throughput for 1 re-readers         =   20054.07 KB/sec
        Min throughput per process                      =   20054.65 KB/sec
        Max throughput per process                      =   20054.65 KB/sec
        Avg throughput per process                      =   20054.65 KB/sec
        Min xfer                                        = 2097152.00 KB
iozone test complete.

And on puffin:

iozone -s 2g -r 64 -i 0 -i 1 -t 1
        Iozone: Performance Test of File I/O
                Version $Revision: 3.308 $
                Compiled for 64 bit mode.
                Build: linux
        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
        Run began: Sun Feb  3 22:54:31 2013
        File size set to 2097152 KB
        Record Size 64 KB
        Command line used: iozone -s 2g -r 64 -i 0 -i 1 -t 1
        Output is in Kbytes/sec
        Time Resolution = 0.000002 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 1 process
        Each process writes a 2097152 Kbyte file in 64 Kbyte records
        Children see throughput for  1 initial writers  =   21821.23 KB/sec
        Parent sees throughput for  1 initial writers   =   16522.62 KB/sec
        Min throughput per process                      =   21821.23 KB/sec
        Max throughput per process                      =   21821.23 KB/sec
        Avg throughput per process                      =   21821.23 KB/sec
        Min xfer                                        = 2097152.00 KB
        Children see throughput for  1 rewriters        =   64297.46 KB/sec
        Parent sees throughput for  1 rewriters         =   44621.53 KB/sec
        Min throughput per process                      =   64297.46 KB/sec
        Max throughput per process                      =   64297.46 KB/sec
        Avg throughput per process                      =   64297.46 KB/sec
        Min xfer                                        = 2097152.00 KB
        Children see throughput for  1 readers          =  341842.59 KB/sec
        Parent sees throughput for  1 readers           =  341666.67 KB/sec
        Min throughput per process                      =  341842.59 KB/sec
        Max throughput per process                      =  341842.59 KB/sec
        Avg throughput per process                      =  341842.59 KB/sec
        Min xfer                                        = 2097152.00 KB
        Children see throughput for 1 re-readers        =  815984.56 KB/sec
        Parent sees throughput for 1 re-readers         =  815463.20 KB/sec
        Min throughput per process                      =  815984.56 KB/sec
        Max throughput per process                      =  815984.56 KB/sec
        Avg throughput per process                      =  815984.56 KB/sec
        Min xfer                                        = 2097152.00 KB
iozone test complete.

I really don't know where to go with this.

hours, totalhours changed

chris — Tue, 05 Feb 2013 14:16:15 GMT

hours changed from 0.0 to 0.5
totalhours changed from 19.14 to 19.64

Time recorded on this comment is for emails on the ttech list and wiki page updates.

hours, totalhours changed

chris — Tue, 05 Feb 2013 23:03:42 GMT

hours changed from 0.0 to 0.7
totalhours changed from 19.64 to 20.34

Installing a munin client on puffin.

Add the server IP to /etc/munin/munin-node.conf:

# https://ecodissident.net/munin/
allow ^93\.95\.226\.170$

Enable some plugins, nginx:

cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/nginx_status
ln -s /usr/share/munin/plugins/nginx_request

Some additional configuration is needed for ngins as the munin plugin isn't getting at stats at http://localhost/status -- these plugins were tested like this:

munin-run nginx_request
  request.value U
munin-run nginx_status
  total.value U
  reading.value U
  writing.value U
  waiting.value U

For php-fpm:

cd /usr/share/munin/plugins
git clone git://github.com/tjstein/php5-fpm-munin-plugins.git
chmod +x php5-fpm-munin-plugins/phpfpm_*
cd /etc/munin/plugins

Again these plugins need some work:

munin-run phpfpm_average
  php_average.value awk: fatal: division by zero attempted
munin-run phpfpm_connections
  accepted.value U
munin-run phpfpm_memory
  ram.value Integer
munin-run phpfpm_processes
  php_processes.value 0
munin-run phpfpm_status
  idle.value U
  active.value U
  total.value U

Mysql:

cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/mysql_bytes
ln -s /usr/share/munin/plugins/mysql_queries
ln -s /usr/share/munin/plugins/mysql_slowqueries
ln -s /usr/share/munin/plugins/mysql_threads

All of these work by default.

Memory usage:

ln -s /usr/share/munin/plugins/multips_memory

And this one need this config in /etc/munin/plugin-conf.d/munin-node:

[multips_memory]
env.names nginx php-fpm mysqld redis-server munin-node

The virtual machines never generate stats for this plugin for some reason:

cd /etc/munin/plugins
rm iostat

However no stats were geing generated on the munin server, so in case it was a firewall issue the following was added to /etc/csf/csf.allow:

tcp:in:d=4949:s=93.95.226.170 # ecodissident.net for munin

And also this was run:

iptables -A INPUT -p tcp -s 93.95.226.170 --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT

Based on suggestions here.

This appears to have done the trick, stats are being generated here:

https://ecodissident.net/munin/webarch.net/puffin.webarch.net/

So my tomorrow we should have some interesting graphs, TODO, sort out nginx and php-fpm stats.

Looking at the running processes I noticed that pure-ftpd is running -- Jim I though this was disabled?

hours, totalhours changed

chris — Wed, 06 Feb 2013 12:11:41 GMT

hours changed from 0.0 to 0.54
totalhours changed from 20.34 to 20.88

This comment is to cover the time just spent on the phone to Jim and Ed discussing the server migration.

hours, totalhours changed

chris — Wed, 06 Feb 2013 13:53:29 GMT

hours changed from 0.0 to 0.5
totalhours changed from 20.88 to 21.38

Picking up on the Munin work done last night on ticket:466#comment:64

Enabling Nginx status for localhost requests, after some trial and error this has been done by creating /etc/nginx/conf.d/localhost.conf with the following in it:

    server {
        listen       80;
        server_name  localhost;
        # for nginx status
        # chris
        location /nginx_status {
                stub_status on;
                access_log   off;
                allow 127.0.0.1;
                deny all;
        }

This comes before the BOA config in /etc/nginx/conf.d/aegir.conf however I need to check that there is other BOA config that need copying into /etc/nginx/conf.d/localhost.conf.

hours, totalhours changed

chris — Wed, 06 Feb 2013 20:49:44 GMT

hours changed from 0.0 to 1.25
totalhours changed from 21.38 to 22.63

We now have some nginx stats:

https://ecodissident.net/munin/webarch.net/puffin.webarch.net/nginx_status.html

The only other reference to localhost I can find in the BOA nginx configuration is in /var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf and it seem that this configuration isn't loaded due to the new config file, and this was causing a default "Welcome to nginx!" page to be served here:

However not at the IP address:

Chive, http://www.chive-project.com/ was being served.

I'm not sure if this will have any unintended consequences but in /etc/nginx/conf.d the following two files were created, with file names to ensure they are loaded before the aegir.conf symlink, 00_puffin.conf to serve the default page for the server, containing:

server {
        listen          80 default_server;
        server_name     puffin.transitionnetwork.org puffin.webarch.net 81.95.52.111;
        access_log      /var/log/nginx/puffin.access.log;
        error_log       /var/log/nginx/puffin.error.log   crit;
        root            /var/www/puffin;
        index           index.html;
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
        location ~ .~$ {
                return 403;
        }
}
server {
        listen          443 default_server;
        server_name     puffin.transitionnetwork.org puffin.webarch.net 81.95.52.111;
        ssl  on;
        ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;
        ssl_certificate_key  /etc/ssl/transitionnetwork.org/transitionnetwork.org.key;
        ssl_protocols  SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers  RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;
        access_log      /var/log/nginx/puffin_ssl.access.log;
        error_log       /var/log/nginx/puffin_ssl.error.log   crit;
        root            /var/www/puffin;
        index           index.html;
}

And for the requests for nginx status and php-fpm status from munin plugins, 01_localhost.conf:

server {
    listen       127.0.0.1:80;
    server_name  localhost;
    # for nginx status
    # chris
    location /nginx_status {
            stub_status on;
            access_log   off;
            allow 127.0.0.1;
            deny all;
    }
    # for munin, see
    # http://tjstein.com/2010/09/php-fpm-plugin-suite-for-munin/
    location ~ ^/(status|ping)$ {
            fastcgi_pass 127.0.0.1:9090;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_intercept_errors on;
            include fastcgi_params;
            access_log off;
            allow 127.0.0.1;
            deny all;
    }
}

To enable the php-fpm status the following was uncommented in /opt/local/etc/php53-fpm.conf:

pm.status_path = /status
ping.path = /ping

The munin php-fpm plugins needed editing as for penguin to hard code the process name, see ticket:470#comment:41

We should now start to get some stats for nginx and php-fpm here:

https://ecodissident.net/munin/webarch.net/puffin.webarch.net/

My TODO:

Look at what other Munin plugins should be enabled
Touch the file to ensure the changes to /opt/local/etc/php53-fpm.conf are not overwritten
Check what tasks in the /var/xdrago/clear.sh cron job are safe to enable
Do a final load test to see if the 502 issue has been totally solved, and if it has close trac:ticket/483

hours, totalhours changed

chris — Wed, 06 Feb 2013 22:19:37 GMT

hours changed from 0.0 to 0.8
totalhours changed from 22.63 to 23.43

The following needed adding to /etc/nginx/conf.d/00_puffin.conf to enable the nginx_request munin plugin:

        location /nginx_status {
                stub_status on;
                access_log   off;
                allow 127.0.0.1;
                allow 81.95.52.103;
                deny all;
        }

I think the munin stats are now adequate, to get some more interesting graphs and to double check that the 502 issue is solved a recursive wget is being run from penguin against puffin.

https://ecodissident.net/munin/webarch.net/puffin.webarch.net/

Some updates have been done on the wiki:PuffinServer page.

Following this advice:

Note: Barracuda will *not* overwrite your changes made to php.ini or php-fpm.conf on the next upgrade if you will touch the empty control file:

$ touch /opt/etc/custom.php.ini

This file was touched.

Looking at /var/xdrago/clear.sh it really doesn't do anything we need, it does things like clobbering log files but we are not so short of space that this is an issue so I don't think the root crontab needs /var/xdrago/clear.sh uncommenting, however after the next BOA upgrade we should check to see that it's not been reinstated.

I think my work on puffin is just about done:

ticket:483 - has been downgraded to trivial rather than a blocker, all that is left is for Jim to raise a ticket upstream about the issue

There are these outstanding tickets of Jim's:

ticket:479 Transfer live transitionnetwork.org site to puffin
ticket:480 Transfer news.transitionnetwork.org to puffin

The migration date looks like it'll be either Monday 11th or Monday 18th Feb 2013.

hours, totalhours changed

chris — Wed, 06 Feb 2013 22:32:55 GMT

hours changed from 0.0 to 0.1
totalhours changed from 23.43 to 23.53

One thing that has surprised me from the munin stats is that despite a lot of tweaks to the quince mysql settings over the years it is only using around 450M of RAM, see:

https://ecodissident.net/munin/webarch.net/quince.webarch.net/multimemory.html

The mysql on puffin is using 1.2GB of RAM:

https://ecodissident.net/munin/webarch.net/puffin.webarch.net/multips_memory.html

hours, totalhours changed

chris — Wed, 13 Feb 2013 10:23:55 GMT

hours changed from 0.0 to 0.6
totalhours changed from 23.53 to 24.13

The problem with Munin stats for nginx and php-fpm raised at ticket:466#comment:64 and then "solved" on ticket:466#comment:66 ticket:466#comment:67 and ticket:466#comment:68 by creating /etc/nginx/conf.d/00_puffin.conf containing:

server {
        listen          80 default_server;
        server_name     puffin.transitionnetwork.org puffin.webarch.net 81.95.52.111;
        access_log      /var/log/nginx/puffin.access.log;
        error_log       /var/log/nginx/puffin.error.log   crit;
        root            /var/www/puffin;
        index           index.html;
        # for nginx status
        # chris
        location /nginx_status {
                stub_status on;
                access_log   off;
                allow 127.0.0.1;
                allow 81.95.52.103;
                deny all;
        }
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
        location ~ .~$ {
                return 403;
        }
}
server {
        listen          443 default_server;
        server_name     puffin.transitionnetwork.org puffin.webarch.net 81.95.52.111;
        ssl  on;
        ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;
        ssl_certificate_key  /etc/ssl/transitionnetwork.org/transitionnetwork.org.key;
        ssl_protocols  SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers  RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;
        access_log      /var/log/nginx/puffin_ssl.access.log;
        error_log       /var/log/nginx/puffin_ssl.error.log   crit;
        root            /var/www/puffin;
        index           index.html;
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
        location ~ .~$ {
                return 403;
        }
}

and /etc/nginx/conf.d/01_localhost.conf containing:

server {
    listen       127.0.0.1:80;
    server_name  localhost;
    # for nginx status
    # chris
    location /nginx_status {
            stub_status on;
            access_log   off;
            allow 127.0.0.1;
            deny all;
    }
    # for munin, see
    # http://tjstein.com/2010/09/php-fpm-plugin-suite-for-munin/
    location ~ ^/(status|ping)$ {
            fastcgi_pass 127.0.0.1:9090;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_intercept_errors on;
            include fastcgi_params;
            access_log off;
            allow 127.0.0.1;
            deny all;
    }
}

Broke the BOA configuration, sorry :-(

Jim fixed BOA by moving 00_puffin.conf and 01_localhost.conf into /root/ and restarting Nginx but we now have the issue of incomplete Nginx and php-fpm munin stats:

There is this suggestion from Jim, however the nginx config file that he suggested should contain the code for /nginx_status does already contain it, /var/aegir/config/server_master/nginx/vhost.d/cgp.master.puffin.webarch.net:

   location /nginx_status {
      stub_status             on;
      access_log              off;
      allow                   81.95.52.103;
      allow                   127.0.0.1;
      deny                    all;
   }

I think the problem probably is that this isn't the default Nginx server, perhaps this one is in /var/aegir/config/server_master/nginx.conf:

server {
  limit_conn   gulag 32; # like mod_evasive - this allows max 32 simultaneous connections from one IP address
  listen       *:80;
  server_name  _;
  location / {
     root   /var/www/nginx-default;
     index  index.html index.htm;
  }
}

So the following was added to that file:

## chris
  location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    allow 81.95.52.103;
    deny all;
  }
  location ~ ^/(status|ping)$ {
    fastcgi_pass 127.0.0.1:9090;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    include fastcgi_params;
    access_log off;
    allow 127.0.0.1;
    deny all;
  }

And the munin plugins were tested:

munin-run nginx_request
  request.value 2
munin-run nginx_status
  total.value 2
  reading.value 0
  writing.value 1
  waiting.value 1
munin-run phpfpm_average
  php_average.value 107542528
munin-run phpfpm_connections
  accepted.value 203
munin-run phpfpm_memory
  ram.value 222187520
munin-run phpfpm_processes
  php_processes.value 3

So as long as the next upgrade of BOA doesn't clobber /var/aegir/config/server_master/nginx.conf this issue is solved. If that file does get clobbered then perhaps this is an issue that need raising as a ticket elsewhere (on which projects ticketing system, Barracuda, Octopus, Aegir or BOA?).

hours, totalhours changed

chris — Mon, 18 Feb 2013 13:17:16 GMT

hours changed from 0.0 to 0.1
totalhours changed from 24.13 to 24.23

DNS update, this is the old zone file:

* 3600 IN A 81.95.52.88
*.newdev 3600 IN A 81.95.52.103
2010.archive 3600 IN A 81.95.52.111
2011.archive 3600 IN A 81.95.52.111
@ 3600 IN A 81.95.52.88
lists 3600 IN A 212.113.133.235
mail 3600 IN A 212.113.133.235
newdev 3600 IN A 81.95.52.103
penguin 3600 IN A 81.95.52.111
puffin 3600 IN A 81.95.52.103
quince 3600 IN A 81.95.52.88
redirects 3600 IN A 81.95.52.111
static 3600 IN A 81.95.52.111
stats 3600 IN A 81.95.52.111
tech 3600 IN A 81.95.52.111
trac 3600 IN A 81.95.52.111
wiki 3600 IN A 81.95.52.111
www 3600 IN A 81.95.52.88
www.wiki 3600 IN A 81.95.52.111
9c086db58fc42046983c43dcd2e6dea0 10800 IN CNAME 8d511978467880c55b9a90e56ff8eb75b8eb813a.comodoca.com.
@ 3600 IN MX 0 mx1.spamfiltering.com.
@ 3600 IN MX 5 mx2.spamfiltering.com.

And this is the new one, the comodo CNAME has been omitted as it was only needed for the SSL cert generation.

* 3600 IN A 81.95.52.103
*.newdev 3600 IN A 81.95.52.103
2010.archive 3600 IN A 81.95.52.111
2011.archive 3600 IN A 81.95.52.111
@ 3600 IN A 81.95.52.103
lists 3600 IN A 212.113.133.235
mail 3600 IN A 212.113.133.235
newdev 3600 IN A 81.95.52.103
penguin 3600 IN A 81.95.52.111
puffin 3600 IN A 81.95.52.103
quince 3600 IN A 81.95.52.88
redirects 3600 IN A 81.95.52.111
static 3600 IN A 81.95.52.111
stats 3600 IN A 81.95.52.111
tech 3600 IN A 81.95.52.111
trac 3600 IN A 81.95.52.111
wiki 3600 IN A 81.95.52.111
www 3600 IN A 81.95.52.103
www.wiki 3600 IN A 81.95.52.111
@ 3600 IN MX 0 mx1.spamfiltering.com.
@ 3600 IN MX 5 mx2.spamfiltering.com.

I think that *.newdev can be omitted but I have left this in for now. If Jim is happy with the above I can make it live!

hours, totalhours changed

chris — Mon, 18 Feb 2013 13:24:52 GMT

hours changed from 0.0 to 0.1
totalhours changed from 24.23 to 24.33

OK, confirmed with Jim in irc, the version above has been made live and the DNS servers should update within 15 mins, you can check them like this:

dig @A.DNS.GANDI.NET www.transitionnetwork.org +short
dig @B.DNS.GANDI.NET www.transitionnetwork.org +short
dig @C.DNS.GANDI.NET www.transitionnetwork.org +short

When 81.95.52.103 is returned rather than 81.95.52.88 then they have updated.

If you want to see the update before the DNS has updated then edit your /etc/hosts file and add this:

81.95.52.103  www.transitionnetwork.org news.transitionnetwork.org

ed — Mon, 18 Feb 2013 14:14:50 GMT

WYSIWYG editor not showing its various options
Production bar still down the side

jim — Mon, 18 Feb 2013 14:32:14 GMT

now fixed, makefile updated
As discussed, this will stay for all Developer role users.. Currently for site admins too as it's a good reminder where a user is in the world of multiple copies of the same site.

hours, totalhours changed

chris — Wed, 06 Mar 2013 14:15:46 GMT

hours changed from 0.0 to 0.25
totalhours changed from 24.33 to 24.58

Replying to chris:

However no stats were geing generated on the munin server, so in case it was a firewall issue the following was added to /etc/csf/csf.allow:
tcp:in:d=4949:s=93.95.226.170 # ecodissident.net for munin
And also this was run:
iptables -A INPUT -p tcp -s 93.95.226.170 --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT
Based on suggestions here.

We have a new Munin server in Sheffield and the above steps were done for the additional IP address, I'll email the list with the details.

hours, totalhours changed

chris — Wed, 06 Mar 2013 14:43:38 GMT

hours changed from 0.0 to 0.1
totalhours changed from 24.58 to 24.68

Replying to chris:

Replying to chris:
However no stats were geing generated on the munin server, so in case it was a firewall issue the following was added to /etc/csf/csf.allow:
tcp:in:d=4949:s=93.95.226.170 # ecodissident.net for munin
And also this was run:
iptables -A INPUT -p tcp -s 93.95.226.170 --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT
Based on suggestions here.
We have a new Munin server in Sheffield and the above steps were done for the additional IP address, I'll email the list with the details.

The above didn't d the trick, still no stats from puffin.webarch.net, I have now tried:

csf -r

hours, status, totalhours changed; resolution set

chris — Tue, 19 Mar 2013 09:47:30 GMT

hours changed from 0.0 to 0.22
status changed from new to closed
resolution set to fixed
totalhours changed from 24.68 to 24.9

I have updated the wiki:PuffinServer documentation wiki:PuffinServer?action=diff&version=29&old_version=27 and moved the list of migration tickets, see wiki:PuffinServer#MigrationTickets, to the end of that page -- I'm closing this ticket, the only outstanding ticket on that list is ticket:483 Nginx 502 Bad Gateway Errors with BOA see the summary of that issue on ticket:483#comment:46.

hours, totalhours changed

chris — Fri, 29 Mar 2013 09:42:05 GMT

hours changed from 0.0 to 0.1
totalhours changed from 24.9 to 25.0

I just noticed that the server clock was out and it didn't have a network time protocol client running so I installed one:

2013-03-29      chris
        *       libopts25{a} ntp rdate : installed