Transition Technology: Ticket Query

#898: Fwd: Access to Drupal

ade — Tue, 26 Jan 2016 17:35:05 GMT

Hi Chris,
The web team at the development agency are requesting access to the
webserver so that they can look at the sites make up.
(Please see below)
Would you please set up an account so that they can get root read access?
I guess this would be done via FTP, but your thoughts greatly appreciated.
best regards
Ade
---------- Forwarded message ----------
From: Ainslie Beattie <ainsliebeattie@transitionnetwork.org>
Date: 26 January 2016 at 17:25
Subject: Fwd: Access to Drupal
To: Sam Rossiter <samrossiter@transitionnetwork.org>, Ade Stuart <
adestuart@transitionnetwork.org>, Yvonne Struthers <yvonne@thisisyoke.com>
Hey both, can you please action this urgently so that Yoke can have access.
Cheers
---------- Forwarded message ----------
From: "Yvonne Struthers" <yvonne@thisisyoke.com>
Date: 26 Jan 2016 10:58
Subject: Access to Drupal
To: <ainsliebeattie@transitionnetwork.org>
Cc:
Hi Ainslie,
Just a quick email as I'm out seeing a client today,but just to say,it
looks like you have only given us access to the database. What we need
please is admin access to the Drupal site and to the code base so that we
can get a sense of how it's all set up.
Thanks in advance!
Yvonne
Sent from my iPhone
--
Ade Stuart
Web Manager - Transition network
07595 331877
The Transition Network is a registered charity
address: 43 Fore St, Totnes, Devon, TQ9 5HN, UK
website: www.transitionnetwork.org
TN company no: 6135675 TN charity no: 1128675

#814: Higher that usual loads on PuffinServer since early September

chris — Wed, 03 Dec 2014 17:12:35 GMT

The following load graph from PuffinServer shows that the load increased substantially in early September 2014, does anyone know why?

When I found I went to Drupal 7.33 and all I got was a slow site I thought that perhaps a Drupal 7 site on the server could be the cause but 7.33 came out on 7th November 2014.

Anyone have any ideas?

#587: Puffin MySQL Tuning

chris — Thu, 05 Sep 2013 12:54:47 GMT

This ticket is to track the tuning we do to MySQL on PuffinServer.

See also previous comments on this issue:

#820: *.transitionnetwork.org 2015 security certificate

chris — Fri, 26 Dec 2014 09:47:49 GMT

The current wild-card *.transitionnetwork.org cert will run out on 24th Jan, this is a ticket to track the time spent renewing it.

See also ticket:795, SHA1 Deprecation: Regenerate all certs using SHA256.

#569: 403s served to editors, admin very slow

ed — Tue, 09 Jul 2013 07:52:32 GMT

Rob is getting 403s when trying to submit his work. Report from 07:12am this morning (Tuesday)

Ed tried to add a blog post at node add:

https://www.transitionnetwork.org/node/add/blog It took nearly 15 seconds to get this published https://www.transitionnetwork.org/blogs/ed-mitchell/2013-07/eds-test-blog-item-check-403

Running admin functions takes ages.This request took well over 30 seconds:

https://www.transitionnetwork.org/admin/content/node/overview

Please advise?

#563: 503 Errors

chris — Wed, 19 Jun 2013 10:59:54 GMT

Original Description

The site is generating a lot of 503 errors, 83 since 6:30am today and there were around 750 yesterday.

#843: 8.8.8.8 (US/United States/google-public-dns-a.google.com) blocked for port scanning

chris — Tue, 07 Apr 2015 23:05:33 GMT

Never seen this before:

Date: Tue,  7 Apr 2015 23:46:09 +0100 (BST)
From: root@puffin.webarch.net
To: chris@webarchitects.co.uk
Subject: lfd on puffin.webarch.net: 8.8.8.8 (US/United States/google-public-dns-a.google.com) blocked for port scanning
Time:    Tue Apr  7 23:46:09 2015 +0000
IP:      8.8.8.8 (US/United States/google-public-dns-a.google.com)
Hits:    20
Blocked: Temporary Block
Sample of block hits:
Apr  7 23:45:36 puffin kernel: [19823338.636822] Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=8.8.8.8 DST=81.95.52.103 LEN=162 TOS=0x00 PREC=0x00 TTL=45 ID=65064 PROTO=UDP SPT=53 DPT=48825 LEN=142

I thought set the Google DNS servers for the machine via /etc/resolv.conf but that contains:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1

There is /etc/resolvconf/resolv.conf.d/original containing:

nameserver 8.8.8.8
nameserver 8.8.4.4

But I don't know what DNS resolver BOA has installed and the server is using.

#610: Aegir database intensive (migrate, clone, restore) tasks hang for larger sites

jim — Tue, 15 Oct 2013 10:18:59 GMT

Large sites (TN.org and variants) will simply not complete their migrate, clone or restore tasks in Aegir.

However, smaller sites are fine, and all tasks work for them.

The process largely completes -- codebase installs, database is cloned, symlinks for sites aliases and files created... BUT the process never completes in Aegir, so the final steps of switching a site's served location never occurs.

Useful links/comments in this issue:

#779: Annesley locked out of puffin?

chris — Wed, 27 Aug 2014 14:28:12 GMT

Looks like Annesley's IP has been blocked on wiki:PuffinServer.

#854: BOA 2.4.3

chris — Tue, 26 May 2015 10:30:41 GMT

BOA 2.4.3 came out 7 days ago (BOA, for an unknown reason, no longer appears to sent out update available emails), from the changelog:

### Stable BOA-2.4.3 Release - Full Edition
### Date: Tue May 19 13:40:40 PDT 2015
### Milestone URL: https://github.com/omega8cc/boa/milestones/2.4.3
  @=> Includes Aegir Hostmaster 2.x-head with improvements
  @=> Includes Aegir Provision 3.x-head with improvements
  @=> Includes Drush 7 customized for BOA
# Release Notes:
  This BOA release is focused on Aegir platforms update with latest Drupal core
  included. There are also a few system updates and bug fixes, as listed below.
# Changes:
  * Redis Integration Module: Update to version mod-08-05-2015
  * Use HTTPS intermediate mode to support legacy systems like XP/IE8 - see #718
# System upgrades:
  * Drush mini-7-08-05-2015
  * MariaDB 10.0.19
  * MariaDB Galera Cluster 10.0.19
  * PHP 5.4.41
  * PHP 5.5.25
  * PHP 5.6.9
  * Redis 3.0.1
# Fixes:
  * CiviCRM known bugs and regressions fixed
  * Improve drush aliases cleanup
  * Redis: sync net.core.somaxconn with tcp-backlog
  * sqlmagic: do not escape backslashes and EOL character - fixes #672

The last BOA upgrade, ticket:844 sent a email, ticket:754#comment:26 saying:

Next BOA upgrade (2.4.3) will force PHP 5.5 on all hosted Aegir
instances, unless you will explicitly opt-out *before* that upgrade
using PHP version control files, as explained in our docs at:
  https://omega8.cc/how-to-quickly-switch-php-to-newer-version-330

And we have been discussing if we should test running the site with PHP 5.5 on PuffinServer or if we should create a development server to test with, this upgrade needs to wait till we have agreed a way forward.

#864: BOA 2.4.5

chris — Sun, 12 Jul 2015 08:36:38 GMT

A new version of BOA came out on Friday:

### Stable BOA-2.4.5 Release - Full Edition
### Date: Fri Jul 10 11:25:43 PDT 2015
### Milestone URL: https://github.com/omega8cc/boa/milestones/2.4.5
### Latest hotfix added on: Fri Jul 10 14:49:11 PDT 2015
  @=> Includes Aegir Hostmaster 2.x-head with improvements
  @=> Includes Aegir Provision 3.x-head with improvements
  @=> Includes Drush 7 customized for BOA
# Release Notes:
  This BOA release includes PHP security upgrade for versions 5.6, 5.5 and 5.4
  plus security upgrade for Redis server and four updated Octopus platforms.
  Support for Drupal 8 is temporarily removed, because now it would require
  an upgrade to Drush 8, which in turn completely removes support for PHP 5.3,
  while it's still more important to support legacy Pressflow 6 sites, if they
  are not ready to move beyond PHP 5.3 yet, than trying to support some
  (too fast) moving targets like Drupal 8 beta, and Drush 8 head.
# Updated Octopus platforms:
  Commerce 2.26 ---------------- https://drupal.org/project/commerce_kickstart
  Commons 3.28 ----------------- https://drupal.org/project/commons
  OpenAtrium 2.43 -------------- https://drupal.org/project/openatrium
  Panopoly 1.25 ---------------- https://drupal.org/project/panopoly
# Changes:
  * Drupal 8 is not supported until we can switch to Drush 8 and remove PHP 5.3
# System upgrades:
  * Nginx 1.9.2
  * PHP 5.4.43
  * PHP 5.5.27
  * PHP 5.6.11
  * Redis 3.0.2

#872: BOA 2.4.6

chris — Tue, 22 Sep 2015 08:50:36 GMT

The Changelog:

### Stable BOA-2.4.6 Release - Full Edition
### Date: Sat Sep 19 11:09:09 PDT 2015
### Milestone URL: https://github.com/omega8cc/boa/milestones/2.4.6
### Latest hotfix added on: Mon Sep 21 05:18:33 PDT 2015
  @=> Includes Aegir Hostmaster 2.x-head with improvements
  @=> Includes Aegir Provision 3.x-head with improvements
  @=> Includes Drush 7 customized for BOA
# Release Notes:
  This BOA release includes several important system upgrades and bug fixes.
  All supported Aegir platforms have been updated with latest Drupal cores.
  @=> Latest Drupal 8.0.0-beta15 works great, but currently only when installed
      as a custom platform in the ~/static directory tree, because it is not
      really symlinks-friendly. We will re-introduce built-in Drupal 8 platforms
      once this situation is improved.
# Changes:
  * Add Twig C extension to PHP - v.1.22.1
  * Allow to customize auto-upgrades mode
  * Disable support for broken OpenScholar and Recruiter
  * Open default Postgres port for outgoing connections
  * Remove support for deprecated Feature Server distro
  * Remove support for deprecated OpenAcademy distro
  * Remove support for deprecated OpenBlog distro
  * Remove support for deprecated OpenChurch v.1 distro
  * Remove support for deprecated OpenDeals distro
  * Use distro specific Drupal core for problematic distros
# System upgrades:
  * cURL 7.44.0 (if installed from sources)
  * Duplicity 0.7.05 (please run 'backboa install' to upgrade)
  * Jetty 7.6.17.v20150415
  * Jetty 8.1.17.v20150415
  * MariaDB 10.0.21
  * MariaDB 5.5.45
  * MariaDB Galera Cluster 10.0.21
  * Nginx 1.9.4
  * OpenSSH 7.1p1 (if installed from sources)
  * PHP 5.6.13, 5.5.29, 5.4.45
  * PHP: ionCube loader 5.0.18
  * Pure-FTPd 1.0.42
  * Redis 3.0.4
  * Ruby 2.2.3, 2.0.0-p647
  * Use pecl-jsmin-1.1.0
# Fixes:
  * Allow to re-install deleted D7/D6 platforms when dev doesn't exist
  * Do not install phpunit -- it adds many PHP tools we don't need
  * Drush requires php-eval to run drush_find_tmp() in sql-sync
  * Fix apache cleanup
  * Fix invalid regex in the INI docs
  * Improve auto-healing for SSHd
  * Improve Nginx DoS an DDoS protection
  * Improve pdnsd auto-healing
  * Improve SSL Docs to add more detail about multidomain certificates #757
  * Issue #766 - Fix for broken boa in-octopus procedure
  * Nginx: Fix support for s3/files/styles (s3fs)
  * Restart PHP-FPM if too many running childs are detected
  * Sync .htaccess with D7 core
  * Sync keywords for exceptions in daily.sh with global.inc
  * Use short sleep on firewall temp blocks cleanup

Previous, un-applied BOA update tickets:

BOA 2.4.5 ticket:864
BOA 2.4.4 ticket:863
BOA 2.4.3 ticket:854

Is there any chance we might be able to discuss what we are going todo regarding the on-going hosting of the Transition Network Drupal site? Some ideas were posted two weeks ago here ticket:754#comment:61

#863: BOA-2.4.4

chris — Thu, 09 Jul 2015 13:36:20 GMT

Last Friday a new version of BOA came out (for unknown reason(s) the automatic notification of new versions stopped working some months ago):

### Stable BOA-2.4.4 Release - Full Edition
### Date: Fri Jul  3 12:08:29 PDT 2015
### Milestone URL: https://github.com/omega8cc/boa/milestones/2.4.4
### Latest hotfix added on: Wed Jul  8 01:28:08 PDT 2015
  @=> Includes Aegir Hostmaster 2.x-head with improvements
  @=> Includes Aegir Provision 3.x-head with improvements
  @=> Includes Drush 7 customized for BOA
# Release Notes:
  This BOA release includes several important system upgrades and bug fixes.
  All supported Aegir platforms have been updated with latest Drupal cores.
  This version automatically switches all hosted sites to PHP 5.5 on systems
  hosted and managed remotely by Omega8.cc support team, unless you have
  explicitly switched your Octopus instance to use PHP version you prefer.
  Using PHP older than 5.5 is strongly discouraged, for security, stability and
  performance reasons.
# Changes:
  * Do not change mysql root password by default -- workaround for #642
  * Enable advagg_async_generation by default
  * Logic update for /root/.high_traffic.cnf
  * Redis Integration Module: Update to version mod-26-06-2015
  * Use modern ssl_ciphers in all templates by default
# System upgrades:
  * cURL 7.43.0 (if installed from sources)
  * Drush mini-7-30-06-2015 -- fixes #734
  * MariaDB 5.5.44
  * MariaDB Galera Cluster 10.0.20
  * Nginx 1.9.1
  * OpenSSH 6.9p1 (if installed from sources)
  * OpenSSL 1.0.1o (if installed from sources)
  * PHP 5.4.42
  * PHP 5.5.26
  * PHP 5.6.10
  * PHPRedis master-27-06-2015
  * Pure-FTPd 1.0.41
  * vnStat 1.14
# Fixes:
  * Add 'grep' to overssh -- a list of commands allowed to execute over SSH
  * Broken pdnsd configuration breaks DNS resolver -- fixes #701
  * Do not force update_agents()
  * Do not modify rkey/debug args in barracuda log/system upgrade mode
  * Don't remove Drupal 6 core themes -- fixes #738
  * Fix for legacy vnStat config
  * Fixed backboa/duobackboa retrieve from remote host -- fixes #741
  * Improve system cron tasks queue
  * Incorrect permissions on /usr/bin/optipng - fixes #722
  * Mitigate LOGJAM - fixes #723
  * Restart Postfix after system DNS update -- #701
  * Skip daily reload on high traffic instances
  * Sync SQL connection limits with _PHP_FPM_WORKERS variable - fixes #699
  * Use _AWS_URL to properly handle us-east-1 exception
  * Use 2048 bit where possible - see #723
  * Use better default value for advagg_cache_level - fixes #726

I no longer know what the Transition Network policy is regarding BOA updates, the last one BOA 2.4.3, on ticket:854 was never applied and as far as I'm aware there is no agreement / policy regarding what to do regarding PHP 5.3, see ticket:754.

#889: BOA-2.4.7

chris — Tue, 08 Dec 2015 17:35:35 GMT

Following is the Changelog, please note this contains:

This BOA release (2.4.7) is the last release which still supports deprecated PHP versions: 5.3 and 5.4

And:

We will support Drupal/Pressflow 6 in all new releases, as long as available PHP versions will allow to use it (we run our own Pressflow 6 based site on PHP 5.6 for many months with zero issues). For more details please check: https://github.com/omega8cc/boa/issues/824

If we are sticking with Drupal 6 and BOA then I would suggest we should test the site on PHP 5.6 and if it is OK then upgrade, perhaps this should be done by spinning up a new virtual server and installing a new version of BOA on it and then copying the site over for testing?

See also the tickets for the previous, un-applied, updates:

BOA 2.4.6 ticket:872
BOA 2.4.5 ticket:864
BOA 2.4.4 ticket:863
BOA 2.4.3 ticket:854

And this ticket on the question of updating PHP: ticket:754

### Stable BOA-2.4.7 Release - Full Edition
### Date: Fri Dec  4 08:09:21 PST 2015
### Milestone URL: https://github.com/omega8cc/boa/milestones/2.4.7
### Latest hotfix added on: Mon Dec  7 15:32:44 PST 2015
  @=> Includes Aegir Hostmaster 2.x-head with improvements
  @=> Includes Aegir Provision 3.x-head with improvements
  @=> Includes Drush 7 customized for BOA
# Release Notes:
  This BOA release includes several important system upgrades and bug fixes,
  with most notable features and changes listed below.
  @=> All supported Aegir platforms have been updated with latest Drupal cores
  @=> Drupal 8 support for custom platforms in the ~/static directory tree
      will be included, along with Drush 8 and PHP 7 in the *upcoming*
      BOA-3.0.0 release: https://github.com/omega8cc/boa/milestones/3.0.0
  @=> This BOA release (2.4.7) is the last release which still supports
      deprecated PHP versions: 5.3 and 5.4 -- You should switch to PHP 5.6
      or at least 5.5 as soon as possible, or you will not be able to upgrade
      to newer BOA versions after 2.4.7 -- https://omega8.cc/node/330
  @=> What are BOA plans for Drupal 6 support after February 24th, 2016?
      We will support Drupal/Pressflow 6 in all new releases, as long as
      available PHP versions will allow to use it (we run our own Pressflow 6
      based site on PHP 5.6 for many months with zero issues). For more details
      please check: https://github.com/omega8cc/boa/issues/824
  @=> SSH keys for root are required by newer OpenSSH versions used in BOA
      BOA installs SSH from sources by default (Debian only). This means that
      password based access for root will not work once BOA is installed or
      upgraded to current stable version. It is a result of OpenSSH changes
      in recent releases and not BOA specific change. BOA will deny the initial
      install and Barracuda will refuse to run upgrade if it detects that system
      root has no SSH keys added and only password based access is available.
      You can still modify this behaviour in /usr/etc/sshd_config but future
      OpenSSH versions may still revert such changes, so it is not recommended.
  @=> BOA switched from SPDY to HTTP/2 + PFS on all supported OS versions
# Changes:
  * Allow to disable SQL monitoring with /root/.no.sql.cpu.limit.cnf -- #799
  * Disable page caching on the fly where needed
  * Disable temporarily support for broken Restaurant distro
  * Do not rebuild features and entities on cache clear
  * Document new requirement: SSH keys for root -- fixes #786 #833
  * Make ioncube_loader optional and disable by default with _PHP_IONCUBE=NO
  * Nginx SSL: enable OCSP stapling by default
  * Nginx SSL: enable OCSP stapling for existing HTTPS vhosts
  * Nginx: Add ssl_dhparam to existing vhosts, if needed
  * Nginx: HTTP/2 replaces SPDY -- fixes #624
  * PHP: Add YAML extension with LibYAML
  * Preserve customized /etc/sysctl.conf -- fixes #789
  * Run modules ON/OFF only weekly -- requires _MODULES_FIX=YES (default is NO)
  * Run most of crontab, install and upgrade tasks with low priority using
    nice and ionice -- fixes #780
# System upgrades:
  * cURL 7.45.0 (if installed from sources)
  * GEOS 3.5.0 (requires _PHP_GEOS=YES)
  * Git 2.6.1 (if installed from sources)
  * MariaDB 10.0.22
  * MariaDB 5.5.46
  * MariaDB Galera Cluster 10.0.22
  * Nginx 1.9.7
  * OpenSSL 1.0.2e (used only in custom built Nginx)
  * PHP 5.5.30
  * PHP 5.6.16
  * Redis 3.0.5
# Fixes:
  * Add /root/.skip_cleanup.cnf support
  * Add feature branch testing in HEAD
  * Avoid load spikes caused by long running tasks
  * Avoid race conditions on multi-line sed replacement -- fixes #806
  * Clean up any remaining procs zombies
  * Clean up postfix queue to get rid of bounced emails
  * Disable ioncube and opcache for HHVM
  * Disable Redis for Hostmaster in the backend
  * Do not allow to install non-standard OpenSSH on Ubuntu
  * Do not break /data/all/cpuinfo permissions on Octopus upgrade
  * Do not run 'apt-get autoremove' automatically
  * Do not use wrapper for dot-files cleanup
  * Document better BOA aggressive installation behavior -- fixes #811
  * Document boa in-octopus command -- fixes #817
  * Don't strip $args from $request_uri in redirects
  * Fix cron schedule for upgrades
  * Fix for broken Git on Ubuntu
  * Fix for not working PHP rebuild check
  * Fix for not working syncpass tool
  * Fix PHP deprecated warning in D8 -- fixes #804
  * Ignore 'env COLUMNS' sent by Drush remotely -- fixes #373
  * Ignore daily.sh in clear.sh
  * Improve _SQUEEZE_TO_WHEEZY procedure -- #627
  * Improve cron tasks schedule
  * Improve daily cleanup performance + support for /root/.giant_traffic.cnf
  * Improve devpts check -- fixes #788
  * Improve docs/MIGRATE.txt
  * Improve resolv.conf auto-recovery procedure
  * Improve system check -- fixes #811
  * Move Redis restart procedure to correct script
  * PHP: Add missing path to open_basedir for CLI
  * Remove debug code to not kill the initial install
  * Remove not working /etc/logrotate.d/lshell -- fixes #823
  * Update advagg auto configuration variables -- fixes #792
  * Update boa/lib/functions/helper.sh.inc with current OS -- fixes #787
  * Update FPM workers autoconf logic
  * Update the cache cleanup logic
  * Use better placeholder for solr_integration_module variable
  * Use correct DPkg::Options for dist-upgrade -- fixes #627
  * Use known MySQLTuner version -- fixes #827
  * Use LibYAML 0.1.6
  * Use opcache.restrict_api
  * Use sha256 for self-signed certs

If we are not going to stick with BOA then can we please consider the suggestions in this comment? ticket:754#comment:61

#589: Blocking spammers at a firewall level

chris — Fri, 06 Sep 2013 09:48:59 GMT

At the meeting on 5th September ticket:585 one thing we discussed was that for August 2013:

More data is transferred for /user/register than the front page, 5.1GB compared to 3.6GB.

Most of this will be spam bots trying to register to post spam. Jim suggested that we could look at blocking some of these spam bots at a firewall level to save on resources. This ticket is to follow up on this suggestion.

#544: CSF / LDF false positive blocks on Puffin

chris — Sat, 04 May 2013 11:02:53 GMT

Ticket to keep track of CSF /LDF issues on Puffin, see wiki:PuffinServer#CSFLDF

#754: Can we upgrade from PHP 5.3?

chris — Sat, 05 Jul 2014 20:52:48 GMT

The Transition Network site is running on PHP 5.3 and it's got a very serious security issue which potentially enables a remote attacked to get the servers private SSL key. Would Drupal work OK with PHP 5.4?

#896: Chive access to TN Drupal DB

chris — Mon, 18 Jan 2016 17:56:44 GMT

Ade would like to give the developers of the new Transition Network WordPress site access to the live database via Chive.

#683: Create Aegir account for Paul

sam — Wed, 22 Jan 2014 12:42:48 GMT

Hi Jim

It looks like i@… will be joining us to pick up some of your work.

Could you create an Aegir account for him please?

I was also wondering about the Transition Network Github repo, do we need to do anything to hand that over?

Thanks

Sam

#218: Debian upgrades and updates

chris — Thu, 06 Jan 2011 13:34:16 GMT

This is a ticket to track debian upgrades to the wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer the time they take.

See:

These updates are generally done using the wiki:AptitudeUpdateScript and this records all the changes in the /root/Changelog and then the contents of the Changelog are pasted into the ticket to document the upgrade.

This ticket was was originally used for the ~~wiki:DevelopmentServer~~ and the ~~wiki:NewLiveServer~~.

#531: Disk usage on puffin

chris — Wed, 10 Apr 2013 13:12:24 GMT

The disk usage on puffing is currently at 85% and it's been going up at around 5% a week, see:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/df.html

This will become a critical issue in a couple of weeks, it would be good to find and address the cause before then.

#574: EFF: How HTTPS Everywhere affects transitionnetwork.org

chris — Thu, 25 Jul 2013 08:09:31 GMT

The following email was sent via the whois contact details for the transitionnetwork.org domain name, see also ticket:571 -- the changes made on ticket:571 has broken the site for anon Firefox users with the EFF HTTPS Everywhere browser extension installed.

Hi,

You're receiving this note because transitionnetwork.org is part of our HTTPS Everywhere browser extension, and an upcoming change to the way Firefox handles HTTPS pages may cause your site to display or function incorrectly. We want to make sure that the nearly 3 million HTTPS Everywhere users have the best possible experience while browsing, so we're asking you to please take a minute and test how your site behaves in Firefox 23. You can find out more about our software at

https://www.eff.org/https-everywhere

To see the rules affecting your site, you can visit the HTTPS Everywhere Atlas at

https://www.eff.org/https-everywhere/atlas/domains/transitionnetwork.org.html

The Atlas shows both rules in the development and stable versions of our extension. Rules in the stable version are used by millions of users, while development rules are used by tens of thousands of users. Development rules are now being tested but will be migrated to the stable version in the future.

An upcoming change (described below) in how the Firefox browser renders HTTPS content makes it especially important that you check that your site is prepared for HTTPS access. We urge you review the rules affecting your site and also to test it using HTTPS Everywhere with the upcoming version of Firefox.

*NEW FIREFOX CONTENT SECURITY POLICY*: In the upcoming Firefox 23 browser release, due out the week of August 6, Firefox will stop loading certain "active" content such as scripts and stylesheets from insecure http:// URLs if they've been included from a secure https:// site. If the HTTPS Everywhere rules send users to the secure version of your site but the secure version includes some content loaded over an insecure connetion, the rendering of your site may become broken for Firefox users with HTTPS Everywhere installed after they upgrade to Firefox 23. You can check this by downloading a preview release of Firefox 23, installing HTTPS Everywhere, and visiting your site. We urge all web site operators to protect their users by making sure that all site content is always loaded over a secure connection. A preview version of Firefox 23 is available now at https://www.mozilla.org/en-US/firefox/beta/ and the HTTPS Everywhere extension is at https://www.eff.org/https-everywhere

HTTPS Everywhere rules instruct browsers to access certain specified resources securely -- over HTTPS -- even if the user typed or followed a non-HTTPS link or even if the resources were included in a page via a non-HTTPS URL. For example, it might automatically rewrite

http://www.transitionnetwork.org/

to

https://www.transitionnetwork.org/

or make some similar change.

The goal of this rewriting is to protect as much as possible of every web site against sniffing and tampering by ensuring that as many site resources as possible are loaded over a secure HTTPS connection.

When web sites are accessed insecurely, users are vulnerable to attacks by other users on their networks. HTTPS Everywhere aims to activate sites' existing HTTPS protection more consistently to make sure users are as well-protected from these attacks as possible -- including attacks like sidejacking and SSL stripping.

http://www.firesheep.org/ http://www.thoughtcrime.org/sslstrip

As a result, we think there's an emerging consensus to make all web sites secure, not just financial sites and login pages. Providing a secure connection helps protect users' login credentials, but also helps protect their privacy and security even when accessing public resources, for example by preventing network operators from injecting malware downloads.

The goal of HTTPS Everywhere is to make the web more secure and help users express their preference to use the secure version of every site automatically, even on sites where this is not the default. We don't want to break sites or harm users' experience. So, we encourage webmasters to test the effect of HTTPS Everywhere on their sites and fix any problems that result -- ideally, by making sure that all resources that make up a site are available over HTTPS, using a current, valid certificate. Although we only include rules that we've been told and believe work properly, we can't always anticipate whether a rule adversely affects a site, especially if the site's URL structure, use of CDNs, or level of HTTPS support changes over time.

We are always happy to receive bug reports, updates, and fixes to HTTPS Everywhere rules. We will also make rules inactive by default if a site operator asks us to. Although we are working for a web where all sites are secure, we are not trying to use this software to force sites to use HTTPS against their operators' wishes. You can send any corrections, updates, or requests to https-everywhere-rules@… (which is a public and publicly-archived mailing list), or by replying to this e-mail address.

Thanks for your attention!

Seth Schoen, Senior Staff Technologist, Electronic Frontier Foundation for the HTTPS Everywhere development team

#895: HTTPS wildcard *.transitionnnetwork.org expires on 22nd January 2016

chris — Mon, 11 Jan 2016 09:56:17 GMT

Unless I hear otherwise I'll renew the *.transitionnnetwork.org cert which is used by PuffinServer, PenguinServer and ParrotServer at a cost of £130.50 on or before the 22nd January 2016 when the current one expires.

An alternative would be to use Free HTTPS certificates from Let's Encrypt but this would take some time to set up as Let's Encrypt don't provide wild card certs.

#717: Heartbleed / Open SSL vunerability

sam — Wed, 09 Apr 2014 20:50:37 GMT

Hi Chris

I don't think I have seen a ticket for this? http://heartbleed.com/

Does it affect us?

Thanks

Sam

#837: Iframe in a panel page

sam — Thu, 12 Mar 2015 11:55:21 GMT

Hi Ben

I'm trying to embed a Eventbrite form into the 'tickets' block on this page: https://www.transitionnetwork.org/conference-2015

It looks like it's going to appear in the preview here: https://www.transitionnetwork.org/node/39195/panel_content

But when I view the actual page it's just a big white space.

Could you estimate how long it would take to get it working?

Thanks

Sam

#673: Install mosh - the mobile shell

chris — Mon, 13 Jan 2014 11:27:40 GMT

The mobile shell enables terminal connections to stay up when using really bad connections, for example on a train, see:

http://mosh.mit.edu/

#846: Load Spikes on BOA PuffinServer

chris — Thu, 16 Apr 2015 11:16:00 GMT

Creating this as a ticket to record load spikes and related site outages.

See wiki:PuffinServer#LoadSpikes for links to historic issues of this nature.

#555: Load spikes causing the TN site to be stopped for 15 min at a time

chris — Wed, 29 May 2013 09:53:52 GMT

The BOA /var/xdrago/second.sh script is run every minute via the root crontab and if it detects a certain load level it changes the nginx config to a "high load" config which results in bots being served 503 errors when they spider the site, see ticket:563. When the load goes higher and hits another threshold the second.sh script kills the webserver applications, nginx and php-fpm, and waits till the load has dropped before starting them up again. This was happening once or twice a day following the increase in traffic around the launch of The Power of Just Doing Stuff. This has been addressed by multiplying the thresholds by 5 in second.sh.

Original Description

This morning at 10:19:24 I received the following alert from puffin:

Subject: lfd on puffin.webarch.net: High 5 minute load average alert - 6.59
Time:                    Wed May 29 10:17:02 2013 +0100
1 Min Load Avg:          23.39
5 Min Load Avg:          6.59
15 Min Load Avg:         2.57
Running/Total Processes: 44/326

At 10:21:57 I got an alert regarding ssh:

Service: SSH
Host: puffin
Address: puffin.webarch.net
State: CRITICAL
Date/Time: Wed May 29 10:21:57 BST 2013
Additional Info:
CRITICAL - Socket timeout after 10 seconds

Then at 10:26:47 ssh appeared to have recovered:

Service: SSH
Host: puffin
Address: puffin.webarch.net
State: OK
Date/Time: Wed May 29 10:26:47 BST 2013
Additional Info:
SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0)

But then pingdom reported at 10:29:07:

www.transitionnetwork.org is down since 29/05/2013  10:24:57.

There was then a report regarding Nginx at 10:32:07:

Notification Type: PROBLEM
Service: HTTP
Host: puffin
Address: puffin.webarch.net
State: CRITICAL
Date/Time: Wed May 29 10:32:07 BST 2013
Additional Info:
Connection refused

So at 10:33:47 I ssh'd in and found that php53-fpm and nginx were not running and it took several attempts to get them running again.

The up email from pingdom reported:

www.transitionnetwork.org is UP again at 29/05/2013  10:36:57, after 12m of downtime.

I can't find anything in the logs to indicate what caused the load spike and php-fpm and nginx to stopp running.

#769: Locked myself out of puffin again

annesley — Mon, 04 Aug 2014 08:40:53 GMT

really sorry. locked my IP out of puffin again. please could you clear it?

#573: MariaDB 5.5.32 is available for Puffin

chris — Thu, 18 Jul 2013 19:11:24 GMT

I could either update this via aptitude and it would be quick and have little down time or I could update it via BOA and it'll take 3 or 4 times as long and involved extra downtime.

Jim -- which way would you like it done?

The MariaDB project is pleased to announce the immediate availability of MariaDB 5.5.32.

This is a bug-fix release. See the Release Notes and Changelog for details.

MariaDB 5.5.32 Stable (GA)

Release Notes: https://kb.askmonty.org/en/mariadb-5532-release-notes
Changelog: https://kb.askmonty.org/en/mariadb-5532-changelog
http://lists.askmonty.org/pipermail/announce/2013-July/000048.html

#593: Migrating Puffin to a ZFS file server

chris — Wed, 11 Sep 2013 15:52:47 GMT

This ticket is for the migration of PuffinServer to a ZFS file server, this will involve some downtime but should result in better IO and easier backups.

#591: Move MySQL temporary directory to tmpfs

jim — Mon, 09 Sep 2013 12:47:41 GMT

Chris, please read: http://2bits.com/articles/reduce-your-servers-resource-usage-moving-mysql-temporary-directory-ram-disk.html

I think we could easily drop a little MySQL memory to give it some in-memory disk space to do the temporary table munching Drupal is causing it. I see there are already some mounted tmpfs partitions.

Related to #590 (proposal L: Review slow query log, explain queries, tweak as necessary/flag poorly behaving modules)

What do you think? Worth doing?

#499: MySQL backup dump error on puffin

chris — Tue, 26 Feb 2013 01:26:53 GMT

Warnings from /etc/backup.d/20.mysql :

Warning: mysqldump: Got error: 1142: "SELECT,LOCK TABL command denied to user 'root'@'localhost' for table 'cond_instances'" when using LOCK TABLES
Warning: Failed to dump mysql databases performance_schema

#760: New BOA-2.2.7 Stable Edition

chris — Thu, 17 Jul 2014 08:34:22 GMT

Email from wiki:PuffinServer:

There is new BOA-2.2.7 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

The Changelog:

### Stable BOA-2.2.5 Release - Full Edition
### Date: Thu May  8 11:59:23 PDT 2014
### Includes Aegir 2.x-boa-custom version.
### Latest hotfix added on: Sat May 10 09:05:19 PDT 2014
# Release Notes:
  This release includes no new features, but does include bug fixes plus latest
  Drupal 7.28.1 and Pressflow 6.31.2 core in all built-in Octopus platforms.
  There are also three updated distributions included, as listed below.
  We also list here all hot-fixes applied to previous stable after its release.
# Important - Read This First! (for self-hosted BOA only)
  If you haven't run full barracuda+octopus upgrade to latest BOA Stable
  Edition yet, don't use any partial upgrade modes explained in docs/UPGRADE.txt
  Once new BOA Stable is released, you must run *full* upgrades with commands:
  $ barracuda up-stable
  $ octopus up-stable all both
  For silent, logged mode with e-mail message sent once the upgrade is
  complete, but no progress is displayed in the terminal window, you can run
  alternatively, starting with screen session to avoid incomplete upgrade
  if your SSH session will be closed for any reason before the upgrade
  will complete:
  $ screen
  $ barracuda up-stable log
  $ octopus up-stable all both log
  Note that the silent, non-interactive mode will automatically say Y/Yes
  to all prompts and is thus useful to run auto-upgrades scheduled in cron.
  If you have skipped some recent BOA releases, and you have new default config
  option: _PERMISSIONS_FIX=NO in your /root/.barracuda.cnf configuration file,
  plus, you are not sure if you follow best practices for managing permissions
  as recommended in our docs: https://omega8.cc/node/116 then we recommend
  that you change it to _PERMISSIONS_FIX=YES temporarily, or even permanently
  if your VPS is fast enough, and then run this powerful script as root:
  $ bash /var/xdrago/daily.sh
  Note that BOA 'legacy' mode is still at version 2.1.3
# Updated Octopus platforms:
  Commons 3.12 ----------------- https://drupal.org/project/commons
  Open Atrium 2.18 ------------- https://drupal.org/project/openatrium
  Open Outreach 1.6 ------------ https://drupal.org/project/openoutreach
# Changes in this release:
  * Add rsyslog/sysklogd to auto-healing procedures.
  * Make the aggressive scan_nginx mode optional and use old mode by default.
  * Nginx: Add HiScan to blocked crawlers list.
  * Nginx: Add Riddler to blocked crawlers list.
  * PHP: Use pm.process_idle_timeout = 10s for speed and RAM optimization.
# System upgrades in this release:
  * MySecureShell 1.33
  * PHP 5.4.28
  * PHP 5.5.12
# Fixes in this release:
  * Always define _PHP_CN variable properly.
  * Firewall: Sync CONNLIMIT for web ports with updated limit_conn in Nginx.
  * Fix for _NGINX_DOS_LIMIT logical error in the scan_nginx template.
  * Force Pure-FTPd server re-install if key files are missing for any reason.
  * Issue #2237167 - Improve authorized IPs detection in all protected vhosts.
  * Issue #2262935 - Modules dir must be group writable in custom platforms.
  * Nginx: Do not overwrite custom symlinks to the Under Construction template.
  * Nginx: Update limit_conn in all instances and vhosts on Barracuda upgrade.
  * PHP: Delete pear in legacy paths, if still exists.
  * PHP: Fix for CVE-2014-0185 privilege escalation in FPM (doesn't affect BOA)
  * Postfix: Force re-install if broken permisions detected on upgrade.
  * Pressflow 6: Fix #GH 84 by using drupal_page_is_cacheable().
  * Pressflow 6: Merge pull request #GH 85 from pressflow/SA-CORE-2014-002-fix.
  * Pressflow 6: Remove duplicate openid_update_6001().
  * Revert "Force MariaDB 5.5 re-install".
  * Set the TERM env variable if missing to avoid errors.
  * Skip packages set on hold when running apticron.
  * The ~/static/control must be writeable by lshell user to manage ctrl files.
  * Add extra cron semaphore to prevent concurrent cron invocations via
    multiple running runner.sh instances.

I can't see any issues that have an immediate impact on us, I'll do the upgrade late one evening.

#765: New BOA-2.2.8 Stable Edition

chris — Sun, 27 Jul 2014 08:08:55 GMT

Email from wiki:PuffinServer:

There is new BOA-2.2.8 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

The Changelog contains:

### Stable BOA-2.2.5 Release - Full Edition
### Date: Thu May  8 11:59:23 PDT 2014
### Includes Aegir 2.x-boa-custom version.
### Latest hotfix added on: Sat May 10 09:05:19 PDT 2014
# Release Notes:
  This release includes no new features, but does include bug fixes plus latest
  Drupal 7.28.1 and Pressflow 6.31.2 core in all built-in Octopus platforms.
  There are also three updated distributions included, as listed below.
  We also list here all hot-fixes applied to previous stable after its release.
# Important - Read This First! (for self-hosted BOA only)
  If you haven't run full barracuda+octopus upgrade to latest BOA Stable
  Edition yet, don't use any partial upgrade modes explained in docs/UPGRADE.txt
  Once new BOA Stable is released, you must run *full* upgrades with commands:
  $ barracuda up-stable
  $ octopus up-stable all both
  For silent, logged mode with e-mail message sent once the upgrade is
  complete, but no progress is displayed in the terminal window, you can run
  alternatively, starting with screen session to avoid incomplete upgrade
  if your SSH session will be closed for any reason before the upgrade
  will complete:
  $ screen
  $ barracuda up-stable log
  $ octopus up-stable all both log
  Note that the silent, non-interactive mode will automatically say Y/Yes
  to all prompts and is thus useful to run auto-upgrades scheduled in cron.
  If you have skipped some recent BOA releases, and you have new default config
  option: _PERMISSIONS_FIX=NO in your /root/.barracuda.cnf configuration file,
  plus, you are not sure if you follow best practices for managing permissions
  as recommended in our docs: https://omega8.cc/node/116 then we recommend
  that you change it to _PERMISSIONS_FIX=YES temporarily, or even permanently
  if your VPS is fast enough, and then run this powerful script as root:
  $ bash /var/xdrago/daily.sh
  Note that BOA 'legacy' mode is still at version 2.1.3
# Updated Octopus platforms:
  Commons 3.12 ----------------- https://drupal.org/project/commons
  Open Atrium 2.18 ------------- https://drupal.org/project/openatrium
  Open Outreach 1.6 ------------ https://drupal.org/project/openoutreach
# Changes in this release:
  * Add rsyslog/sysklogd to auto-healing procedures.
  * Make the aggressive scan_nginx mode optional and use old mode by default.
  * Nginx: Add HiScan to blocked crawlers list.
  * Nginx: Add Riddler to blocked crawlers list.
  * PHP: Use pm.process_idle_timeout = 10s for speed and RAM optimization.
# System upgrades in this release:
  * MySecureShell 1.33
  * PHP 5.4.28
  * PHP 5.5.12
# Fixes in this release:
  * Always define _PHP_CN variable properly.
  * Firewall: Sync CONNLIMIT for web ports with updated limit_conn in Nginx.
  * Fix for _NGINX_DOS_LIMIT logical error in the scan_nginx template.
  * Force Pure-FTPd server re-install if key files are missing for any reason.
  * Issue #2237167 - Improve authorized IPs detection in all protected vhosts.
  * Issue #2262935 - Modules dir must be group writable in custom platforms.
  * Nginx: Do not overwrite custom symlinks to the Under Construction template.
  * Nginx: Update limit_conn in all instances and vhosts on Barracuda upgrade.
  * PHP: Delete pear in legacy paths, if still exists.
  * PHP: Fix for CVE-2014-0185 privilege escalation in FPM (doesn't affect BOA)
  * Postfix: Force re-install if broken permisions detected on upgrade.
  * Pressflow 6: Fix #GH 84 by using drupal_page_is_cacheable().
  * Pressflow 6: Merge pull request #GH 85 from pressflow/SA-CORE-2014-002-fix.
  * Pressflow 6: Remove duplicate openid_update_6001().
  * Revert "Force MariaDB 5.5 re-install".
  * Set the TERM env variable if missing to avoid errors.
  * Skip packages set on hold when running apticron.
  * The ~/static/control must be writeable by lshell user to manage ctrl files.
  * Add extra cron semaphore to prevent concurrent cron invocations via
    multiple running runner.sh instances.

I'll do this update tonight, following wiki:PuffinServer#UpgradingBOA

#775: New BOA-2.2.9 Stable Edition available

chris — Thu, 07 Aug 2014 08:39:18 GMT

Email from wiki:PuffinServer:

There is new BOA-2.2.9 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/boa-changes

The Changelog:

### Stable BOA-2.2.9 Release - Full Edition
### Date: Wed Aug  6 17:08:10 PDT 2014
### Includes Aegir 2.x-boa-custom version.
# Release Notes:
  This release includes updated versions of all supported Drupal platforms to
  provide latest Drupal 7 and Pressflow 6 core, plus some changes, improvements,
  bug fixes, and five (5) updated Octopus platforms.
  NOTE: Since the first Edition in the BOA-2.3.x series is not ready for release
  yet, and new Drupal core has been released to fix security issues, followed
  by yet another release to fix serious regressions, followed by yet another
  security release, we have decided to make it available to everyone and release
  yet another stable BOA-2.2.x Edition.
  IMPORTANT! This is the last Edition in the 2.2.x series, which marks the end
  of Drupal 5, PHP 5.2 and Drush 4 support. Next Edition will open 2.3.x series,
  which will allow us to provide newer Aegir version with built-in Drush 6
  support, sites in subdirectories, and many Aegir User Interface improvements.
  If you still host any Drupal 5 sites or you are using PHP 5.2 for D6 sites,
  you will not be able to upgrade to the next 2.3.x Edition and you will have to
  stay on the 'legacy' BOA 2.2.x version, which will receive only system
  security upgrades, but no further feature nor bugfix releases.
  This also means that from now on the 'legacy' 2.2.x version will no longer
  receive Drupal core upgrades, even if there will be security core releases.
  It is time to upgrade away from Drupal 5 and away from PHP 5.2, if still used.
# Updated Octopus platforms:
  aGov 1.2 --------------------- https://drupal.org/project/agov
  Guardr 1.10 ------------------ https://drupal.org/project/guardr
  Open Outreach 1.9 ------------ https://drupal.org/project/openoutreach
  OpenPublic 1.0-rc4 ----------- https://drupal.org/project/openpublic
  Panopoly 1.10 ---------------- https://drupal.org/project/panopoly
# New features and enhancements in this release:
  * RVM: Add exceptions for gems which can't be installed in Limited Shell.
  * Shell: Compass Tools: Allow to access guard.
  * Shell: Improve config to better support advanced Drush commands over SSH.
  * Shell: Improve Drush over SSH experience
# Changes in this release:
  * Drush: Upgrade command line version 6 to mini-6-06-08-2014
# System upgrades in this release:
  * MariaDB 5.5.39
  * Nginx 1.7.4
  * OpenSSL 1.0.1i (if installed from sources)
# Fixes in this release:
  * Add cleanup for .tmp in sub-accounts.
  * Add cleanup for drush-backups leftovers.
  * Add cleanup for various /var/backups/* leftovers.
  * Add daily auto-cleanup for ghost vhosts, platforms and drush aliases.
  * Add exception for symlinked /data/all
  * Add hint for HTTPS-only mode forced in local.settings.php
  * Fix -mtime expected values.
  * Fix cleanup for .restore vhost leftovers.
  * Fix Nginx monitor to respect all whitelisted POST requests in both modes.
  * Fix permissions on sites/all/{modules,libraries,themes} globally.
  * Improve RVM cleanup.
  * Make sure that local IPs are never blocked by mistake.
  * Never touch websh wrapper to avoid high load because of redirect loop.
  * Nginx: Fix limreq also for some really old vhosts.
  * Nginx: Modify only vhosts known as included in the protected mode.
  * Remove debugging mode in old codebases cleanup.
  * Restore default websh wrapper symlink as fast as possible.
  * Run manage_ltd_users every 3 minutes instead of every minute.
  * Update regex for exceptions in Nginx monitoring.

I'll do this update after the meeting tonight.

#784: New BOA-2.3.0

chris — Tue, 09 Sep 2014 08:52:22 GMT

These are the updates from the Changelog:

### Stable BOA-2.3.0 Release - Full Edition
### Date: Mon Sep  8 08:42:01 PDT 2014
### Includes Aegir 2.1 with improvements
# Release Notes:
  This new BOA Edition introduces latest Aegir 2.1 stable version with newest
  Drush 6 in the backend and with support for Drupal sites in subdirectories
  enabled by default, among many other improvements included in this version,
  like tasks list per site, ability to search in the sites list per domain name
  and/or profile, to schedule tasks in batches, to select any existing domain
  alias as a redirect target, but without the need to rename the site, etc.
  While Barracuda 2.3.0 can continue to run and even upgrade if needed also
  the very old PHP 5.2 version, only Octopus instances running at least PHP 5.3
  or newer in both FPM and CLI mode can be upgraded to Octopus 2.3.0 Edition.
  If you are still using PHP 5.2 in your Octopus instance, you will not
  receive Aegir nor Drupal Platforms upgrade, but the Barracuda part of your
  system will receive upgrade to 2.3.0 anyway, so it will be ready to support
  your outdated Octopus instance upgrade as soon as you will switch it to
  modern and secure PHP version -- which is easy!
  Let's quote the original how-to for reference:
#-### Support for PHP FPM/CLI version safe switch per Octopus instance
  This allows to easily switch PHP version by the instance owner w/o system
  admin (root) help. All you need to do is to create ~/static/control/fpm.info
  and ~/static/control/cli.info file with a single line telling the system
  which available PHP version should be used (if installed): 5.5 or 5.4 or 5.3
  Only one of them can be set, but you can use separate versions for web access
  (fpm.info) and the Aegir backend (cli.info). The system will switch versions
  defined via these control files in 5 minutes or less. We use external control
  files and not any option in the Aegir interface to make sure you will never
  lock yourself by switching to version which may cause unexpected problems.
#-### Legacy mode moves to 2.2.x branch
  From now on, the 'legacy' install and upgrade mode available in all meta-
  installers will utilize branch 2.2.x instead of deprecated 2.1.x series.
# Updated Octopus platforms:
  aGov 1.4 --------------------- https://drupal.org/project/agov
  Guardr 1.12 ------------------ https://drupal.org/project/guardr
  Open Academy 1.1 ------------- https://drupal.org/project/openacademy
  Restaurant 1.0-b9 ------------ https://drupal.org/project/restaurant
# New features and enhancements in this release:
  * It is now possible to add stable Octopus instances w/o forcing Barracuda
    upgrade, plus optionally with no platforms added by default -- usage:
    $ boa {in-octopus} {email} {o2} {mini|max|none}
  * Add default aggressive php-fpm monitoring + /root/.no.fpm.cpu.limit.cnf
  * Allow to define always disabled modules via _MODULES_FORCE variable.
  * Better wait limits on connection testing for slow network / long distance.
  * Issue #362 - Add imageapi_optimize binaries via IMG in _XTRAS_LIST
  * Make firewall management faster with randomized schedule.
  * Procs monitor runs every 3 seconds.
  * Run mysql_proc_control every 5 seconds for better results.
# Changes in this release:
  * Delete default profiles in the hostmaster platform.
  * Disable _DEBUG_MODE if not enabled on the fly.
  * Drush: Upgrade command line version 6 to mini-6-06-09-2014
  * Nginx: Remove deprecated code - _HTTP_WILDCARD is already used by default.
  * Nginx: Use limit_conn protection only for known dynamic requests.
  * Remove dependency on Update Manager globally.
  * Set hosting_default_profile to 'minimal' to improve Ubercart 3 visibility.
  * Use Provision CiviCRM boa-2.3.0-dev
# System upgrades in this release:
  * Git 2.1.0 (if installed from sources)
  * PHP 5.3.29 EOL! Please read: http://php.net/archive/2014.php#id2014-08-14-1
  * PHP 5.4.32
  * PHP 5.5.16
  * Redis 2.8.14
# Fixes in this release:
  * Add cleanup for _GIT_FORCE_REINSTALL if added in .barracuda.cnf
  * Add missing drush cache-clear drush to improve upgrade path.
  * Allow to clear drush cache without directory restrictions.
  * Always set correct TMP path for supported users.
  * Cleanup for cron pid files in user specific .tmp dirs.
  * Count properly also symlinked files directories (improved).
  * D6 colorbox module requires old 1.3.18 library.
  * Delete drush_make leftovers.
  * Delete duplicate menu items on upgrade.
  * Do not allow to install SSH from sources on Trusty to avoid problems.
  * Do not skip daily.sh during barracuda system only update.
  * Eldir theme: Use max width for buttons, if possible.
  * Fix cleanup for drush aliases in sub-accounts.
  * Fix daily cleanup for user specific .tmp directories.
  * Fix docs/HINTS.txt
  * Fix for broken mariadb.list
  * Fix for ghost dirs cleanup.
  * Fix for ghost vhosts cleanup.
  * Fix for missing symlinks to existing platforms.
  * Fix for not working protection from blocking local IPs on multi-IP systems.
  * Fix for subdirs_support universal check.
  * Fix for unreliable _IS_OLD check on Octopus instances upgrade.
  * Fix for warning "Could not create directory ." on Hostmaster site Verify.
  * Fix the fields order in the site edit form.
  * Fix the regex to not whitelist unexpected IP ranges inadvertently.
  * Force cURL rebuild if installed with outdated OpenSSL version.
  * Guard against destructive or insecure tasks run on the hostmaster site.
  * Improve cleanup for empty platforms directories.
  * Improve monitoring to protect against convert trying to overload the system.
  * Issue #2330781 - Use Drush dt() wrapper instead of not always available t()
  * Issue #357 - Fix the logic for Git (re)install from sources.
  * Issue #360 - Exclude special --CDN vhosts from daily cleanup.
  * Issue #361 - Update and improve docs/FAQ.txt
  * Issue #369 - Automatically download and fix /bin/websh if missing.
  * Issue #369 - Restore classic /bin/sh symlink automatically if needed.
  * Issue #373 - Set correct TMP, TEMP, TMPDIR env variables in limited shell.
  * Issue #373 - Too restrictive lshell forbidden list breaks drush sql-sync.
  * Issue #380 - Nameserver / pdnsd problem -- Fixes also Issue #2007990.
  * Issue #381 - Zend OPcache forced adds useless noise in the log.
  * Make it clear that subdomain and subdirectory name must be identical.
  * Make sure that keys subdirectory exists to avoid active platforms cleanup.
  * Nginx: Add config symlinks only on legacy instances.
  * Nginx: Add cron access support for subdir sites.
  * Nginx: Disable monitoring for POST requests related to cart/checkout URI.
  * Nginx: Remove deprecated code and config templates.
  * Nginx: Sanitize aliases in vhost_disabled.tpl.php to avoid warnings.
  * Nginx: Update config includes to match optional BOA features improvements.
  * Nginx: Update unified configuration templates in Provision to unfork BOA.
  * Nginx: Update vhosts templates to match BOA improvements.
  * PHP: Avoid unintended duplicate rebuilds.
  * Protect sites/all/drush
  * Provision: Backport provision_hosting_feature_enabled()
  * Provision: Remove legacy subdir code and update checks.
  * Redis config should sync with PHP-CLI, not PHP-FPM.
  * Remove legacy procs monitoring code.
  * Remove no longer needed limreq global fixes.
  * Remove no longer needed/used contrib updates.
  * Remove redundant file_exists() if is_readable() is also used.
  * Restart pdnsd before running barracuda upgrade.
  * Restore BOA formatting for tasks log to improve readability.
  * Restore BOA naming convention and docs in Hostmaster.
  * Restore BOA naming convention for Installation profiles in Hostmaster.
  * Restore BOA strict _hosting_valid_fqdn* testing procedures in Hostmaster.
  * Restore BOA weight defaults in the form in Hostmaster.
  * Restore punycode in Hostmaster.
  * Restore tasks sort to always show tasks scheduled and running at the top.
  * Sanitize cli.info and fpm.info
  * Set _PLATFORMS_LIST properly.
  * Simplify colorbox-1.3.18 download.
  * Simplify colorbox-1.5.13 download.
  * Switch branch on the fly and add support for Aegir vanilla mode.
  * Sync /tmp access restrictions.
  * Update for the Hostmaster welcome page.
  * Update FPM monitoring settings.
  * Use as short labels on the site node as possible.
  * Use correct paths to platform level drushrc.php file.
  * Use Drush6 with @hostmaster.
  * Use is_dir() instead of file_exists() when checking directory existence.
  * Use is_file() and is_link() instead of file_exists() before trying unlink()
  * Use is_readable() and file_exists() instead of file_exists() for backup.
  * Use is_readable() check instead of insufficient file_exists() for includes.
  * Use is_readable() instead of file_exists() when checking alias existence.
  * Install latest Git even if not specified via _XTRAS_LIST but previous
    version built from sources is detected.
  * Issue #2278847 - Derivatives can't be created on install with Drush and
    Aegir or when no vhost is available yet (Drupal Commons)

I can't see any issues that directly impact on us apart from the new version of PHP, we are running PHP Version 5.3.28, the release notes for PHP 5.3.29:

14 Aug 2014

The PHP development team announces the immediate availability of PHP 5.3.29. This release marks the end of life of the PHP 5.3 series. Future releases of this series are not planned. All PHP 5.3 users are encouraged to upgrade to the current stable version of PHP 5.5 or previous stable version of PHP 5.4, which are supported till at least 2016 and 2015 respectively.

PHP 5.3.29 contains about 25 potentially security related fixes backported from PHP 5.4 and 5.5.

More information in the Changelog.

I'll apply this update one evening soon.

#788: New BOA-2.3.3 Stable Edition available

chris — Mon, 15 Sep 2014 09:21:34 GMT

Changelog: http://bit.ly/boa-changes

### Stable BOA-2.3.1 Release - Full Edition
### Date: Sun Sep 14 15:53:25 SGT 2014
### Includes Aegir 2.1 with improvements
### Latest hotfix added on: Mon Sep 15 05:30:37 SGT 2014
# Release Notes:
  This major BOA Edition introduces many new features, changes and fixes.
  You should carefully read about some caveats further below **before** running
  this major upgrade on your system. Please secure a fresh system backup first.
  If you haven't run full barracuda+octopus upgrade to latest BOA Stable
  Edition yet, don't use any partial/system upgrade modes.
  Once new BOA Stable is released, you must run *full* upgrades with commands:
  $ cd;wget -q -U iCab http://files.aegir.cc/BOA.sh.txt;bash BOA.sh.txt
  $ barracuda up-stable
  $ octopus up-stable all both
  @=> Key new features:
  * BOA-2.3.1 comes with shiny Aegir 2.1 stable version, finally!
  * Support for Drupal sites in subdirectories is enabled by default
  * Solr 4 cores can be added/updated/deleted via site level INI settings
  * Super-easy to use New Relic support with per Octopus license key
  * Ability to add new Octopus instances with new, simple command syntax
  @=> Aegir control panel new features:
  * The list of sites is searchable by name or installation profile
  * You can schedule tasks against filtered sites in batches
  * Scheduling tasks in batches is available also on the platform view
  * Scheduling tasks in batches is available also on the profile view
  * Each site has its own tasks list available from the site view tab
  * You can schedule tasks also against platforms in batches
  * You can safely apply db updates via 'Run db updates' task on any site
  * It is now possible to choose any existing alias or the main site name
    as a redirect target, but without the need to rename the site --
    it will just re-verify the site and create new vhost automatically
  @=> Other important changes:
  * Support for PHP 5.2 has been officially deprecated
  * The www53 PHP-FPM pool has been switched from port to default socket mode
  * All existing vhosts must use wildcard in the Nginx 'listen' directive
  * Legacy mode for Install and Upgrade moves to 2.2.x branch
  * DB credentials are no longer in settings.php, only in drushrc.php
  * Latest Drush 6 version is used in the Aegir backend by default
  But what if you are not ready for this major upgrade and you would like
  to have more time for testing, but still be able to run system upgrades,
  thus effectively still using previous version 2.2.9 ?
#-### Legacy mode for Install and Upgrade moves to 2.2.x branch
  From now on, the 'legacy' install and upgrade mode available in all meta-
  installers will utilize branch 2.2.x instead of deprecated 2.1.x series.
  This means that starting with meta installers updated to use BOA-2.3.1
  version you can use commands like shown below to update Barracuda, Octopus
  and also to install more Octopus instances, while still using version 2.2.9:
  $ boa in-legacy public server.mydomain.org my@email o1
  $ barracuda up-legacy system
  $ octopus up-legacy o1
  $ boa in-legacy public server.mydomain.org my@email o2 mini
  etc.
  Remember to update your meta-installers first!
  $ cd;wget -q -U iCab http://files.aegir.cc/BOA.sh.txt;bash BOA.sh.txt
  Note also that if you will upgrade to current 'stable', it is not possible
  to downgrade back to the 'old stable' with 'legacy' mode, so please proceed
  with care!
  Remember also that current legacy version will not receive any further
  updates, even for security issues (besides those provided as packages by
  your OS vendor - Debian or Ubuntu, which will still work), because it is
  already different enough from current 2.3.1 stable, so we can't reliably
  maintain both with working upgrade path.
#-### Caveats: This upgrade will force wildcard in the Nginx 'listen' directive
  If you have old enough BOA system which still uses legacy IP mode and not
  a wildcard in the Nginx 'listen' directive, which is both Aegir and BOA
  standard for a long time already, this upgrade will fix the problem and
  update directives only in vhosts known and controlled by BOA.
  If you have any other vhosts, located in standard or non-standard Nginx/BOA
  directories for vhosts, you have to update them manually after upgrade to
  BOA-2.3.0 or newer, or they will take over all other vhosts on the system
  and cause redirects to /install.php which results with Nginx error 403 or 404,
  depending on the prior configuration.
  It will happen because IP based 'listen' directive in Nginx has higher
  priority, and will mess things horribly if there are vhosts using wildcard
  and some using the main system IP address.
  What and how to replace? Here are commands you need to run as root:
    $ sed -i "s/.*listen.*:80;/  listen  \*:80;/g" /path/to/vhost.file
    $ sed -i "s/.*listen.*:443/  listen  \*:443/g" /path/to/vhost.file
    $ service nginx reload
  Note: this **doesn't** affect special vhosts for SSL enabled sites, if used,
  because they are designed to use IP based 'listen' directives to provide
  separation between SSL enabled IPs and their associated certificates,
  while their associated 'upstream' block may even point to either local or
  remote IP address, so there is no wildcard to use in this case, and it will
  not conflict with all other vhosts managed by Aegir, because all SSL enabled
  vhosts listen on other IP addresses than the main system IP, which is
  by default used by all vhosts with wildcard in the 'listen' directive.
  The problem may happen only when you have vhosts using wildcard and also
  some vhosts using **main** system IP address in the 'listen' directive,
  which may happen also unintentionally during upgrade to BOA-2.3.0 or never,
  if there are either vhosts BOA doesn't control, or there are ghost vhosts
  not yet purged if you didn't upgrade to BOA-2.2.9 before, or there are
  some disabled sites, so their vhosts will not be re-created by Aegir
  during this major upgrade (because only active sites can be re-verified).
  While BOA will fix also any such ghost vhosts anyway, it will not be able
  to detect and fix vhosts outside of the standard directories managed by Aegir.
#-### Ability to add new Octopus instances with new, simple command syntax
  It is now possible to add stable Octopus instances w/o forcing Barracuda
  upgrade, plus optionally with no platforms added by default -- usage:
    $ boa {in-octopus} {email} {o2} {mini|max|none}
#-### The www53 PHP-FPM pool has been switched from port to default socket mode.
  Note that we are breaking backward compatibility here, so it will cause
  downtime on upgrade from any too old BOA version, until you will upgrade also
  Octopus instance(s) and update any other non-standard vhosts or includes
  still using legacy port mode for 'fastcgi_pass' Nginx directive.
  If you have 'fastcgi_pass 127.0.0.1:9090;' in any custom vhost or Nginx
  include file on the Octopus instance, you should replace it with:
    fastcgi_pass unix:/var/run/o1.fpm.socket;
  where 'o1' is your corresponding Octopus system username.
  Note that if you have custom vhosts or includes in the Aegir Master Instance,
  you should instead replace 'fastcgi_pass 127.0.0.1:9090;' with:
    fastcgi_pass unix:/var/run/www53.fpm.socket;
  where '53' is related to PHP version defined via _PHP_FPM_VERSION in your
  /root/.barracuda.cnf file. Note that while variable has a dot, the socket
  name doesn't.
#-### Support for PHP 5.2 has been officially deprecated
  While Barracuda 2.3.1 can continue to run and even upgrade if needed also
  the very old PHP 5.2 version, only Octopus instances running at least PHP 5.3
  or newer in both FPM and CLI mode can be upgraded to Octopus 2.3.1 Edition.
  If you are still using PHP 5.2 in your Octopus instance, you will not
  receive Aegir nor Drupal Platforms upgrade, but the Barracuda part of your
  system will receive upgrade to 2.3.1 anyway, so it will be ready to support
  your outdated Octopus instance upgrade as soon as you will switch it to
  modern and secure PHP version -- which is easy!
  Let's quote the original how-to for reference:
#-### Support for PHP FPM/CLI version safe switch per Octopus instance
  This allows to easily switch PHP version by the instance owner w/o system
  admin (root) help. All you need to do is to create ~/static/control/fpm.info
  and ~/static/control/cli.info file with a single line telling the system
  which available PHP version should be used (if installed): 5.5 or 5.4 or 5.3
  Only one of them can be set, but you can use separate versions for web access
  (fpm.info) and the Aegir backend (cli.info). The system will switch versions
  defined via these control files in 5 minutes or less. We use external control
  files and not any option in the Aegir interface to make sure you will never
  lock yourself by switching to version which may cause unexpected problems.
#-### Support for New Relic monitoring with per Octopus instance license key
  This new feature will disable global New Relic monitoring by deactivating
  server-level license key, so it can safely auto-enable or auto-disable it
  every 5 minutes, but per Octopus instance -- for all sites hosted on
  the given instance -- when a valid license key is present in the special
  new ~/static/control/newrelic.info control file.
  Please note that valid license key is a 40-character hexadecimal string
  that New Relic provides when you sign up for an account.
  To disable New Relic monitoring for the Octopus instance, simply delete
  its ~/static/control/newrelic.info control file and wait a few minutes.
  Please note that on a self-hosted BOA you still need to add your valid
  license key as _NEWRELIC_KEY in the /root/.barracuda.cnf file and run
  system upgrade with at least 'barracuda up-stable' first. This step is
  not required on Omega8.cc hosted service, where New Relic agent is already
  pre-installed for you.
#-### Solr 4 cores can be added/updated/deleted via site level INI settings
;;
;;  This option allows to activate Solr 4 core configuration for the site.
;;
;;  Only Solr 4 powered by Jetty server is available. Supported integration
;;  modules are limited to latest versions of either search_api_solr (D7 only)
;;  or apachesolr (will use Drupal core specific version automatically).
;;
;;  Currently used versions are listed below:
;;
;;    http://ftp.drupal.org/files/projects/search_api_solr-7.x-1.6.tar.gz
;;    http://ftp.drupal.org/files/projects/apachesolr-7.x-1.7.tar.gz
;;    http://ftp.drupal.org/files/projects/apachesolr-6.x-3.0-rc2.tar.gz
;;
;;  Note that you still need to add preferred integration module along with
;;  any its dependencies in your codebase since this feature doesn't modify
;;  your platform or site - it only creates Solr core with configuration
;;  files provided by integration module: schema.xml and solrconfig.xml
;;
;;  This setting affects only the running daily maintenance system behaviour,
;;  so you need to wait until next morning to be able to use new Solr 4 core.
;;
;;  Once the Solr core is ready to use, you will find a special file in your
;;  site directory: sites/foo.com/solr.php with details on how to access
;;  your new Solr core with correct credentials.
;;
;;  The site with enabled Solr core can be safely migrated between platforms,
;;  integration module can be moved within your codebase and even upgraded,
;;  as long as it is using compatible schema.xml and solrconfig.xml files.
;;
;;  Supported values for the solr_integration_module variable:
;;
;;    apachesolr
;;    search_api_solr
;;
;;  To delete existing Solr core simply comment out this line.
;;  The system will cleanly delete existing Solr core next morning.
;;
;solr_integration_module = NO
;;
;;  This option allows to auto-update your Solr 4 core configuration files:
;;
;;    schema.xml
;;    solrconfig.xml
;;
;;  If there is new release for either apachesolr or search_api_solr, your
;;  Solr core will not be automatically upgraded to use newer schema.xml and
;;  solrconfig.xml, unless allowed by switching solr_update_config to YES.
;;
;;  This option will be ignored if you will set solr_custom_config to YES.
;;
;solr_update_config = NO
;;
;;  This option allows to protect custom Solr 4 core configuration files:
;;
;;    schema.xml
;;    solrconfig.xml
;;
;;  To use customized version of either schema.xml or solrconfig.xml, you need
;;  to switch solr_custom_config to YES below and if you are using hosted
;;  Aegir service, submit a support ticket to get these files updated with
;;  your custom versions. On self-hosted BOA simply update these files directly.
;;
;;  Please remember to use Solr 4 compatible config files.
;;
;solr_custom_config = NO
# Updated Octopus platforms:
  aGov 1.4 --------------------- https://drupal.org/project/agov
  Guardr 1.12 ------------------ https://drupal.org/project/guardr
  Open Academy 1.1 ------------- https://drupal.org/project/openacademy
  Restaurant 1.0-b9 ------------ https://drupal.org/project/restaurant
  Ubercart 3.7 ----------------- https://drupal.org/project/ubercart
# New features and enhancements in this release:
  * Ability to add new Octopus instances with new, simple command syntax
  * Add default aggressive php-fpm monitoring + /root/.no.fpm.cpu.limit.cnf
  * Allow to define always disabled modules via _MODULES_FORCE variable.
  * Better wait limits on connection testing for slow network / long distance.
  * Issue #1927522 - Add support for easy Solr cores self-management.
  * Issue #362 - Add imageapi_optimize binaries via IMG in _XTRAS_LIST
  * Issue #376 - Add New Relic support with per Octopus instance license key.
  * Make firewall management faster with randomized schedule.
  * Procs monitor runs every 3 seconds.
  * Run mysql_proc_control every 5 seconds for better results.
  * You can safely apply db updates via 'Run db updates' task on any site.
# Changes in this release:
  * DB credentials are no longer visible in settings.php, only in drushrc.php
  * Delete default profiles in the hostmaster platform.
  * Disable _DEBUG_MODE if not enabled on the fly.
  * Disable newrelic-sysmond unless /root/.enable.newrelic.sysmond.cnf exists.
  * Drush: Upgrade command line version 6 to mini-6-14-09-2014
  * Nginx: Remove deprecated code - _HTTP_WILDCARD is already used by default.
  * Nginx: Use limit_conn protection only for known dynamic requests.
  * Redis Integration Module (cache_backport): Update to version 6.x-1.0-rc2
  * Redis Integration Module: Update to version mod-12-09-2014
  * Remove _ALLOW_UNSUPPORTED legacy and no longer working properly feature.
  * Remove dependency on Update Manager globally.
  * Remove deprecated multi-instance labels in the New Relic configuration.
  * Replace old hosting_civicrm_cron with newer hosting_civicrm module.
  * Set hosting_default_profile to 'minimal' to improve Ubercart 3 visibility.
  * The www53 PHP-FPM pool has been switched from port to default socket mode.
  * Use Provision CiviCRM boa-2.3.1-dev
# System upgrades in this release:
  * cURL 7.38.0 (if installed from sources)
  * Git 2.1.0 (if installed from sources)
  * Jetty 7.6.16.v20140903
  * Jetty 8.1.16.v20140903
  * Jetty 9.2.3.v20140905
  * PHP 5.3.29 EOL! Please read: http://php.net/archive/2014.php#id2014-08-14-1
  * PHP 5.4.32
  * PHP 5.5.16
  * Redis 2.8.14
# Fixes in this release:
  * Add cleanup for _GIT_FORCE_REINSTALL if added in .barracuda.cnf
  * Add missing drush cache-clear drush to improve upgrade path.
  * Add new features in the README.txt
  * Add wheezy to the exceptions list where required.
  * Allow to clear drush cache without directory restrictions.
  * Always set correct TMP path for supported users.
  * Cleanup for cron pid files in user specific .tmp dirs.
  * Count properly also symlinked files directories (improved).
  * D6 colorbox module requires old 1.3.18 library.
  * Delete drush_make leftovers.
  * Delete duplicate menu items on upgrade.
  * Do not allow to install SSH from sources on Trusty to avoid problems.
  * Do not skip daily.sh during barracuda system only update.
  * Eldir theme: Use max width for buttons, if possible.
  * Explain why installing RVM may take longer than expected.
  * Fix cleanup for drush aliases in sub-accounts.
  * Fix daily cleanup for user specific .tmp directories.
  * Fix docs/HINTS.txt
  * Fix for broken mariadb.list
  * Fix for broken, way too aggressive PHP-FPM monitoring.
  * Fix for ghost dirs cleanup.
  * Fix for ghost vhosts cleanup.
  * Fix for missing symlinks to existing platforms.
  * Fix for not working protection from blocking local IPs on multi-IP systems.
  * Fix for subdirs_support universal check.
  * Fix for unreliable _IS_OLD check on Octopus instances upgrade.
  * Fix for warning "Could not create directory ." on Hostmaster site Verify.
  * Fix the fields order in the site edit form.
  * Fix the regex to not whitelist unexpected IP ranges inadvertently.
  * Force cURL rebuild if installed with outdated OpenSSL version.
  * Guard against destructive or insecure tasks run on the hostmaster site.
  * Improve cleanup for empty platforms directories.
  * Improve monitoring to protect against convert trying to overload the system.
  * Issue #2330781 - Use Drush dt() wrapper instead of not always available t()
  * Issue #357 - Fix the logic for Git (re)install from sources.
  * Issue #360 - Exclude special --CDN vhosts from daily cleanup.
  * Issue #361 - Update and improve docs/FAQ.txt
  * Issue #369 - Automatically download and fix /bin/websh if missing.
  * Issue #369 - Restore classic /bin/sh symlink automatically if needed.
  * Issue #373 - Set correct TMP, TEMP, TMPDIR env variables in limited shell.
  * Issue #373 - Too restrictive lshell forbidden list breaks drush sql-sync.
  * Issue #380 - Nameserver / pdnsd problem -- Fixes also Issue #2007990.
  * Issue #381 - Zend OPcache forced adds useless noise in the log.
  * Issue #388 - Version 6.x-2.x of provision_civicrm requires hosting_civicrm
  * Issue #389 - hosting_civicrm breaks site install form with confusing error.
  * Issue #390 - Duplicate platforms nodes are created after upgrade to 2.3.0
  * Issue #395 - Validate username isn't reserved before running install script.
  * Issue #396 - Locale isn't getting set properly.
  * Issue #397 - Not actually prompted for platforms during installation.
  * Issue #398 - Make locales setup/fix for Debian always OS compatible.
  * Issue #399 - The hitimes gem needs to be pre-installed to support Omega4.
  * Issue #400 - CiviCRM is not installed on 2.3.0
  * Issue #401 - Create sites/all/* subdirs in Hostmaster early enough.
  * Issue #402 - Fix for ghost or disabled vhosts which still listen on IP.
  * Issue #405 - Installer hangs due to yes/no dialog - "Untrusted packages"
  * Issue #406 - Force keyring reinstall also upon 'GPG error'.
  * Issue #407 - Fix for 'username is already taken' error on a local VM install
  * Issue #408 - Fix for multiple funny typos. Thanks ar-jan!
  * Make it clear that subdomain and subdirectory name must be identical.
  * Make sure that keys subdirectory exists to avoid active platforms cleanup.
  * Make the PHP-FPM processes monitor less aggressive by default.
  * Nginx: Add config symlinks only on legacy instances.
  * Nginx: Add cron access support for subdir sites.
  * Nginx: Convert all vhosts to wildcard mode on Barracuda upgrade.
  * Nginx: Disable monitoring for POST requests related to cart/checkout URI.
  * Nginx: Do not touch nginx_wild_ssl.conf during this upgrade.
  * Nginx: Improve wildcard conversion procedure on some really old instances.
  * Nginx: Remove deprecated code and config templates.
  * Nginx: Sanitize aliases in vhost_disabled.tpl.php to avoid warnings.
  * Nginx: Update config includes to match optional BOA features improvements.
  * Nginx: Update unified configuration templates in Provision to unfork BOA.
  * Nginx: Update vhosts templates to match BOA improvements.
  * PHP: Avoid unintended duplicate rebuilds.
  * PHP: Sync disable_functions list.
  * Protect sites/all/drush
  * Provision: Backport provision_hosting_feature_enabled()
  * Provision: Remove legacy subdir code and update checks.
  * Redis config should sync with PHP-CLI, not PHP-FPM.
  * Remove legacy procs monitoring code.
  * Remove no longer needed limreq global fixes.
  * Remove no longer needed/used contrib updates.
  * Remove redundant file_exists() if is_readable() is also used.
  * Replace old hosting_civicrm_cron with newer hosting_civicrm module.
  * Restart pdnsd before running barracuda upgrade.
  * Restore BOA formatting for tasks log to improve readability.
  * Restore BOA naming convention and docs in Hostmaster.
  * Restore BOA naming convention for Installation profiles in Hostmaster.
  * Restore BOA strict _hosting_valid_fqdn* testing procedures in Hostmaster.
  * Restore BOA weight defaults in the form in Hostmaster.
  * Restore punycode in Hostmaster.
  * Restore tasks sort to always show tasks scheduled and running at the top.
  * Sanitize cli.info and fpm.info
  * Set _PLATFORMS_LIST properly.
  * Silence early sed replacements to avoid confusion.
  * Simplify colorbox-1.3.18 download.
  * Simplify colorbox-1.5.13 download.
  * Switch branch on the fly and add support for Aegir vanilla mode.
  * Sync /tmp access restrictions.
  * The hosting_civicrm_cron is now a submodule and should be also auto-enabled.
  * The wildcard transition **doesn't** affect vhosts for SSL enabled sites.
  * There is no need to force backend clone from GitHub on initial upgrade.
  * Update for the Hostmaster welcome page.
  * Update FPM monitoring settings.
  * Use as short labels on the site node as possible.
  * Use control files properly to not run redundant Jetty/Solr upgrade.
  * Use correct paths to platform level drushrc.php file.
  * Use correct Provision version on initial upgrade to 2.3.0
  * Use Drush6 with @hostmaster.
  * Use is_dir() instead of file_exists() when checking directory existence.
  * Use is_file() and is_link() instead of file_exists() before trying unlink()
  * Use is_readable() and file_exists() instead of file_exists() for backup.
  * Use is_readable() check instead of insufficient file_exists() for includes.
  * Use is_readable() instead of file_exists() when checking alias existence.
  * Install latest Git even if not specified via _XTRAS_LIST but previous
    version built from sources is detected.
  * Issue #2278847 - Derivatives can't be created on install with Drush and
    Aegir or when no vhost is available yet (Drupal Commons)

Having read through the above it is good to see the switch to use sockets rather than TCP/IP for php-fpm, not sure if there are any other changes that would effect us. I'll do the upgrade one evening this week.

#529: New Barracuda BOA-2.0.7 Edition available

chris — Fri, 05 Apr 2013 08:57:09 GMT

There is new BOA-2.0.7 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

#530: New Barracuda BOA-2.0.8 Edition available

chris — Mon, 08 Apr 2013 07:44:51 GMT

If I had known this was about to come out I would have waited before doing the BOA-2.0.7 upgrade last night on ticket:529

There is new BOA-2.0.8 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

Is mostly to fix a problem for people using Percona, https://drupal.org/node/1962690 and as we are using MariaDB this isn't an issue for us.

Other updates in this version:

Allow to use [a-z0-9] subdomains and not only [www] for IDN domain names.
Change the interval between platforms builds from 5 to 3 seconds.
Forced 1s Speed Booster TTL for vhosts behind local proxy is deprecated.
Move old firewall logs to backups to avoid crazy load after upgrade.
Nginx: Better exceptions handling in the Abuse Guard for js/shs modules.
PHP: CLI is at 5.3 since BOA-2.0.4, so symlink old 5.2 binary path to 5.3
Update _LENNY_TO_SQUEEZE major upgrade procedure.
Update contrib with login_security-7.x-1.2
Use static downloads for all distros in stable edition.

I'll do this update tonight unless there are any objections, hopefully it should be quite quick.

#547: New Barracuda BOA-2.0.9 Edition available

chris — Fri, 10 May 2013 11:28:17 GMT

Do we want to upgrade to Wheezy, see ticket:535 at the same time as upgrading to BOA-2.0.9? Or should we wait a few weeks, as Jim suggested here ticket:535#comment:3

I think we should schedule this update (to BOA-2.0.9) for an evening when there isn't much traffic on the site, perhaps Saturday or Sunday night. I don't think this update is super urgent as we have already upgraded Nginx, see ticket:218#comment:80

There is new BOA-2.0.9 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

The Changelog contains:

# This is the first Barracuda-only Edition, released to address important
  security issue with Nginx server and provide system level upgrades.
  This Edition will not upgrade Aegir Master nor Aegir Satellite Instances,
  because there was no new Drupal core released since BOA-2.0.8 Edition and
  there were not enough updates to built-in platforms or contrib accumulated.
  Releasing Barracuda-only Edition separately from full Edition allows us
  to address system/services security issues without any extra delay,
  while releasing Octopus-only Edition will allow us to provide Drupal core
  or Aegir version upgrades, without affecting system level services.
  There is also another reason while separate releases will be useful.
  BOA-2.0.9 is the last Edition where Aegir 2.x uses still old Drush 4.6
  in the backend. We need to sync BOA specific Aegir 2.x with upstream
  and finally switch to Drush 5, or even Drush 6, if possible.
  This change, however, may cause issues if you still host legacy Drupal 5
  or some old Drupal 6 sites, with either core or contrib not compatible
  with PHP 5.3, which is now used by default.
  That is why we plan to introduce ability to install older/previous
  Barracuda and/or Octopus release, if you need more time to upgrade.
# New features and enhancements in this release:
  * Debian 7.0 Wheezy support.
  * Automated upgrade from Squeeze with _SQUEEZE_TO_WHEEZY=YES option.
  * Added config template with inline how-to in docs/cnf/barracuda.cnf
  * Added config template with inline how-to in docs/cnf/octopus.cnf
  * Added passwords encryption how-to in docs/BLOWFISH.txt
  * Added the list of symbols used on install in docs/PLATFORMS.txt
  * Forced mysql restart if there are too many high CPU mysqld processes.
  * Improved docs/NOTES.txt
  * Improved docs/README.txt
  * Install libpam-unix2 and libxcrypt1 by default.
  * Install s3cmd by default.
  * Issue #1974640 - Allow to use Midnight Commander for limited shell users.
  * Limited Shell Logs Monitor enabled by default.
  * Nginx: Check for Linux/Cdorked.A malware and delete if discovered.
  * Re-generate and sync Aegir passwords before and after instance upgrade.
  * The silent 'system' mode documented in docs/UPGRADE.txt
  * Allow to exclude platform from otherwise forced `drush en entitycache -y`
    if sites/all/modules/entitycache_dont_enable.info control file is present.
# Changes in this release:
  * Nginx 1.5.0 - security upgrade for CVE-2013-2028
  * PHP 5.3.25
  * Redis 2.6.13
  * Do not disable update module in platforms known to include it as required.
  * Firewall: Open port 1129 for outgoing connections (some gateways need it).
  * Force syslog module as disabled by default and save some disk I/O.
  * Tune kernel to always use max RAM and not swap, if possible.
# Fixes in this release:
  * Add outgoing port 25 SMTP to the list of requirements.
  * Firewall: Add truly permanent block for heavy abusers.
  * Fix for mytop support, available again on systems with MariaDB.
  * Fix permissions in the /data/all tree if required.
  * Fix the order of checks - they scan only the last (current) minute.
  * Force _STRONG_PASSWORDS=NO if locales still look broken on second check.
  * Improve detecting no longer running drush.php and/or cron PHP processes.
  * Improve fix_locales logic.
  * Improve global.inc symlinking on initial install and upgrade.
  * Improve messages displayed when fix_locales discovers broken locales.
  * Improve monitoring to avoid duplicate entries on low traffic systems.
  * Improve sanitize_string() filtering to avoid issues with strong passwords.
  * Improve syncpass tool - Update system user passwd and flush privileges.
  * Issue #1961226 - Warning: Could not change permissions of sites/all to 751.
  * Issue #1962458 - 403 for anonymous users on node/add.
  * Issue #1963044 - Force UTF-8 locales if not present/configured properly.
  * Issue #1974542 - Use /root/.home.no.wildcard.chmod.cnf control file.
  * Issue #1987936 - Restore ability to install PHP 5.2 for FPM and CLI.
  * Make sure that /dev/null is writable for everyone.
  * Make sure that all drushrc.php files are owned by Aegir system user.
  * Make sure that all expected sites/all/{modules,themes,libraries} dirs exist.
  * Make sure that DB server is restarted on upgrade after config tuning.
  * Make sure that pdnsd and resolvconf are properly installed.
  * Nginx: Remove duplicate Vary: Accept-Encoding headers.
  * Percona no longer supports older Ubuntu non-LTS releases.
  * PHP: Do not reload FPM every hour - it may cause error 502.
  * PHP: Fix paths depending on CLI version used.
  * PHP: Fix the extensions installation and upgrade logic.
  * PHP: Make sure that the FPM port is set correctly for D6 sites with 5.2
  * PHP: Properly uninstall all related packages when using source build.
  * PHP: Start more FPM workers on systems with enough RAM by default.
  * Purge bin logs before disabling them.
  * Run NewRelic re-install early enough to avoid locking full-upgrade.
  * Sync the load limits for spiders and backend tasks.
  * The Java/Jetty monitor should use higher allowed limits by default.
  * Update apticron message to recommend system mode instead of full upgrade.
  * Update docs for _BUILD_FROM_SRC option.
  * Use aggressive enough Jetty restart procedure on nightly services reload.
  * Use correct status messages on install and upgrade.
  * Use installer and not Aegir version download on stable install/upgrade.

#586: New Relic Monitoring for BOA

chris — Thu, 05 Sep 2013 08:35:17 GMT

This is a ticket to document the consideration, and possible installation, of https://newrelic.com/ on the PuffinServer

#483: Nginx 502 Bad Gateway Errors with BOA

chris — Mon, 28 Jan 2013 11:23:45 GMT

The Barracuda Octopus Aegir server on puffin.webarch.net is getting generating a lot of Nginx 502 Bad Gateway errors, see ticket:466#comment:31 and ticket:466#comment:38

The Nginx logs contain 58 502 errors:

cd /var/log/nginx
grep " 502 " * | wc -l
58

I expect the cause of these is Nginx asking PHP-FPM for a page and not getting one, an answer might be to either speed up PHP-FPM response time or adjust the Nginx settings so that it waits longs for a response from PHP-FPM.

There are several threads about this on http://groups.drupal.org/ but none of them have any useful suggestions (eg 1 2 3 4).

#797: POODLE: SSLv3.0 vulnerability (CVE-2014-3566)

chris — Wed, 15 Oct 2014 12:49:39 GMT

Check which serives are available with SSLv3.0, see:

and disable SSLv3.0 where it is being offered.

#470: Penguin install and configuration

chris — Sat, 15 Dec 2012 16:05:16 GMT

penguin.webarch.net is a new 2GB RAM virtual server which will replace NewLiveServer and DevelopmentServer for running all non-Drupal sites and is due to go live in early 2013. Drupal sites from the old servers will be migrated to PuffinServer.

This ticket has been created for tracking time and tasks done during the install.

See https://tech.transitionnetwork.org/trac/wiki/PenguinServer and also the corresponding PuffinServer.

#552: Puffin Downtime 23rd May 2013

chris — Thu, 23 May 2013 20:54:15 GMT

There was a massive load spike on wiki:PuffinServer this afternoon, the load average peaked at over 60 and at exactly the same time there was also one on the virtual server serving http://webarch.net/.

Pingdom reported:

www.transitionnetwork.org down since 23/05/2013 12:11:57
www.transitionnetwork.org is UP again at 23/05/2013 12:19:57, after 8m of downtime
www.transitionnetwork.org is down since 23/05/2013 12:26:57
www.transitionnetwork.org is UP again at 23/05/2013 12:58:57, after 32m of downtime

When I did managed to get back into the server php-fpm and nginx were not running and it took several attempt to get them to start up.

Attached is the munin graps of the load from today, munin didn't record the start of the spike but it did start recording it as it came down.

This ticket has been created to report the issue and also to record anything that might be found in the logs to explain what happened.

#543: Puffin Load Spike

chris — Fri, 03 May 2013 20:48:16 GMT

Puffin just had a massive load spike up to 65 and then it calmed down and it's up in the 70's again:

uptime
 21:46:43 up 91 days,  3:01,  4 users,  load average: 73.45, 30.76, 28.09

I'm trying to find the cause, it's making the TN site unresponsive.

#862: Puffin locked

chris — Sat, 27 Jun 2015 15:31:27 GMT

PuffinServer is not responding, I got a Munin email alert, on the Xen console:

[2008077.910371] BUG: soft lockup - CPU#1 stuck for 61s! [munin-node [::f:25444]
[2008077.910371] Modules linked in: joydev sg st sd_mod crc_t10dif sr_mod scsi_mod ide_gd_mod ide_cd_mod ide_core cdrom xt_recent xt_tcpudp xt_connlimit nf_nat_ftp ipt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp ipt_REJECT ipt_LOG xt_limit xt_multiport iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext4 crc16 jbd2 mbcache dm_mod xen_netfront xen_blkfront
[2008077.910371] CPU 1:
[2008077.910371] Modules linked in: joydev sg st sd_mod crc_t10dif sr_mod scsi_mod ide_gd_mod ide_cd_mod ide_core cdrom xt_recent xt_tcpudp xt_connlimit nf_nat_ftp ipt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp ipt_REJECT ipt_LOG xt_limit xt_multiport iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext4 crc16 jbd2 mbcache dm_mod xen_netfront xen_blkfront
[2008077.910371] Pid: 25444, comm: munin-node [::f Not tainted 2.6.32-5-xen-amd64 #1
[2008077.910371] RIP: e030:[<ffffffff8100922a>]  [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
[2008077.910371] RSP: e02b:ffff8800988f7ba8  EFLAGS: 00000246
[2008077.910371] RAX: 0000000000040000 RBX: ffffea0006c2eb88 RCX: ffffffff8100922a
[2008077.910371] RDX: 00000000ffffff00 RSI: 0000000000000000 RDI: 0000000000000000
[2008077.910371] RBP: 0000000000000002 R08: 0000000000000002 R09: ffff8801ffc1dd00
[2008077.910371] R10: 0000000000000002 R11: 0000000000000246 R12: ffff88000000ad00
[2008077.910371] R13: ffff880000008000 R14: 0000000000000200 R15: 000000000000000e
[2008077.910371] FS:  00007fefd0bde700(0000) GS:ffff88000bb20000(0000) knlGS:0000000000000000
[2008077.910371] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[2008077.910371] CR2: 00007fefce124380 CR3: 0000000001001000 CR4: 0000000000000660
[2008077.910371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2008077.910371] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[2008077.910371] Call Trace:
[2008077.910371]  [<ffffffff810baa62>] ? free_hot_cold_page+0x1a2/0x1af
[2008077.910371]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[2008077.910371]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[2008077.910371]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[2008077.910371]  [<ffffffff8130f142>] ? _spin_unlock_irqrestore+0xd/0xe
[2008077.910371]  [<ffffffff810bd9ca>] ? release_pages+0x16a/0x18d
[2008077.910371]  [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
[2008077.910371]  [<ffffffff810da555>] ? free_pages_and_swap_cache+0x57/0x73
[2008077.910371]  [<ffffffff810cd5bf>] ? unmap_vmas+0x6cb/0x959
[2008077.910371]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[2008077.910371]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[2008077.910371]  [<ffffffff810d1bca>] ? exit_mmap+0xc4/0x148
[2008077.910371]  [<ffffffff8104cd95>] ? mmput+0x3c/0xdf
[2008077.910371]  [<ffffffff81050a2e>] ? exit_mm+0x102/0x10d
[2008077.910371]  [<ffffffff81052453>] ? do_exit+0x1f8/0x6c9
[2008077.910371]  [<ffffffff8105299a>] ? do_group_exit+0x76/0x9d
[2008077.910371]  [<ffffffff810529d3>] ? sys_exit_group+0x12/0x16
[2008077.910371]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[2008381.335662] Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=122.172.30.208 DST=81.95.52.103 LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=30880 DF PROTO=TCP SPT=60400 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
[2008384.465057] Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=122.172.30.208 DST=81.95.52.103 LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=30881 DF PROTO=TCP SPT=60400 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
[2008390.255448] Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=122.172.30.208 DST=81.95.52.103 LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=30882 DF PROTO=TCP SPT=60400 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0

#674: Puffin locked up

chris — Tue, 14 Jan 2014 09:59:40 GMT

This is on the console, no response via web or ssh :

[4478428.894563]  [<ffffffff810f3634>] ? sys_readlinkat+0x25/0x8d
[4478428.894571]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[4478428.894581] INFO: task cron:46009 blocked for more than 120 seconds.
[4478428.894587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[4478428.894595] cron          D ffff8801fb270e20     0 46009   8829 0x00000080
[4478428.894606]  ffff8801fb270e20 0000000000000286 0000000000000001 00000001810423b0
[4478428.894617]  ffff88000bcf87c0 000000000000000b 000000000000f9e0 ffff8800cbb9dfd8
[4478428.894628]  00000000000157c0 00000000000157c0 ffff8801fa458000 ffff8801fa4582f8
[4478428.894640] Call Trace:
[4478428.894648]  [<ffffffff811976a6>] ? vsnprintf+0x40a/0x449
[4478428.894656]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
[4478428.894666]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
[4478428.894673]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
[4478428.894681]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
[4478428.894689]  [<ffffffff810f826a>] ? do_follow_link+0x1fa/0x317
[4478428.894697]  [<ffffffff810f86e0>] ? __link_path_walk+0x359/0x6f5
[4478428.894705]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
[4478428.894713]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
[4478428.894721]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
[4478428.894729]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[4478428.894738]  [<ffffffff8130dda8>] ? thread_return+0x79/0xe0
[4478428.894746]  [<ffffffff811901fb>] ? _atomic_dec_and_lock+0x33/0x50
[4478428.894755]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
[4478428.894763]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
[4478428.894770]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[4478669.197427] INFO: task vnstatd:1920 blocked for more than 120 seconds.
[4478669.197448] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[4478669.197458] vnstatd       D 0000000000000000     0  1920      1 0x00000000
[4478669.197469]  ffff8801ff18bf90 0000000000000282 ffff8801fa651e10 ffff880000009680
[4478669.197482]  0000000000000000 000000000000000a 000000000000f9e0 ffff8801fa651fd8
[4478669.197494]  00000000000157c0 00000000000157c0 ffff8801fe335bd0 ffff8801fe335ec8
[4478669.197505] Call Trace:
[4478669.197520]  [<ffffffff8130f6fa>] ? error_exit+0x2a/0x60
[4478669.197531]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[4478669.197542]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
[4478669.197551]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
[4478669.197561]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
[4478669.197569]  [<ffffffff810f86aa>] ? __link_path_walk+0x323/0x6f5
[4478669.197577]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
[4478669.197585]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
[4478669.197593]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
[4478669.197602]  [<ffffffff8130f054>] ? _spin_lock_irqsave+0x15/0x34
[4478669.197612]  [<ffffffff81068fa6>] ? hrtimer_try_to_cancel+0x3a/0x43
[4478669.197622]  [<ffffffff8102de30>] ? pvclock_clocksource_read+0x3a/0x8b
[4478669.197631]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
[4478669.197641]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
[4478669.197649]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b

Rebooting...

#588: RSS feed caching

chris — Fri, 06 Sep 2013 09:43:57 GMT

At the meeting on 5th September ticket:585 one thing we discussed was that:

The biggest single source of bandwidth by URL is Rob's RSS feed, 34.5GB of data, 150k hits. This represents half the total site data transfer, the RSS files is 0.4MB but will hopefully be mostly served gzipped at 0.1MB.

This ticket is to follow up on that -- are the RSS feeds served directly by Nginx?

#730: Redis Munin stats for puffin

chris — Fri, 23 May 2014 08:38:33 GMT

The Redis stats for wiki:PuffinServer are not being generated:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/index.html#redis

#545: Registration page: 502

ed — Wed, 08 May 2013 08:38:13 GMT

http://www.transitionnetwork.org/user/register

found a bunch of these yesterday - this one is still showing a 502...

#831: Rob is having Image upload issues

ade — Tue, 17 Feb 2015 11:37:09 GMT

Hi Paul, Rob in Totnes is creating a blog post and is getting an error whilst trying to upload an image. Initially I thought it could have been a harddrive full issue, but we can see that we have drive room. Sam and I have had a quick look to see if anything obvious, but we are now stumbling a bit. We can replicate the issue by trying to upload an image to a blog post, but cannot see where to find an event log to see if any logs are being generated that may aid in a diagnosis. Could you have a look and let us know your thoughts please? Rob is currently waiting to post a blog so is quite urgent.

Many thanks Ade

#670: Roll back performance customisations and use stock BOA settings where possible

jim — Sat, 11 Jan 2014 20:51:30 GMT

Issue Given so much has changed since the initial issues on the server, I now strongly recommend reverting all settings changes that do not add features back to stock BOA settings after the next BOA release.

These would include all MySQL, PHP, FPM, Redis and other settings that have been for performance reasons, or to combat the situation where there was hardware/IO issues with the underlying server. I'm most interested in FPM and MySQL settings.

The next version of BOA will include some improvements we need (see 629: Upgrades to BOA which should handle load on our server with a lot of CPU cores. With this in place we'll be able to revert more easily to stock settings.

Rationale I'm not talking about rolling back changes that provide us with features or mission-critical capabilities, just the changes to the subsystems I list above for performance reasons.

It's my belief that these enhancements no longer match the needs of the server since the changes to filesystem and underlying hardware fixes have been completed. They also represent an ongoing risk around updates, future planning -- plus it's possible they might mean Puffin's web services need more memory than it otherwise would, costing TN more than it should need to spend on hardware.

Proposed solution

Await the next version of BOA, and setup the enhanced load settings per the documentation.
Revert all other changes to conf files for MySQL, PHP, FPM, Redis that do not add a feature or are not mission-critical.
Review /root/.barracuda.cnf and turn off any overrides and customisations we don't now need as a result of 2).
Run the BOA BOND.sh script to do the tuning of the server appropriate to the memory requirements. This will tune for the current levels on first pass.
Review Munin and site performance. If we need to make any tweaks then we can do a minimal set as required -- keeping an eye on memory usage.
Once a few days have gone by I would hope that the overall memory use will be lower, OR with more cached data. At this point we can either re-run the barracuda installer with the _RESERVED_RAM set to 1-4 Gb, or simply reduce the memory available to Puffin.
Repeat from 4, using BOND.sh to optimise for the new memory footprint.

Clearly, it's possible no memory savings can be made, or just 1Gb or so is sensible. Either way, rolling back the changes made for a system that has changed immensely is worth attempting to compare current (tweaked) performance to the stock system. Since current settings are all documented and can be backed up, we should be able to test this with no risk and the ability to roll back as needed.

Next steps

Chris and Ed to give their thoughts.
Ed to green-light before we proceed in too much detail or take any action.
Chris and Jim to establish the changes and outcomes.
Chris, Jim and whoever to do the new optimisation process.

#795: SHA1 Deprecation: Regenerate all certs using SHA256

chris — Fri, 10 Oct 2014 20:37:49 GMT

SHA1 SSL certs and chains are now flagged at SSLLabs, see SHA1 Deprecation: What You Need to Know, however Gandi doesn't yet support SHA256, see SHAAAAAAAAAAAAA which links to this tweet, when they do support SHA256 all the keys, certs and chains will need updating.

#685: SSL certificate about to expire?

sam — Thu, 23 Jan 2014 12:31:06 GMT

Hi Chris this dropped into my inbox this morning.

Can you tell from the following if it's a certificate we still use? Does it need to be renewed?

Thanks

Sam

---

Hello,

This mail is to inform you that your certificate SSL Standard (*.transitionnetwork.org) expires today, on 2014-01-24 00:59.

Warning: for Pro and Business certificates, web browsers have increased security. It can now take up to several weeks, and so we strongly recommend that you perform the operation as soon as possible.

If you would like to keep your certificate, we recommend renewing it today. For this you must launch the renewal process from the following page: https://www.gandi.net/admin/ssl/renew/26873

If you do not want to keep your certificate, then no further action is necessary on your part. It will be automatically revoked by our services and rendered useless.

#920: SSL weirdness?

sam — Thu, 14 Jul 2016 20:20:19 GMT

Hi Chris

So Paul put the site into maintenance mode, took a database dump and then tried to re-enable live mode using the drush command.

It seems it came out of maintenance mode OK, but we're now getting this certificate error.

I have changed the Zone file on Gandi in the meantime, but this doesn't seem to be propagating.

Any ideas?

Thanks

Sam

#599: Server time drift

chris — Sun, 29 Sep 2013 21:57:59 GMT

The servers are not keeping good time at the moment.

#576: Site down

ed — Thu, 01 Aug 2013 09:42:44 GMT

Been down for 15 minutes by now I reckon

#828: Site down due to massive load spike 2015-01-29

chris — Thu, 29 Jan 2015 17:09:07 GMT

Seems to be recovering now, this ticket is to try to find the cause.

#554: Site slow down and MySQL load increase

chris — Tue, 28 May 2013 18:11:51 GMT

Since the upgrade to MariaDB 5.5.31, done on ticket:218#comment:93 (and fixed on ticket:548#comment:32) there appears to have been been a noticeable slowdown in the time for pages to be generated, measuring with http://tools.pingdom.com/fpt/ the front page alone takes around 5 seconds to generate.

There is a clear increase in the amount of MySQL/MariaDB activity measured by Munin, see the attached graphs, the upgrade was done around midday on 24th May 2013. There has also been a increase in traffic according to the firewall graphs. The memory usage of redis has also dropped right down and the database memory usage has significantly increased.

It's not totally clear if the cause of this change in behaviour of the site is related to the MySQL/MariaDB upgrade or if there was a coincidental change in the traffic to the site at the same time. There is no noticeable change in the visitors recorded in the Piwik stats.

According to pingdom the front page of the site is now "slower than 77% of all tested websites" with a total load time of around 6 seconds, almost all of this is down to the wait of around 5 seconds for the index.php file. This can also be tested from parrot with Apache bench, sometimes cached pages are served up and these appear in an instance, if the front page is generated it takes around 5 seconds, see wiki:LoadTimes#a2013-05-28

#677: Spike in MyISAM (search) database activity, Redis unable to cache such requests

chris — Wed, 15 Jan 2014 10:49:33 GMT

Since Sunday there has been a marked increase in MySQL activity and also a drop in Redis memory use, I suspect these things are related.

I'll post some Munin graphs below to illustrate this.

#844: Stable BOA 2.4.2 Release

chris — Fri, 10 Apr 2015 10:34:56 GMT

Looks like all the tickets for BOA 2.4.2 have been closed and it is due to be released today:

https://github.com/omega8cc/boa/milestones

The Changelog has not yet been updated:

https://github.com/omega8cc/boa/blob/master/CHANGELOG.txt

For some unknown reason BOA no longer sends emails when a new version is out so the Changelog will need checking manually.

#827: Stable BOA-2.4.0 Release

chris — Mon, 19 Jan 2015 20:11:09 GMT

No release date yet, but lots of detail in the Changelog:

https://github.com/omega8cc/boa/blob/master/CHANGELOG.txt

2.4.0 issues:

https://github.com/omega8cc/boa/milestones/2.4.0

#839: Stable BOA-2.4.1 Release

chris — Tue, 24 Mar 2015 10:44:41 GMT

BOA appears to have stopped sending email to notify that new versions are available, I just manually checked the Changelog and discovered that BOA-2.4.1 came out on 8th March 2015:

### Stable BOA-2.4.1 Release - Full Edition
### Date: Sun Mar  8 14:56:51 PDT 2015
### Milestone URL: https://github.com/omega8cc/boa/milestones/2.4.1
### Latest hotfix added on: Wed Mar 11 11:58:52 PDT 2015
  @=> Includes Aegir Hostmaster 2.x-head with improvements
  @=> Includes Aegir Provision 3.x-head with improvements
  @=> Includes Drush 7.0.0-alpha9 customized for BOA
# Release Notes:
  This new BOA release includes one new and 12 updated Aegir platforms,
  8 new features and enhancements, 15 new software versions, 10 other changes,
  plus over 38 bug fixes, with most notable features and changes listed below:
  @=> Add duobackboa with /root/.duobackboa.cnf file to run duplicate backups
  @=> Add SSL with TLS/SNI on server with one IP, multiple certificates support
  @=> Add support for Octopus batch migration - see docs/MIGRATE.txt for details
  @=> Allow to use _PHP_GEOS=YES with all PHP versions
# New Octopus platforms:
  OpenAid 2.0 ------------------ https://drupal.org/project/openaid
# Updated Octopus platforms:
  Commerce 1.33 ---------------- https://drupal.org/project/commerce_kickstart
  Commerce 2.21 ---------------- https://drupal.org/project/commerce_kickstart
  Commons 2.22 ----------------- https://drupal.org/project/commons
  Commons 3.22 ----------------- https://drupal.org/project/commons
  Drupal 8.0.0-b7 -------------- https://drupal.org/drupal-8.0
  Guardr 2.8 ------------------- https://drupal.org/project/guardr
  OpenAtrium 2.32 -------------- https://drupal.org/project/openatrium
  OpenChurch 2.1-b5 ------------ https://drupal.org/project/openchurch
  OpenOutreach 1.16 ------------ https://drupal.org/project/openoutreach
  OpenScholar 3.20.0 ----------- http://theopenscholar.org
  Panopoly 1.18 ---------------- https://drupal.org/project/panopoly
  Recruiter 1.5 ---------------- https://drupal.org/project/recruiter
# New features and enhancements:
  * Add compatibility with latest VS beng kernel
  * Add duobackboa with /root/.duobackboa.cnf file to run duplicate backups
  * Add support for multivalued fields in SOLR 4 - pull request #626
  * Add support for mysqladmin proc logging
  * Add support for Octopus batch migration - see docs/MIGRATE.txt for details
  * Add support for scout/mysql monitoring
  * CSF: Add popular ports 222 and 2222 to TCP_OUT by default
  * SSL with TLS/SNI on server with one IP, multiple certificates - fixes #465
# Changes:
  * Allow to run automated SQL conversion only weekly
  * Allow to use _PHP_GEOS=YES with all PHP versions
  * Do not send extra nocache cookie on GET requests
  * Drush mini-7-07-03-2015
  * Make barracuda wrapper available on initial install to avoid confusion
  * Nginx: Update for crawlers exceptions list
  * Redis Integration Module: Update to version mod-05-03-2015
  * Remove dependency on legacy Drush 4
  * Use latest Apache Solr Search 6.x-3.x config
  * Use latest Apache Solr Search 7.x-1.x config
# System upgrades:
  * Apache Solr 4.9.1
  * cURL 7.41.0 (if installed from sources)
  * Git 2.3.0 (if installed from sources)
  * Jetty 9.2.7.v20150116
  * MariaDB 10.0.17
  * MariaDB 5.5.42
  * MariaDB Galera Cluster 10.0.17
  * Nginx 1.7.10
  * OpenSSL 1.0.2 (if installed from sources)
  * PHP 5.4.38
  * PHP 5.5.22
  * PHP 5.6.6
  * PHP: ionCube loader 4.7.4
  * Pure-FTPd 1.0.37
  * Ruby 2.2.1
  * Use duplicity 0.7.01 and boto 2.36.0 - fixes #630
  * Vnstat 1.13
# Fixes:
  * [provision] False "load on system too heavy" messages - fixes #619
  * [provision] Issue #2350695 - Profile is registered twice, also as a module
  * [provision] Nginx: Remove webform keyword from regex locations - fixes #599
  * Add also manage_ltd_users to the list - fixes #616
  * Avoid installing New Relic with no valid license key provided - fixes #608
  * Do not add no longer used symlink
  * Do not create conflicting plain HTTP proxy for single IP mode - fixes #465
  * Do not delete backboa while duplicity is running
  * Do not replace any contrib in latest OA - fixes #2420131
  * Do not run D7 core hotfix on already fixed instances
  * Fix for legacy systems autoupdate logic
  * Fix for missing chattr -i on web user update
  * Fix for missing datestamp
  * Fix for too dangerous pdnsd auto-config logic
  * Fix pdnsd restarts procedures - fixes #610
  * Fix permissions for pdnsd if needed
  * Fix variable in autoupboa - pull request #629
  * Force php.ini update
  * Hotfix for cluster instances
  * Hotfix for OpenSSL/cURL versions out of sync
  * How to enable permanent redirect to HTTPS with single IP - #465
  * Issue #2425963 - Broken slider in Commerce Kickstart 2.21
  * Make sure that @hostmaster alias works after migration
  * Provide a patch for older civicrm versions to make them Drush 7 compatible
  * Randomize backups schedule to avoid issues with AWS limits
  * Reload nginx service automatically - #465
  * Remove conflicting pdnsd restarts to avoid race conditions - fixes #610
  * Remove deprecated sysctl options
  * Remove post-install leftovers if needed
  * Single PHP-version installation fails - fixes #598
  * Typo - fixes #539
  * Unable to connect to SOLR on latest head - fixes #623
  * Update installers as expected, also with _SKYNET_MODE=OFF - fixes #644
  * Update meta-installers for new stable
  * Update the upgrade procedure how-to - fixes ##616
  * Use civicrm-4.5.6 compatible with Drush 7
  * Use correct AWS Endpoint when us-east-1 Region is specified
  * Use correct open_basedir for lshell user - fixes #603
  * Use separate loops for symlinks and ghost cleanup
  * Workaround for EntityMalformedException in Open Outreach - fixes #229
  * Workaround for missing interface/lo.pdnsd on legacy systems
  * Workaround for SA-CONTRIB-2015-063 - Webform - Cross Site Scripting

I'll run the upgrade one evening this week.

#585: TTech Meeting 5th September 2013

chris — Thu, 05 Sep 2013 07:47:16 GMT

This ticket has been created for the TTech Skype meeting due to take place today.

#604: Times for admin tasks

ed — Thu, 03 Oct 2013 13:10:00 GMT

I am timing some of the admin tasks and keeping a log here for reference. Front of the site is great, backend is still like hauling ghosts through jelly for me.

58 seconds: https://www.transitionnetwork.org/admin/user/user/create

#845: Unneeded FTP server on PuffinServer

chris — Mon, 13 Apr 2015 11:03:12 GMT

The BOA stack installs and runs a FTP server, which we don't need as we use SSH/SFTP and it therefore causes a unneeded load, see ticket:692#comment:177 for an example.

#900: Unusal High Load on Puffin

chris — Sun, 31 Jan 2016 12:24:40 GMT

For the first time since the BOA cron jobs were commented out we have had some very high loads on PuffinServer, this is shown in the Munin graphs, I'll post some specific ones in comments to this ticket.

#567: Update BOA for new Redis 2.6.14

chris — Tue, 02 Jul 2013 11:10:27 GMT

A suggestion from Jim:

BOA now includes Redis 2.6.14 <https://raw.github.com/antirez/redis/2.6/00-RELEASENOTES> if you do a 'barracda up-stable system'... What interests me about this is these lines from the changelog:

UPGRADE URGENCY: HIGH because of the following two issues:

Lua scripting + Replication + AOF in slaves problem (see Issue #1164).
AOF + expires possible race condition (see Issue #1079).

It's a long shot, but that could maybe be part of the issue we've seen recently.

I'm not sure if this is best done now or later tonight when the site is less busy?

#535: Upgrade Puffin, Penguin and Parrot from Debian Squeeze to Wheezy

chris — Fri, 26 Apr 2013 18:49:16 GMT

Debian 7 is due out on 5th May 2013 and wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer will need upgrading.

#612: Upgrade to BOA-2.1.1 Stable Edition

chris — Sun, 03 Nov 2013 11:11:24 GMT

I have been sent this email from wiki:PuffinServer:

There is new BOA-2.1.0 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

I could potentially do this upgrade tonight as it's a quite time for the server -- Jim, any reason not to apply this update tonight?

#629: Upgrade to BOA-2.1.3 Stable Edition

chris — Sun, 24 Nov 2013 11:47:23 GMT

This ticket is for BOA-2.1.3, the previous version was BOA-2.1.1 ticket:612, see wiki:PuffinServer#Upgradetickets The steps to follow when upgrading wiki:PuffinServer are documented at wiki:PuffinServer#UpgradingBOA

#707: Upgrade to BOA-2.2.2

chris — Wed, 26 Mar 2014 19:25:52 GMT

I have created a new ticket for this as I have found having one ticket (see ticket:629) for all BOA upgrades makes it really hard to review past upgrades.

Upgrades from BOA-2.0.7 to BOA-2.1.1 did have their own tickets, see wiki:PuffinServer#Upgradetickets and unless there is a convincing reason not to have one ticket per upgrade I'd rather do it like this.

Jim has pointed out on the Ttech list that:

the v2.2.0 changelog is up as of a few days ago: http://drupalcode.org/project/barracuda.git/blob/HEAD:/CHANGELOG.txt

The Changelog starts:

Stable BOA-2.2.0 Release - Full Edition
Date: TBD
Includes Aegir 2.x-boa-custom version.
Release Notes:

There are many important changes and improvements in this release you should be aware of *before* running your BOA system upgrade.

Even if you are on a hosted BOA system with upgrades managed for you, it is very important to read at least this extensive release notes.

And if you are more curious, read also the big changelog further below, which covers only a small number of over 530 commits since BOA-2.1.3

I have yet to read the rest of the Changelog.

There is also a task to copy the proposed changes to the BOA configuration in ticket:629 over to this ticket.

Should people other than chris and ed be CC's for this ticket?

#721: Upgrade to BOA-2.2.3 Stable Edition

chris — Sat, 19 Apr 2014 05:58:00 GMT

wiki:PuffinServer has sent this email:

There is new BOA-2.2.3 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

The Changelog contains:

### Stable BOA-2.2.3 Release - Full Edition
### Date: Fri Apr 18 12:57:40 PDT 2014
### Includes Aegir 2.x-boa-custom version.
# Release Notes:
  This release includes several bug fixes and security upgrades both for the
  system services and Drupal core, along with three updated platforms and new
  features, including support for MariaDB 10.0 and Ubuntu 14.04 LTS Trusty.
# Important - Read This First! (for self-hosted BOA only)
  If you haven't run full barracuda+octopus upgrade to latest BOA Stable
  Edition yet, don't use any partial upgrade modes explained in docs/UPGRADE.txt
  Once new BOA Stable is released, you must run *full* upgrades with commands:
  $ barracuda up-stable
  $ octopus up-stable all both
  For silent, logged mode with e-mail message sent once the upgrade is
  complete, but no progress is displayed in the terminal window, you can run
  alternatively, starting with screen session to avoid incomplete upgrade
  if your SSH session will be closed for any reason before the upgrade
  will complete:
  $ screen
  $ barracuda up-stable log
  $ octopus up-stable all both log
  Note that the silent, non-interactive mode will automatically say Y/Yes
  to all prompts and is thus useful to run auto-upgrades scheduled in cron.
# Updated Octopus platforms:
  ### Drupal 7.27.1
  Guardr 1.3 ------------------- https://drupal.org/project/guardr
  Open Atrium 2.17 ------------- https://drupal.org/project/openatrium
  Recruiter 1.2 ---------------- https://drupal.org/project/recruiter
# New features and enhancements in this release:
  * Add docs/FAQ.txt
  * Add support for MariaDB 10.0 or 5.5 install via _DB_SERIES variable.
  * Add support for Ubuntu 14.04 LTS Trusty.
  * Improve auto-healing for multi-version PHP-FPM setup.
  * Improve docs/UPGRADE.txt
  * Improve health check for protected vhosts during live SSH-auth update.
# Changes in this release:
  * Issue #GH-299 - Force disable LESS developer mode on production sites.
  * Move custom scripts to /opt/local/bin/
  * Normalize localhost entry in /etc/hosts to avoid FQDN mapped to 127.0.0.1
  * PHP: Do not use separate FPM pool for cron if _PHP_FPM_DENY is empty.
# System upgrades in this release:
  * MariaDB 5.5.37
# Fixes in this release:
  * Add 'exit 0' line if missing.
  * Add /opt/local/bin to PATH by default.
  * Add symlinks for wrappers only temporarily.
  * Better gem uninstall options.
  * Compass: Multiple fixes for various expected gems versions install/upgrades.
  * Do not override lshell env_path in websh wrapper.
  * Do not use monitored bin path for custom scripts to avoid LFD false alarms.
  * Extra db GRANT for 127.0.0.1 not added when migrating site.
  * Improve auto-healing to create required directories in /var/run/ if missing.
  * Issue #2230269 - New Jetty 9 version overrides JETTY_PORT=8099 with 8080.
  * Issue #2235991 - Drush make needs better exceptions in websh wrapper.
  * Issue #2236475 - Clarify what the Legacy mode really means.
  * Issue #2238965 - Add missing path to switch_to_bash().
  * Issue #2241013 - Git commands should be whitelisted in websh wrapper.
  * Issue #2241495 - wkhtmltopdf stopped working after upgrade.
  * Issue #GH-301 - Update the list of restricted keywords for Octopus username.
  * Make sure that permissions on Chive Manager dir/files are correct.
  * Note: _SSL_FROM_SOURCES=YES is ignored and not needed on Wheezy and Precise.
  * Remove the line with header TABLE_NAME (sqlmagic).
  * Reset PATH to avoid RVM overrides after Compass Tools install/upgrade.
  * Shell: Allow to run 'drush cache-clear drush' in any directory.
  * The _PHP_MODERN_ONLY variable is no longer used.
  * Ubuntu 14.04 LTS Trusty requires MariaDB 10.0
  * Use hostname -b instead of deprecated hostname -v.

Note that we are already running the latest MariaDB, see ticket:692#comment:31

See also the last BOA upgrade ticket, BOA-2.2.2 ticket:707 and also ticket:670

#725: Upgrade to BOA-2.2.5

chris — Fri, 02 May 2014 08:49:22 GMT

Note this ticket was opened to upgrade to BOA-2.2.4 but when the upgrade was done BOA-2.2.5 was out so BOA-2.2.4 was skipped

From the Changelog at http://bit.ly/newboa

### Stable BOA-2.2.4 Release - Full Edition
### Date: Wed Apr 30 17:03:36 PDT 2014
### Includes Aegir 2.x-boa-custom version.
# Release Notes:
  This release includes several bug fixes along with five updated platforms,
  plus some hot-fixes applied to previous stable after its release. We have
  also added a fix for known problem is recent Drupal 7.27 [#2245331] hence
  the change from Drupal 7.27.1 to 7.27.2 in all D7 platforms.
# Important - Read This First! (for self-hosted BOA only)
  If you haven't run full barracuda+octopus upgrade to latest BOA Stable
  Edition yet, don't use any partial upgrade modes explained in docs/UPGRADE.txt
  Once new BOA Stable is released, you must run *full* upgrades with commands:
  $ barracuda up-stable
  $ octopus up-stable all both
  For silent, logged mode with e-mail message sent once the upgrade is
  complete, but no progress is displayed in the terminal window, you can run
  alternatively, starting with screen session to avoid incomplete upgrade
  if your SSH session will be closed for any reason before the upgrade
  will complete:
  $ screen
  $ barracuda up-stable log
  $ octopus up-stable all both log
  Note that the silent, non-interactive mode will automatically say Y/Yes
  to all prompts and is thus useful to run auto-upgrades scheduled in cron.
  If you have skipped some recent BOA releases, and you have new default config
  option: _PERMISSIONS_FIX=NO in your /root/.barracuda.cnf configuration file,
  plus, you are not sure if you follow best practices for managing permissions
  as recommended in our docs: https://omega8.cc/node/116 then we recommend
  that you change it to _PERMISSIONS_FIX=YES temporarily, or even permanently
  if your VPS is fast enough, and then run this powerful script as root:
  $ bash /var/xdrago/daily.sh
  Note that BOA 'legacy' mode is still at version 2.1.3
# Updated Octopus platforms:
  ### Drupal 7.27.2
  Commerce 1.25 ---------------- https://drupal.org/project/commerce_kickstart
  Commerce 2.14 ---------------- https://drupal.org/project/commerce_kickstart
  Commons 3.11 ----------------- https://drupal.org/project/commons
  Panopoly 1.5 ----------------- https://drupal.org/project/panopoly
  ### Pressflow 6.31.1
  Commons 2.17 ----------------- https://drupal.org/project/commons
  Note: Always read and follow upgrade procedure if explained in the distro
  release notes, like for Panopoly 1.5 at https://drupal.org/node/2255133
# New o_contrib modules:
  * print-6.x-1.19 (includes patch to auto-detect /usr/bin/wkhtmltopdf)
  * print-7.x-2.0  (includes patch to auto-detect /usr/bin/wkhtmltopdf)
# New features and enhancements in this release:
  * Support for session.gc_maxlifetime configurable via INI files.
  You can control session garbage collector (EOL) per site and per platform.
  The value (in seconds) of the session_gc_eol variable is used as
  session.gc_maxlifetime value and specifies the number of seconds after which
  data will be seen as 'garbage' and potentially cleaned up, resulting with
  $_SESSION variable discarded and affected authenticated users logged out.
  BOA default defined in the system level global.inc file is 86400 == 24h.
# Changes in this release:
  * Drush: Upgrade command line version 6 to mini-6-26-04-2014
  * Nginx: Use higher defaults for limit_conn to avoid error 503 (CloudFlare)
  * Nginx: Use more aggressive limits against spambots trying to rgstr accounts.
  * Redis: Integration module (the modern variant) upgrade to 7.x-2.x-o8-2.6-B
# System upgrades in this release:
  * Nginx 1.7.0
  * PHP 5.5.12
  * Redis 2.8.9
# Fixes in this release:
  * Add symlinks in the home directory if missing (every 5 minutes).
  * Add warning that Compass Tools install and upgrade may take a LONG time.
  * Always define _PHP_CN variable properly.
  * Do not delete symlinks to wrappers to avoid false LFD alarms.
  * Fix for 'Force backward compatible SERVER_SOFTWARE'.
  * Fix in websh for _IN_PATH logic to not break backend Drush tasks.
  * Fix the logic for wrappers update and symlinks.
  * Force MariaDB 5.5 re-install if installed version doesn't match latest.
  * Improve status messages to display when silent mode is used on upgrade.
  * Improve whitelisting in the websh wrapper.
  * Issue #2238805 - Command filtering - no word containing *drush* is allowed.
  * Issue #2241495 - wkhtmltopdf stopped working after upgrade.
  * Issue #2247997 - Update docs/REMOTE.txt with workaround for websh issue.
  * Issue #2250397 - Always follow (limited) redirects in cURL requests.
  * Issue #GH-304  - [rvm] use $_RUBY_VERSION as default.
  * Issue #GH-305  - Check disk usage before running install/upgrade.
  * Issue #GH-306  - Allow ruby 1.8 to remain installed.
  * Nginx: Allow to configure keywords for aggressive requests rate monitoring.
  * Nginx: Sync FastCGI timeouts with other Nginx and PHP-FPM defaults.
  * PHP: Add /opt/local/bin/php tmp symlink on barracuda/octopus upgrade.
  * PHP: Allow to set custom _PHP_FPM_TIMEOUT but not lower than 60 (in seconds)
  * PHP: Always respect _PHP_FPM_WORKERS variable if set to numeric value > 0
  * PHP: Better defaults for realpath_cache_ttl and realpath_cache_size.
  * PHP: Fix for CVE-2014-0185 privilege escalation in FPM (doesn't affect BOA)
  * PHP: pm.max_children was not properly updated on FPM version self-switch.
  * PHP: Sync incorrect default_socket_timeout with max_execution_time (180s).
  * PHP: Use 30s for pm.process_idle_timeout - it prevents too high RAM usage.
  * PHP: Variable _PROCESS_MAX_FPM is not used on the Satellite Instance level.
  * Postfix: Force re-install if broken permisions detected on upgrade.
  * Prevent duplicate cron invocations with more strict delays.
  * Shell: Proper fix for wildcard in the path (cd command only)
  * Standardize install and upgrade for Chive, SQL Buddy and CGP.
  * Sync Redis timeout with default FPM timeout (180s).
  * Sync SQL connect_timeout with default mysql.connect_timeout in PHP (60s).
  * Update the logic for multi-version PHP support in BOND.
  * Update the logic for multi-version PHP support in docs/REMOTE.txt

#745: Upgrade to BOA-2.2.6 Stable Edition

chris — Mon, 23 Jun 2014 09:32:31 GMT

Email from wiki:PuffinServer:

There is new BOA-2.2.6 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

Reading through the change log there doesn't appear to be anything of note that directly effects us, the new php versions:

PHP 5.4.28
PHP 5.5.12

Don't as we are still on PHP Version 5.3.28.

I think it would be worth doing this update one evening this week just to keep upto date with BOA.

Previous updates: wiki:PuffinServer#Upgradetickets

#762: cannot log in to Puffin

annesley — Sat, 19 Jul 2014 08:46:53 GMT

hi, i think i've locked myself out of puffin again. i successfully logged in and navigated to ~ and /. but it wouldn't let me access /data/disk/tn/static/transition-network-d6-s012 it asked for more authentication and then stopped responding.

thanks, a

#698: intransitionmovie.com returns 405 on submit

sam — Thu, 06 Mar 2014 11:45:39 GMT

http://www.intransitionmovie.com/screenings/share-your-screening/

vid Atcheson (david@…) sent a message using the contact form at https://www.transitionnetwork.org/contact.

I couldn't find any contact info at http://www.intransitionmovie.com/, but am hoping you can help. When I visit http://www.intransitionmovie.com/screenings/share-your-screening/index.html and put in my email address and hit Enter, I get:

Hi I checked this and it returns a 405 error for me too.

http://www.intransitionmovie.com/screenings/share-your-screening/

405 Not Allowed nginx/1.4.4

Hoping this can be fixed so we can let you know about our screening on April 8 at the Hawaii State Capitol auditorium.

Aloha,

David Atcheson, Transition Oahu

I'll do some initial investigation

#580: php5-fpm starting when puffin boots

chris — Mon, 12 Aug 2013 16:03:06 GMT

When wiki:PuffinServer is booted /etc/init.d/php5-fpm is started and this causes errors with the site.

#487: robots.txt files for development sites

chris — Tue, 29 Jan 2013 13:24:35 GMT

All the sites other than www.transitionnetwork.org on wiki:PuffinServer should have a robots.txt file to exclude them from being crawled and indexed to prevent the development versions of sites being included in search results.

#836: "Date is invalid" on film content type

sam — Thu, 05 Mar 2015 14:30:22 GMT

Hi Paul

Don't spend more than half an hour on this, if it takes longer I'll just remove the date field instead.

If I edit: https://www.transitionnetwork.org/node/35510/edit

Or add https://www.transitionnetwork.org/node/add/films

A film, the website it returns a "Year is invalid." error.

In the settings it's set to 'Y' https://www.transitionnetwork.org/admin/content/node-type/films/fields/field_film_year

I'm entering a four digit date, eg 2010

Any ideas?

Thanks

Sam

#644: AWstats Nginx config breaks aegir

jim — Mon, 09 Dec 2013 16:46:05 GMT

Since the last update we've had a silent ngnix error that means http://tn.puffin.webarch.net was not available.

I restarted nginx and got:

[  ok  ] Stopping Nginx Server...:
[ .... ] Starting Nginx Server...:nginx: [emerg] "log_format" directive is not allowed here in /etc/nginx/nginx.conf:28

Which equates to the AWstats entry which is now commented out per:

# log for awstats
#log_format apache '$remote_addr - $remote_user [$time_local] "$request" '
#                   '$status $body_bytes_sent "$http_referer" '
#                   '"$http_user_agent"';
#access_log         /var/log/nginx/awstats.log apache;

I/we need access to aegir more than AWStats, so I've commented out the lines above and restarted nginx. Aegir is back and working well.

This ticket is to find the correct log_format for modern nginx versions and reinstate AWstats -- assigning to Chris as a low priority thing.

#626: Add redirect from an old CMS to a new URL

ed — Wed, 20 Nov 2013 12:11:14 GMT

Can Ed add a redirect from:

https://www.transitionnetwork.org/cms/haddenham

to another URL easily? Or does he need to ask Chris to do it?

#824: Analysis of the 2014 maintenance ticket time

chris — Wed, 07 Jan 2015 15:48:14 GMT

Ed has ask that I spend up to 2 hours on an analysis of the 2014 maintenance ticket time for our meeting tomorrow in Bristol.

#790: Annesley locked out of puffin

chris — Tue, 23 Sep 2014 14:05:18 GMT

Email from lfd:

Time:     Tue Sep 23 13:47:01 2014 +0100
IP:       XX.XX.XX.XX (HU/Hungary/XXXXXX.catv.pool.telekom.hu)
Failures: 5 (sshd)
Interval: 300 seconds
Blocked:  Permanent Block
Log entries:
Sep 23 13:46:28 puffin sshd[6056]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=XX.XX.XX.XX  user=tn.ftp
Sep 23 13:46:30 puffin sshd[6056]: Failed password for tn.ftp from XX.XX.XX.XX port 54327 ssh2
Sep 23 13:46:33 puffin sshd[6056]: Failed password for tn.ftp from XX.XX.XX.XX port 54327 ssh2
Sep 23 13:46:56 puffin sshd[6409]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=XX.XX.XX.XX  user=anewholm
Sep 23 13:46:58 puffin sshd[6409]: Failed password for anewholm from XX.XX.XX.XX port 54328 ssh2

#893: BOA Cron Jobs

chris — Thu, 24 Dec 2015 11:39:51 GMT

All the BOA cron jobs were stopped on ticket:846#comment:88. This ticket is for looking at them all and deciding which, if any, are needed.

#692: Debian Updates

chris — Tue, 25 Feb 2014 15:16:17 GMT

This is a ticket to track debian upgrades to the wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer the time they take.

See:

This ticket took over from ticket:218 on 2014-02-25.

#689: Duplicate comments

sam — Fri, 14 Feb 2014 12:21:23 GMT

Hi I got the below message from Mike. Paul could you take a look if you have a minute?

Thanks

Sam

I am noticing that many of the comments are being duplicated quite often - sometimes once and Rob's last comments was added twice. I've been deleting them but will be offline from now over the weekend.

This article is getting lots of comments https://www.transitionnetwork.org/blogs/rob-hopkins/2014-02/open-letter-bbc-lord-lawsons-today-programme-appearance

#901: Enable SSH access to PuffinServer for Ade

chris — Wed, 03 Feb 2016 13:25:27 GMT

This is a ticket to track the time spent sorting out SSH access for Ade to PuffinServer.

#875: Free HTTPS certificates from Let's Encrypt

chris — Mon, 05 Oct 2015 10:48:11 GMT

From mid November 2015 Let's Encrypt should be live, providing free SSL/TLS certificates. Currently the TN pays for a Gandi wild card cert, costing £130.50 a year, in addition most the WordPress sites on ParrotServer don't have certs due to the cost, see ticket:540.

The Let's Encrypt code is designed to be set up to run automatically -- certs are only valid for 90 days and the automatic renewal process runs when the cert is 60 days old.

We should consider if we want to use Let's Encrypt and what things would need to be put in place to use it, the wild card cert is due to expire on 22/01/16.

PuffinServer -- are we still going to be running PuffinServer in January 2016? Is there any chance that we might be able to consider the suggestions in ticket:754#comment:61? I'm not sure if I want to spend time trying to get Let's Encrypt working with a old version of BOA, up to date versions of BOA might support it out of the box.
PenguinServer -- this site hosts a lot of sites, see the listing, automating Let's Encrypt would probably be a hour or two of work, it might makes sense to upgrade it to Debian Jessie at the same time.
ParrotServer -- I suggest we rebuild this server from scratch, this would enable it to have the latest version of the Webarch Secure Hosting scripts and this include support for fail2ban for WordPress and phpMyAdmin, thus solving ticket:871 and includes automatic provisioning of Let's Encrypt certs for sites.

What do people think?

#716: Heartbleed

chris — Wed, 09 Apr 2014 08:53:58 GMT

Following on from ticket:692#comment:18 we should undertake the steps Drupal have taken: https://drupal.org/news/2014-04-08-security-update

#897: Hosting information/requirements for 2016

chris — Tue, 19 Jan 2016 10:14:57 GMT

This is a ticket to track the time spent on an email thread with Ade.

#904: Issues to consider in the migration from Drupal to WordPress

chris — Fri, 19 Feb 2016 10:41:04 GMT

A few weeks ago Ade said he though it would be worth me opening a ticket to use to flag up some issues to be considered in the migration of the Transition Network site from Drupal 6 to WordPress.

#903: Large load spike on PuffinServer

chris — Mon, 08 Feb 2016 08:46:37 GMT

There was a large load spike this morning on PuffinServer, which appears to have been caused by 12k requests for pages (Nginx doesn't log requests for anything other than PHP generated pages) from one IP address, this IP address has been blocked and I'll post some details below.

#884: RE: http://news.transitionnetwork.org

paul — Thu, 03 Dec 2015 12:40:03 GMT

Hi Chris, All
Can you help me to reset my password for paulbooker for
news.transitionnetwork.org? I just tried to use the reset password form but
I never received an email and when I Iooked at the settings,php file for
the website (generated by Aegir) I couldn't see immediately where to find
the database.
I think I may have missed some recent updates to news.transitionnetwork.org
so urgently need to resolve this today.
Not sure how this has fallen of my radar, but, I just noticed that
news.transitionnetwork.org is no longer mentioned on the platform page on
Aegir so may have got into thinking that this site no longer exists.
http://news.transitionnetwork.org
https://tn.puffin.webarch.net/hosting/platforms
--
Paul Booker
Drupal Support for Websites and Linux Servers
Website: http://www.paulbooker.co.uk
Tel: +44 01922 861636

#763: Server Backups

chris — Mon, 21 Jul 2014 17:09:21 GMT

Two weeks ago annesley asked:

what off-site data storage, file backup and quick setup do we have?

I answered:

The 3 virtual servers have their file system mounted off a BSD/NFS/ZFS file server and the whole file system is backed up and stored onto another BSD/ZFS server in the same data centre. We did have backups also being copied to a server in Manchester but this is currently off-line as the Manchester server needs a disk swapping and rebuilding as a BSD/ZFS server.

A problem with this is that it's only me and Alan that have access to these backups, so I'd like to suggest I set up a new account for backups on our backup server and sort out cron jobs to rsync data to this account and document how people can access these backups.

The result would be that everybody would have SFTP access to 60 days worth of snapshots of backups from all three servers whenever needed without any need for my or Alan's intervention.

I expect this would take abount an hour to set up and another hour to document and help people understand it.

There would be no additional cost to the TN because backup space is already paid for.

#834: Slovenian State info missing again

sam — Thu, 26 Feb 2015 10:41:47 GMT

Hi Paul

The change that you made in this ticket: https://trac.transitionnetwork.org/trac/ticket/802

Seems to have been lost. I am no longer able to edit https://www.transitionnetwork.org/node/37435/edit

As the state/province information is missing.

Could you re-do the change please?

Thanks

Sam

#742: Stg site to play with

sam — Thu, 12 Jun 2014 14:35:42 GMT

Hi Paul

I'm trying to set up a stage site, just to test rearranging the homepage blocks.

I created a site on the "Transition Network D6 S012 Booker" Platform, but I just get an empty pressflow site: http://stgsam.transitionnetwork.org/

Can I use your stg site to test the block arrangement instead: https://booker-stage-20140501.transitionnetwork.org/

Or could you let me know what might be going wrong?

Thanks

Sam

#859: Subscription emails broken

sam — Tue, 16 Jun 2015 13:08:44 GMT

Hi just got this mail

"For some reason I realized I wasn't hearing from Rob. You might want to check your system because mine hasn't changed as far as I know."

Had a look in my inbox & the last mail from Drupal subscription system was on 27th of May.

I may be the guilty party, as I did go in to edit the message around this time.

I'll investigate via the Drupal admin interface, but has anything else happened/ been done that could have stopped the mails?

#905: TN site down due to redis not running

chris — Thu, 25 Feb 2016 10:28:40 GMT

I'm working on this...