Transition Technology: Ticket Query

#97: Hardware upgrade to GH tier2 servers

ed — Thu, 10 Jun 2010 20:13:29 GMT

Gaiahost have given us their 'tier2' servers so we will need to move across to them.

#101: 27th June 2010 Site Downtime

chris — Mon, 28 Jun 2010 08:03:59 GMT

On Sunday 27th June the site was down for about 10 hours, see the attached graph of MySQL throughput.

Following are some extract from the emails sent from Gaia regarding this issue.

We need to investigate the cause of this to see if we can avoid it happening in the future...

The issue was the 'variable' table in the "live" drupal database had crashed. I took the web server fully offline for a few minutes just now as I repaired that table and then brought it back online. The homepage is now working fine, but other pages are still generating errors. There's a backup of the database I took just now at:

/web/transitionnetwork.org/live.20100627.sql

I had to take the site back down again and manually emptied all 'cache*' tables in the database.

we received emails into support@… from two people browsing the site early this morning. The default server error notice includes a reference to our support email.

So what can be done to mitigate this in the future: You should review Drupal related logs to see if it gives any clues as to what caused the crash. An action on the 'variable' table in the database could be a clue, as that as the sole table crashed in the db. There would have been a faster response to the outage on our side if our monitors had been specifically watching the drupal database named "live" as well as a more content specific monitor for an actual page generated by Drupal. We can coordinate on monitor customization, just let us know.

it looks like the internal server errors starting happening right after /cron.php was run at 23:00 UTC:

transitiontowns.gaiahost.coop - - [26/Jun/2010:23:00:00 +0000] "GET /cron.php HTTP/1.0" 200 - "-" "ApacheBench/2.3"

It specifically didn't look like a corruption related to a regular system or database backup process. The other traffic at this time was normal.

#103: Managing news install

ed — Mon, 05 Jul 2010 15:35:43 GMT

install managing news on the gaiahost servers once we're on the new hardware... trial on graham's box here: http://mn.mc3.coop/

#104: Media page: add map to listings

ed — Mon, 05 Jul 2010 15:55:25 GMT

The media page: http://www.transitionnetwork.org/media

Please include a map - with the markers being those initiatives (rather than the people) - which is linked from the media page with the following text:

"<link>Click here to see these initiatives on a map</link>"

#105: Initiative profile editors locked out of their profiles

ed — Wed, 07 Jul 2010 12:14:16 GMT

Some Initiative profile editors are locked out of editing their profiles.

It appears that the 'Further information' free text field for some of the initiatives has turned itself to 'full html', locking the initiative profile editors out of their pages (they don't have access to full html).

This happened with http://www.transitionnetwork.org/initiatives/bristol http://www.transitionnetwork.org/initiatives/paredes-em-transição

but not http://www.transitionnetwork.org/initiatives/hkd-transition

not sure how to proceed with this one, but it's important

#106: extra text for initiative addition form

ed — Wed, 07 Jul 2010 13:56:43 GMT

The initiatives are still giving themselves silly names and putting 'transition' in the front of their locations.

This URL: http://www.transitionnetwork.org/node/add/initiative-profile

Update the text at the top of the initiative adding form to this:

"Please choose a name for your initiative that reflects your location. As well as this making sense to everyone else, the name affects how it shows up in the alphabetical directory search. We recommend that you do not add "Transition" or "Transition Town" in front of your name as it will only appear in the "T" section of the directory. Thank you.

#109: Commenting process needs tidying up

ed — Thu, 08 Jul 2010 09:44:55 GMT

When you make a comment, the 'preview' stage does not show your identity (photo etc.). Your identity only appears once the comment is saved.

Please include the user's identity during this middle 'preview' step of the commenting process

#110: Remove RHS blocks from events calendar page

ed — Thu, 08 Jul 2010 09:55:43 GMT

Please remove RHS blocks from events calendar page: http://www.transitionnetwork.org/events/calendar

and make the calendar as big as possible!

if that is the biggest it gets (currently), then don't remove blocks.

ALSO: is this something to do with Ed - I think it might be?

#111: Adding a map block for directory pages

ed — Thu, 08 Jul 2010 09:57:40 GMT

Q: how much time is it to create a block to show the map (which links to the big map) for the directory pages:

initiatives
projects
people

#112: Swap mollom for reCatpcha on registration

ed — Thu, 08 Jul 2010 10:41:26 GMT

many users having problems with mollom on registration. not seeing the captcha and not being able to hear the audio version. please swap mollom for recaptcha on registration (where we don't need mollom's sophistication)

#113: A forum list of all the latest topics

ed — Wed, 14 Jul 2010 10:44:00 GMT

Q: how do I go about creating one page where the latest forum topics are listed? Like the active forum topics, but in a page. User requests that the 'more' link from the active forum topics block goes to a list of latest topics, not the forum homepage as current.

#114: Change web host, use better Drupal stack, save CO2 with modern VPS + server setup

jim — Wed, 14 Jul 2010 11:20:42 GMT

Gaia host are not very good - they're too busy with their other big clients for us. Their server isn't much cop and neither is their support turnaround. I don't know how much they charge but MANY other hosts can provide us with a VPS (not a whole, mostly wasted but horribly underpowered dedicated server) with lots of flexibility and speed.

E.g. Linode (http://www.linode.com/) has unmanaged London-based hosting and can provide us a very Fast (according to several reviews I've read) VPS with 1Gb memory, 32Gb storage and 400GB of bandwidth for ~£25 per month. IF we needed more juice (and I see no reason why we would), memory and CPU can be extended easily. Linode are one of the best I've found so far, though there are literally hundreds to choose from.

I'm pretty certains a modern, low-power server split into many virtual machines is much better for the environment than as single machine - it's basic maths.

Finally, by getting a Pantheon Mercury Drupal (http://getpantheon.com/mercury/what-is-mercury) VPS image we can have a Drupal system that is tuned, supported, AND VERY VERY VERY FAST up and running in a few minutes. It's really worth your time to read about it.

Feel free to close/delete this ticket if it's inappropriate, but frankly Gaia's service and performance on our current setup has been embarrassing.

#115: Update Drupal core to 6.19, plus the theme and modules to latest

jim — Wed, 14 Jul 2010 15:21:51 GMT

#116: Update all Drupal modules

jim — Wed, 14 Jul 2010 15:27:56 GMT

Security and other updates include: Chaos tools, Contact, CCK, content profile, context, Custom Breadcrumbs, Features, Feeds, FileField?, GMap, Hierarchical Select, Image Resize Filter, ImageAPI, ImageField?, IMCE, Link, Location, LoginToboggan?, Menu block, Mime Mail, Mollom, Node Access User Reference, Node Limit Number, Nodequeue, Organic groups, Panels, Reroute Email, Rules, Simplenews, Terms of Use, Token and Views currently.

I would suggest this needs the #97 live server upgrade comes first to allow us to do the SVN process quickly. The above will likely fix some minor issues too.

#117: Forum PathAuto settings not working

jim — Thu, 15 Jul 2010 12:16:05 GMT

Currently using incorrect tag as path and/or term lineage not working.

#118: Newsletter - integrate MailChimp with Drupal name fields Content Profile fields

ed — Thu, 15 Jul 2010 17:23:27 GMT

install mailchimp integration module
and other things that you need to do to get back integrated with mailchimp

#119: Initiatives by number table needs UK not broken

ed — Thu, 15 Jul 2010 17:39:47 GMT

please make the table so that 'United Kingdom' remains on one line, so that the table is one line per ini (on the whole)

http://www.transitionnetwork.org/initiatives/by-number

#120: Patterns Directory: start a new directory

ed — Mon, 19 Jul 2010 14:18:37 GMT

Create a new directory on Tn.org for the 'Patterns' as per the document: 'Pattern language online facilitation report and early project scope FINAL.doc'

Please review document and make estimate for directory build and test

Delivery: August 2010!

#121: email alerts for new projects and project updates

ed — Mon, 19 Jul 2010 15:05:29 GMT

please either:

set up an email alert that sends a message to ed whenever a new project is added or a project is updated

tell ed how to do it - he can't work it out!

#122: email alerts for moderation actions

ed — Tue, 20 Jul 2010 10:37:47 GMT

please either:

set up an email alert to be sent to Ed when a users flags an item as nonsense and invokes the moderation module

tell ed how to do it!

#124: Live database backups

chris — Tue, 20 Jul 2010 13:06:52 GMT

Automate the nightly syncing of a database dump on the live server to the dev server and have an easy to import the live db, minus email addresses, into the dev and test sites.

#125: 'Facilitator' and 'Speaker' to the user profile editing options

ed — Thu, 22 Jul 2010 15:36:25 GMT

We need to be able to identify and list all the users who have self-identified as either/and 'Facilitator' and 'Speaker' on the people profiles. This requires:

adding these options ('Speaker' 'Facilitator') as tick boxes to the 'Involvement and interest with the transition network movement' section of the user profile.

creating views like the /media views with:

2a. list of names (with links to their profiles but not the initiatives) 2b. a map

#128: Design: links in some blocks wrong colour

ed — Mon, 26 Jul 2010 11:44:05 GMT

The links in some blocks in some regions are grey instead of the nice warm red-y colour. I've added blocks to the homepage (in footer to get four) and the links are grey: http://www.transitionnetwork.org/

Please adjust

#130: Remove 'promote' function on items

ed — Thu, 29 Jul 2010 16:20:25 GMT

we have nodequeue for front page news and blogs - so don't need the 'promote' option in the publishing bit of node editing pages. it's confusing the editor so please remove it.

#131: nodequeue: front page add broken

ed — Wed, 04 Aug 2010 10:48:58 GMT

it seems that nodequeue's add to front page is broken: http://www.transitionnetwork.org/node/12157/nodequeue

have tried on a couple of items today and it's not happening.

pretty important this one so sticking to critical...

#132: Documentation

ed — Fri, 06 Aug 2010 16:18:20 GMT

please write an item describing the features and functions behind the site - as you did earlier here: http://transitiontowns.org/webprojectblog/2010/01/25/features/

It is to help outsiders understand the site architecture, modules and features - the early piece was really popular.

#147: Migration of live server

chris — Fri, 15 Oct 2010 14:31:20 GMT

This is a ticket to track the migration of wiki:LiveServer (transitionnetwork.gaiahost.coop) to wiki:NewLiveServer (quince.webarch.net), even though we haven't got the final go ahead, due to the desire to migrate by the end of the month, I have made a "at risk" start on this.

The hosting side of things is all done apart from sorting out the backups.

The virtual server has been set up, accounts have been created for jim and john and their public keys have been installed so they should be able to connect to it.

A copy of the live site has been set up here and it's now available for testing:

http://live.quince.webarch.net/

It has the same settings as the dev and test sites to ensure that it doesn't send out unwanted emails to users.

One issue I found is that the cacherouter module isn't in svn any more, so this was manually copied across, there are also some drupal errors and I think they might also be due to missing modules but I haven't bottomed this out yet (more on these issues here: wiki:NewLiveServer).

One thing we need to decide is what php accelerator to use -- no apache/php/mysql optimisations at all have been done so far.

#165: Security certificate warning on new server

ed — Thu, 04 Nov 2010 20:46:31 GMT

Users are getting site untrusted warnings: is all well with the security certificate on new servers?

"Just one comment - when I try to connect to the login page, my browser is telling me that the site is untrusted and that the connection can not be verified. Are there some additional steps you need to take to ensure that the site connects securely?

The browser reports that your site is using an invalid security certificate."

#218: Debian upgrades and updates

chris — Thu, 06 Jan 2011 13:34:16 GMT

This is a ticket to track debian upgrades to the wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer the time they take.

See:

These updates are generally done using the wiki:AptitudeUpdateScript and this records all the changes in the /root/Changelog and then the contents of the Changelog are pasted into the ticket to document the upgrade.

This ticket was was originally used for the ~~wiki:DevelopmentServer~~ and the ~~wiki:NewLiveServer~~.

#221: Adding Big Blue Button to TN.org: QUOTE PLEASE

ed — Tue, 11 Jan 2011 09:41:55 GMT

please give a quote for adding Big Blue Button virtual conferencing to TN.org, integrating it with the user database so that users can add and join teleconferences with their Transition logins. Please consider likely hosting and bandwidth costs etc. for a year.

#227: Set up mirroring capability for core data types

ed — Mon, 31 Jan 2011 16:06:08 GMT

Action point from the web strategy meeting, while discussing security:

Technical discussion track needed on ongoing security with these priorities: 1. how to keep safe copies, 2. how to avoid vandalism

Need to explore 'Mirrors' for TN.org: core data: all Directories, secondary data: forums, blogs, news etc.

This ticket for setting up 'mirroring' capability for core data as part of the under the waterline element of phase 4

#287: Live Server Load

chris — Fri, 01 Jul 2011 10:19:01 GMT

Currently it's though the roof (anything over 4 and it's maxed out, it has 4 processors):

load average: 43.63, 33.84, 18.85

This could be because of things Jim is doing combined with the GA import that is running but I'm seeing a *lot* of apache processes so it seems to be combined with a spike in traffic.

I'm keen to migrate the site onto a faster server :-)

#301: Upgrade LIVE server to Debian Squeeze

jim — Wed, 17 Aug 2011 20:32:44 GMT

Many of the libraries and programs available to us are limited to older versions because LIVE is on Debian Etch. Upgrading to Squeeze will allow many components to be upgraded, and hopefully provide other nice side effects.

Elements upgraded:

Memcached (important)
PHP/MySQL/Apache (probably?)
Linux kernel an all associated stuff

#369: Drupal-level performance enhancements

jim — Sun, 04 Dec 2011 23:23:27 GMT

Per TTech discussion, some easy/quick things we can do to improve the site's performance... In rough order of ease technically and politically:

Trawl SQL logs for slow queries, find out what's causing them and eradicate the cause -- tough job but potentially huge returns.
Add flood control/DoS blocking and other temporary blocks at the server level if an IP address or crawler bot is taking the piss. A job for Chris.
~~Do config/performance enhancement ideas from DB Tuner (was in mysqltuner.log, JK to run again around xmas)~~
~~Disable Piwik on LIVE and use Google Analytics instead OR move Piwik to another server. Chris is opposed to this but server resources are at a premium unfortunately -- See #371~~
Massively shrink site CSS -- Small performance increase for 'normal' users, BIG gains for older computers and mobile devices - As Laura points out, better than lots of images, but it still needs to re-use CSS rules/selectors where possible and throw out legacy code.
Force HTTP only for non-logged in users (explained above) -- BIG boost, quick settings change, though might need Chris to alter htaccess/varnish setup.
~~Enable Throttle module to auto-disable modules during heavy loads -- Throttle gets a bad press and can cause issues.~~
Go on a module cull -- This is hard and slow, and will result in a loss of functionality. But some modules could be dropped if there's a will.
~~Switch to NGINX -- Chris thinks this is a lot of work, see #357~~

#370: Problem with nightly MySQL backup load

chris — Mon, 05 Dec 2011 13:26:39 GMT

Since the start of December 2011 the nightly backupninja mysql backup is causing load spikes which result in the server being unresponsive.

#385: Install new SSL certificate on TN.org

laura — Fri, 06 Jan 2012 15:02:22 GMT

The SSL certificate has been renewed today via Gandi and needs to be set up. (Chris see email sent earlier today!)

#386: Domain redirect for the InTransitionmovie domain

laura — Fri, 06 Jan 2012 15:04:06 GMT

As a temporary redirect for the intransitionmovie.com in gandi, the domain is currently pointed to the home page of TN. Until we get the new micro site up and running, we would like to direct the domain to www.transitionnetwork.org/transition-2 for now.

#390: Apache pcre segfaults

chris — Thu, 02 Feb 2012 10:55:58 GMT

Yesterday apache / pcre was segfaulting, an extract from /var/log/messages, follows, in all there were 51 segfault log lines, starting at Feb 1 17:44:34 and ending at Feb 1 19:21:08, I haven't been able to work out the cause of this, I'm opening this ticket in case we see more segfaults and do want to follow it up.

Feb  1 17:44:34 quince kernel: [11153359.760960] apache2[8193]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:34 quince drupal: https://workspaces.transitionnetwork.org|1328118274|user|XXX.XXX.XXX.XXX|https://workspaces.transitionnetwork.org/customerror/403?destination=node%2F116|https://workspaces.transitionnetwork.org/ws/national-hubs|0||Login attempt failed for James Samuel.
Feb  1 17:44:35 quince kernel: [11153361.102389] apache2[8182]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:36 quince kernel: [11153362.535735] apache2[8198]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:38 quince kernel: [11153364.036616] apache2[7160]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:43 quince drupal: https://workspaces.transitionnetwork.org|1328118283|user|XXX.XXX.XXX.XXX|https://workspaces.transitionnetwork.org/customerror/403?destination=node%2F116|https://workspaces.transitionnetwork.org/customerror/403?destination=node%2F116|0||Login attempt failed for James Samuel.
Feb  1 17:44:46 quince kernel: [11153372.256252] apache2[7212]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:48 quince kernel: [11153374.450117] apache2[8214]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:49 quince kernel: [11153375.778416] apache2[8215]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]

An extract from the /var/log/apache/eeeor.log:

[Wed Feb 01 17:44:34 2012] [notice] child pid 8193 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:36 2012] [notice] child pid 8182 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:37 2012] [notice] child pid 8198 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:39 2012] [notice] child pid 7160 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:47 2012] [notice] child pid 7212 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:49 2012] [notice] child pid 8214 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:50 2012] [notice] child pid 8215 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:52 2012] [notice] child pid 8217 exit signal Segmentation fault (11)

The corresponding apache access log for the first min:

127.0.0.1 - - [01/Feb/2012:17:44:00 +0000] "GET /tags/behaviour-change HTTP/1.1" 200 7542 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:01 +0000] "GET /cron.php HTTP/1.0" 302 - "-" "Wget/1.12 (linux-gnu)"
127.0.0.1 - - [01/Feb/2012:17:44:02 +0000] "GET /notifications/anonymous/subscribe/author/author/10548?destination=node%2F17790&timestamp=1327498024&signature=a1db08a827cdbaf608b5ae75df760481 HTTP/1.1" 200 5881 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:04 +0000] "GET /blogs/energy-bulletin?page=73 HTTP/1.1" 200 8935 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:06 +0000] "GET /notifications/anonymous/subscribe/author/author/8220?destination=node%2F13477&timestamp=1328117196&signature=92f5e293767d5afa02d60fae9a72ae0c HTTP/1.1" 200 5878 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:07 +0000] "GET /people/monika-kirsch HTTP/1.1" 200 6504 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:12 +0000] "GET /query/entity_ids-location/json/1/52.511012,-1.469337_0.5/10 HTTP/1.1" 200 17 "-" "Drupal (+http://drupal.org/)"
127.0.0.1 - - [01/Feb/2012:17:44:12 +0000] "GET /initiatives/nuneaton-transition-town HTTP/1.1" 200 9355 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
127.0.0.1 - - [01/Feb/2012:17:44:13 +0000] "GET /favicon.ico HTTP/1.1" 302 272 "-" "Safari/7534.52.7 CFNetwork/520.2.5 Darwin/11.2.0 (x86_64) (MacBookPro8%2C2)"
127.0.0.1 - - [01/Feb/2012:17:44:15 +0000] "GET /node/all/by_subject/1538?order=title&sort=asc HTTP/1.1" 200 6145 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:15 +0000] "GET /news/2011-01-26/send-rob-your-posters HTTP/1.1" 200 7851 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:17 +0000] "GET /node/all/by_subject/428?order=comment_count&sort=desc HTTP/1.1" 200 5992 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:17 +0000] "GET /query/entity_ids-location/json/1/50.633501,3.788273_1/10?page=3 HTTP/1.1" 200 9142 "-" "Drupal (+http://drupal.org/)"
127.0.0.1 - - [01/Feb/2012:17:44:19 +0000] "GET /people/facilitators?page=64 HTTP/1.1" 200 7488 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:19 +0000] "GET /sources/uk/deal-it?page=%2C7%2C9%2C2%2C2%2C7 HTTP/1.1" 200 5364 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:22 +0000] "GET /node/all/by_subject/1798?order=last_comment_timestamp&sort=desc HTTP/1.1" 200 5904 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:23 +0000] "GET /people/mike-miller HTTP/1.1" 200 6540 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:24 +0000] "GET /node/all/by_subject/1350?order=type&sort=desc HTTP/1.1" 200 5689 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:26 +0000] "GET /node/all/by_subject/236?order=last_comment_timestamp&sort=asc HTTP/1.1" 200 5711 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:27 +0000] "GET /cms/bellingen HTTP/1.1" 200 8468 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
127.0.0.1 - - [01/Feb/2012:17:44:28 +0000] "GET /node/all/by_subject/3341?order=last_comment_timestamp&sort=desc HTTP/1.1" 200 5692 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:29 +0000] "GET /people/alex-cicelsky HTTP/1.1" 200 6249 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:34 +0000] "GET /piwik.js HTTP/1.1" 200 7779 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.1; .NET4.0C)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:34 +0000] "GET /piwik.php?action_name=Welcome%20%7C%20Transition%20Network&idsite=1&rec=1&r=680344&h=11&m=44&s=34&url=http%3A%2F%2Fwww.transitionnetwork.org%2F&urlref=http%3A%2F%2Fwww.theglobeandmail.com%2Fnews%2Fopinions%2Fopinion%2Four-peak-oil-premium%2Farticle2321815%2Fcomments%2F&_id=218ab4fa03a9a340&_idts=1328118274&_idvc=1&_idn=1&_refts=1328118274&_viewts=1328118274&_ref=http%3A%2F%2Fwww.theglobeandmail.com%2Fnews%2Fopinions%2Fopinion%2Four-peak-oil-premium%2Farticle2321815%2Fcomments%2F&java=1&res=1069x855&cookie=1 HTTP/1.1" 200 50 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.1; .NET4.0C)"
::1 - - [01/Feb/2012:17:44:35 +0000] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [01/Feb/2012:17:44:35 +0000] "GET /notifications/anonymous/subscribe/author/author/4545?destination=node%2F4907&timestamp=1327729023&signature=7de36d2b702fd311d9e29c0b07839c66 HTTP/1.1" 200 5896 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:36 +0000] "GET /people/abduselam-abdai-mohammed HTTP/1.1" 200 7457 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:37 +0000] "GET /node/all/by_subject/2400?order=last_comment_timestamp&sort=asc HTTP/1.1" 200 5707 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:40 +0000] "GET /node/15409/by_subject/3007?order=comment_count&sort=desc HTTP/1.1" 200 5857 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:41 +0000] "GET /people/sophy-banks HTTP/1.1" 200 7352 "http://www.transitionnetwork.org/news" "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"
127.0.0.1 - - [01/Feb/2012:17:44:43 +0000] "GET /user/register?destination=comment%2Freply%2F21723%23comment-form HTTP/1.1" 301 307 "http://www.transitionnetwork.org/news" "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"
127.0.0.1 - - [01/Feb/2012:17:44:42 +0000] "GET /notifications/anonymous/subscribe/author/author/3016?destination=node%2F3378&timestamp=1328116976&signature=135af8f9b16253b1f60ca6161cac7141 HTTP/1.1" 200 5882 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:44 +0000] "GET /blogs/energy-bulletin/2011-12-23?page=22 HTTP/1.1" 200 9119 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:44 +0000] "GET /sharing/widget_news HTTP/1.1" 200 1095 "http://transitionnorwichnews.blogspot.com/2012/01/where-are-all-fruit-and-nut-trees.html" "Mozilla/5.0 (Windows NT 5.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /node/all/by_subject/2409?order=title&sort=asc HTTP/1.1" 200 5818 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /query/entity_ids-location/rss/1/50.431213,-3.685455_1/10 HTTP/1.1" 200 2016 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; Microsoft Outlook 14.0.6109; ms-office; MSOffice 14)"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /notifications/anonymous/subscribe/author/author/1064?destination=node%2F1427&timestamp=1327506623&signature=1b484e663e414fc697ce6363edf7e2b4 HTTP/1.1" 200 5878 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /notifications/anonymous/subscribe/author/author/3461?destination=node%2F3823&timestamp=1322606638&signature=483b9c34a74cbf27a739746b8ea9588a HTTP/1.1" 200 5874 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:46 +0000] "GET /tags/bus-transportation HTTP/1.1" 200 6911 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:47 +0000] "GET /resources/branding-guidelines?order=field_asset_type_value&sort=asc&page=1&themes_nid[0]=13 HTTP/1.1" 200 10076 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:51 +0000] "GET /piwik.js HTTP/1.1" 200 7779 "http://www.transitionnetwork.org/blogs/ben-brangwyn/2011-03/faith-movie-weve-all-been-waiting" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB7.2; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET4.0C; .NET CLR 3.0.30729; AskTB5.6)"
::1 - - [01/Feb/2012:17:44:52 +0000] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:51 +0000] "GET /piwik.php?action_name=The%20faith%20movie%20we've%20all%20been%20waiting%20for!%20%7C%20Transition%20Network&idsite=1&rec=1&r=327050&h=12&m=44&s=53&url=http%3A%2F%2Fwww.transitionnetwork.org%2Fblogs%2Fben-brangwyn%2F2011-03%2Ffaith-movie-weve-all-been-waiting&urlref=http%3A%2F%2Fwww.google.ca%2Fimgres%3Fimgurl%3Dhttp%3A%2F%2Ffr.academic.ru%2Fpictures%2Ffrwiki%2F71%2FGreat_Synagogue_Plzen_CZ.jpg%26imgrefurl%3Dhttp%3A%2F%2Fwww.transitionnetwork.org%2Fblogs%2Fben-brangwyn%2F2011-03%2Ffaith-movie-weve-all-been-waiting%26h%3D1556%26w%3D1387%26sz%3D346%26tbnid%3DDyxrYgfcASljuM%3A%26tbnh%3D90%26tbnw%3D80%26prev%3D%2Fsearch%253Fq%253D(synagogue)%2526tbm%253Disch%2526tbo%253Du%26zoom%3D1%26q%3D(synagogue)%26docid%3D4kDZHm-yl2nChM%26hl%3Dfr%26sa%3DX%26ei%3DvnkpT--mG4rG0AHM1dXUAg%26ved%3D0CFYQ9QEwBw%26dur%3D1076&_id=4ecf13956b5e2ee4&_idts=1328118293&_idvc=1&_idn=1&_refts=1328118293&_viewts=1328118293&_ref=http%3A%2F%2Fwww.google.ca%2Fimgres%3Fimgurl%3Dhttp%3A%2F%2Ffr.academic.ru%2Fpictures%2Ffrwiki%2F71%2FGreat_Synagogue_Plzen_CZ.jpg%26imgrefurl%3Dhttp%3A%2F%2Fwww.transitionnetwork.org%2Fblogs%2Fben-brangwyn%2F2011-03%2Ffaith-movie-weve-all-been-waiting%26h%3D1556%26w%3D1387%26sz%3D346%26tbnid%3DDyxrYgfcASljuM%3A%26tbnh%3D90%26tbnw%3D80%26prev%3D%2Fsearch%253Fq%253D(synagogue)%2526tbm%253Disch%2526tbo%253Du%26zoom%3D1%26q%3D(synagogue)%26docid%3D4kDZHm-yl2nChM%26hl%3Dfr%26sa%3DX%26ei%3DvnkpT--mG4rG0AHM1dXUAg%26ved%3D0CFYQ9QEwBw%26dur%3D1076&java=1&res=1366x768&cookie=1 HTTP/1.1" 200 50 "http://www.transitionnetwork.org/blogs/ben-brangwyn/2011-03/faith-movie-weve-all-been-waiting" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB7.2; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET4.0C; .NET CLR 3.0.30729; AskTB5.6)"
127.0.0.1 - - [01/Feb/2012:17:44:51 +0000] "GET /tags/reedbed HTTP/1.1" 200 6483 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:52 +0000] "GET /news/network/feed HTTP/1.1" 200 6004 "-" "Apple-PubSub/65.28"
127.0.0.1 - - [01/Feb/2012:17:44:52 +0000] "GET /people/mim-meilland HTTP/1.1" 200 6555 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:53 +0000] "GET /tags/may-celebration HTTP/1.1" 200 6412 "https://www.transitionnetwork.org/user/login" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/5.0)"
127.0.0.1 - - [01/Feb/2012:17:44:53 +0000] "GET /people/irene-lim HTTP/1.1" 404 44619 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:55 +0000] "GET /notifications/anonymous/subscribe/author/author/2791?destination=node%2F3153&timestamp=1328109179&signature=bb861cf6a1a3d856bc959f44c72f017e HTTP/1.1" 200 5887 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:55 +0000] "GET /user/login?destination=comment%2Freply%2F14129%23comment-form HTTP/1.1" 301 307 "http://tones.transitionnetwork.org/tags/may-celebration" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/5.0)"
127.0.0.1 - - [01/Feb/2012:17:44:57 +0000] "GET /sources/uk/deal-it?page=%2C7%2C9%2C2%2C2%2C5 HTTP/1.1" 200 5364 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:59 +0000] "GET /user/12203/contact HTTP/1.1" 301 278 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
127.0.0.1 - - [01/Feb/2012:17:44:59 +0000] "GET /tags/arts-and-culture HTTP/1.1" 200 7813 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

#391: PSE tracking, moderation and security

chris — Thu, 02 Feb 2012 11:30:00 GMT

This ticket is track time spent on the Tracking, Moderation and Security discussion which has a wiki page here: https://wiki.transitionnetwork.org/Sharing_Engine/Tracking,_Moderation_and_Security

#392: PSE Server Upgrade

chris — Thu, 02 Feb 2012 12:53:41 GMT

This ticket has been created to discuss and implement and upgrade to the wiki:NewLiveServer

#396: Migrate MySQL Databases from MyISAM to InnoDB

chris — Thu, 23 Feb 2012 15:32:33 GMT

This should result in things being faster, however there are some potential new issues that it would create, see for example The perils of InnoDB with Debian and startup scripts.

There is also an issue with the one !InnoDB table on kiwi which will need investigation and fixing, the dev Mediawiki database backup fails each night, this is the error:

Warning: mysqldump: Got error: 1286: Unknown table engine 'InnoDB' when using LOCK TABLES
Warning: Failed to dump mysql databases transwiki

#397: Live server RAM and disk upgrade

chris — Thu, 23 Feb 2012 15:54:04 GMT

The live server is all set to have the additional 1GB of RAM added to it, it just needs a reboot at an appropriate time for this to become available. Once it's available configurations changes will be needed to Apache, Varnish etc to enable applications to make use of it.

In addition to upgrading the RAM we have made a new 15GB partition available to the virtual machine (it'll only appear after a reboot) with the intention of using this for the !MySQL databases. This partition is on a pair of mirrored SCSI disks, the rest of the virtual machine is on a pair of SATA disks.

A quick test of the new partition disk speeds:

 hdparm -tT /dev/FastDisk/quince-var-lib-mysql
/dev/FastDisk/quince-var-lib-mysql:
 Timing cached reads:   3626 MB in  2.00 seconds = 1815.54 MB/sec
 Timing buffered disk reads:  408 MB in  3.01 seconds = 135.60 MB/sec

Compared to the SATA disks:

hdparm -tT /dev/xvda2
/dev/xvda2:
 Timing cached reads:   1998 MB in  2.00 seconds = 999.00 MB/sec
 Timing buffered disk reads:  92 MB in  3.03 seconds =  30.34 MB/sec

The faster partition and the switch to !InodeDB, see ticket:396 should, hopefully, result in dramatic performance improvement for !MySQL.

#398: Host Upgrade to Debian Squeeze

chris — Wed, 29 Feb 2012 21:31:14 GMT

While the host that the virtual machines are on is upgraded to squeeze we will point the IP address at another server, this ticket is for this temp redirect of the site to a splash page.

Also ticket:396 will be done during the downtime.

#401: Intransitionmovie.com errors with Google and Paypal

laura — Fri, 16 Mar 2012 12:19:53 GMT

Having two issues currently on the tnmovie site, and you may be able to guide -

1 - We're using a simple plugin that checks the address input with Google to validate it for the 'share your screenings' section on the site. Each time an address is input it comes up with an error. 'Error encountered during lookup'. Worked all fine on my demo server, but not on the new hosting. I've enabled an API key to be able to refer to the tnmovie site to see if that helps (the plugin doesn't require it), and also on the one item that was okay (transferred from the demo site) - and the google map doesn't show. Not sure if a settings on the webserver thing, or I need to add anything extra to the files.

2 - not sure if paypal has had some hiccups today and late yesterday. We're trying to find out from customers who seems to add an order but not be able to pay, to find out whether the problem is on our side, or when trying to complete the transaction in paypal. The orders show in the admin section of the site as pending, but no payment went through in paypal, nor any paypal logs showing in the wp-plugins/woocommerce/logs/paypal.txt - anything you spot in the server logs that would highlight any errors all welcomed. Some orders have gone through - but a few have tried several times and not managed to pay. (I had a quick look at the logs files via sftp, but I need a lesson in understanding all of those!) Thanks

#403: Wild card domain names - *.transitionnetwork.org

chris — Sat, 31 Mar 2012 10:51:02 GMT

Currently http://everything.transitionnetwork.org/ and http://anything.transitionnetwork.org/ will serve the live site -- would it be OK to redirect all requests that are not prefixed with www to http://www.transitionnetwork.org/ in order to make varnish (and nginx in the future) more efficient?

#404: Wild card domain names - *.transitionnetwork.org

chris — Sat, 31 Mar 2012 10:54:57 GMT

#405: Live server APC settings

chris — Tue, 17 Apr 2012 10:52:39 GMT

Since the changes to the APC settings were made for Piwki, ticket:393#comment:20 the APC purge rate is now running at over 50%, see https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/php_apc_purge.html -- a review of the APC configuration is probably needed.

#408: MySQL InnoDB Changes

chris — Tue, 24 Apr 2012 08:45:32 GMT

!MySQL suggestions from Jim, update:

InnoDB to use 1 file per table, and also to ensure cache clears aren't slow, leave (or convert back) the menu_router and search_* tables to MyISAM as D6 batch updates these tables in a retardedly slow manner. (confirmed though personal experience, see http://groups.drupal.org/node/18177 and others for more).

ticket:406#comment:4

#409: HTTPS Security Issues

chris — Mon, 30 Apr 2012 09:49:55 GMT

There are some issues that are highlighted here:

Overall Rating: F Zero

Chain issues Incorrect order

This server is vulnerable to the BEAST attack (more info https://community.qualys.com/blogs/securitylabs/2011/10/17/mitigating-the-beast-attack-on-tls )

https://www.ssllabs.com/ssltest/analyze.html?d=transitionnetwork.org

That should be fixed on both servers and the documentation should be updated to match:

#417: Images issue on site

laura — Thu, 20 Sep 2012 14:13:00 GMT

Not sure if a Chris or Jim one. At times over the past few months, images are not always displaying on the site. Seems to fix itself quite quickly usually. But has been more noticeable in occurance over past few weeks. Today though this is an ongoing issue and received loads of emails from staff, vols and users. The images are there in the html, just not showing to the end user. (popups working fine when clicked on). I've just done a clear cache in settings/performance which seems to have helped.

Presume some sort of caching issue going on?

(also site quite sluggish for editing today too)

#420: Varnish Downtime

chris — Tue, 09 Oct 2012 11:49:56 GMT

Varnish fell over early this morning, this is the entry in /var/log/messages:

Oct  9 02:55:30 quince varnishd[8635]: Child (2879) died signal=3
Oct  9 02:55:54 quince varnishd[8635]: child (11098) Started
Oct  9 02:56:24 quince varnishd[8635]: Child (11098) said Child starts
Oct  9 02:56:27 quince varnishd[8635]: Child (11098) said SMF.s0 mmap'ed 536870912 bytes of 536870912
Oct  9 02:56:27 quince varnishd[8635]: Child (11098) said Child dies

#421: Subdomains: list from user: normal?

ed — Thu, 18 Oct 2012 08:04:22 GMT

This from a user: could you read/think/check/let us know what you think Chris?

Recently, Google announced a change in the page rank mechanism: http://googlewebmastercentral.blogspot.de/2012/10/a-new-tool-to-disavow-...

I've noticed that I get hundreds of links to our site because transitionnetwork.org can be prefixed by litteraly anything, thus duplicating its contents and unique URLs. This kind of deduplication is not good since google will consider the links to be of a poorer quality. As an example, I get a list of links with the following originating URLs:

222.transitionnetwork.org 2ftotnes.transitionnetwork.org 2fwww.transitionnetwork.org 2www.transitionnetwork.org dir.transitionnetwork.org edmitchell.transitionnetwork.org httpstatic.transitionnetwork.org jims-test-presence.transitionnetwork.org maail.transitionnetwork.org mean.www.transitionnetwork.org news.transitionnetwork.org newstats.transitionnetwork.org smtp.transitionnetwork.org stories.transitionnetwork.org w3ww.transitionnetwork.org www.transitionnetwork.org wwww.transitionnetwork.org I understand you want to catchall typos like wwww instead of www, but I think the better way would be to redirect all traffic to www instead of actually allow to browse the site with wwww.

#463: Ttech Skype Meeting 22nd November 2012

chris — Thu, 22 Nov 2012 15:20:06 GMT

Skype meeting at 8pm tonight, see ticket:456 for the last meeting.

#466: Puffin install and configuration

chris — Wed, 28 Nov 2012 13:42:56 GMT

This ticket is to track the install and configuration of the new wiki:PuffinServer which is to replace wiki:NewLiveServer and wiki:DevelopmentServer and is due to go live in early 2013.

#468: Load problems on kiwi and quince

chris — Mon, 03 Dec 2012 13:40:53 GMT

The load on the live server today was such that it ended up in off-line mode.

It's now right down again, after these things were done:

Munin has been stopped on kiwi, there are alternative stats here:
- https://ecodissident.net/munin/webarch.net/kiwi.webarch.net/
- https://ecodissident.net/munin/webarch.net/quince.webarch.net/
"DTS Agent" (which appears to be a spam bot) was requesting a huge number of pages so it's been blocked at a varnish level following the suggestion here http://omninoggin.com/web-development/block-unwanted-spam-bots-using-varnish-vcl/
We have increased the amount of CPU slices quince get's to address the steal issue show here https://ecodissident.net/munin/webarch.net/quince.webarch.net/cpu.html

#470: Penguin install and configuration

chris — Sat, 15 Dec 2012 16:05:16 GMT

penguin.webarch.net is a new 2GB RAM virtual server which will replace NewLiveServer and DevelopmentServer for running all non-Drupal sites and is due to go live in early 2013. Drupal sites from the old servers will be migrated to PuffinServer.

This ticket has been created for tracking time and tasks done during the install.

See https://tech.transitionnetwork.org/trac/wiki/PenguinServer and also the corresponding PuffinServer.

#471: Ttech Skype Meeting 17th December 2012

chris — Mon, 17 Dec 2012 20:04:44 GMT

Skype meeting tonight, now, the last meeting was ticket:463

#472: Quince to Puffin rsync script

chris — Tue, 08 Jan 2013 13:04:54 GMT

Request from Jim:

"can you set up an rsync script on the new box that can:

Grab all the files from sites/default/files on the old site recursively
Exclude the directories 'ctools', 'js', 'css' and 'tmp' folders
Put them in the /data/disk/[o1 OR tn depending on setup]/static/sites/transitionnetwork.org/files (make the folder)
Be able to be run any time to bring this up to date, including deleting files that are gone and replacing all changed (except those in the ignored folders)
Set the permissions and owners afterwards (I'll get the correct values for this afterwards, just leave a space in the script at the end)

If you can put this in /data/disk/[o1 OR tn depending on setup]/scripts/pull-files-from-quince.sh or similar then that would be awesome. This will allow us to get all the files from the other site whenever and will be useful many times, including when we finally do the go-live."

#475: Generate a new SSL certificate

chris — Tue, 22 Jan 2013 16:11:08 GMT

We need a new certificate for *.transitionnetwork.org and we need to document how to get one and how to set it up and we might as well do this on the on the wiki:SecurityInfo page.

See the notes for the existing live server wiki:NewLiveServer#HTTPS

#478: Import TN.org site from Quince to Puffin

jim — Sat, 26 Jan 2013 19:47:02 GMT

This ticket is to log/document steps in the process of importing TN.org onto the new Puffin server.

Current imported site at: http://newlive.puffin.webarch.net/

Import Process

This needs Part A of Transfer live TN.org site to Puffin before continuing.

Run Quince to Puffin rsync script to get Drupal files.
Make 'Transition Network XXX' platform from the latest makefile per BOA Server page.
Create an empty site on platform called 'www.transitionnetwork.org' with aliases for dev.newlive.puffin.webarch.net, newlive.puffin.webarch.net, and dev.www.transitionnetwork.org (dev.newlive... ones will go after launch).

SSH to Puffin and replace the default files folder with symlink to the one imported:

totn
cd static/transition-network-d6-XXX/sites/transitionnetwork.org
rm -R files
ln -s ~/static/sites/transitionnetwork.org-PROD/files/

Use one-time login link sent to me but replacing transitionnetwork.org with dev.newlive.puffin.webarch.net in url.
Enable Backup and Migrate module, use to import DB backup from Quince TN.org present in the Backup & Migrate manual folder.
```
drush @www.transitionnetwork.org en backup_migrate
```
(white screen of death, expected)

Run following as 'tn' to remove unneeded modules, then clear registry and caches, then update database for module/core updates:

drush @www.transitionnetwork.org rr
drush @www.transitionnetwork.org updb
drush @www.transitionnetwork.org en reroute_email environment_indicator robotstxt

Check site is ok.
Double-migrate per http://omega8.cc/import-your-sites-to-aegir-in-8-easy-steps-109#hint-8 (migrate site to 'rename.transitionnetwork.org', then back to 'www.transitionnetwork.org')
add puffin_server_override_settings_set_environment('Production'); to the local.settings.php file to set environment to test mode (no emails).
Set logo and favicon admin/build/themes/settings/transition2 (replace 'default' with 'www.transitionnetwork.org')
Change 443 Session settings to force HTTPS for logged in users.
Run developer tests on site, email all to begin testing before decision on DNS switch.

#481: Puffin tweaks

jim — Sun, 27 Jan 2013 18:26:45 GMT

Further to #466 there are some things to tweak on Puffin. Thought I'd put the little ones in this ticket...

Bash autocompletes don't really work with tab for apt-get, service, some filename options etc... miss that a lot!
Users don't seem to be auto-logged out once in via ssh. I've had my pc standby and the connection drop, but the logins I had don't seem to have been killed. According to http://cgp.master.puffin.webarch.net/host.php?h=puffin.webarch.net there are 3 users when I have only 1 session active. Are these yours or from old screens?

#483: Nginx 502 Bad Gateway Errors with BOA

chris — Mon, 28 Jan 2013 11:23:45 GMT

The Barracuda Octopus Aegir server on puffin.webarch.net is getting generating a lot of Nginx 502 Bad Gateway errors, see ticket:466#comment:31 and ticket:466#comment:38

The Nginx logs contain 58 502 errors:

cd /var/log/nginx
grep " 502 " * | wc -l
58

I expect the cause of these is Nginx asking PHP-FPM for a page and not getting one, an answer might be to either speed up PHP-FPM response time or adjust the Nginx settings so that it waits longs for a response from PHP-FPM.

There are several threads about this on http://groups.drupal.org/ but none of them have any useful suggestions (eg 1 2 3 4).

#487: robots.txt files for development sites

chris — Tue, 29 Jan 2013 13:24:35 GMT

All the sites other than www.transitionnetwork.org on wiki:PuffinServer should have a robots.txt file to exclude them from being crawled and indexed to prevent the development versions of sites being included in search results.

#489: Problems with SSL?

ed — Thu, 07 Feb 2013 07:32:35 GMT

User having problems with tn.org login: "I enter my username and password, get a message: The identity of this website has not been verified, server's certificate has expired"

do we need to attend to the SSL certificate in anyway?

#499: MySQL backup dump error on puffin

chris — Tue, 26 Feb 2013 01:26:53 GMT

Warnings from /etc/backup.d/20.mysql :

Warning: mysqldump: Got error: 1142: "SELECT,LOCK TABL command denied to user 'root'@'localhost' for table 'cond_instances'" when using LOCK TABLES
Warning: Failed to dump mysql databases performance_schema

#500: Quince shutdown

chris — Tue, 26 Feb 2013 20:07:46 GMT

Can people check that there is nothing they need on quince -- I think it's safe to shut it down at the end of the month?

There will still be the backups, I suggest we keep them for a few month just in case.

#503: Widget owners cannot see project moderation tab

ed — Fri, 01 Mar 2013 15:41:08 GMT

Test user1 cannot see project moderation tab. Admin can. Must be a rights issue. Can't test!

#522: Uninstall 'collectd' as redundant in face of Munin setup

chris — Sat, 16 Mar 2013 09:50:01 GMT

Collectd is no longer needed as Munin is set up and being used by wider infrastructure at Web Architects.

Uninstall Collectd, remove from any documentation.

-- was ---

Every evening at 17:36 these errors are written to the /var/log/daemon.log on wiki:PuffinServer

Mar 15 17:36:41 puffin mysqld: 130315 17:36:41 [Warning] Access denied for user 'collectd'@'localhost' (using password: YES)
Mar 15 17:36:41 puffin collectd[57763]: mysql plugin: Failed to connect to database <none> at server localhost: Access denied for user 'collectd'@'localhost' (using password: YES)
Mar 15 17:36:41 puffin collectd[57763]: read-function of plugin `mysql' failed. Will suspend it for 86400 seconds.
Mar 15 18:18:10 puffin mysqld: cron, s.conditions, s.lang

#529: New Barracuda BOA-2.0.7 Edition available

chris — Fri, 05 Apr 2013 08:57:09 GMT

There is new BOA-2.0.7 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

#530: New Barracuda BOA-2.0.8 Edition available

chris — Mon, 08 Apr 2013 07:44:51 GMT

If I had known this was about to come out I would have waited before doing the BOA-2.0.7 upgrade last night on ticket:529

There is new BOA-2.0.8 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

Is mostly to fix a problem for people using Percona, https://drupal.org/node/1962690 and as we are using MariaDB this isn't an issue for us.

Other updates in this version:

Allow to use [a-z0-9] subdomains and not only [www] for IDN domain names.
Change the interval between platforms builds from 5 to 3 seconds.
Forced 1s Speed Booster TTL for vhosts behind local proxy is deprecated.
Move old firewall logs to backups to avoid crazy load after upgrade.
Nginx: Better exceptions handling in the Abuse Guard for js/shs modules.
PHP: CLI is at 5.3 since BOA-2.0.4, so symlink old 5.2 binary path to 5.3
Update _LENNY_TO_SQUEEZE major upgrade procedure.
Update contrib with login_security-7.x-1.2
Use static downloads for all distros in stable edition.

I'll do this update tonight unless there are any objections, hopefully it should be quite quick.

#531: Disk usage on puffin

chris — Wed, 10 Apr 2013 13:12:24 GMT

The disk usage on puffing is currently at 85% and it's been going up at around 5% a week, see:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/df.html

This will become a critical issue in a couple of weeks, it would be good to find and address the cause before then.

#535: Upgrade Puffin, Penguin and Parrot from Debian Squeeze to Wheezy

chris — Fri, 26 Apr 2013 18:49:16 GMT

Debian 7 is due out on 5th May 2013 and wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer will need upgrading.

#543: Puffin Load Spike

chris — Fri, 03 May 2013 20:48:16 GMT

Puffin just had a massive load spike up to 65 and then it calmed down and it's up in the 70's again:

uptime
 21:46:43 up 91 days,  3:01,  4 users,  load average: 73.45, 30.76, 28.09

I'm trying to find the cause, it's making the TN site unresponsive.

#544: CSF / LDF false positive blocks on Puffin

chris — Sat, 04 May 2013 11:02:53 GMT

Ticket to keep track of CSF /LDF issues on Puffin, see wiki:PuffinServer#CSFLDF

#545: Registration page: 502

ed — Wed, 08 May 2013 08:38:13 GMT

http://www.transitionnetwork.org/user/register

found a bunch of these yesterday - this one is still showing a 502...

#547: New Barracuda BOA-2.0.9 Edition available

chris — Fri, 10 May 2013 11:28:17 GMT

Do we want to upgrade to Wheezy, see ticket:535 at the same time as upgrading to BOA-2.0.9? Or should we wait a few weeks, as Jim suggested here ticket:535#comment:3

I think we should schedule this update (to BOA-2.0.9) for an evening when there isn't much traffic on the site, perhaps Saturday or Sunday night. I don't think this update is super urgent as we have already upgraded Nginx, see ticket:218#comment:80

There is new BOA-2.0.9 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

The Changelog contains:

# This is the first Barracuda-only Edition, released to address important
  security issue with Nginx server and provide system level upgrades.
  This Edition will not upgrade Aegir Master nor Aegir Satellite Instances,
  because there was no new Drupal core released since BOA-2.0.8 Edition and
  there were not enough updates to built-in platforms or contrib accumulated.
  Releasing Barracuda-only Edition separately from full Edition allows us
  to address system/services security issues without any extra delay,
  while releasing Octopus-only Edition will allow us to provide Drupal core
  or Aegir version upgrades, without affecting system level services.
  There is also another reason while separate releases will be useful.
  BOA-2.0.9 is the last Edition where Aegir 2.x uses still old Drush 4.6
  in the backend. We need to sync BOA specific Aegir 2.x with upstream
  and finally switch to Drush 5, or even Drush 6, if possible.
  This change, however, may cause issues if you still host legacy Drupal 5
  or some old Drupal 6 sites, with either core or contrib not compatible
  with PHP 5.3, which is now used by default.
  That is why we plan to introduce ability to install older/previous
  Barracuda and/or Octopus release, if you need more time to upgrade.
# New features and enhancements in this release:
  * Debian 7.0 Wheezy support.
  * Automated upgrade from Squeeze with _SQUEEZE_TO_WHEEZY=YES option.
  * Added config template with inline how-to in docs/cnf/barracuda.cnf
  * Added config template with inline how-to in docs/cnf/octopus.cnf
  * Added passwords encryption how-to in docs/BLOWFISH.txt
  * Added the list of symbols used on install in docs/PLATFORMS.txt
  * Forced mysql restart if there are too many high CPU mysqld processes.
  * Improved docs/NOTES.txt
  * Improved docs/README.txt
  * Install libpam-unix2 and libxcrypt1 by default.
  * Install s3cmd by default.
  * Issue #1974640 - Allow to use Midnight Commander for limited shell users.
  * Limited Shell Logs Monitor enabled by default.
  * Nginx: Check for Linux/Cdorked.A malware and delete if discovered.
  * Re-generate and sync Aegir passwords before and after instance upgrade.
  * The silent 'system' mode documented in docs/UPGRADE.txt
  * Allow to exclude platform from otherwise forced `drush en entitycache -y`
    if sites/all/modules/entitycache_dont_enable.info control file is present.
# Changes in this release:
  * Nginx 1.5.0 - security upgrade for CVE-2013-2028
  * PHP 5.3.25
  * Redis 2.6.13
  * Do not disable update module in platforms known to include it as required.
  * Firewall: Open port 1129 for outgoing connections (some gateways need it).
  * Force syslog module as disabled by default and save some disk I/O.
  * Tune kernel to always use max RAM and not swap, if possible.
# Fixes in this release:
  * Add outgoing port 25 SMTP to the list of requirements.
  * Firewall: Add truly permanent block for heavy abusers.
  * Fix for mytop support, available again on systems with MariaDB.
  * Fix permissions in the /data/all tree if required.
  * Fix the order of checks - they scan only the last (current) minute.
  * Force _STRONG_PASSWORDS=NO if locales still look broken on second check.
  * Improve detecting no longer running drush.php and/or cron PHP processes.
  * Improve fix_locales logic.
  * Improve global.inc symlinking on initial install and upgrade.
  * Improve messages displayed when fix_locales discovers broken locales.
  * Improve monitoring to avoid duplicate entries on low traffic systems.
  * Improve sanitize_string() filtering to avoid issues with strong passwords.
  * Improve syncpass tool - Update system user passwd and flush privileges.
  * Issue #1961226 - Warning: Could not change permissions of sites/all to 751.
  * Issue #1962458 - 403 for anonymous users on node/add.
  * Issue #1963044 - Force UTF-8 locales if not present/configured properly.
  * Issue #1974542 - Use /root/.home.no.wildcard.chmod.cnf control file.
  * Issue #1987936 - Restore ability to install PHP 5.2 for FPM and CLI.
  * Make sure that /dev/null is writable for everyone.
  * Make sure that all drushrc.php files are owned by Aegir system user.
  * Make sure that all expected sites/all/{modules,themes,libraries} dirs exist.
  * Make sure that DB server is restarted on upgrade after config tuning.
  * Make sure that pdnsd and resolvconf are properly installed.
  * Nginx: Remove duplicate Vary: Accept-Encoding headers.
  * Percona no longer supports older Ubuntu non-LTS releases.
  * PHP: Do not reload FPM every hour - it may cause error 502.
  * PHP: Fix paths depending on CLI version used.
  * PHP: Fix the extensions installation and upgrade logic.
  * PHP: Make sure that the FPM port is set correctly for D6 sites with 5.2
  * PHP: Properly uninstall all related packages when using source build.
  * PHP: Start more FPM workers on systems with enough RAM by default.
  * Purge bin logs before disabling them.
  * Run NewRelic re-install early enough to avoid locking full-upgrade.
  * Sync the load limits for spiders and backend tasks.
  * The Java/Jetty monitor should use higher allowed limits by default.
  * Update apticron message to recommend system mode instead of full upgrade.
  * Update docs for _BUILD_FROM_SRC option.
  * Use aggressive enough Jetty restart procedure on nightly services reload.
  * Use correct status messages on install and upgrade.
  * Use installer and not Aegir version download on stable install/upgrade.

#549: Support with publishing process

ed — Wed, 15 May 2013 16:09:13 GMT

Mark supporting Ben with publishing process

#552: Puffin Downtime 23rd May 2013

chris — Thu, 23 May 2013 20:54:15 GMT

There was a massive load spike on wiki:PuffinServer this afternoon, the load average peaked at over 60 and at exactly the same time there was also one on the virtual server serving http://webarch.net/.

Pingdom reported:

www.transitionnetwork.org down since 23/05/2013 12:11:57
www.transitionnetwork.org is UP again at 23/05/2013 12:19:57, after 8m of downtime
www.transitionnetwork.org is down since 23/05/2013 12:26:57
www.transitionnetwork.org is UP again at 23/05/2013 12:58:57, after 32m of downtime

When I did managed to get back into the server php-fpm and nginx were not running and it took several attempt to get them to start up.

Attached is the munin graps of the load from today, munin didn't record the start of the spike but it did start recording it as it came down.

This ticket has been created to report the issue and also to record anything that might be found in the logs to explain what happened.

#554: Site slow down and MySQL load increase

chris — Tue, 28 May 2013 18:11:51 GMT

Since the upgrade to MariaDB 5.5.31, done on ticket:218#comment:93 (and fixed on ticket:548#comment:32) there appears to have been been a noticeable slowdown in the time for pages to be generated, measuring with http://tools.pingdom.com/fpt/ the front page alone takes around 5 seconds to generate.

There is a clear increase in the amount of MySQL/MariaDB activity measured by Munin, see the attached graphs, the upgrade was done around midday on 24th May 2013. There has also been a increase in traffic according to the firewall graphs. The memory usage of redis has also dropped right down and the database memory usage has significantly increased.

It's not totally clear if the cause of this change in behaviour of the site is related to the MySQL/MariaDB upgrade or if there was a coincidental change in the traffic to the site at the same time. There is no noticeable change in the visitors recorded in the Piwik stats.

According to pingdom the front page of the site is now "slower than 77% of all tested websites" with a total load time of around 6 seconds, almost all of this is down to the wait of around 5 seconds for the index.php file. This can also be tested from parrot with Apache bench, sometimes cached pages are served up and these appear in an instance, if the front page is generated it takes around 5 seconds, see wiki:LoadTimes#a2013-05-28

#555: Load spikes causing the TN site to be stopped for 15 min at a time

chris — Wed, 29 May 2013 09:53:52 GMT

The BOA /var/xdrago/second.sh script is run every minute via the root crontab and if it detects a certain load level it changes the nginx config to a "high load" config which results in bots being served 503 errors when they spider the site, see ticket:563. When the load goes higher and hits another threshold the second.sh script kills the webserver applications, nginx and php-fpm, and waits till the load has dropped before starting them up again. This was happening once or twice a day following the increase in traffic around the launch of The Power of Just Doing Stuff. This has been addressed by multiplying the thresholds by 5 in second.sh.

Original Description

This morning at 10:19:24 I received the following alert from puffin:

Subject: lfd on puffin.webarch.net: High 5 minute load average alert - 6.59
Time:                    Wed May 29 10:17:02 2013 +0100
1 Min Load Avg:          23.39
5 Min Load Avg:          6.59
15 Min Load Avg:         2.57
Running/Total Processes: 44/326

At 10:21:57 I got an alert regarding ssh:

Service: SSH
Host: puffin
Address: puffin.webarch.net
State: CRITICAL
Date/Time: Wed May 29 10:21:57 BST 2013
Additional Info:
CRITICAL - Socket timeout after 10 seconds

Then at 10:26:47 ssh appeared to have recovered:

Service: SSH
Host: puffin
Address: puffin.webarch.net
State: OK
Date/Time: Wed May 29 10:26:47 BST 2013
Additional Info:
SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0)

But then pingdom reported at 10:29:07:

www.transitionnetwork.org is down since 29/05/2013  10:24:57.

There was then a report regarding Nginx at 10:32:07:

Notification Type: PROBLEM
Service: HTTP
Host: puffin
Address: puffin.webarch.net
State: CRITICAL
Date/Time: Wed May 29 10:32:07 BST 2013
Additional Info:
Connection refused

So at 10:33:47 I ssh'd in and found that php53-fpm and nginx were not running and it took several attempts to get them running again.

The up email from pingdom reported:

www.transitionnetwork.org is UP again at 29/05/2013  10:36:57, after 12m of downtime.

I can't find anything in the logs to indicate what caused the load spike and php-fpm and nginx to stopp running.

#563: 503 Errors

chris — Wed, 19 Jun 2013 10:59:54 GMT

Original Description

The site is generating a lot of 503 errors, 83 since 6:30am today and there were around 750 yesterday.

#567: Update BOA for new Redis 2.6.14

chris — Tue, 02 Jul 2013 11:10:27 GMT

A suggestion from Jim:

BOA now includes Redis 2.6.14 <https://raw.github.com/antirez/redis/2.6/00-RELEASENOTES> if you do a 'barracda up-stable system'... What interests me about this is these lines from the changelog:

UPGRADE URGENCY: HIGH because of the following two issues:

Lua scripting + Replication + AOF in slaves problem (see Issue #1164).
AOF + expires possible race condition (see Issue #1079).

It's a long shot, but that could maybe be part of the issue we've seen recently.

I'm not sure if this is best done now or later tonight when the site is less busy?

#569: 403s served to editors, admin very slow

ed — Tue, 09 Jul 2013 07:52:32 GMT

Rob is getting 403s when trying to submit his work. Report from 07:12am this morning (Tuesday)

Ed tried to add a blog post at node add:

https://www.transitionnetwork.org/node/add/blog It took nearly 15 seconds to get this published https://www.transitionnetwork.org/blogs/ed-mitchell/2013-07/eds-test-blog-item-check-403

Running admin functions takes ages.This request took well over 30 seconds:

https://www.transitionnetwork.org/admin/content/node/overview

Please advise?

#573: MariaDB 5.5.32 is available for Puffin

chris — Thu, 18 Jul 2013 19:11:24 GMT

I could either update this via aptitude and it would be quick and have little down time or I could update it via BOA and it'll take 3 or 4 times as long and involved extra downtime.

Jim -- which way would you like it done?

The MariaDB project is pleased to announce the immediate availability of MariaDB 5.5.32.

This is a bug-fix release. See the Release Notes and Changelog for details.

MariaDB 5.5.32 Stable (GA)

Release Notes: https://kb.askmonty.org/en/mariadb-5532-release-notes
Changelog: https://kb.askmonty.org/en/mariadb-5532-changelog
http://lists.askmonty.org/pipermail/announce/2013-July/000048.html

#574: EFF: How HTTPS Everywhere affects transitionnetwork.org

chris — Thu, 25 Jul 2013 08:09:31 GMT

The following email was sent via the whois contact details for the transitionnetwork.org domain name, see also ticket:571 -- the changes made on ticket:571 has broken the site for anon Firefox users with the EFF HTTPS Everywhere browser extension installed.

Hi,

You're receiving this note because transitionnetwork.org is part of our HTTPS Everywhere browser extension, and an upcoming change to the way Firefox handles HTTPS pages may cause your site to display or function incorrectly. We want to make sure that the nearly 3 million HTTPS Everywhere users have the best possible experience while browsing, so we're asking you to please take a minute and test how your site behaves in Firefox 23. You can find out more about our software at

https://www.eff.org/https-everywhere

To see the rules affecting your site, you can visit the HTTPS Everywhere Atlas at

https://www.eff.org/https-everywhere/atlas/domains/transitionnetwork.org.html

The Atlas shows both rules in the development and stable versions of our extension. Rules in the stable version are used by millions of users, while development rules are used by tens of thousands of users. Development rules are now being tested but will be migrated to the stable version in the future.

An upcoming change (described below) in how the Firefox browser renders HTTPS content makes it especially important that you check that your site is prepared for HTTPS access. We urge you review the rules affecting your site and also to test it using HTTPS Everywhere with the upcoming version of Firefox.

*NEW FIREFOX CONTENT SECURITY POLICY*: In the upcoming Firefox 23 browser release, due out the week of August 6, Firefox will stop loading certain "active" content such as scripts and stylesheets from insecure http:// URLs if they've been included from a secure https:// site. If the HTTPS Everywhere rules send users to the secure version of your site but the secure version includes some content loaded over an insecure connetion, the rendering of your site may become broken for Firefox users with HTTPS Everywhere installed after they upgrade to Firefox 23. You can check this by downloading a preview release of Firefox 23, installing HTTPS Everywhere, and visiting your site. We urge all web site operators to protect their users by making sure that all site content is always loaded over a secure connection. A preview version of Firefox 23 is available now at https://www.mozilla.org/en-US/firefox/beta/ and the HTTPS Everywhere extension is at https://www.eff.org/https-everywhere

HTTPS Everywhere rules instruct browsers to access certain specified resources securely -- over HTTPS -- even if the user typed or followed a non-HTTPS link or even if the resources were included in a page via a non-HTTPS URL. For example, it might automatically rewrite

http://www.transitionnetwork.org/

to

https://www.transitionnetwork.org/

or make some similar change.

The goal of this rewriting is to protect as much as possible of every web site against sniffing and tampering by ensuring that as many site resources as possible are loaded over a secure HTTPS connection.

When web sites are accessed insecurely, users are vulnerable to attacks by other users on their networks. HTTPS Everywhere aims to activate sites' existing HTTPS protection more consistently to make sure users are as well-protected from these attacks as possible -- including attacks like sidejacking and SSL stripping.

http://www.firesheep.org/ http://www.thoughtcrime.org/sslstrip

As a result, we think there's an emerging consensus to make all web sites secure, not just financial sites and login pages. Providing a secure connection helps protect users' login credentials, but also helps protect their privacy and security even when accessing public resources, for example by preventing network operators from injecting malware downloads.

The goal of HTTPS Everywhere is to make the web more secure and help users express their preference to use the secure version of every site automatically, even on sites where this is not the default. We don't want to break sites or harm users' experience. So, we encourage webmasters to test the effect of HTTPS Everywhere on their sites and fix any problems that result -- ideally, by making sure that all resources that make up a site are available over HTTPS, using a current, valid certificate. Although we only include rules that we've been told and believe work properly, we can't always anticipate whether a rule adversely affects a site, especially if the site's URL structure, use of CDNs, or level of HTTPS support changes over time.

We are always happy to receive bug reports, updates, and fixes to HTTPS Everywhere rules. We will also make rules inactive by default if a site operator asks us to. Although we are working for a web where all sites are secure, we are not trying to use this software to force sites to use HTTPS against their operators' wishes. You can send any corrections, updates, or requests to https-everywhere-rules@… (which is a public and publicly-archived mailing list), or by replying to this e-mail address.

Thanks for your attention!

Seth Schoen, Senior Staff Technologist, Electronic Frontier Foundation for the HTTPS Everywhere development team

#576: Site down

ed — Thu, 01 Aug 2013 09:42:44 GMT

Been down for 15 minutes by now I reckon

#580: php5-fpm starting when puffin boots

chris — Mon, 12 Aug 2013 16:03:06 GMT

When wiki:PuffinServer is booted /etc/init.d/php5-fpm is started and this causes errors with the site.

#585: TTech Meeting 5th September 2013

chris — Thu, 05 Sep 2013 07:47:16 GMT

This ticket has been created for the TTech Skype meeting due to take place today.

#586: New Relic Monitoring for BOA

chris — Thu, 05 Sep 2013 08:35:17 GMT

This is a ticket to document the consideration, and possible installation, of https://newrelic.com/ on the PuffinServer

#588: RSS feed caching

chris — Fri, 06 Sep 2013 09:43:57 GMT

At the meeting on 5th September ticket:585 one thing we discussed was that:

The biggest single source of bandwidth by URL is Rob's RSS feed, 34.5GB of data, 150k hits. This represents half the total site data transfer, the RSS files is 0.4MB but will hopefully be mostly served gzipped at 0.1MB.

This ticket is to follow up on that -- are the RSS feeds served directly by Nginx?

#589: Blocking spammers at a firewall level

chris — Fri, 06 Sep 2013 09:48:59 GMT

At the meeting on 5th September ticket:585 one thing we discussed was that for August 2013:

More data is transferred for /user/register than the front page, 5.1GB compared to 3.6GB.

Most of this will be spam bots trying to register to post spam. Jim suggested that we could look at blocking some of these spam bots at a firewall level to save on resources. This ticket is to follow up on this suggestion.

#591: Move MySQL temporary directory to tmpfs

jim — Mon, 09 Sep 2013 12:47:41 GMT

Chris, please read: http://2bits.com/articles/reduce-your-servers-resource-usage-moving-mysql-temporary-directory-ram-disk.html

I think we could easily drop a little MySQL memory to give it some in-memory disk space to do the temporary table munching Drupal is causing it. I see there are already some mounted tmpfs partitions.

Related to #590 (proposal L: Review slow query log, explain queries, tweak as necessary/flag poorly behaving modules)

What do you think? Worth doing?

#593: Migrating Puffin to a ZFS file server

chris — Wed, 11 Sep 2013 15:52:47 GMT

This ticket is for the migration of PuffinServer to a ZFS file server, this will involve some downtime but should result in better IO and easier backups.

#599: Server time drift

chris — Sun, 29 Sep 2013 21:57:59 GMT

The servers are not keeping good time at the moment.

#604: Times for admin tasks

ed — Thu, 03 Oct 2013 13:10:00 GMT

I am timing some of the admin tasks and keeping a log here for reference. Front of the site is great, backend is still like hauling ghosts through jelly for me.

58 seconds: https://www.transitionnetwork.org/admin/user/user/create

#610: Aegir database intensive (migrate, clone, restore) tasks hang for larger sites

jim — Tue, 15 Oct 2013 10:18:59 GMT

Large sites (TN.org and variants) will simply not complete their migrate, clone or restore tasks in Aegir.

However, smaller sites are fine, and all tasks work for them.

The process largely completes -- codebase installs, database is cloned, symlinks for sites aliases and files created... BUT the process never completes in Aegir, so the final steps of switching a site's served location never occurs.

Useful links/comments in this issue:

#612: Upgrade to BOA-2.1.1 Stable Edition

chris — Sun, 03 Nov 2013 11:11:24 GMT

I have been sent this email from wiki:PuffinServer:

There is new BOA-2.1.0 Stable Edition available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

I could potentially do this upgrade tonight as it's a quite time for the server -- Jim, any reason not to apply this update tonight?

#629: Upgrade to BOA-2.1.3 Stable Edition

chris — Sun, 24 Nov 2013 11:47:23 GMT

This ticket is for BOA-2.1.3, the previous version was BOA-2.1.1 ticket:612, see wiki:PuffinServer#Upgradetickets The steps to follow when upgrading wiki:PuffinServer are documented at wiki:PuffinServer#UpgradingBOA

#670: Roll back performance customisations and use stock BOA settings where possible

jim — Sat, 11 Jan 2014 20:51:30 GMT

Issue Given so much has changed since the initial issues on the server, I now strongly recommend reverting all settings changes that do not add features back to stock BOA settings after the next BOA release.

These would include all MySQL, PHP, FPM, Redis and other settings that have been for performance reasons, or to combat the situation where there was hardware/IO issues with the underlying server. I'm most interested in FPM and MySQL settings.

The next version of BOA will include some improvements we need (see 629: Upgrades to BOA which should handle load on our server with a lot of CPU cores. With this in place we'll be able to revert more easily to stock settings.

Rationale I'm not talking about rolling back changes that provide us with features or mission-critical capabilities, just the changes to the subsystems I list above for performance reasons.

It's my belief that these enhancements no longer match the needs of the server since the changes to filesystem and underlying hardware fixes have been completed. They also represent an ongoing risk around updates, future planning -- plus it's possible they might mean Puffin's web services need more memory than it otherwise would, costing TN more than it should need to spend on hardware.

Proposed solution

Await the next version of BOA, and setup the enhanced load settings per the documentation.
Revert all other changes to conf files for MySQL, PHP, FPM, Redis that do not add a feature or are not mission-critical.
Review /root/.barracuda.cnf and turn off any overrides and customisations we don't now need as a result of 2).
Run the BOA BOND.sh script to do the tuning of the server appropriate to the memory requirements. This will tune for the current levels on first pass.
Review Munin and site performance. If we need to make any tweaks then we can do a minimal set as required -- keeping an eye on memory usage.
Once a few days have gone by I would hope that the overall memory use will be lower, OR with more cached data. At this point we can either re-run the barracuda installer with the _RESERVED_RAM set to 1-4 Gb, or simply reduce the memory available to Puffin.
Repeat from 4, using BOND.sh to optimise for the new memory footprint.

Clearly, it's possible no memory savings can be made, or just 1Gb or so is sensible. Either way, rolling back the changes made for a system that has changed immensely is worth attempting to compare current (tweaked) performance to the stock system. Since current settings are all documented and can be backed up, we should be able to test this with no risk and the ability to roll back as needed.

Next steps

Chris and Ed to give their thoughts.
Ed to green-light before we proceed in too much detail or take any action.
Chris and Jim to establish the changes and outcomes.
Chris, Jim and whoever to do the new optimisation process.

#673: Install mosh - the mobile shell

chris — Mon, 13 Jan 2014 11:27:40 GMT

The mobile shell enables terminal connections to stay up when using really bad connections, for example on a train, see:

http://mosh.mit.edu/