TN.org platform and sites

[jim]

The TN.org platform and Drupal site updates are to be tracked in this ticket.

Current PROD platform build = P009
Current STG platform build = S010

Updates pending:

• SECURITY UPDATE - NO RISK: Pressflow core 6.30 is due, but the security holes fixed do not affect us, low priority. Platforms: present in S010, but not in P009.

[jim]

This is blocking ​Ticket 578: Comments subscriptions: enable user alerts if a user has made a comment.

Adding time.

[ed]

anything about a software as a service to analyse TN.org to find the load spikes?

[jim]

Replying to ed:

anything about a software as a service to analyse TN.org to find the load spikes?

New Relic integration is at the BOA/server level, and all Drupal sites hosted will automatically start reporting their stats once set up... So that's unrelated to this.

[ed]

open atrium?

[jim]

Per ticket:590#comment:28, we need to apply the patch for blockcache_alter (no grants) to Drupal core...

[jim]

Regarding Location module patch -- our old version with the changes to the French Departments est ici: ​https://github.com/i-jk/OBSOLETE-Transition-Network/blob/610d54daa7bd5bc1480157241b7f5f273c6c3ba8/sites/all/modules/location/supported/location.fr.inc

Will compare current Locaton 6.x-dev with our old version and combine the best.

[jim]

Adding (per Proposal N on #590): need to get latest admin menu 3.x branch.

And these are the changes in functions I patched in LoginToboggan? -- need to make a patch in our repo and submit '@' part back to the project:

---- [Removed LOWER() in the queries ] ----
function logintoboggan_user($op, &$edit, &$user_edit, $category = NULL) {
[[snip]]
      if (isset($edit['name']) && db_result(db_query("SELECT uid FROM {users} WHERE mail = '%s' AND uid <> %d", $edit['name'], $uid))) {
        form_set_error('name', t('This name has already been taken by another user.'));
      }
      // Check that no user is using this email address for their name.
      if (isset($edit['mail']) && db_result(db_query("SELECT uid FROM {users} WHERE name = '%s' AND uid <> %d", $edit['mail'], $uid))) {
        form_set_error('mail', t('This e-mail has already been taken by another user.'));
[[snip]]
}


---- [ added '@' check and removed LOWER() in query] ----
function logintoboggan_user_login_validate($form, &$form_state) {
if (isset($form_state['values']['name']) && $form_state['values']['name'] &&
    strpos($form_state['values']['name'], '@') > 0) {
    if ($name = db_result(db_query("SELECT name FROM {users} WHERE mail = '%s'", $form_state['values']['name']))) {
      form_set_value($form['name'], $name, $form_state);
    }
  }
}

[jim]

As part of Proposal N #590, I'll be adding ​https://drupal.org/project/advagg to the site, too.

[jim]

Include Term Node Count too...

Given the downtime today and the meeting for tomorrow I'll wait for the all-clear before doing these updates. Pushed back 1 week for now...

[jim]

I've changed to use comment_notify instead of notifications_extra - better, simpler module.

Committed: ​Advagg, comment_notify and node_term_count added, admin_menu updated to 3.x-dev.

[jim]

Checked modules & update, all good, no changes needed to makefile and all relevant contrib updates will be brought into the next platform.

TO DO here:

• GMap revert to our old version
• Location and LoginToboggan? patches created and applied

Postponing captions issue on #596 until after this is settled - that'll happen on that ticket.

[jim]

Also: add views_content_cache!

[jim]

Update:

• Views content cache added for use in #590.
• ​Committed change to fix French Departments... This uses latest dev of Location plus the patch I submitted at ​https://drupal.org/node/635958#comment-7961467

Now just the GMap module to look into...

[jim]

description update.

[jim]

Platform now ready to build (tonight):

• ​LoginToboggan patched per comment 7
• Reverting back to our ​custom Gmap 1.x version as 2.x is still not ready :(

[jim]

Have corrected a couple of minor issues/typos in makefile and am building the updated platform for testing with STG...

[jim]

Had a couple more issues that were my fault (typos) in the patch files I made, now fixed.

STG is NOT migrating though... opening a separate ticket for this as I think we need Octopus updating in line with the recent Barracuda updates.

[jim]

Blocked by ​#610 - Aegir migrate tasks hang on Drushrc load.

[jim]

Please note the ​patch I submitted before for the French Departments has been accepted and applied! Next time we can just use the stock version of the Location module.

[jim]

FYI I'm waiting on the BOA update #629 now.

[jim]

This is no longer blocked by #629, but seems still affected by #610 -- more over there on that.

Regarding updates:

• Makefile updated to use latest D6 Pressflow core.
• Minor tweaks too to clean up patches not needed.
• Hang occurred on provision verify of 'Transition Network D6 S008' platform -- seems intact though, deleted and tried again
• New platform 'Transition Network D6 S008' created manually (see command below), will clone Stg2 site into it now to test it.

The commands for making a new TN D6 platform is:

[log in as Aegir user, or sudo to root and run totn]
cd static
rm transitionnetwork.org-d6.make ; wget https://raw.github.com/transitionnetwork/transitionnetwork.org-d6.profile/master/transitionnetwork.org-d6.make ; drush make transitionnetwork.org-d6.make transition-network-d6-s008

The second line above removes any old makefile, gets the latest, runs it with Drush Make into directory in ~/static/transition-network-d6-s008 <-- edit the directory name increment number!

[jim]

Since #610 is now fixed, I'll now build a new TN PROD platform via Aegir, then migrate news and www sites to it. The platform and process was tested over on #610 with STG being cloned from PROD then migrated, so no further tests are required on the near-identical PROD sites.

For the benefit of future Aegir users, I'll quickly outline the process as I do it...

1. Add new platform: Name = Transition Network D6 P009, makefile = ​https://raw.github.com/transitionnetwork/transitionnetwork.org-d6.profile/master/transitionnetwork.org-d6.make, Drush make option = Normal -> 'Save'.
2. When built go to news.transitionnetwork.org -> Migrate -> Transition Network D6 P009 -> Migrate.
3. Test news works ok (it does).
4. Go to www.transitionnetwork.org -> Migrate -> Choose Transition Network D6 P009 -> Migrate.
5. Test www works ok (it does).

In addition to the above sites, I've also migrated the Space site, and cleaned up the various STG sites, plus the old (now empty) platforms.

Note that this work is actually around 15 mins to complete, but once I'd set up all the tasks then left Aegir to do the work, it took around 1/2 hour to complete. Hence the process for future Aegir users is:

1. Do the prep/planning/makefile tweaks
2. Build the platforms for STG and PROD, migrate STG, test
3. Set the PROD sites to migrate.
4. Set the (soon to be) empty platform to delete -- best to do STG only to start, so it's still possible to roll back PROD if needs be.
5. Make tea, eat a sandwich, have a doobie, cook your scag etc.
6. Final testing/tweaks/cleanup. Done.

[jim]

Hmm... hit an interesting 'Gotcha' moment during this update:

1. Everything worked as planned...
2. BUT the platform was newly-created, and therefore did not have the [platform root]/modules/o_contrib folder symlinked into it -- this happens as part of the /var/xdrago/daily.sh script which is run at 6.45 every day and does a bunch of things.
3. This meant that though the site was ok and ran fine for the User1 account, anonymous users got a white screen that was caused by the lack of Block Cache Alter module (provided by the o_contrib folder).
4. Rebuilding the registry fixed the issue, then I ran /var/xdrago/daily.sh when logged in as root to fix the problem.
5. I then manually re-enabled Block Cache Alter to fix the issue.

There is an issue over on Drupal.org for this, and I added my comment: ​Add o_contrib modules and Boost cache dir on new platforms when created.

The simple (but slightly annoying) solution (in lieu of a proper fix) is to include the required modules in the Makefile...

---

So left on this ticket is:
A) Add Block Cache Alter (any any other required modules in o_contrib) to the makefile.
B) Apply Patch in comment:7 to LoginToboggan.
C) Possibly make a GMap patch and add to our makefile so our custom markers appear ok -- per #615.

[jim]

Downgrading, only task A, B and C in comment:24 required at this point, all other updates done.

[ed]

where will you document teh simple but annoying solution in lieu of a proper fix ?

[jim]

Hmmm... Drupal core has just had a security update! 6.30 is now available.

But according to ​https://drupal.org/SA-CORE-2014-001, none of the risks affect us. We can therefore wait for the next sensible time to update.

I also note that

• I'd missed Elysia Cron from the makefile, causing #677, now fixed.
• I've left the old GMap (our fork of 1.x) in the makefile too. This means part (C) of comment:24 is necessary to have a proper makefile meet our GMap needs.
• And the LoginToboggan? patch in (B) is already done in comment:15.
• And I've ​added Block Cache Alter to the makefile.

Updating summary...

[jim]

Better title/description.

Also cc Sam FYI.

[jim]

FYI I've added a note to the TN.org Update Status admin page and told it to ignore 6.30 as the security risk/fix simply does not apply to us.

Also, these notes are provided by Update Status Advanced module, and should be kept up to date and mirror any key pins/ignores in the makefile.

I'll try to do the GMap patch and associated makefile tweaks later tonight so we can then have just 'normal' update tracking on this ticket.

[jim]

I've made a few commits to the makefile and patches folder that:

• ​Comments out reference to our old custom Gmap module, uses 6.x-2.x-dev branch with our patch (see below).
• Includes ​a new patch for the custom marker set we use on TN.org for some map views
• ​Renamed the patches to start "tnXX_" where XX is the number used in the makefile to help people find the right ones.
• Updated the core to the Aegir Project's recommended 6.30.1 version (per instruction at top of makefile, using this to find current best version: ​https://github.com/omega8cc/nginx-for-drupal/search?q=pressflow).

Note to make a patch that includes the binary files (marker PNGs) I had to follow the instructions here: ​http://mediatribe.net/en/node/63 -- e.g.

[after I had made the changes to a freshly cloned gmap-6.x-2.x-dev branch]
git diff --full-index --binary > tn03_gmap-tn-markers.patch

The key bits being the --full-index and --binary to tell Git to include non-text files in the patch.

I'll now try building the new platform to test.

[jim]

The S010 platform built properly, and the 'transition' folder with our markers is present here: /data/disk/tn/static/transition-network-d6-s010/sites/all/modules/contrib/gmap/markers

So the last 'non-standard' thing with our modules has been resolved and this ticket is able to just track the updates as they come in now. Reassigning to Sam for now, and updating main description so the current/best practice TN.org platform build numbers are documented.

Sam: I'll stay CCed on this ticket so can answer questions and help out as needed. Otherwise we're in great shape so it's all yours! (or delegate as you see fit!)

[jim]

There was a simple Tagadelic module security update. The latest would have been covered by the makefile anyway, but I've done a hotfix on PROD anyway as it was a drop-in replacement (no db update). So there is no need to update PRoD now...

STG and STG2 should be migrated to the S010 platform when possible, I've updated the module there too. (they're on S008 now).