Create a new stgX.transitionnetwork.org site

[sam]

Hi Paul

I have been trying to build a staging site using your Github repository with the changes you made for ticket : ​/trac/ticket/693

I have edited the D6 s008 platform: ​https://tn.puffin.webarch.net/node/1157/edit to use your makefile.

It builds a site, but I just get an empty pressflow site at the end.

Could you build a staging site using your makefile?

Thanks

Sam

[paul]

Hi Sam, 

I'll give this a go. 


-- 



Best



Paul Booker


Drupal Developer & Systems Administrator


Website: http://www.paulbooker.co.uk


Tel: +44 01922 861636





On Tuesday, 1 April 2014 at 16:03, Transiton Technology Trac wrote:

> #712: Create a new stgX.transitionnetwork.org (http://stgX.transitionnetwork.org) site
> -------------------------------------+-------------------------------------
> Reporter: sam | Owner: paul
> Type: | Status: new
> maintenance | Milestone: Maintenance
> Priority: major | Keywords:
> Component: Drupal | Add Hours to Ticket: 0
> modules & settings | Total Hours: 0
> Estimated Number of Hours: 0 |
> Billable?: 1 |
> -------------------------------------+-------------------------------------
> Hi Paul
> 
> I have been trying to build a staging site using your Github repository
> with the changes you made for ticket :
> /trac/ticket/693
> 
> I have edited the D6 s008 platform:
> https://tn.puffin.webarch.net/node/1157/edit to use your makefile.
> 
> It builds a site, but I just get an empty pressflow site at the end.
> 
> Could you build a staging site using your makefile?
> 
> Thanks
> 
> Sam
> 
> -- 
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/712>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
> 
> 

[paul]

I just tried to build a new platform using the latest version of the makefile from my [*] Git account; but it failed with some errors, that don't make sense right now as the core and api versions were specified.

Investigating ..

[*] ​https://github.com/paulbooker/transitionnetwork.org-d6.profile/blob/master/transitionnetwork.org-d6.make

[paul]

Forgot to add a time update ..

[paul]

I need to use the raw version.

​https://raw.githubusercontent.com/paulbooker/transitionnetwork.org-d6.profile/master/transitionnetwork.org-d6.make

I have now built the platform Transition Network D6 S011 from the latest profile. I'll later look at migrating the TN site to this platform.

[paul]

Hi Sam,

I have cloned the stage site stg.transitionnetwork.org on the same platform (pb-stage-20140403.transitionnetwork.org)
and migrated the site to our new platform.

The site update looks to have worked successfully:

​https://pb-stage-20140403.transitionnetwork.org/admin/reports/updates
​https://pb-stage-20140403.transitionnetwork.org/admin/reports/dblog

Let me know if you're happy for production to be updated.

I'll go back to the wiki and read up on how to do this ..

Time taken includes listening to the first 46-00+ minutes of Jim's talk on BOA and taking some notes.

[sam]

Hey Paul nice one :)

Be great to see any notes you made as you went.

I see that the content dates back to January. Any idea why it's not getting
a more up to date database?

Thanks

Sam




On 3 April 2014 13:03, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #712: Create a new stgX.transitionnetwork.org site
> -------------------------------------+-------------------------------------
>            Reporter:  sam            |                      Owner:  paul
>                Type:  maintenance    |                     Status:  new
>            Priority:  major          |                  Milestone:
>           Component:  Drupal         |  Maintenance
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  1.75           |                  Billable?:  1
>         Total Hours:  1.0            |
> -------------------------------------+-------------------------------------
> Changes (by paul):
>
>  * hours:  0.0 => 1.75
>  * totalhours:  1.0 => 2.75
>
>
> Comment:
>
>  Hi Sam,
>
>  I have cloned the stage site  stg.transitionnetwork.org on the same
>  platform  (pb-stage-20140403.transitionnetwork.org)
>  and migrated the site to our new platform.
>
>  The site update looks to have worked successfully:
>
>  https://pb-stage-20140403.transitionnetwork.org/admin/reports/updates
>  https://pb-stage-20140403.transitionnetwork.org/admin/reports/dblog
>
>  Let me know if you're happy for production to be updated.
>
>  I'll go back to the wiki and read up on how to do this ..
>
>  Time taken includes listening  to the first 46-00+ minutes of  Jim's talk
>  on BOA and taking some notes.
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/712#comment:5>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>



-- 
~~~~
Sam Rossiter: Transition Network Web and Communications operations
web: http://transitionnetwork.org
mobile: 07503386283
office hours: Tues - Fri 9am - 5pm
company no. 6135675  charity no. 1128675
Subscribe to Transition news: http://tinyurl.com/transitionregister
~~~~

[paul]

Be great to see any notes you made as you went.

Sorry Sam. My notes are not any more detailed than above:

Doing a core update …

Update makefile,
Build new platform using an updated makefile,
Clone stage site on its platform,
Migrate new stage site to the new platform

What I'll do later is create an internal blog post for each particular high level task. e.g Updating core - with plenty of screenshots followed by a quick screencast.

I see that the content dates back to January. Any idea why it's not getting
a more up to date database?

I think we need to clone the current site on the production platform, and migrate this site over to out stage platform. Investigating ..

[paul]

According to the wiki, we will next need to create a new production platform and then migrate the www.transitionnetwork.org and news.transitionnetwork.org sites to the new production platform. If there is a problem we should be able to go back to the previous platform.

Maybe we should be try this - immediately after - a server backup has been taken?

From the Wiki ..

​https://wiki.transitionnetwork.org/BOA_Server/Development_workflow

Updating sites in PROD
You need to build a PROD version of the platform using the approach in BOA_Server/Building_platforms, making sure you follow the naming convention.
If you've done your testing in STG, carry on -- if not STOP and do the 'TEST THE CHANGES' steps above!
Migrate the www.transitionnetwork.org and news.transitionnetwork.org sites to the new PROD platform. There will a few minutes of downtime, so do this out of hours.

...

Rolling back a failed migration
@TODO CHECK IF THIS PROCESS IS CORRECT

If there's a problem and only a short time (hours at most) has gone by, you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

Time taken: includes finishing Jim's talk and taking some further notes.

Tomorrow, I'll have a read over the wiki pages.

[sam]

Hi Paul

Are you up for having a go at building a clone with an up to date database?

Or do we need to hold off until the BOA/ loadspike issue is resolved?

Thanks

Sam

[ed]

sorry to butt in lads, but i'm happy for you to give this a try atm; you'll see the various emails on list; we really need to ascertain how/if this works ....

[paul]

If I have time later this afternoon; I'll give this a go. But if not today. I'll get to this tomorrow afternoon.

[ed]

Great news - looking forward to seeing how it goes

[sam]

Hi Paul

How are you getting on with this? Have you had a chance to try a clone yet?

Thanks

Sam

[paul]

Sorry Guys,

I'll have a look at this either Monday or Tuesday afternoon, when I'm back at my desk.

Best, Paul

[sam]

Hi Paul

Ed and I had a chat yesterday. I think we are both getting increasingly concerned about running the site with overdue security updates.

We talked about a timeline that would get us to a fully upgraded secure site by the end of next week; May 9th.

As I understand it it would be something like;

• Clone the existing site on it's platform - Thurs 1st/Fri 2nd
• Migrate the clone to your platform & apply the changes -Thurs 1st/Fri 2nd
• Test Mon 5th/Tues 6th
• Fix the gotcha's Tues 6th/Weds 7th
• Migrate the live site to use the new build Thurs 8th/ Fri 9th

Obviously if we can bring that timeline forward then that's all good too.

Does that seem reasonable to you? Do you have the time available to make it happen?

Thanks

Sam

[paul]

Sam,

Sounds good. I have the time to do this within the timeline you indicated. I'll work on this later this afternoon.

Best, Paul

[paul]

Hi Sam,

I have rebuilt the site again on my local machine, with the latest version of pressflow. No problems to report. I have updated the makefile on my fork. I'll now:

1) build a new stage platform on the server with the updated makefile.
2) clone the production site on the production platform
3) migrate the clone of the production site to the new stage platform

​https://raw.githubusercontent.com/paulbooker/transitionnetwork.org-d6.profile/master/transitionnetwork.org-d6.make

Problems that I came across while re-building the site on my local machine -- that I didn't document previously:

1) Need to switch your web server over to PHP 5.2 for Drupal 6

2) Look our for errors like "Unable to create CTools CSS cache directory. Check the permissions on your files directory." If you have your vhosts file configured, as per attachment, you will need to ensure that sites/www.transitionnetwork.org directory is writable by the web server.

[paul]

Building a new platform not working

[paul]

Building a new platform is not working. Is this related to a scheduled BOA update?

I'll continue tomorrow afternoon.

[ed]

Adding Chris - who is doing an update see #724 and #721

[paul]

I have completed step 1 to 3 and the [*] site look to be working ok.

​https://booker-stage-20140501.transitionnetwork.org/admin/reports/status
​https://booker-stage-20140501.transitionnetwork.org/admin/reports/updates

[*] ​https://www.booker-stage-20140501.transitionnetwork.org/ (some images are missing as they are hard coded to point at www.transitionnetwork.org, but are looking for a subdirectory that only exists on my stage site)

I'll now make a new production platform, and advise the remaining steps that need to be done. I'll NOT migrate the production site to the new production platform until we're clear what needs to be done (I'll refer again to the wiki), that it's clear that we can recover from any potential problems, and that I have clearance to update the production site.

​https://tn.puffin.webarch.net/hosting/c/booker-stage-20140501.transitionnetwork.org
​https://tn.puffin.webarch.net/hosting/c/platform_TransitionNetworkD6S012

[paul]

The new production platform is now ready:

Transition Network D6 P010 Booker
​https://tn.puffin.webarch.net/hosting/c/platform_TransitionNetworkD6P010Booker

From the wiki ..

Migrate the www.transitionnetwork.org and news.transitionnetwork.org sites to the new PROD platform. There will a few minutes of downtime, so do this out of hours.

Question for Jim:

When creating a platform we never specify explicitly that this platform is a production platform, only indirectly by our naming convention PXXX; Is this correct? To migrate the websites to the new production platform, do we simply do a migrate, and everything that needs to happen is taken care of in the background?

From the wiki ..

If there's a problem and only a short time (hours at most) has gone by, you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

Question for Jim:

Has this been tested? Can you provide more detailed instructions?

Question for Sam:

When we finally do the migration, should this be done on the weekend?

[paul]

Adjusting todays hours, my total should be 2.5 (1.25 to 1.75)

[sam]

Hi Paul

A user sent a contact email that appears to have come from the staging account:
Mary (xxxx@…) sent a message using the contact form at
​https://booker-stage-20140501.transitionnetwork.org/contact.

Do you think she has found the stage site through Google? Or is it just seeming to come from the staging site, but actually coming from production?

Thanks

Sam

[ed]

Adding Jim as cc as there could be some issues around turning email alerts on and off on staging server - I think there are notes in the documentation about this?

[chris]

Replying to ed:

Adding Jim as cc as there could be some issues around turning email alerts on and off on staging server - I think there are notes in the documentation about this?

There are, also staging sites should have a robots.txt file that stops them from being indexed, this isn't the case for this site:

• ​https://booker-stage-20140501.transitionnetwork.org/robots.txt

See ticket:487 for info on robots.txt on dev sites.

See ticket:136 for info on dev sites and email.

[sam]

I just updated the robots.txt through the frontend to:

User-agent: *
Disallow: /

[paul]

Thanks Sam.

In the future I'll be sure to update the robot.txt file on the server.

[paul]

Thanks Sam. I see you have to update at admin/settings/robotstxt.

It would be good to get this configuration into features and under version control in the next version of the site.

[paul]

@Sam

Did you try out the new staged production site:

​https://booker-stage-20140501.transitionnetwork.org/

Earlier question:

When we finally do the migration, should this be done on the weekend? When it's quiet.

@Jim

Would you reply to the above questions (in bold) on the remaining steps to update the production site. Thanks.

[sam]

Hi Paul

I have had a look around today. I can't spot any obvious problems, but I wouldn't call it a comprehensive test!

Does anyone know what features/parts of the site ctools has been most heavily used in so I can take a careful look at those area's?

Thanks

Sam

[chris]

Replying to paul:

When we finally do the migration, should this be done on the weekend? When it's quiet.

FWIW, in the past, things like this have been done late at night and / or at weekends as the site traffic does drop off a lot at those times.

[paul]

Replying to chris:

Replying to paul:

When we finally do the migration, should this be done on the weekend? When it's quiet.

FWIW, in the past, things like this have been done late at night and / or at weekends as the site traffic does drop off a lot at those times.

I can do any night over the weekend -- I think this should be done only when your around too, just in case anything goes wrong.

[sam]

1, I'm finding I get some 504 timeout errors on the site, is this just because the site has less resources than the production site?

2, Just looking at this:https://booker-stage-20140501.transitionnetwork.org/admin/reports/status

Settings.php is not setup correctly. With the current configuration of 443 Session module, the following lines must be in settings.php.

if (!empty($_SERVERHTTPS?) && $_SERVERHTTPS? != 'off') {

ini_set('session.cookie_secure', 1);

}

However I note that this error also appears on the production site, not sure if it's an issue?

Things that seem to work:

Test Webform: ​https://booker-stage-20140501.transitionnetwork.org/node/35475

Test Blog: ​https://booker-stage-20140501.transitionnetwork.org/blogs/rob-hopkins/2014-05/test-blog-entry

Test event: ​https://booker-stage-20140501.transitionnetwork.org/events/2014-12-17/test-2

[sam]

Another submission from the stage site came in:

---

xxx (xxxxx@…) sent a message using the contact form at
​https://booker-stage-20140501.transitionnetwork.org/contact.

10 emails I'm one day-- for the second time. Cease unsubscribe me AT ONCE.

---

Putting two and two together I think we may have sent the email backlog from both the live site and the stage site.

It would explain how people are finding the site..

Thanks

Sam

[paul]

Replying to sam:

Hi Paul

I have had a look around today. I can't spot any obvious problems, but I wouldn't call it a comprehensive test!

Does anyone know what features/parts of the site ctools has been most heavily used in so I can take a careful look at those area's?

Thanks

Sam

Views has ctools as a dependency.

[paul]

Replying to sam:

1, I'm finding I get some 504 timeout errors on the site, is this just because the site has less resources than the production site?

2, Just looking at this:https://booker-stage-20140501.transitionnetwork.org/admin/reports/status

Settings.php is not setup correctly. With the current configuration of 443 Session module, the following lines must be in settings.php.

if (!empty($_SERVERHTTPS?) && $_SERVERHTTPS? != 'off') {

ini_set('session.cookie_secure', 1);

}

However I note that this error also appears on the production site, not sure if it's an issue?

I brought this up when I did a full review of the site. I think we agreed nothing urgent needs to be done here. I'll refresh my memory ..

Things that seem to work:

Test Webform: ​https://booker-stage-20140501.transitionnetwork.org/node/35475

Test Blog: ​https://booker-stage-20140501.transitionnetwork.org/blogs/rob-hopkins/2014-05/test-blog-entry

Test event: ​https://booker-stage-20140501.transitionnetwork.org/events/2014-12-17/test-2

[chris]

Replying to sam:

I think we may have sent the email backlog from both the live site and the stage site.

Oh dear, I guess this step was missed?

Reroute Email. You should also turn on the Reroute Email module, which will force all email generated by Drupal to be rerouted to a single, configurable address (ie. yours). This module should already be installed and enabled, and you'll just need to add permission to use it to the developer role ([url]admin/user/permissions#module-reroute_email). Once you've got the permission to use it, go to Admin > Site Configuration > Reroute Email, check " Enable rerouting", enter your own email address, and click "Save configuration".

wiki:website/devEnvironment/workstation#Finishingoffimagesteaetc

• ​https://drupal.org/project/reroute_email

See ticket:136 for some history of this issue.

[ed]

oh fuck me. I think a bit of grovelling may be in order. I'll write a short apology and stuff it on my blog tomorrow.

[chris]

Replying to sam:

1, I'm finding I get some 504 timeout errors on the site, is this just because the site has less resources than the production site?

No, that shouldn't happen, dev sites have exactly the same resources as the live site as far as I'm aware.

[chris]

Replying to chris:

This module should already be installed and enabled

wiki:website/devEnvironment/workstation#Finishingoffimagesteaetc

• ​https://drupal.org/project/reroute_email

My understanding was that, as quoted above, reroute email was installed and enabled by default? Is this not the case?

[chris]

Replying to ed:

oh fuck me. I think a bit of grovelling may be in order.

:-(

How many dev sites are there on the server -- did they all send emails out?

[paul]

I thought that emails on a stage site, would not be sent out, as the cron would need to be set up on the server -- and run. And that it wouldn't be executable from the browser because that wouldn't happen on the production site from where it's cloned.

Investigating ..

Stopwatch time for this and 2 previous comments.

[chris]

See also:

I've added Reroute Email, and Environment Indicator to the makefile, and to the platform.

ticket:487#comment:3

[chris]

Replying to paul:

cron would need to be set up on the server

This is done by BOA -- it's not set up using a conventional crontab, as far as I'm aware.

[paul]

I see I hadn't completed the steps required. Fail.

It looks as though in the past because I have only been cloning from stage I had failed to check all the steps in the wiki. I did carry out these steps on my local server.

It's strange that BOA would set up a stage server and send out all the emails by default.

[paul]

So the problem should only be with:

​https://booker-stage-20140501.transitionnetwork.org/

I'll just check that everything is now setup correctly ..

[chris]

Replying to paul:

I had failed to check all the steps in the wiki.

The wiki documentation which refers to reroute email is for "Develop directly on your local workstation" wiki:website/devEnvironment/workstation#Finishingoffimagesteaetc the "Develop directly on the Transition Network Aegir server" wiki:website/devEnvironment/aegir documentation makes no mention of reroute email.

This wiki page could do with the robots.txt and reroute email info being added to it: wiki:website/devEnvironment/aegir -- it's not been worked on since Mark was working with us: wiki:website/devEnvironment/aegir?action=history

[jim]

Hi all, a few things:

• I'd guess that when the cron started running again after the 2.2.4 update, the backlog of notifications started going out.
• If any sites were cloned from PROD, or any that did not have Reroute Email enabled, then they'd start to send mails.
• All the STG/DEV sites I made had these modules enabled by a line in the settings.local.php file (see ticket:487#comment:3) which enabled them. That was documented, but I can't see my old BOA dev wiki pages so not sure where -- are they still about or are they gone?
• ALL non-PROD sites need Reroute Email enabled and running, plus Robots.txt, and ideally Environment Indicator. Obviously if you clone an older site it'll inherit these already enabled.
• The Robots.txt module relies on the /robots.txt file NOT existing, otherwise ALL sites on the platform will get the standard Drupal one.

In terms of setting up new STG sites... Decide your source: clone either an existing STG (quick and safest) OR clone PROD (more faff and needs care as demonstrated by the issues on this ticket).

If cloning PROD:

• Ensure you change the site/[sitename]/files symlink to one of the STG files folders in /data/disk/tn/static.
• Ensure you DISABLE these: drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom session443 moopapi botcha UNLESS you're specifically working on notifications etc.
• Enable Reroute Email, Robots.txt, and Environment Indicator ensure they're all doing what they should (for non-PROD you should disallow all robots).

If cloning STG:

• Ideally you'll be cloning from the STG site, which will have all the above PROD stuff done. Otherwise, you'll need to do it.
• There is nothing else.

Finally, I'd suggest unless there's some pretty wild work/development/changes happening to a given STG build, there's no need to create new ones unless there's a major update coming out. In which case just clone a known working STG (the STG site) and then migrate that to the new platform that has the updates.

What we don't need is lots of STG & DEV sites without an owner, and a purpose. So best to have as few as needed for the tasks at hand.

Any other questions, please direct them to me with an obvious 'JIM' label. Thanks.

Best,

Jim

[chris]

Replying to paul:

It's strange that BOA would set up a stage server and send out all the emails by default.

It appears the work Jim did to automate this was never totally completed?

See this comment from jim ticket:487#comment:3

So the only thing we need do now is enforce reroute_email and environment_indicator to be enabled on every site. There are a number of ways to do this:

• BOA does it daily for modules here: ​https://github.com/omega8cc/nginx-for-drupal/blob/master/aegir/tools/system/usage.sh (FYI lives in /var/xdrago/)
• Add a new 'Transition Environment Settings' feature that does some goodies (still needs force-loading for sites)
• Other ways, I'm sure.

[chris]

Replying to jim:

I can't see my old BOA dev wiki pages so not sure where -- are they still about or are they gone?

Some things are on the ​https://wiki.transitionnetwork.org/ site, eg:

• ​http://wiki.transitionnetwork.org/BOA_Server/Development_workflow

Is that what you were thinking of?

[paul]

Catching up with previous comments shortly.

All modules are still enabled. I can't access drush as paul or root. I'll disable these through the UI.

​https://booker-stage-20140501.transitionnetwork.org/admin/build/modules

[chris]

Replying to paul:

I can't access drush as paul or root

My understand is that BOA creates ssh accounts for each site, as root you should be able to su to the account for the site or set that accounts passwd or add your ssh public key(s) to the account and then connect to it directly.

[jim]

Replying to chris:

• ​http://wiki.transitionnetwork.org/BOA_Server/Development_workflow

Is that what you were thinking of?

YES! Thanks Chris -- it's amazing how the details slowly buy surely fall out of one's mind after a prolonged period away...

So anyway, yes, it's all here: ​http://wiki.transitionnetwork.org/BOA_Server/Development_workflow#Cloning_PROD_to_STG

[paul]

Disabled the modules [*] through the UI as per the drush command:

drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom search

​https://wiki.transitionnetwork.org/BOA_Server/Development_workflow (Cloning PROD to STG)

Notice that notifications has a typo in the drush command.

[*] ​https://booker-stage-20140501.transitionnetwork.org/admin/build/modules

Catching up on comments ..

[paul]

Replying to jim:

Hi all, a few things:

• I'd guess that when the cron started running again after the 2.2.4 update, the backlog of notifications started going out.
• If any sites were cloned from PROD, or any that did not have Reroute Email enabled, then they'd start to send mails.
• All the STG/DEV sites I made had these modules enabled by a line in the settings.local.php file (see ticket:487#comment:3) which enabled them. That was documented, but I can't see my old BOA dev wiki pages so not sure where -- are they still about or are they gone?
• ALL non-PROD sites need Reroute Email enabled and running, plus Robots.txt, and ideally Environment Indicator. Obviously if you clone an older site it'll inherit these already enabled.
• The Robots.txt module relies on the /robots.txt file NOT existing, otherwise ALL sites on the platform will get the standard Drupal one.

In terms of setting up new STG sites... Decide your source: clone either an existing STG (quick and safest) OR clone PROD (more faff and needs care as demonstrated by the issues on this ticket).

If cloning PROD:

• Ensure you change the site/[sitename]/files symlink to one of the STG files folders in /data/disk/tn/static.
• Ensure you DISABLE these: drush dis -y notifictions messaging piwik google_analytics mailchimp captcha mollom session443 moopapi botcha UNLESS you're specifically working on notifications etc.
• Enable Reroute Email, Robots.txt, and Environment Indicator ensure they're all doing what they should (for non-PROD you should disallow all robots).

If cloning STG:

• Ideally you'll be cloning from the STG site, which will have all the above PROD stuff done. Otherwise, you'll need to do it.
• There is nothing else.

Finally, I'd suggest unless there's some pretty wild work/development/changes happening to a given STG build, there's no need to create new ones unless there's a major update coming out. In which case just clone a known working STG (the STG site) and then migrate that to the new platform that has the updates.

What we don't need is lots of STG & DEV sites without an owner, and a purpose. So best to have as few as needed for the tasks at hand.

Any other questions, please direct them to me with an obvious 'JIM' label. Thanks.

Best,

Jim

Sounds good.

I had a first pass over the referenced ticket (very interesting) and bookmarked that and the current ticket for future reference.

[ed]

Couple of points:

1. Can we find out how many emails have been sent to what type of subscribee? So I know how to handle this comms-wise.

2. My understanding is that this event was triggered by a lack of RTFM, but I see the cause behind it as the whole situation; so:

2a. how can we make sure that it does not happen again and
2b. how can we make sure that similar fuckups don't happen?

3. What do we need to do to roll to LIVE? And when? Is JIM required?

[chris]

Replying to ed:

Couple of points:

1. Can we find out how many emails have been sent to what type of subscribee? So I know how to handle this comms-wise.

I don't know if the Drupal site logs outgoing email, but the mailserver does, the best thing we have to match on is the ID of the sender, we have entries like this:

May  1 00:42:17 puffin postfix/pickup[47077]: C5E3C80CE4: uid=116 from=<site@transitionnetwork.org>
May  1 00:50:41 puffin postfix/pickup[47077]: 10283811CF: uid=116 from=<www53>
May  1 00:52:58 puffin postfix/pickup[47077]: 130F7811CF: uid=116 from=<www53>
May  1 00:56:40 puffin postfix/pickup[47077]: 23150811D7: uid=0 from=<root>
May  1 01:01:03 puffin postfix/pickup[24304]: 4A71D811D7: uid=0 from=<chris@webarchitects.co.uk>
May  1 01:04:34 puffin postfix/pickup[24304]: 3CD768121D: uid=999 from=<site@transitionnetwork.org>

But I don't think this will help as php-fpm is running as UID 116 for all sites, from /etc/passwd:

www53:x:116:125::/var/www/www53:/bin/false

And the From: field seems to be set to <site@transitionnetwork.org> for all sites -- so a count of the number of emails will have to come from Drupal I think.

2a. how can we make sure that it does not happen again and
2b. how can we make sure that similar fuckups don't happen?

Good questions... not sure what the answer is apart from learning the hard way... this has happened before... ticket:136 ...

[chris]

Replying to ed:

I'll write a short apology and stuff it on my blog tomorrow.

FWIW this is what you wrote last time:

• ​https://www.transitionnetwork.org/blogs/ed-mitchell/2010-09/weird-messages-our-development-server

[ed]

I like that. That's cheered me right up. The cyclical nature of things, eh? Anyone for some metaphysics?

Anyway - my answer to number 2 is:

1. Better planning for TNv3 upfront
2. Standardised version control with documentation styles and standards and locations agreed upfront.
3. Formally include technical project management into the lead developer's role

But my question 3 is still open - what do we need to do to roll LIVE and do we need Jim

[paul]

Replying to ed:

Couple of points:

1. Can we find out how many emails have been sent to what type of subscribee? So I know how to handle this comms-wise.

The database / syslog logging are disabled on the site, so I think we will need to look at mail log for answers. Chris?

2. My understanding is that this event was triggered by a lack of RTFM, but I see the cause behind it as the whole situation;

That's right. I could of prevented the problem by reading the wiki. I'll not make the mistake again. However If I had read the wiki on trac I could of hit this problem when setting up my local server.

After completing the update of the production site I'll be in a good position to take a look at revising the wiki notes.

so:

2a. how can we make sure that it does not happen again and

In the near term, I'll manually enable the reroute email module. Later I'll look at automating this step as discussed in a previous ticket.

2b. how can we make sure that similar fuckups don't happen?

I think we all need to watch out for each other at the moment, and when carrying out tasks on the server to ask yourself the question: what could go wrong here?

3. What do we need to do to roll to LIVE? And when? Is JIM required?

The new production platform is now ready. We now need to migrate the production sites to the new production platform. I think we should do this over the weekend. There are a couple of questions that I have for Jim : ​https://tech.transitionnetwork.org/trac/ticket/712?replyto=60#comment:25

[chris]

Replying to paul:

The database / syslog logging are disabled on the site, so I think we will need to look at mail log for answers. Chris?

Sorry, no answers there, see ticket:712#comment:61

[paul]

So currently we have a system that takes a production website, puts it on a stage server, sends out emails by default to all of its users and doesn't turn on logging!

[chris]

Replying to paul:

So currently we have a system that takes a production website, puts it on a stage server, sends out emails by default to all of its users and doesn't turn on logging!

In the days before BOA we did have a script that did everything for you, see wiki:DevelopmentServer#live2dev but since BOA, as far as I'm aware, it's only been Jim who has copied sites on the server, until you tried it...

[paul]

Thanks Chris, I have bookmarked that.

Now that we're ready to do the last step (migrate the production sites to the new production platform) it would be good if JIm could have a quick look over things - and give us the green light, and also reaffirm that if we should hit a problem, that we can rewind.

[chris]

Replying to paul:

doesn't turn on logging!

I think that is correct, last time I looked there was no Drupal level logging for any sites on BOA:

The dblog issues did get saved in the database and were available via ​​https://www.transitionnetwork.org/ some years ago, then on wiki:NewLiveServer it was changed so they were written to a syslog file, but on wiki:PuffinServer they don't appear to be anywhere?

ticket:208#comment:7

[paul]

The syslog module is currently disabled on www.transitionnetwork.org.

@Jim
How are we currently managing our website log files on stage/ production?

[chris]

Replying to paul:

The syslog module is currently disabled on www.transitionnetwork.org.

@Jim
How are we currently managing our website log files on stage/ production?

I don't know why syslog is disabled, I did search this trac site for an answer to this and the only thing I found was ticket:208#comment:7.

There are some notes on the Nginx weblogs wiki:WebServerLogs

[jim]

Hi All, my answers to the various things:

When creating a platform we never specify explicitly that this platform is a production platform, only indirectly by our naming convention PXXX; Is this correct?

There is nothing specifically telling anything this is a PROD platform, apart from the function call in the settings.local.php file.

To migrate the websites to the new production platform, do we simply do a migrate, and everything that needs to happen is taken care of in the background?

Yes, however you should clone a recent STG to the platform first to ensure the updates are ok.

If there's a problem and only a short time (hours at most) has gone by, you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

Yes, kinda. Every site is backed up first before migrate (it's what takes the most time). If there's an issue that needs a roll back, you can restore the site in situ, or to a new site before migrating it back to the live domain.

Has this been tested? Can you provide more detailed instructions?

Has what been tested? There's plenty of documentation in our proper Wiki, and on the Aegir Project site. Please be specific, and check what you need isn't already documented somewhere by the pros!

a) What do we need to do to roll to LIVE? b) And when? c) Is JIM required?

a) Prepare the new platform, migrate a STG, test, migrate LIVE.
b) Your call.
c) Hopefully not! The migrate STG step will catch 95% of issues, so don't skip that!

How are we currently managing our website log files on stage/ production?

Again, all in the Wiki docs, but in a nutshell: both STG and PROD symlink the the sites/www.transtitionnetwork.org/files folder to somewhere in /data/disk/tn/static. This means the file move part is instant, which is good for a site with as many assets as TN. If you looking the static folder, you'll see there's separate files folders for STG and PROD. Again the wiki docs I wrote explain about re-creating this (only needed if you're spinning up a STG from scratch, or making one from PROD).

The syslog module is currently disabled on www.transitionnetwork.org.

Correct, by design and for performance. You can enable that or DB Log (neither recommended for PROD use) and they'll stay on until 3am-ish. If you guys really want a logging module enabled, you can alter one of the .sh files in /var/log/xdrago... Do a grep for 'syslog'. I think its there! Do a search of Omega8cc's github acc if you need to confirm.

[paul]

Thanks for the replies Jim. Hopefully just a few more Y/N questions.

Replying to jim:

Hi All, my answers to the various things:

When creating a platform we never specify explicitly that this platform is a production platform, only indirectly by our naming convention PXXX; Is this correct?

There is nothing specifically telling anything this is a PROD platform, apart from the function call in the settings.local.php file.

I have created a new platform: Transition Network D6 P010 Booker. Will this platform automatically become the production platform, when we migrate the production sites to the new production platform?

​https://tn.puffin.webarch.net/hosting/c/platform_TransitionNetworkD6P010Booker

To migrate the websites to the new production platform, do we simply do a migrate, and everything that needs to happen is taken care of in the background?

Yes, however you should clone a recent STG to the platform first to ensure the updates are ok.

I have already created a new stage platform Transition Network D6 S012 Booker with the same makefile and migrated the production site to the new stage platform and tested the updated here. I think we don't need to do any further testing?

​https://tn.puffin.webarch.net/hosting/c/platform_TransitionNetworkD6S012

If there's a problem and only a short time (hours at most) has gone by, you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

Yes, kinda. Every site is backed up first before migrate (it's what takes the most time). If there's an issue that needs a roll back, you can restore the site in situ, or to a new site before migrating it back to the live domain.

Has this been tested? Can you provide more detailed instructions?

Has what been tested? There's plenty of documentation in our proper Wiki, and on the Aegir Project site. Please be specific, and check what you need isn't already documented somewhere by the pros!

Sorry, In the wiki page that discusses failed migrations, there are no notes on this. So my question should have been: have you tried to simulate a recovery on the server, and do you have any bookmarked notes that you would have refereed to if something had of gone wrong.

​https://wiki.transitionnetwork.org/BOA_Server/Development_workflow :
Rolling back a failed migration
@TODO CHECK IF THIS PROCESS IS CORRECT

a) What do we need to do to roll to LIVE? b) And when? c) Is JIM required?

a) Prepare the new platform, migrate a STG, test, migrate LIVE.
b) Your call.
c) Hopefully not! The migrate STG step will catch 95% of issues, so don't skip that!

I think we're ready to do the migration, but not ready at the moment, if something goes wrong?

How are we currently managing our website log files on stage/ production?

Again, all in the Wiki docs, but in a nutshell: both STG and PROD symlink the the sites/www.transtitionnetwork.org/files folder to somewhere in /data/disk/tn/static. This means the file move part is instant, which is good for a site with as many assets as TN. If you looking the static folder, you'll see there's separate files folders for STG and PROD. Again the wiki docs I wrote explain about re-creating this (only needed if you're spinning up a STG from scratch, or making one from PROD).

The syslog module is currently disabled on www.transitionnetwork.org.

Correct, by design and for performance. You can enable that or DB Log (neither recommended for PROD use) and they'll stay on until 3am-ish. If you guys really want a logging module enabled, you can alter one of the .sh files in /var/log/xdrago... Do a grep for 'syslog'. I think its there! Do a search of Omega8cc's github acc if you need to confirm.

[paul]

Replying to paul:

Thanks for the replies Jim. Hopefully just a few more Y/N questions.

Replying to jim:

Hi All, my answers to the various things:

When creating a platform we never specify explicitly that this platform is a production platform, only indirectly by our naming convention PXXX; Is this correct?

There is nothing specifically telling anything this is a PROD platform, apart from the function call in the settings.local.php file.

I have created a new platform: Transition Network D6 P010 Booker. Will this platform automatically become the production platform, when we migrate the production sites to the new production platform?

​https://tn.puffin.webarch.net/hosting/c/platform_TransitionNetworkD6P010Booker

To migrate the websites to the new production platform, do we simply do a migrate, and everything that needs to happen is taken care of in the background?

Yes, however you should clone a recent STG to the platform first to ensure the updates are ok.

I have already created a new stage platform: Transition Network D6 S012 Booker with the same makefile -- and migrated the production site to the new stage platform, and tested the updates here. I think we don't need to do any further testing?

​https://tn.puffin.webarch.net/hosting/c/platform_TransitionNetworkD6S012

If there's a problem and only a short time (hours at most) has gone by, you can disable the current site in PROD, then restore the backup that was made just before the migration onto the the older platform.

Yes, kinda. Every site is backed up first before migrate (it's what takes the most time). If there's an issue that needs a roll back, you can restore the site in situ, or to a new site before migrating it back to the live domain.

Has this been tested? Can you provide more detailed instructions?

Has what been tested? There's plenty of documentation in our proper Wiki, and on the Aegir Project site. Please be specific, and check what you need isn't already documented somewhere by the pros!

Sorry, In the wiki page that discusses failed migrations, there are no notes on this. So my question should have been: have you tried to simulate a recovery on the server, and do you have any bookmarked notes that you would have refered to, if something had of gone wrong.

​https://wiki.transitionnetwork.org/BOA_Server/Development_workflow :
Rolling back a failed migration
@TODO CHECK IF THIS PROCESS IS CORRECT

a) What do we need to do to roll to LIVE? b) And when? c) Is JIM required?

a) Prepare the new platform, migrate a STG, test, migrate LIVE.
b) Your call.
c) Hopefully not! The migrate STG step will catch 95% of issues, so don't skip that!

I think we're ready to do the migration, but not ready at the moment, if something goes wrong?

How are we currently managing our website log files on stage/ production?

Again, all in the Wiki docs, but in a nutshell: both STG and PROD symlink the the sites/www.transtitionnetwork.org/files folder to somewhere in /data/disk/tn/static. This means the file move part is instant, which is good for a site with as many assets as TN. If you looking the static folder, you'll see there's separate files folders for STG and PROD. Again the wiki docs I wrote explain about re-creating this (only needed if you're spinning up a STG from scratch, or making one from PROD).

The syslog module is currently disabled on www.transitionnetwork.org.

Correct, by design and for performance. You can enable that or DB Log (neither recommended for PROD use) and they'll stay on until 3am-ish. If you guys really want a logging module enabled, you can alter one of the .sh files in /var/log/xdrago... Do a grep for 'syslog'. I think its there! Do a search of Omega8cc's github acc if you need to confirm.

[paul]

I think we're ready to do the migration. Before doing the actual migration - I'll take a manual backup, and download it off the server, on to my local machine. When we start the migration, Aegir should should create a further backup, which it should be possible to restore from if anything does go wrong.

​https://tn.puffin.webarch.net/node/621/backups

Let me know when you would like me to do the migration?

[ed]

Good work Paul.

The timing question is still there - who needs to be present? By knowing that, we can organise.

[paul]

Thanks Ed,

I think it would be best to have Chris available via email for a couple of hours over the weekend, should I need to share any problems.

[chris]

Replying to jim:

The syslog module is currently disabled on www.transitionnetwork.org.

Correct, by design and for performance. You can enable that or DB Log (neither recommended for PROD use) and they'll stay on until 3am-ish. If you guys really want a logging module enabled, you can alter one of the .sh files in /var/log/xdrago... Do a grep for 'syslog'. I think its there! Do a search of of Omega8cc's github acc if you need to confirm.

If we need this log, and in the past it was, at times, useful, then we shound enable it -- the server does have the capacity to record this log, I very much doubt it'll have a significant impact on performance -- it's effect would probably be dramatically less that the effect of BOA halfing the RAM available to MySQL, see ticket:587#comment:11.

Replying to paul:

I think it would be best to have Chris available via email for a couple of hours over the weekend, should I need to share any problems.

I don't have any plans and should be around. I'll do the outstanding BOA upgrade now, ticket:725.

[ed]

Good work lads. Exciting times. I will be away from computer, but on phone if I'm needed (not that I will, but you know...)

[paul]

Giving this a go shortly.

[paul]

My new production platform is not showing under the list of possible migration platforms for our sites.

Investigating ..

[paul]

Building another production platform - following the naming convention, i.e dropping 'Booker'. This should work, as earlier I could see my stage platform, and noticed that I had only added 'Booker' to the stage platform name, after building the platform.

[paul]

Still not seeing the new production platform. I couldn't find anything in our wiki that helps to resolve this problem.

​https://wiki.transitionnetwork.org/BOA_Server/Development_workflow

Jim said in a earlier comment that, "There is nothing specifically telling anything this is a PROD platform, apart from the function call in the settings.local.php file."

I did a search for that file ..

puffin:/data/disk/tn# find . -name settings.local.php

..but couldn't find it.

I'll have a look at the Aegir documentation ..

[paul]

Not clear how to proceed.

There is something that I need to do - to get the new "production" platform, to show up, on the the modalframe dialog on the migrate form.

@Jim

Would you advise how to complete the migration? or point me at the wiki, trac or aegir documentation if I have missed something.

Migrating/upgrading and renaming websites
​http://community.aegirproject.org/node/23

[paul]

I tried deleting some of the platforms, that I had created earlier, but no longer need, to see if that helps. Fail.

I guess I could use my latest "stage" platform - which does show up on the migrate form, and migrate our sites there. Changing the name of the platform, before or later? But I think it will be best to resolve our current problem, before trying possible work arounds.

I think there may be a bug - with new platforms not now showing on migration form; so I'm going to see if Chris / Jim have any thoughts on how to proceed?

[chris]

Replying to paul:

I think there may be a bug - with new platforms not now showing on migration form; so I'm going to see if Chris / Jim have any thoughts on how to proceed?

I don't I'm afraid.

[jim]

Hi guys, it's almost certainly because there is something amiss with the new platform, rather than a bug - what are the differences between it and the working STG platform?

I can't do much at the mo because my home broadband is (still) out of action... I'll try to grab 5 to look at it tomorrow at work.

[paul]

Replying to jim:

Hi guys, it's almost certainly because there is something amiss with the new platform, rather than a bug - what are the differences between it and the working STG platform?

None, that I am aware of.

I can't do much at the mo because my home broadband is (still) out of action... I'll try to grab 5 to look at it tomorrow at work.

Thanks Jim.

[jim]

OK so a few things:

A) It's not called settings.local.php, it's local.settings.php! My bad, sorry! Please see the /sites/[domain]/local.settings.php files for STG and PROD to see how they work.

B) I've taken a look at the platform in Aegir, and nothing is wrong... However when I log in as admin I certainly can migrate a site to Paul's new platform. See this shot:

SOOO... this is a permissions issue for the Aegir user, site or platform. Basically Paul either doens't have permission to migrate tn.org, or TN doesn't have permission to use the platform he made.

I checked the platform edit page (​https://tn.puffin.webarch.net/node/1318/edit) and saw that 'Transition Network' was not checked. So I did that:

It think, therefore, it's all working as expected... So Paul, please try again, and if it's not appearing check the perms for the site, platform, client (Transition Network) and yourself.

C) Still no broadband at home... :-(

[paul]

@Jim

I can now see the platform. Thanks a lot for that Jim. I would probably not have tried ticking the checkbox for "Transition Network".

@Ed

Shall I have a go at the upgrade at 1am tonight ?

Note:
The wiki will need updating to include: ticking the checkbox for "Transition Network".
​https://wiki.transitionnetwork.org/BOA_Server/Building_platforms

[ed]

@Paul - no time like the present mate - if you're set, let's go. 01:00 might find you quite alone though - if it would be more suitable I'm happy to wait until the weekend when you might have some company.

Wiki comment - is that you saying you will update the wiki?

[paul]

@Ed

I'm ready to give it a go now, so here goes ..

I'll update the wiki.

[paul]

I can't see the platforms again. Investigating ..

[paul]

I was able to associate the admin user with my account - that resolved the problem.

[paul]

Ok, www.transitionnetwork.org is queued for migration...

[paul]

@Ed,

Upgrade looks to have been successful.

No errors generated as a result of the upgrade:

​https://www.transitionnetwork.org/admin/reports/updates
​https://www.transitionnetwork.org/admin/reports/status

I have done some random clicking around and not seen any problems.

Best, Paul

[paul]

Updated the wiki.

[ed]

Can't find any problems yet from this end.

[sam]

Hi Paul

Nice one :) Great to see those security updates applied.

I have not run into any issues, and no users have raised any so I think we can call that a success.

I'll also close: ​/trac/ticket/693 as that ticket has resolved it.

Thanks

Sam

[sam]

Hi Paul

I was wondering if there are any changes/ clarifications/ simplifications you could make to the documentation for this process here:

​https://wiki.transitionnetwork.org/BOA_Server/Development_workflow#Module_.26_core_updates_for_sites

Maybe pull together a bullet point section for each of the main steps with links to further info as needed?

Or is it as clear as it possibly can be already?

Thanks

Sam

[paul]

Hi Sam,

I had a read through that section of the wiki, and the steps are documented clearly, so the wiki probably doesn't need editing. But let me know if you have any questions.

Best, Paul

[paul]

@Sam Do you have any further questions? Do you still think the wiki documentation needs refactoring. If so, I'll see what I can do to make things clearer. Thanks Sam.