Import TN.org site from Quince to Puffin

[jim]

This ticket is to log/document steps in the process of importing TN.org onto the new Puffin server.

Current imported site at: ​http://newlive.puffin.webarch.net/

Import Process

This needs Part A of ​Transfer live TN.org site to Puffin before continuing.

1. Run ​Quince to Puffin rsync script to get Drupal files.
2. Make 'Transition Network XXX' platform from the latest makefile per ​BOA Server page.
3. Create an empty site on platform called 'www.transitionnetwork.org' with aliases for dev.newlive.puffin.webarch.net, newlive.puffin.webarch.net, and dev.www.transitionnetwork.org (dev.newlive... ones will go after launch).
4. SSH to Puffin and replace the default files folder with symlink to the one imported:

   totn
   cd static/transition-network-d6-XXX/sites/transitionnetwork.org
   rm -R files
   ln -s ~/static/sites/transitionnetwork.org-PROD/files/
   

5. Use one-time login link sent to me but replacing transitionnetwork.org with dev.newlive.puffin.webarch.net in url.
6. Enable Backup and Migrate module, use to import DB backup from Quince TN.org present in the Backup & Migrate manual folder.

   drush @www.transitionnetwork.org en backup_migrate
   

7. (white screen of death, expected)
8. Run following as 'tn' to remove unneeded modules, then clear registry and caches, then update database for module/core updates:

   drush @www.transitionnetwork.org rr 
   drush @www.transitionnetwork.org updb
   drush @www.transitionnetwork.org en reroute_email environment_indicator robotstxt
   

9. Check site is ok.
10. Double-migrate per ​http://omega8.cc/import-your-sites-to-aegir-in-8-easy-steps-109#hint-8 (migrate site to 'rename.transitionnetwork.org', then back to 'www.transitionnetwork.org')
11. add puffin_server_override_settings_set_environment('Production'); to the local.settings.php file to set environment to test mode (no emails).
12. Set logo and favicon admin/build/themes/settings/transition2 (replace 'default' with 'www.transitionnetwork.org')
13. Change 443 Session settings to force HTTPS for logged in users.
14. Run developer tests on site, email all to begin testing before decision on DNS switch.

[jim]

Steps to get site onto platform:

• Created "Transition Network D6 002" platform based on latest Drush Make file
• Create an empty site called 'transitionnetwork.org' with aliases for dev.newlive.puffin.webarch.net, newlive.puffin.webarch.net, and www.transitionnetwork.org
• Use one-time login link sent to me but with dev.newlive.puffin.webarch.net as host in instead of transitionnetwork.org.
• Enable Backup and Migrate module
• SSH to Puffin and replace the default files folder with symlink to the one imported:

  totn
  cd static/transition-network-d6-002/sites/transitionnetwork.org
  rm -R files
  ln -s ~/static/sites/transitionnetwork.org-PROD/files/
  

[jim]

• Use Backup and Migrate to restore the DB snapshot
• (white screen of death, expected)
• Run drush transitionnetwork.org rr to clear registry and caches
• Run drush transitionnetwork.org updb to do updates present in new platform. This also disables the old modules we won't use like Memcache, Varnish, Session 443

Platform/site seem healthy this time...

NOTES/TO FIX:
1) Views Slideshow: is 2.4 on current site and platform is 3.0, checking to see any issues/damage on homepage slideshow. Needs libraries (now added to Make file and 002 platform)
2) Colorbox: on status page says "Colorbox plugin must be at least 1.3.18"... It should be, will check sites/all/libraries/colorbox folder
3) Broken links due to change in sites/default/files to sites/transitionnetwork.org/files can be dealt with by migrating the site twice in Aegir.

[jim]

Fixes:
1) Views Slideshow major version must match Views, I've updated Makefile to manage this.
2) Colorbox needs to be in colorbox/colorbox, Makefile reflects this.
3) Will try double-migrate per ​http://omega8.cc/import-your-sites-to-aegir-in-8-easy-steps-109#hint-8 now.

[jim]

Updates:
2) Colorbox needs specific version downloaded, rather than latest. Set to 1.3.18 in makefile.
3) Double migrate worked AWESOMELY. (Migrated to 'rename.tn.org', then back to 'tn.org'. Most file references are fixed, but some links refer to the original domain and are therefore broken until DNS changes are done.

New issues:
4) Warning: file_get_contents(sites/all/modules/contrib/gmap/thirdparty/markerclusterer_packed.js): failed to open stream: No such file or directory in _locale_parse_js_file() (line 1710 of /data/disk/tn/static/transition-network-d6-002/includes/locale.inc). -- This is because we need the TN Gmap version instead of the standard one. Will add to Github
5) Logo is set to wrong place at admin/build/themes/settings/transition2 sites/default/files/transition2_logo.jpg needs bold bit moved to transitionnetwork.org

[jim]

4) Will add our Gmap (rather than patching the shit out of the standard version) to Github and changing the makefile to use our version.
5) also affects favicon -- easy to fix.

So all good so far!!!

To do on this ticket:

• Resolve issues 1-5 (mostly done)
• Rewrite process in ticket description to final verison
• Run though final version with final (hopefully) makefile

[chris]

Replying to jim:

• Get the site secured using Aegir's inbuilt SSL handing

Anything I need to help with on this?

I note I need to tweak some Nginx settings regarding HTTPS:

This server is vulnerable to the BEAST attack.
Chain issues: Incomplete

​https://www.ssllabs.com/ssltest/analyze.html?d=newdev.transitionnetwork.org

[jim]

@Chris:

• SSL should just need the NGINX SSL feature on master.puffin.webarch.net to be enabled, then the site's SSL setting put to 'automatic'. I'll try it on
• I'll leave you to work out a fix for BEAST, but if we have found something that needs changing BOA stuff we should raise a ticket. How best to fix?

[jim]

Need run this Drush command to disable the modules that are installed on current but not needed on the new platform:

drush dis memcache memcache_admin session443 varnish

And do files import... Fleshing out process in description.

[jim]

Fixed:
4) Our copy of GMap in the repo, makefile updated.
5) Logo/favicon details added to process

As for 6) SSL:

• Trying this now: ​http://community.aegirproject.org/content/content/administrator/post-install-configuration/using-ssl -- Will add to process or raise another ticket depending on sucess/fail when setting this up on ​http://newlive.puffin.webarch.net/
• Chris does this look useful or needed?: ​http://drupalcode.org/project/barracuda.git/blob/HEAD:/docs/SSL.txt -- seems to be for an extra IP, does that match our needs?

NOTE: the 'Aborted connection' on the DB causing/caused by lots of 502 Bad Gateways is back... going back to #466 now.

[jim]

Tweaks to process.

[jim]

Domain name corrections as www is to be used rather than naked domain.

[jim]

Re SSL -- ​http://drupalcode.org/project/barracuda.git/blob_plain/HEAD:/docs/SSL.txt <-- this is the 'proper' way of doing things. DO NOT use aegirproject.org link posted in comment 9... Correct link says:

1. Use existing or deploy a new site as usual - don't enable SSL features in Aegir.
2. Create two extra configuration files with contents as shown further below.

    * Replace YO.UR.AEGIR.IP with your Aegir Hostmaster main IP address.
    * Replace YO.UR.EXTRA.IP1,2,3 etc with correct extra IP addresses.
    * Paste your SSL key in the file /etc/ssl/private/abc-ssl-enabled-domain.key
    * Paste your SSL certificate and all intermediate certificates (bundles)
      in the file /etc/ssl/private/abc-ssl-enabled-domain.crt

3. Restart Nginx with `service nginx reload` or `service nginx restart`. Done!

SOOOooo.... I have broken the aegir panel by enabling the SSL/Nginx SSL features, so I'm now backing out of this per ​http://drupal.org/node/1882078. Bugger. Once it's back to normal I'll follow the instructions above.

---

ALSO: corrections to Drush commands in process

AND: See ​http://tn.i-jk.co.uk/ <-- platform and site (minus files) installed per this process with no issues except for one Colorbox path thing which I've now fixed in the makefile -- WOOP! Process and makefile good and ready for L-Day!

SSL to go...

[chris]

Replying to jim:

Re SSL -- ​http://drupalcode.org/project/barracuda.git/blob_plain/HEAD:/docs/SSL.txt <-- this is the 'proper' way of doing things.

We are not adding an additional IP address since we don't need one, the directory suggested for creating the two new Nginx config files, /var/aegir/config/server_master/nginx/pre.d/ contains nginx_wild_ssl.conf which has the following:

  ssl_certificate              /etc/ssl/private/nginx-wild-ssl.crt;
  ssl_certificate_key          /etc/ssl/private/nginx-wild-ssl.key;

These are symlinks:

/etc/ssl/private/nginx-wild-ssl.crt -> ../transitionnetwork.org/transitionnetwork.org.crt
/etc/ssl/private/nginx-wild-ssl.key -> ../transitionnetwork.org/transitionnetwork.org.key

But the cert is wrong, Nginx needs the chained one, so I have fixed it:

cd /etc/ssl/private/
rm nginx-wild-ssl.crt
ln -s /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem nginx-wild-ssl.crt

However the certificate used at ​https://newlive.puffin.webarch.net/ is still wrong, I'm still not clear where the certs are that the server is using?

[jim]

OK I think I now understand much more about the guts of Aegir after that goose chase!

Nginx SSL entry removed manually from /data/disk/tn/config/server_master/nginx/vhost.d/transitionnetwork.org, which I backed up to /root/scratch/transitionnetwork.org. In there at the top is this section:

server {
  include      /data/disk/tn/config/includes/fastcgi_ssl_params.conf;
  limit_conn   gulag 32; # like mod_evasive - this allows max 32 simultaneous connections from one IP address
  listen       81.95.52.103:443;
  server_name  transitionnetwork.org dev.newlive.puffin.webarch.net newlive.puffin.webarch.net www.transitionnetwork.org;
  root         /data/disk/tn/static/transition-network-d6-002;
  ssl                        on;
  ssl_certificate            /data/disk/tn/config/server_master/ssl.d/transitionnetwork.org/openssl.crt;
  ssl_certificate_key        /data/disk/tn/config/server_master/ssl.d/transitionnetwork.org/openssl.key;
  ssl_protocols              SSLv3 TLSv1;
  ssl_ciphers                HIGH:!ADH:!MD5;
  ssl_prefer_server_ciphers  on;
  keepalive_timeout          70;
  # Extra configuration from modules:
  include      /data/disk/tn/config/includes/nginx_octopus_include.conf;
}

Which answers a number of questions, like where the cert was coming from.

Chris, I would be tempted to revert the wildcard certs symlinks and look around in /data/disk/tn/config/ for all the relevant config. Note that this is all auto-created by Aegir & Octopus, so I'd avoid editing things directly... Extra conf can be added here and there, but best to check the docs here: ​http://drupalcode.org/project/barracuda.git/tree/HEAD:/docs

So my mistake is fixed... SSL now needs setting up though. I'll research the RIGHT WAY now.

[jim]

AND Since I restarted Nginx after removing the broken HTTPS stuff, the cert at ​https://newlive.puffin.webarch.net is correct! Ghandi... WOOP! (or was that your doing Chris?)

Moving SSL stuff to ticket: #484

[jim]

reroute_email and environment_indicator notes.

[jim]

A few tweaks...

[jim]

robots! and testing environment

[jim]

Done for real, no issues found...

Tweaks for Prod vs Test/Dev?:

• puffin_server_override_settings_set_environment('Production');

[jim]

Done, closing.