Transition Technology: Ticket Query

#582: TN.org platform and sites

jim — Mon, 02 Sep 2013 09:30:02 GMT

The TN.org platform and Drupal site updates are to be tracked in this ticket.

Current PROD platform build = P009 Current STG platform build = S010

Updates pending:

SECURITY UPDATE - NO RISK: Pressflow core 6.30 is due, but the security holes fixed do not affect us, low priority. Platforms: present in S010, but not in P009.

#264: Context changes

ed — Fri, 17 Jun 2011 12:23:53 GMT

As discussed at meeting. Contexts not really working. Blocks on a right all over the place. Perhaps need some new page layouts (?) or other piece of work with panels (?) to sort out.

One page to start with is the Training page (and subsection). Ed to meet Trainers and their marketing consultant on 21/6.

Jim please advise Ed what Jim needs to

#326: Usability changes as per the usability report

ed — Mon, 12 Sep 2011 11:10:37 GMT

Report has been circulated around ttech - to READ and then discuss - some will be phase 5 work, some will be the webmaster's responsibility...

#384: Enhance Project Profile Content Type and directory

ed — Tue, 20 Dec 2011 23:25:32 GMT

Laura to re-design IA and interface for admin and public profiles for project profile CT and directory.

Considered as part of PSE Project, hence design in Jan, build in Feb 2012

#520: Session 443 config in settings.php

chris — Fri, 15 Mar 2013 23:16:49 GMT

There is this warning displaying at https://www.transitionnetwork.org/admin/reports/status

Settings.php is not setup correctly. With the current configuration of 443 Session module, the following lines must be in settings.php.
      if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] != 'off') {
        ini_set('session.cookie_secure', 1);
      }

Based on the check of what is happening with cookies done on ticket:371#comment:34 and ticket:371#comment:36 things are currently working OK, session cookies do have the secure flag set, so I'm a bit confused by this warning message. I also think that the PHP suggested to add to settings.php looks perfectly sensible and should be included, I'm sure we did have it on the old server, however there are 33 settings.php files on wiki:PuffinServer and I'm not clear which one the live site uses.

#521: MySQL Unsafe statement warnings in the daemon.log

chris — Sat, 16 Mar 2013 09:46:57 GMT

I don't know if these matter?

I found them when hunting for 502 errors.

grep "Unsafe statement written to the binary log" /var/log/daemon.log | wc -l
343

Some examples:

Mar 16 09:28:20 puffin mysqld: 130316  9:28:20 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave. Statement: DELETE FROM notifications_event WHERE created < 1363426040 AND eid < (SELECT MIN(eid) FROM notifications_queue)

Mar 16 05:52:12 puffin mysqld: 130316  5:52:12 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave. Statement: INSERT INTO notifications_queue (uid, mdid, send_method, sid, module, eid, send_interval, language, cron, created, conditions) SELECT DISTINCT s.uid, s.mdid, s.send_method, s.sid, s.module, 61233, s.send_interval, s.language, s.cron, 1363413132, s.conditions FROM notifications s LEFT JOIN notifications_fields f ON s.sid = f.sid WHERE (s.status = 1) AND (s.event_type = 'node') AND (s.send_interval >= 0) AND ((f.field = 'nid' AND f.intval = 30718) OR (f.field = 'type' AND f.value = 'profile') OR (f.field = 'author' AND f.intval = 16908)) GROUP BY s.uid, s.mdid, s.send_method, s.sid, s.module, s.send_interval, s.

#457: Projects form - Enhance form entry

laura — Thu, 08 Nov 2012 18:15:44 GMT

1 - Entry Form:

Set up new fields and permissions and groupings
Enhance ‘helper’ texts and any links to other parts of TN listed on form to enhance usability and context.
CSS and potential of custom templating/panels if needed for style and layouts

#461: Spam account war

ed — Wed, 21 Nov 2012 16:19:10 GMT

Aim:

tell drupal (and server level stuff?) to sniff out and destroy spam accounts without them knowing we did it, and ban them from doing it again

Wiki page:

https://wiki.transitionnetwork.org/Spam_accounts

#606: Site upgrade tasks -- pre-migration cleanup

jim — Fri, 11 Oct 2013 12:00:13 GMT

This ticket is to track the issues left over from #590 that need to be considered and tackled prior to migrating the site from D6 to D7 (or 8).

Please feel free to add as needed, but sticky to the

C) Cleanup: List of features we don't really need

Ed to add his items to following list... Need rational and alternative approaches for each.

C.1) Remove 'Geographic region' and related taxonomy and Hierarchical Select modules 1 hour, low reward, low risk -- never really been used and is effectively a duplicate of the location field. let's kill it!
C.2) Kill Microsites and the Forums -- The handful of people using the CMS feature should be migrated to Open Atrium if they need such features.
C.3) Remove forums -- We could migrate the forum to a simpler setup (not using forum module) that leverages normal commenting, or even Disqus or other services to offload comments and moderation. Also encourage user-submitted ocontent and promote that if it's good or gets interesting debate.

D) Key development tasks

D.1) All inline PHP must be moved to modules and features -- This has great benefit for management, maintenance and developers. Eval()uated code is much slower than PHP in files, especially since it can't be accelerated by APC or Zend Opcode cache... We have a few blocks and many views that are loaded from the database and evaluated. Ideally the blocks would be moved to the 'Transition Extras' module, and the views would be pushed into features. This work is good to do for maintainability and D7 upgrades, too. See: http://2bits.com/api/abuse-drupal-best-practices-your-own-peril-poor-performance.html and http://2bits.com/articles/free-your-content-php-moving-php-code-out-blocks-views-and-nodes.html

D.2) Build in ESI (Edge Side Includes) support from the outset, ensure Drupal renders only what it needs to -- BOA packages the ESI (Edge Side Includes integration) module, which makes NginX cache the whole page (as it does now), but also for user-logged in pages (which it does for 5 seconds since the page data changes). This means Drupal renders the ESI component (blocks, panels panes) that are have user-specific data in. Potential boost quickly, but will need time to tweak settings to get best from this across whole site. See comments in 4 & 5 below for discussion~~, should be done after proposal F, above~~.

E) Key editorial tasks

E.1) More Taxonomy cleanup -- try to merge terms with the same names, clear out spammy terms, general spit-and-polish. Ed plus team of busy interns to do this when the time is right.

Z) old stuff for reference; tasks from #590 rendered pointless by move

Z.1) Find Variable table writes and kill them -- seeing plenty of SELECT * FROM variable calls, which imply a cache clear due to a variable being set. In normal use variables shouldn't be set (admin screens tend to do this), so I'd like to try to see what module it causing this and patch/remove it. Will need to run grep -R "variable_set() * > ~/static/variable_set-calls.txt" in the {{{sites/all directory to generate a list, then trawl though it to find candidates/bad modules practice.

#671: Replace core Search module with Apache Solr

jim — Sat, 11 Jan 2014 21:11:14 GMT

Issue & background During work on #610, it was discovered that of a 1/4GB database dump for TN.org, ~80% (180Mb) of it was related to the Drupal 6 core Search module.

It's worth noting this was raised when we migrated the site to the Puffin server in March 2013, but it's generally the case that the core Search module does not scale easily beyond a few thousand nodes.

www.transitionnetwork.org has 23,803 nodes at time of writing -- this is probably approaching the sensible limit of the core module's capability.

Note also, any future D7 or D8 version of the site would also hugely benefit from using Solr, so the server config part is time well spent.

Proposed solution

Add the Apache Solr option to BOA, re-run the installer to get it installed and configured automatically.
Add the ApacheSolr module and any related required modules to the TN D6 makefile -- it's not clear if the 6.x-3.x branch or 6.x-1.x branch is the right choice at present.
Build a new platform containing these modules, migrate a clone of STG to it.
Enable the modules, configure them, disable core Search.
Create a feature that wraps up config for Solr and required modules. Add to Git, add reference to feature to makefile
Test, tweak, repeat 3 & 4 & 5 as needed.
Migrate PROD to the new plaform, enabled feature, index site.

This could be parked until D7/8 migration, or not... Ed's call.

#519: Fixing various URL in the Database

chris — Fri, 15 Mar 2013 13:47:21 GMT

This page:

http://transitionnetwork.org/support/what-transition-initiative

Contains this HTML:

<p><img alt="TransitionSantaCruz" src="http://transitionsc.org/sites/www.transitionnetwork.org/files/pixture_reloaded_logo.png" align="right" height="69" width="150"></p>

The image is a 404:

http://transitionsc.org/sites/www.transitionnetwork.org/files/pixture_reloaded_logo.png

The correct location for the image is:

http://transitionsc.org/sites/default/files/pixture_reloaded_logo.png

Looking at the Internet Archive this was correct back in October 2012,

http://web.archive.org/web/20121022030350/http://www.transitionnetwork.org/support/what-transition-initiative

Their munged HTML contains the correct URL:

<p><a href="/web/20121022030350/http://transitionsc.org/sites/default/files/pixture_reloaded_logo.png" class="colorbox initColorbox-processed cboxElement">

It appears to me that an edit must have been done on the database something like:

s;/sites/default/files/;/sites/www.transitionnetwork.org/files/;

There might well be other URLs to other Drupal sites that were changed when they shouldn't have been?

I have had a quick look at the database dump and couldn't find any examples of this problem, but there are 113 lines to check:

grep "sites/www.transitionnetwork.org/files" /var/backups/mysql/sqldump/transitionnetwor.sql | wc -l
113

I did notice that there are a lot of URLs in the database like this:

src=\"http://www.transitionnetwork.org/sites/www.transitionnetwork.org/files/uploaded/u5857/Map-TransitionNetworkOffice.jpg\"

And

src=\"https://www.transitionnetwork.org/sites/www.transitionnetwork.org/files/uploaded/u4/transition%20companion%20cover.jpg\"

Both the above links would be better starting with / or //www.transitionnetwork.org/ as this would avoid people getting HTTPS content when using HTTP and also getting HTTP content when using HTTPS.

I think it would be worth putting the site into maintenance mode, doing a dump of the database, checking these 113 lines for issues like those above, correcting them all and then reinserting the data, however this would need to be done at a suitable time.

I'd be happy to do this task. Ed, Jim, any thoughts about when would be a good time to do it?

#626: Add redirect from an old CMS to a new URL

ed — Wed, 20 Nov 2013 12:11:14 GMT

Can Ed add a redirect from:

https://www.transitionnetwork.org/cms/haddenham

to another URL easily? Or does he need to ask Chris to do it?

#644: AWstats Nginx config breaks aegir

jim — Mon, 09 Dec 2013 16:46:05 GMT

Since the last update we've had a silent ngnix error that means http://tn.puffin.webarch.net was not available.

I restarted nginx and got:

[  ok  ] Stopping Nginx Server...:
[ .... ] Starting Nginx Server...:nginx: [emerg] "log_format" directive is not allowed here in /etc/nginx/nginx.conf:28

Which equates to the AWstats entry which is now commented out per:

# log for awstats
#log_format apache '$remote_addr - $remote_user [$time_local] "$request" '
#                   '$status $body_bytes_sent "$http_referer" '
#                   '"$http_user_agent"';
#access_log         /var/log/nginx/awstats.log apache;

I/we need access to aegir more than AWStats, so I've commented out the lines above and restarted nginx. Aegir is back and working well.

This ticket is to find the correct log_format for modern nginx versions and reinstate AWstats -- assigning to Chris as a low priority thing.

#676: Alternative to Skype for TTech Meetings

chris — Tue, 14 Jan 2014 13:33:51 GMT

Jim has pointed out that:

Skype costs us 15-30 minutes of grinding pain every time we do this!

So what are the alternatives and what are our requirements?

#716: Heartbleed

chris — Wed, 09 Apr 2014 08:53:58 GMT

Following on from ticket:692#comment:18 we should undertake the steps Drupal have taken: https://drupal.org/news/2014-04-08-security-update