Transition Technology: Ticket Query

#483: Nginx 502 Bad Gateway Errors with BOA

chris — Mon, 28 Jan 2013 11:23:45 GMT

The Barracuda Octopus Aegir server on puffin.webarch.net is getting generating a lot of Nginx 502 Bad Gateway errors, see ticket:466#comment:31 and ticket:466#comment:38

The Nginx logs contain 58 502 errors:

cd /var/log/nginx
grep " 502 " * | wc -l
58

I expect the cause of these is Nginx asking PHP-FPM for a page and not getting one, an answer might be to either speed up PHP-FPM response time or adjust the Nginx settings so that it waits longs for a response from PHP-FPM.

There are several threads about this on http://groups.drupal.org/ but none of them have any useful suggestions (eg 1 2 3 4).

#487: robots.txt files for development sites

chris — Tue, 29 Jan 2013 13:24:35 GMT

All the sites other than www.transitionnetwork.org on wiki:PuffinServer should have a robots.txt file to exclude them from being crawled and indexed to prevent the development versions of sites being included in search results.

#545: Registration page: 502

ed — Wed, 08 May 2013 08:38:13 GMT

http://www.transitionnetwork.org/user/register

found a bunch of these yesterday - this one is still showing a 502...

#548: All Admin functions broken on TN.org

ed — Tue, 14 May 2013 07:50:15 GMT

Admins, editors, social reporters cannot create content on TN.org. Choosing to create content leads to homepage. Can't do anything on the admin menu.

Emergency.

#553: Invalid response from server ERROR message

ed — Tue, 28 May 2013 15:13:43 GMT

Trying to add a user. Get error message: "Received an Invalid response from the server" then served a blank screen. Same with project profile:

https://www.transitionnetwork.org/admin/user/user/create https://www.transitionnetwork.org/node/add/project-profile

This has also happened for a user trying to add a project.

Ed can add an event and blog post, ingredient, panel page.

#562: Users being subscribed to widgets newsletter on registratoin

ed — Mon, 17 Jun 2013 15:11:22 GMT

Ed added new users to system via user-->add user. They are being automatically subscribed to the widget owners newsletter. Please investigate and stop this.

#566: Profile photo does not work

ed — Wed, 26 Jun 2013 14:17:35 GMT

Using this user account: https://www.transitionnetwork.org/users/patricia-benson https://www.transitionnetwork.org/people/patricia-benson

I can't get the photo to sit correctly. I have already deleted her first user account as the first photo she put in was sideways - the system presented that the right size, but sideways. Then I couldn't get the system to forget the original photo so had to delete the account and start again. Now I've tried adding bigger, smaller pictures and they all come out far too big - the system is enlarging them up to wider than the space alloted.

Very odd. Too tired and busy to do this now - will revisit in July after holiday.

#568: TC blog: can't not send notifications

ed — Mon, 08 Jul 2013 10:20:16 GMT

I can't see how to do a node edit and *not* send a notification - here seen on the Transition Culture blog:

https://www.transitionnetwork.org/node/32863/edit

please advise - there's been some kerfuffle about the TC blog move so swift resolution much appreciated

#569: 403s served to editors, admin very slow

ed — Tue, 09 Jul 2013 07:52:32 GMT

Rob is getting 403s when trying to submit his work. Report from 07:12am this morning (Tuesday)

Ed tried to add a blog post at node add:

https://www.transitionnetwork.org/node/add/blog It took nearly 15 seconds to get this published https://www.transitionnetwork.org/blogs/ed-mitchell/2013-07/eds-test-blog-item-check-403

Running admin functions takes ages.This request took well over 30 seconds:

https://www.transitionnetwork.org/admin/content/node/overview

Please advise?

#575: User system is blocking everyone - needs fixing urgently

ed — Wed, 31 Jul 2013 14:39:11 GMT

Everyone who registers for the last two weeks is being blocked - check user list and see screengrab. Needs fixing promptly.

#576: Site down

ed — Thu, 01 Aug 2013 09:42:44 GMT

Been down for 15 minutes by now I reckon

#583: tmp files on parrot exceeded inode limit

ed — Mon, 02 Sep 2013 12:28:46 GMT

Users are attempting to purchase a DVD and the cart always zeros out even after attempts to update cart etc.

Laura suggests that this may be a server issue and to start with Chris:

The prob could v likely be on the hosting side, so be worth contacting Chris first (and mention can add to cart all okay when logged in, view cart and alter and get to billing page, so could be a session setting that's changed) -

Does the server have disk space left on /tmp. Session data is most likely written there. (sometimes this cart faff issue can happen because of that.)

Are the PHP Session settings correct/ have they changed recently? Chris can fix this if needing a tweakette in php.ini or other places in the underworld of those types of files.

DB could potentially need check/repair.

Another one is the shop element relies on cookies so if a user has set browser to not accept cookies may poss not work! (but think thats something not to think about here, as I tested as both logged in and logged out with cookies set to work on my browser, logged out got the 'cart empty' faff factor like other users will be getting.)

If Chris can't spot the bug (does sound server side and sessions related from a quick test here, haven't looked at the server logs, but if get a mo later will see if I can), poss good to chat and I can think some more on it, but first thoughts alert me mostly to server side, sometimes a roll back to php 5.2 clicks things back into place (but shouldn't be needed). Be good to see if there is a correlation with any server changes/tweaks happening around same timing that the cart stopped working. Keep me in the loop if it is a server fix-y thing as always interested to hear results on bug fixing on sites.

#584: News CTs and featured images: not handling as expected

ed — Wed, 04 Sep 2013 08:33:12 GMT

The news CT has a featured image, but it doesn't seem to be handling it as expected (succesfully on the blogs). See this news item:

https://www.transitionnetwork.org/news/2013-09-03/new-economy-20-enterprises-report-released

The featured image isn't visible anywhere - either:

in the news item itself
in the /news view

Please investigate and resolve

#592: Many panes on homepage are in italics

ed — Wed, 11 Sep 2013 07:29:50 GMT

they shouldn't be, but they are: ingredient of the day latest tc post featured training why do transition featured resrouce featured project social reporters all automated listings at bottom

please resolve - ben are you around to do this or is it more suitable to ask jim atm?

#602: RSS problems

ed — Thu, 03 Oct 2013 09:14:43 GMT

Blogs feed not working:

https://www.transitionnetwork.org/blogs/feed/

Rob's blog not working:

https://www.transitionnetwork.org/blogs/feed/rob-hopkins

Getting complaints from users who are getting multiple copies of each blog post. Can't check this as the feed is down, but I've got 2 or 3 complaints in my inbox.

#608: User profile pictures are too big

ed — Mon, 14 Oct 2013 08:37:38 GMT

The user profile pictures on the user profile nodes are too big.

e.g. https://transitionnetwork.org/people/tina-clarke

Please investigate and restrain them into a suitable maximum size

NB: Ed is keeping a number of small UI-ish tickets back for Ben to do in a lump when we've built up enough so don't do anything yet.

#609: Videos not showing if added via video link

ed — Tue, 15 Oct 2013 08:33:43 GMT

e.g. of project profile: https://www.transitionnetwork.org/projects/new-forest-transition-school-community-energy-saving-project

none showing on video page: https://www.transitionnetwork.org/video

#610: Aegir database intensive (migrate, clone, restore) tasks hang for larger sites

jim — Tue, 15 Oct 2013 10:18:59 GMT

Large sites (TN.org and variants) will simply not complete their migrate, clone or restore tasks in Aegir.

However, smaller sites are fine, and all tasks work for them.

The process largely completes -- codebase installs, database is cloned, symlinks for sites aliases and files created... BUT the process never completes in Aegir, so the final steps of switching a site's served location never occurs.

Useful links/comments in this issue:

#622: Maps not working on TN.org

ed — Mon, 18 Nov 2013 17:06:59 GMT

https://www.transitionnetwork.org/initiatives/map https://www.transitionnetwork.org/initiatives/exeter-nh-transition-town

please attend

#654: TIs appearing in 'TIs by list' view without a number

ed — Thu, 12 Dec 2013 12:43:12 GMT

Some TIs are appearing in the 'Initiatives by number' view - but don't have numbers:

https://www.transitionnetwork.org/initiatives/by-number

for Sam to invesitage and solve or put to techs in January

#657: Homepage slideshow: can't order the images

ed — Mon, 16 Dec 2013 14:30:49 GMT

Big picture slideshow on TN homepage: Rob can't get it to go in the order he wants - please investigate:

It is this nodequeue: https://www.transitionnetwork.org/admin/content/nodequeue/9/view/8

Made up with these CTs: https://www.transitionnetwork.org/admin/content/node-type/slide

#658: Check caching on Social Reporters views

ed — Mon, 16 Dec 2013 14:59:38 GMT

The SRs are reporting very slow site updates - is this to do with not switching them over to the new views content caching? please check:

https://www.transitionnetwork.org/admin/build/views/edit/blogs?destination=stories#views-tab-panel_pane_2

https://www.transitionnetwork.org/admin/build/block/configure/views/blogs-block_1?destination=newhome

And elsewhere there may be SR latest views?

#659: Featured story on homepage not updating

ed — Mon, 16 Dec 2013 17:03:41 GMT

Rob cleared the nodequeue for the featured story on the homepage about three hours ago, removed all but one story. No change has been observed. Please investigate:

Nodequeue: https://www.transitionnetwork.org/admin/content/nodequeue/6/view/5

This view: https://www.transitionnetwork.org/admin/build/views/edit/featured_content?destination=newhome#views-tab-panel_pane_1

This is using content cache

#679: Filter Initiative by Country not working

sam — Fri, 17 Jan 2014 17:15:58 GMT

Filtering the initiatives by country is not working

https://www.transitionnetwork.org/initiatives?themes=All&community_type=All&status_value=All&country=at&field_title_search=

I had a play with the view on stg2.transitionnetwork.org but I couldn't get it working.

It should display only initiatives from the country selected in the filter. It is displaying initiatives from all countries & not respecting the filter.

This view is used a lot by international initiatives.

#695: File upload problem with TTT WordPress site

chris — Sun, 02 Mar 2014 19:46:50 GMT

Laura has reported:

Had a report this evening that the TTT website isn't letting any uploads happen (adding of a new plugin by another admin, and also media/image uploads not being able to uploaded - no real error message as such just doesn't upload).

I just logged in quickly to see if I could upload a pic to test and wouldn't upload. Just wanted to check if anything had changed server side re permissions.

#698: intransitionmovie.com returns 405 on submit

sam — Thu, 06 Mar 2014 11:45:39 GMT

http://www.intransitionmovie.com/screenings/share-your-screening/

vid Atcheson (david@…) sent a message using the contact form at https://www.transitionnetwork.org/contact.

I couldn't find any contact info at http://www.intransitionmovie.com/, but am hoping you can help. When I visit http://www.intransitionmovie.com/screenings/share-your-screening/index.html and put in my email address and hit Enter, I get:

Hi I checked this and it returns a 405 error for me too.

http://www.intransitionmovie.com/screenings/share-your-screening/

405 Not Allowed nginx/1.4.4

Hoping this can be fixed so we can let you know about our screening on April 8 at the Hawaii State Capitol auditorium.

Aloha,

David Atcheson, Transition Oahu

I'll do some initial investigation

#699: Update Core & Plugins on transitionculture.org

sam — Mon, 10 Mar 2014 11:46:33 GMT

This site needs an update.

I'll do a database backup then update it.

#702: Attachments not being deleted from Trustees page

ed — Fri, 21 Mar 2014 17:11:14 GMT

I can't remove the attachments from the Trustees page: https://www.transitionnetwork.org/about/people/trustees

please investigate and sort if you can. We need to remove all the attachments apart from those in 2013.

#703: Image not scaling on project

ed — Wed, 26 Mar 2014 13:06:15 GMT

The REconomy logo is not scaling and being re-presented well on the project page: https://www.transitionnetwork.org/projects/reconomy

Is this a theme/design issue or a problem with image re-sizing?

What can we do about it?

#704: Image not scaling on project

ed — Wed, 26 Mar 2014 13:51:48 GMT

The REconomy logo is not scaling and being re-presented well on the project page: https://www.transitionnetwork.org/projects/reconomy

Is this a theme/design issue or a problem with image re-sizing?

What can we do about it?

#708: Map not showing Indian initiative

sam — Thu, 27 Mar 2014 13:44:29 GMT

https://www.transitionnetwork.org/initiatives/heal-soil-csa-community-supported-agriculture Address is correct, but doesn’t appear. The person who put it there doesn’t appear on the people map either, so wondering what’s afoot.

#718: REconomy site showing adverts randomly

ed — Thu, 10 Apr 2014 08:19:34 GMT

Load http://www.reconomy.org and the first time you get it, and other times at random and you get spam.

URGENT check please - on REconomy and all WP sites...

#724: Subscription emails from Rob's blog not arriving.

sam — Tue, 29 Apr 2014 09:21:12 GMT

A user has got in touch to let us know he is not getting Rob's updates by email. He's still subscribed.

I just checked my mail and the last one I received was on the 11/4. I'll have a look and see if I can work it out.

#748: Urgent: Fiona having problems sending email

sam — Wed, 25 Jun 2014 12:42:27 GMT

Hi Chris, Fiona is having problems sending email. I have checked the quota in the web interface and she is not over her quota.

She initially sent this:

The message could not be sent because the server rejected the sender's email address. The sender's email address was 'fionaward@…'.

Subject 'Re: filming' Server Error: 550 Server Response: 550 Access denied - Invalid HELO name (See RFC2821 4.1.1.1) Server: 'mail.transitionnetwork.org' Windows Live Mail Error ID: 0x800CCC78 Protocol: SMTP Port: 26 Secure(SSL): No

Which looked to me like the email had been sent and rejected by the receiving server.

So I sent this to Fiona:

On 24 June 2014 16:11, Sam Rossiter <samintransition@…> wrote: Hi Fiona I'll take a look now. It could have been rejected for a number of reasons. There's no keywords in the email that might trigger a spam rejection? Or have you Cc'd a large number of people?

As a workaround until we get to the bottom of it is it something a collegue could forward? Or does it have to come from you?

Thanks

Sam

Fiona replied:

Hi Sam I don;t think it's that email, as there are 13 that won't send frm my outbox even if I remove this email - same problem with the next one! Thanks Fiona

---

Chris I'm not working until next Tuesday, so I was wondering if you could pick this up and liase with Fiona? fionawardttt [AT] googlemail.com

Thanks

Sam

#751: Email alert changes

ed — Tue, 01 Jul 2014 11:14:42 GMT

Change the email alert template for news items to include the term 'news item' so it is:

"New news item: [title]"

When users click on the subs links at the bottom of their email alerts, and they are not logged in they get the ‘access denied’ screen. This is not good. Please investigate

(a) can this be changed (with small time investment) (b) can we change the access denied blurb to include something human encouraging the user to login to continue the journey:

"We are sorry for the inconvenience, but if you are seeing this screen having followed a link, you will probably need to login to continue with your request" (NB if they are coming from an email link with their id in it, how do we keep that journey so they get to the destinateion they wanted?)

When user clicks on the general subs link at the bottom of an email alert (and is not logged in), they get a 403 forbidden page. Not good - pls investigage

#755: Site administrator's file upload capacity: not responding to changes

ed — Tue, 08 Jul 2014 09:21:16 GMT

Rob Hopkins reached the limit for file uploads and could not upload any more images (not good for him as he writes posts with lots of pictures daily).

Sam raised the limit. This change in the limit does not seem to have affected Rob's situation - he still can't upload any images.

Looking at these settings: https://www.transitionnetwork.org/admin/settings/uploads

Rob should be able to upload more - he is a site editor and admin - but he can't.

Please investigate

#771: cannot access puffin over sftp again

annesley — Tue, 05 Aug 2014 09:24:10 GMT

not sure what happened this time. my connection to other servers is fine. i am using:

sftp://tn.ftp@puffin.transitionnetwork.org/static/transition-network-d6-32-s002-booker/

the password was accepted first time and i successfully navigated to /static

which worked until i tried to get to the transition-network-d6-32-s002-booker directory. now ssh fails also.

when i ssh in i get plenty of warnings and potential lock outs when i try various things. like trying to access /static.

#777: Comments to blog post only showing up when logged in

ed — Mon, 25 Aug 2014 09:18:39 GMT

Most of the comments to this blog post are only showing up when you are logged in: https://www.transitionnetwork.org/blogs/rob-hopkins/2014-07/fiona-ward-learning-celebrate-10000-failure#comment-17492

Logged in: 5 comments Not logged in: 1 comment

Handing this to Sam but happy for it to escalate. Presumably this issue won't be on this one post only - and it is important.

#780: parrot

annesley — Wed, 27 Aug 2014 14:35:51 GMT

can't access parrot again also: it re-requests my password very quickly. what is the session timeout?

also: i cannot include my files. seems that the website is located under the /home/annesley directory. is this some sort of strange setup because the PHP cannot see that area. and i cannot cd to the root through SFTP...

#837: Iframe in a panel page

sam — Thu, 12 Mar 2015 11:55:21 GMT

Hi Ben

I'm trying to embed a Eventbrite form into the 'tickets' block on this page: https://www.transitionnetwork.org/conference-2015

It looks like it's going to appear in the preview here: https://www.transitionnetwork.org/node/39195/panel_content

But when I view the actual page it's just a big white space.

Could you estimate how long it would take to get it working?

Thanks

Sam

#862: Puffin locked

chris — Sat, 27 Jun 2015 15:31:27 GMT

PuffinServer is not responding, I got a Munin email alert, on the Xen console:

[2008077.910371] BUG: soft lockup - CPU#1 stuck for 61s! [munin-node [::f:25444]
[2008077.910371] Modules linked in: joydev sg st sd_mod crc_t10dif sr_mod scsi_mod ide_gd_mod ide_cd_mod ide_core cdrom xt_recent xt_tcpudp xt_connlimit nf_nat_ftp ipt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp ipt_REJECT ipt_LOG xt_limit xt_multiport iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext4 crc16 jbd2 mbcache dm_mod xen_netfront xen_blkfront
[2008077.910371] CPU 1:
[2008077.910371] Modules linked in: joydev sg st sd_mod crc_t10dif sr_mod scsi_mod ide_gd_mod ide_cd_mod ide_core cdrom xt_recent xt_tcpudp xt_connlimit nf_nat_ftp ipt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp ipt_REJECT ipt_LOG xt_limit xt_multiport iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext4 crc16 jbd2 mbcache dm_mod xen_netfront xen_blkfront
[2008077.910371] Pid: 25444, comm: munin-node [::f Not tainted 2.6.32-5-xen-amd64 #1
[2008077.910371] RIP: e030:[<ffffffff8100922a>]  [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
[2008077.910371] RSP: e02b:ffff8800988f7ba8  EFLAGS: 00000246
[2008077.910371] RAX: 0000000000040000 RBX: ffffea0006c2eb88 RCX: ffffffff8100922a
[2008077.910371] RDX: 00000000ffffff00 RSI: 0000000000000000 RDI: 0000000000000000
[2008077.910371] RBP: 0000000000000002 R08: 0000000000000002 R09: ffff8801ffc1dd00
[2008077.910371] R10: 0000000000000002 R11: 0000000000000246 R12: ffff88000000ad00
[2008077.910371] R13: ffff880000008000 R14: 0000000000000200 R15: 000000000000000e
[2008077.910371] FS:  00007fefd0bde700(0000) GS:ffff88000bb20000(0000) knlGS:0000000000000000
[2008077.910371] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[2008077.910371] CR2: 00007fefce124380 CR3: 0000000001001000 CR4: 0000000000000660
[2008077.910371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2008077.910371] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[2008077.910371] Call Trace:
[2008077.910371]  [<ffffffff810baa62>] ? free_hot_cold_page+0x1a2/0x1af
[2008077.910371]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[2008077.910371]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[2008077.910371]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[2008077.910371]  [<ffffffff8130f142>] ? _spin_unlock_irqrestore+0xd/0xe
[2008077.910371]  [<ffffffff810bd9ca>] ? release_pages+0x16a/0x18d
[2008077.910371]  [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
[2008077.910371]  [<ffffffff810da555>] ? free_pages_and_swap_cache+0x57/0x73
[2008077.910371]  [<ffffffff810cd5bf>] ? unmap_vmas+0x6cb/0x959
[2008077.910371]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[2008077.910371]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[2008077.910371]  [<ffffffff810d1bca>] ? exit_mmap+0xc4/0x148
[2008077.910371]  [<ffffffff8104cd95>] ? mmput+0x3c/0xdf
[2008077.910371]  [<ffffffff81050a2e>] ? exit_mm+0x102/0x10d
[2008077.910371]  [<ffffffff81052453>] ? do_exit+0x1f8/0x6c9
[2008077.910371]  [<ffffffff8105299a>] ? do_group_exit+0x76/0x9d
[2008077.910371]  [<ffffffff810529d3>] ? sys_exit_group+0x12/0x16
[2008077.910371]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[2008381.335662] Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=122.172.30.208 DST=81.95.52.103 LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=30880 DF PROTO=TCP SPT=60400 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
[2008384.465057] Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=122.172.30.208 DST=81.95.52.103 LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=30881 DF PROTO=TCP SPT=60400 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
[2008390.255448] Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:19:68:02:00:12:1e:13:6c:db:08:00 SRC=122.172.30.208 DST=81.95.52.103 LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=30882 DF PROTO=TCP SPT=60400 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0

#888: Adverts on Transition Network Front Page loaded via flickrit.com embedded content

chris — Sun, 06 Dec 2015 12:25:49 GMT

It it intentional or accidental that adverts from https://secureads.bitbillions.com/ are being loaded on the front page of https://www.transitionnetwork.org/ via the embedded content from flickrit.com?

#895: HTTPS wildcard *.transitionnnetwork.org expires on 22nd January 2016

chris — Mon, 11 Jan 2016 09:56:17 GMT

Unless I hear otherwise I'll renew the *.transitionnnetwork.org cert which is used by PuffinServer, PenguinServer and ParrotServer at a cost of £130.50 on or before the 22nd January 2016 when the current one expires.

An alternative would be to use Free HTTPS certificates from Let's Encrypt but this would take some time to set up as Let's Encrypt don't provide wild card certs.

#908: Please enable Google Analytics

sam — Thu, 10 Mar 2016 15:43:56 GMT

Hi all

The web project board has decided to move from Piwik to Google Analytics.

I realise this has significant privacy implications, these were discussed, but it was decided that the move would go ahead.

I have discussed with Ainslie adding an opt out in the form of a link to EFF's Privacy badger: https://www.eff.org/privacybadger This will be added to the cookie pop up bar in the new website.

Could we enable the Google Analytics module on the existing site so I can add in the tracking code.

Thanks

Sam

#911: Disk space for /home on Parrot is running out

chris — Mon, 30 May 2016 20:58:53 GMT

Getting this alert from ParrotServer every 5 mins:

transitionnetwork.org :: parrot.transitionnetwork.org :: Disk usage in percent
        WARNINGs: /home is 96.06 (outside range [:96]).
        OKs: /run/shm is 0.00, /run is 0.09, /dev is 0.00, / is 95.94, / is 95.94, /run/lock is 0.00.

#502: patterns sub-domain for transitionresearchnetwork.org, was: Setting up specific pointer for URL

ed — Wed, 27 Feb 2013 07:10:03 GMT

Can we set this URL:
www.transitionresearchnetwork.org/research-guidelines
to point to this location:
https://cldstr.com/docs/configure-dns
While this URL:
www.transitionresearchnetwork.org/
Still points to the current location.
~~~~
Ed Mitchell: Transition Network Web manager
web: http://transitionnetwork.org
project blog: http://transitionnetwork.org/blogs/ed-mitchell
mobile: +44 (0)7807 141 828
office hours: Mon, Tues, Wed, Thurs
company no. 6135675  charity no. 1128675
Subscribe to Transition news: http://tinyurl.com/transitionregister
~~~~

#560: Install drupal-based project management system onto our servers

ed — Tue, 11 Jun 2013 17:00:28 GMT

Please can you install a suitable project management drupal-based tool onto the suitable server? I am thinking of Open Atrium. It's for staff and partners to use for management stuff (ie not a ticketing system or mediawiki).

Open Atrium?

Can you let me know how long it would take to install to a point where I can manage it and get a pilot project up and running?

#571: Force HTTP for anonymous, HTTPS for logged in users

jim — Tue, 16 Jul 2013 11:59:54 GMT

To further reduce load and leverage the caching I've changed the Session 443 settings to force anon users to HTTP and logged in to HTTPS. The benefit of allowing a handful of users to choose is tiny compared the downsides of outages, 503s and higher load.

The "User state" setting on the config page is now " Redirect authenticated users to HTTPS and redirect anonymous users to HTTP (with the exception of login/registration pages).", was "Redirect authenticated users to HTTPS and redirect anonymous users on login/registration pages to HTTPS. Anonymous users visiting other pages may use HTTP or HTTPS."

I've also set user and site-wide contact forms, plus the mailchimp subs page force secure-only per the "Additional pages to make secure" setting.

We can see if this makes a difference, and this ticket is to track comments and see if it results in any improvement in performance/stability.

#579: Special email alert for Rob Hopkins blog posts on TN.org

ed — Mon, 12 Aug 2013 13:49:30 GMT

Can we create a specific email alert for new blog posts from Rob Hopkins' blog (/blogs/rob-hopkins)?
Can Ed do this?
See below:

Proposed new alert: ~

From: [Rob Hopkins at Transition Network <site@…>] Subject: New blog post from Rob Hopkins' Transition Culture To: [subscriber email] Reply to: [<site@…>]

Hello. I've written a new blog Transition Culture blog post

You can read it and leave comments here: [LINK]

[BLOG POST TITLE] [LINE BREAK - FULL CLEAR LINE] [BLOG POST TEASER] [LINE BREAK - FULL CLEAR LINE]

Best regards, Rob

-- [TN EMAIL SUBSCRIPTION AUTO-BLURB] This is an automatic message from Transition Network To manage your subscriptions, browse to http://www.transitionnetwork.org/user/285/notifications You can unsubscribe at http://www.transitionnetwork.org/notifications/unsubscribe/subscription/...

#614: Subscription option for Social Reporters CT

ed — Wed, 06 Nov 2013 16:49:49 GMT

We want a button on the Social Reporters section that enables users to easily subscribe to the Social Reporters' blog posts CT. Offers the option to all types of user:

Users may not be logged in
Users may not be registered

This is a bit like the sub button on Transition Culture section
I can't find it in TRAC but have this memory of enabling email-only subscriptions to not-registered-users from olden days - can you enlighten me about that? I'm not sure it's the answer - we probably want them to register so they can add comments - but this request has brought up that old function

#621: Check and speed up updates to homepage sections: TC and slideshow

ed — Mon, 18 Nov 2013 09:22:59 GMT

Two bits of the homepage are taking ages to update once Rob has made the changes and it's driving him spare. He's waiting up to 30 mins for them to update.

Please speed up the times that it takes the system to update these two parts of the homepage to *instant* if possible:

Slideshow on homepage (AKA: https://www.transitionnetwork.org/admin/build/views/edit/slideshows?destination=newhome#views-tab-panel_pane_1).

Latest Transition Culture Blog pane on Homepage (AKA https://www.transitionnetwork.org/admin/build/views/edit/tc_latest_blog?destination=newhome#views-tab-block_1)

#643: Add paginator for /films view

ed — Mon, 09 Dec 2013 11:40:08 GMT

Please add pages links for this view: https://www.transitionnetwork.org/admin/build/views/edit/Films?destination=films#views-tab-page_1

10 per page should do it

#744: Add CSS Injector module to the D6 mix

annesley — Thu, 19 Jun 2014 10:50:03 GMT

CSS Injector module allows admin dynamic adding of CSS in to a live production server without code updates. this is to respond quickly to client needs whilst implementing the requests in the correct place in the themes on the next promotion from dev.

#207: Logwatch Issues

chris — Fri, 17 Dec 2010 14:19:50 GMT

This is a ticket to track items that show up in the logwatch emails to root and often just take a few mins of reading time and response.

Any issues that look like they might take longer than a few mins should have their own tickets opened.

#208: Dblog Issues

chris — Fri, 17 Dec 2010 14:53:29 GMT

"The dblog module monitors your website, capturing system events in a log to be reviewed by an authorized individual at a later time. The dblog log is simply a list of recorded events containing usage data, performance data, errors, warnings and operational information. It is vital to check the dblog report on a regular basis as it is often the only way to tell what is going on."

https://www.transitionnetwork.org/admin/reports/dblog

This is a ticket to be used to flag up issues that are not going to take up enough time that they justify their own ticket -- if an issue is raised in a comment on this ticket that does look like it's going to take some significant time then best start a new ticket for it.

#218: Debian upgrades and updates

chris — Thu, 06 Jan 2011 13:34:16 GMT

This is a ticket to track debian upgrades to the wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer the time they take.

See:

These updates are generally done using the wiki:AptitudeUpdateScript and this records all the changes in the /root/Changelog and then the contents of the Changelog are pasted into the ticket to document the upgrade.

This ticket was was originally used for the ~~wiki:DevelopmentServer~~ and the ~~wiki:NewLiveServer~~.

#371: Piwik Hosting

chris — Mon, 05 Dec 2011 21:34:51 GMT

Another option to consider for Piwik would be be to use a instance on another server, and there happens to be one here that could be used:

https://stats.webarch.net/

Could this be considered?

#382: Measurement and tracking requirements for TN and PSE

ed — Tue, 13 Dec 2011 04:57:12 GMT

to list the measurement and tracking requirements for TN and PSE so that we can assess piwik vs google and decide which way to go.

#470: Penguin install and configuration

chris — Sat, 15 Dec 2012 16:05:16 GMT

penguin.webarch.net is a new 2GB RAM virtual server which will replace NewLiveServer and DevelopmentServer for running all non-Drupal sites and is due to go live in early 2013. Drupal sites from the old servers will be migrated to PuffinServer.

This ticket has been created for tracking time and tasks done during the install.

See https://tech.transitionnetwork.org/trac/wiki/PenguinServer and also the corresponding PuffinServer.

#494: Email account for TRAC

ed — Thu, 21 Feb 2013 10:16:37 GMT

Set up: 'trac@…' Password: going to Chris separately Secure SSL/TLS Settings Username: trac@… Password: Use the email account’s password. Incoming Server: mail.xssl.net IMAP: Port 993 POP3: Port 995 Outgoing Server: mail.xssl.net SMTP: Port 465 Authentication is required for IMAP, POP3, and SMTP.

#529: New Barracuda BOA-2.0.7 Edition available

chris — Fri, 05 Apr 2013 08:57:09 GMT

There is new BOA-2.0.7 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

#530: New Barracuda BOA-2.0.8 Edition available

chris — Mon, 08 Apr 2013 07:44:51 GMT

If I had known this was about to come out I would have waited before doing the BOA-2.0.7 upgrade last night on ticket:529

There is new BOA-2.0.8 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

Is mostly to fix a problem for people using Percona, https://drupal.org/node/1962690 and as we are using MariaDB this isn't an issue for us.

Other updates in this version:

Allow to use [a-z0-9] subdomains and not only [www] for IDN domain names.
Change the interval between platforms builds from 5 to 3 seconds.
Forced 1s Speed Booster TTL for vhosts behind local proxy is deprecated.
Move old firewall logs to backups to avoid crazy load after upgrade.
Nginx: Better exceptions handling in the Abuse Guard for js/shs modules.
PHP: CLI is at 5.3 since BOA-2.0.4, so symlink old 5.2 binary path to 5.3
Update _LENNY_TO_SQUEEZE major upgrade procedure.
Update contrib with login_security-7.x-1.2
Use static downloads for all distros in stable edition.

I'll do this update tonight unless there are any objections, hopefully it should be quite quick.

#531: Disk usage on puffin

chris — Wed, 10 Apr 2013 13:12:24 GMT

The disk usage on puffing is currently at 85% and it's been going up at around 5% a week, see:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/df.html

This will become a critical issue in a couple of weeks, it would be good to find and address the cause before then.

#532: Upgrade to Mediawiki 1.19.5

chris — Mon, 15 Apr 2013 13:40:34 GMT

New version out tonight:

This is a notice that on Monday, April 15th between 20:00-21:00 UTC (1-2pm PDT) Wikimedia Foundation will release security updates for current and supported branches of the MediaWiki software. Downloads and patches will be available at that time, with the git repositories updated later that afternoon. CVSS scores are between 4.3 and 7.1, most users will want to update.

http://lists.wikimedia.org/pipermail/mediawiki-announce/2013-April/000126.html

#534: Add accounts for Ben

chris — Fri, 26 Apr 2013 17:16:23 GMT

Ben is a new member of the TTech team and accounts need setting up for him, this is a ticket to track the time taken to do this.

#535: Upgrade Puffin, Penguin and Parrot from Debian Squeeze to Wheezy

chris — Fri, 26 Apr 2013 18:49:16 GMT

Debian 7 is due out on 5th May 2013 and wiki:PuffinServer, wiki:PenguinServer and wiki:ParrotServer will need upgrading.

#536: Upgrade Mediawiki to 1.19.6

chris — Mon, 29 Apr 2013 20:35:32 GMT

A new version of Mediawiki is due out tomorrow:

This is a notice that on Tuesday, April 30th between 20:00-21:00 UTC (1-2pm PDT) Wikimedia Foundation will release security updates for current and supported branches of the MediaWiki software. Downloads and patches will be available at that time, with the git repositories updated later that afternoon.

http://lists.wikimedia.org/pipermail/mediawiki-announce/2013-April/000128.html

The upgrade should simply be a matter of following the steps taken last time, see ticket:532#comment:2

#538: intransitionmovie.com checks and updates

chris — Tue, 30 Apr 2013 20:02:03 GMT

The http://www.intransitionmovie.com/ site has been migrated to wiki:ParrotServer, the DNS might not have yet updated for everybody.

Laura -- could you check that the site is working OK if you get a chance, especially the ecommerce elements?

There are some outstanding updates, WordPress is version 3.3.2 need updating to 3.5.1 and there are 12 plugins with updates available:

18 installed plugins:
 UA affiliates
 UA affiliates-woocommerce-light
 UA akismet
  A backupwordpress
 UI bad-behavior
 UA bwp-minify
  A contact-form-7
 UA event-o-matic
  A transposh-translation-filter-for-wordpress
 UA usernoise
  A woocommerce
 UA woocommerce-export-csv
  A woocommerce-currency-converter
 UA woocommerce-dynamic-pricing
  A woocommerce-delivery-notes
 UA woocommerce-shipping-table-rate
 UA wp-customer-reviews
 UI wp-super-cache
Legend: A = Active, I = Inactive, U = Update Available

I'm happy to do all these updates but I wouldn't know what exactly to check to see that the updates don't break anything -- would Laura be able to help with a quick check that updates haven't broken anything?

#539: REconomy site migration and updates

chris — Wed, 01 May 2013 10:01:00 GMT

This ticket is to keep track of the tasks undertaken and the time spent on the migration of the http://www.reconomy.org/ site to wiki:ParrotServer.

#542: Parrot RAM

chris — Wed, 01 May 2013 20:35:50 GMT

I'm concerned that wiki:ParrotServer might not be able to cope with load spikes due to it only having 1GB of RAM, these are the Munin graphs we need to key an eye on:

If the server runs out of RAM it will basically stop responding.

#543: Puffin Load Spike

chris — Fri, 03 May 2013 20:48:16 GMT

Puffin just had a massive load spike up to 65 and then it calmed down and it's up in the 70's again:

uptime
 21:46:43 up 91 days,  3:01,  4 users,  load average: 73.45, 30.76, 28.09

I'm trying to find the cause, it's making the TN site unresponsive.

#544: CSF / LDF false positive blocks on Puffin

chris — Sat, 04 May 2013 11:02:53 GMT

Ticket to keep track of CSF /LDF issues on Puffin, see wiki:PuffinServer#CSFLDF

#546: Trac documentation

chris — Wed, 08 May 2013 18:47:09 GMT

Better documentation of things like how to set billing times etc

#547: New Barracuda BOA-2.0.9 Edition available

chris — Fri, 10 May 2013 11:28:17 GMT

Do we want to upgrade to Wheezy, see ticket:535 at the same time as upgrading to BOA-2.0.9? Or should we wait a few weeks, as Jim suggested here ticket:535#comment:3

I think we should schedule this update (to BOA-2.0.9) for an evening when there isn't much traffic on the site, perhaps Saturday or Sunday night. I don't think this update is super urgent as we have already upgraded Nginx, see ticket:218#comment:80

There is new BOA-2.0.9 Edition of Barracuda and Octopus available.

Please review the changelog and upgrade as soon as possible to receive all security updates and new features.

Changelog: http://bit.ly/newboa

The Changelog contains:

# This is the first Barracuda-only Edition, released to address important
  security issue with Nginx server and provide system level upgrades.
  This Edition will not upgrade Aegir Master nor Aegir Satellite Instances,
  because there was no new Drupal core released since BOA-2.0.8 Edition and
  there were not enough updates to built-in platforms or contrib accumulated.
  Releasing Barracuda-only Edition separately from full Edition allows us
  to address system/services security issues without any extra delay,
  while releasing Octopus-only Edition will allow us to provide Drupal core
  or Aegir version upgrades, without affecting system level services.
  There is also another reason while separate releases will be useful.
  BOA-2.0.9 is the last Edition where Aegir 2.x uses still old Drush 4.6
  in the backend. We need to sync BOA specific Aegir 2.x with upstream
  and finally switch to Drush 5, or even Drush 6, if possible.
  This change, however, may cause issues if you still host legacy Drupal 5
  or some old Drupal 6 sites, with either core or contrib not compatible
  with PHP 5.3, which is now used by default.
  That is why we plan to introduce ability to install older/previous
  Barracuda and/or Octopus release, if you need more time to upgrade.
# New features and enhancements in this release:
  * Debian 7.0 Wheezy support.
  * Automated upgrade from Squeeze with _SQUEEZE_TO_WHEEZY=YES option.
  * Added config template with inline how-to in docs/cnf/barracuda.cnf
  * Added config template with inline how-to in docs/cnf/octopus.cnf
  * Added passwords encryption how-to in docs/BLOWFISH.txt
  * Added the list of symbols used on install in docs/PLATFORMS.txt
  * Forced mysql restart if there are too many high CPU mysqld processes.
  * Improved docs/NOTES.txt
  * Improved docs/README.txt
  * Install libpam-unix2 and libxcrypt1 by default.
  * Install s3cmd by default.
  * Issue #1974640 - Allow to use Midnight Commander for limited shell users.
  * Limited Shell Logs Monitor enabled by default.
  * Nginx: Check for Linux/Cdorked.A malware and delete if discovered.
  * Re-generate and sync Aegir passwords before and after instance upgrade.
  * The silent 'system' mode documented in docs/UPGRADE.txt
  * Allow to exclude platform from otherwise forced `drush en entitycache -y`
    if sites/all/modules/entitycache_dont_enable.info control file is present.
# Changes in this release:
  * Nginx 1.5.0 - security upgrade for CVE-2013-2028
  * PHP 5.3.25
  * Redis 2.6.13
  * Do not disable update module in platforms known to include it as required.
  * Firewall: Open port 1129 for outgoing connections (some gateways need it).
  * Force syslog module as disabled by default and save some disk I/O.
  * Tune kernel to always use max RAM and not swap, if possible.
# Fixes in this release:
  * Add outgoing port 25 SMTP to the list of requirements.
  * Firewall: Add truly permanent block for heavy abusers.
  * Fix for mytop support, available again on systems with MariaDB.
  * Fix permissions in the /data/all tree if required.
  * Fix the order of checks - they scan only the last (current) minute.
  * Force _STRONG_PASSWORDS=NO if locales still look broken on second check.
  * Improve detecting no longer running drush.php and/or cron PHP processes.
  * Improve fix_locales logic.
  * Improve global.inc symlinking on initial install and upgrade.
  * Improve messages displayed when fix_locales discovers broken locales.
  * Improve monitoring to avoid duplicate entries on low traffic systems.
  * Improve sanitize_string() filtering to avoid issues with strong passwords.
  * Improve syncpass tool - Update system user passwd and flush privileges.
  * Issue #1961226 - Warning: Could not change permissions of sites/all to 751.
  * Issue #1962458 - 403 for anonymous users on node/add.
  * Issue #1963044 - Force UTF-8 locales if not present/configured properly.
  * Issue #1974542 - Use /root/.home.no.wildcard.chmod.cnf control file.
  * Issue #1987936 - Restore ability to install PHP 5.2 for FPM and CLI.
  * Make sure that /dev/null is writable for everyone.
  * Make sure that all drushrc.php files are owned by Aegir system user.
  * Make sure that all expected sites/all/{modules,themes,libraries} dirs exist.
  * Make sure that DB server is restarted on upgrade after config tuning.
  * Make sure that pdnsd and resolvconf are properly installed.
  * Nginx: Remove duplicate Vary: Accept-Encoding headers.
  * Percona no longer supports older Ubuntu non-LTS releases.
  * PHP: Do not reload FPM every hour - it may cause error 502.
  * PHP: Fix paths depending on CLI version used.
  * PHP: Fix the extensions installation and upgrade logic.
  * PHP: Make sure that the FPM port is set correctly for D6 sites with 5.2
  * PHP: Properly uninstall all related packages when using source build.
  * PHP: Start more FPM workers on systems with enough RAM by default.
  * Purge bin logs before disabling them.
  * Run NewRelic re-install early enough to avoid locking full-upgrade.
  * Sync the load limits for spiders and backend tasks.
  * The Java/Jetty monitor should use higher allowed limits by default.
  * Update apticron message to recommend system mode instead of full upgrade.
  * Update docs for _BUILD_FROM_SRC option.
  * Use aggressive enough Jetty restart procedure on nightly services reload.
  * Use correct status messages on install and upgrade.
  * Use installer and not Aegir version download on stable install/upgrade.

#550: Transition Town Totnes migration

chris — Sun, 19 May 2013 14:59:44 GMT

It has been agreed to migrate Transition Town Totnes, http://transitiontowntotnes.org/ to wiki:ParrotServer and this ticket has been created to record this process and the time spent on it and the documentation of the migration.

There is also an issue regarding what, if anything, to do with the old Drupal site at http://archive.transitiontowntotnes.org/

#551: Mediawiki 1.19.7 upgrade

chris — Tue, 21 May 2013 08:25:51 GMT

From MediaWiki-announce:

This is a notice that on Tuesday, May 21st between 20:00-21:00 UTC (1-2pm PDT) Wikimedia Foundation will release security updates for current and supported branches of the MediaWiki software. Downloads and patches will be available at that time, with the git repositories updated later that afternoon. Although MediaWiki does not have the vulnerable feature enabled by default, most wiki using common advanced features will want to patch for this issue.

#552: Puffin Downtime 23rd May 2013

chris — Thu, 23 May 2013 20:54:15 GMT

There was a massive load spike on wiki:PuffinServer this afternoon, the load average peaked at over 60 and at exactly the same time there was also one on the virtual server serving http://webarch.net/.

Pingdom reported:

www.transitionnetwork.org down since 23/05/2013 12:11:57
www.transitionnetwork.org is UP again at 23/05/2013 12:19:57, after 8m of downtime
www.transitionnetwork.org is down since 23/05/2013 12:26:57
www.transitionnetwork.org is UP again at 23/05/2013 12:58:57, after 32m of downtime

When I did managed to get back into the server php-fpm and nginx were not running and it took several attempt to get them to start up.

Attached is the munin graps of the load from today, munin didn't record the start of the spike but it did start recording it as it came down.

This ticket has been created to report the issue and also to record anything that might be found in the logs to explain what happened.

#554: Site slow down and MySQL load increase

chris — Tue, 28 May 2013 18:11:51 GMT

Since the upgrade to MariaDB 5.5.31, done on ticket:218#comment:93 (and fixed on ticket:548#comment:32) there appears to have been been a noticeable slowdown in the time for pages to be generated, measuring with http://tools.pingdom.com/fpt/ the front page alone takes around 5 seconds to generate.

There is a clear increase in the amount of MySQL/MariaDB activity measured by Munin, see the attached graphs, the upgrade was done around midday on 24th May 2013. There has also been a increase in traffic according to the firewall graphs. The memory usage of redis has also dropped right down and the database memory usage has significantly increased.

It's not totally clear if the cause of this change in behaviour of the site is related to the MySQL/MariaDB upgrade or if there was a coincidental change in the traffic to the site at the same time. There is no noticeable change in the visitors recorded in the Piwik stats.

According to pingdom the front page of the site is now "slower than 77% of all tested websites" with a total load time of around 6 seconds, almost all of this is down to the wait of around 5 seconds for the index.php file. This can also be tested from parrot with Apache bench, sometimes cached pages are served up and these appear in an instance, if the front page is generated it takes around 5 seconds, see wiki:LoadTimes#a2013-05-28

#555: Load spikes causing the TN site to be stopped for 15 min at a time

chris — Wed, 29 May 2013 09:53:52 GMT

The BOA /var/xdrago/second.sh script is run every minute via the root crontab and if it detects a certain load level it changes the nginx config to a "high load" config which results in bots being served 503 errors when they spider the site, see ticket:563. When the load goes higher and hits another threshold the second.sh script kills the webserver applications, nginx and php-fpm, and waits till the load has dropped before starting them up again. This was happening once or twice a day following the increase in traffic around the launch of The Power of Just Doing Stuff. This has been addressed by multiplying the thresholds by 5 in second.sh.

Original Description

This morning at 10:19:24 I received the following alert from puffin:

Subject: lfd on puffin.webarch.net: High 5 minute load average alert - 6.59
Time:                    Wed May 29 10:17:02 2013 +0100
1 Min Load Avg:          23.39
5 Min Load Avg:          6.59
15 Min Load Avg:         2.57
Running/Total Processes: 44/326

At 10:21:57 I got an alert regarding ssh:

Service: SSH
Host: puffin
Address: puffin.webarch.net
State: CRITICAL
Date/Time: Wed May 29 10:21:57 BST 2013
Additional Info:
CRITICAL - Socket timeout after 10 seconds

Then at 10:26:47 ssh appeared to have recovered:

Service: SSH
Host: puffin
Address: puffin.webarch.net
State: OK
Date/Time: Wed May 29 10:26:47 BST 2013
Additional Info:
SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0)

But then pingdom reported at 10:29:07:

www.transitionnetwork.org is down since 29/05/2013  10:24:57.

There was then a report regarding Nginx at 10:32:07:

Notification Type: PROBLEM
Service: HTTP
Host: puffin
Address: puffin.webarch.net
State: CRITICAL
Date/Time: Wed May 29 10:32:07 BST 2013
Additional Info:
Connection refused

So at 10:33:47 I ssh'd in and found that php53-fpm and nginx were not running and it took several attempts to get them running again.

The up email from pingdom reported:

www.transitionnetwork.org is UP again at 29/05/2013  10:36:57, after 12m of downtime.

I can't find anything in the logs to indicate what caused the load spike and php-fpm and nginx to stopp running.

#556: Upgrade Piwik to version 1.12

chris — Thu, 30 May 2013 21:40:48 GMT

A new version of Piwik came out today with some interesting looking features:

#563: 503 Errors

chris — Wed, 19 Jun 2013 10:59:54 GMT

Original Description

The site is generating a lot of 503 errors, 83 since 6:30am today and there were around 750 yesterday.

#565: Blogs breadcrumbs incorrect on listings views

ed — Tue, 25 Jun 2013 13:21:56 GMT

Blogs breadcrumbs are wrong at main all blogs view and one author's full list view: https://www.transitionnetwork.org/blog https://www.transitionnetwork.org/blogs/ed-mitchell

Jim can you sort this in under 15 minutes (which is what you have left)

#567: Update BOA for new Redis 2.6.14

chris — Tue, 02 Jul 2013 11:10:27 GMT

A suggestion from Jim:

BOA now includes Redis 2.6.14 <https://raw.github.com/antirez/redis/2.6/00-RELEASENOTES> if you do a 'barracda up-stable system'... What interests me about this is these lines from the changelog:

UPGRADE URGENCY: HIGH because of the following two issues:

Lua scripting + Replication + AOF in slaves problem (see Issue #1164).
AOF + expires possible race condition (see Issue #1079).

It's a long shot, but that could maybe be part of the issue we've seen recently.

I'm not sure if this is best done now or later tonight when the site is less busy?

#572: Design changes for homepage and TC section

ed — Wed, 17 Jul 2013 14:17:25 GMT

Design changes for TN homepage and TC section done by Ben to be part of maintenance budget.

list here: https://docs.google.com/spreadsheet/ccc?key=0An7ZaZdq6UfJdDhxS0JxbXZLYnhLRkN4X3h5aVV3R1E#gid=0

#573: MariaDB 5.5.32 is available for Puffin

chris — Thu, 18 Jul 2013 19:11:24 GMT

I could either update this via aptitude and it would be quick and have little down time or I could update it via BOA and it'll take 3 or 4 times as long and involved extra downtime.

Jim -- which way would you like it done?

The MariaDB project is pleased to announce the immediate availability of MariaDB 5.5.32.

This is a bug-fix release. See the Release Notes and Changelog for details.

MariaDB 5.5.32 Stable (GA)

Release Notes: https://kb.askmonty.org/en/mariadb-5532-release-notes
Changelog: https://kb.askmonty.org/en/mariadb-5532-changelog
http://lists.askmonty.org/pipermail/announce/2013-July/000048.html

#577: Transition Streets Wordpress Migration

chris — Tue, 06 Aug 2013 21:28:25 GMT

This is a ticket to document the migration of the Transition Totnes http://www.transitionstreets.org.uk/ wiki:WordPress site to wiki:ParrotServer.

#580: php5-fpm starting when puffin boots

chris — Mon, 12 Aug 2013 16:03:06 GMT

When wiki:PuffinServer is booted /etc/init.d/php5-fpm is started and this causes errors with the site.

#581: Remove 'add to newsletter' option for site editors/admins

ed — Mon, 26 Aug 2013 13:24:17 GMT

Please remove the option for site editors/admins to 'add this to the newsletter' (which would send it to the newsletter nodequeue). Please also for now make the newsletter nodequeue non-active.

It's been confusing for staff members, and was for the simmplemail service really, so we're going to trial using a google doc for staff to add their stories to the newsletter instead.

#585: TTech Meeting 5th September 2013

chris — Thu, 05 Sep 2013 07:47:16 GMT

This ticket has been created for the TTech Skype meeting due to take place today.

#586: New Relic Monitoring for BOA

chris — Thu, 05 Sep 2013 08:35:17 GMT

This is a ticket to document the consideration, and possible installation, of https://newrelic.com/ on the PuffinServer

#588: RSS feed caching

chris — Fri, 06 Sep 2013 09:43:57 GMT

At the meeting on 5th September ticket:585 one thing we discussed was that:

The biggest single source of bandwidth by URL is Rob's RSS feed, 34.5GB of data, 150k hits. This represents half the total site data transfer, the RSS files is 0.4MB but will hopefully be mostly served gzipped at 0.1MB.

This ticket is to follow up on that -- are the RSS feeds served directly by Nginx?

#589: Blocking spammers at a firewall level

chris — Fri, 06 Sep 2013 09:48:59 GMT

At the meeting on 5th September ticket:585 one thing we discussed was that for August 2013:

More data is transferred for /user/register than the front page, 5.1GB compared to 3.6GB.

Most of this will be spam bots trying to register to post spam. Jim suggested that we could look at blocking some of these spam bots at a firewall level to save on resources. This ticket is to follow up on this suggestion.

#591: Move MySQL temporary directory to tmpfs

jim — Mon, 09 Sep 2013 12:47:41 GMT

Chris, please read: http://2bits.com/articles/reduce-your-servers-resource-usage-moving-mysql-temporary-directory-ram-disk.html

I think we could easily drop a little MySQL memory to give it some in-memory disk space to do the temporary table munching Drupal is causing it. I see there are already some mounted tmpfs partitions.

Related to #590 (proposal L: Review slow query log, explain queries, tweak as necessary/flag poorly behaving modules)

What do you think? Worth doing?

#593: Migrating Puffin to a ZFS file server

chris — Wed, 11 Sep 2013 15:52:47 GMT

This ticket is for the migration of PuffinServer to a ZFS file server, this will involve some downtime but should result in better IO and easier backups.

#594: WordPress 3.6.1 Maintenance and Security Release

chris — Wed, 11 Sep 2013 22:05:01 GMT

The following is from http://wordpress.org/news/2013/09/wordpress-3-6-1/

WordPress 3.6.1 is also a security release for all previous WordPress versions and we strongly encourage you to update your sites immediately. It addresses three issues fixed by the WordPress security team:

Block unsafe PHP unserialization that could occur in limited situations and setups, which can lead to remote code execution. Reported by Tom Van Goethem.
Prevent a user with an Author role, using a specially crafted request, from being able to create a post “written by” another user. Reported by Anakorn Kyavatanakij.
Fix insufficient input validation that could result in redirecting or leading a user to another website. Reported by Dave Cummo, a Northrup Grumman subcontractor for the U.S. Centers for Disease Control and Prevention.
Additionally, we’ve adjusted security restrictions around file uploads to mitigate the potential for cross-site scripting.

#595: Upgrade Mediawiki to 1.19.8 from 1.19.7

chris — Wed, 11 Sep 2013 22:18:05 GMT

From https://git.wikimedia.org/blob/mediawiki%2Fcore.git/REL1_19/RELEASE-NOTES-1.19

MediaWiki 1.19.8

This is a security and maintenance release of the MediaWiki 1.19 branch.

Changes since 1.19.7

SECURITY: Sanitize ResourceLoader exception messages
SECURITY: Token-getting functions will fail when using jsonp callbacks.
SECURITY: Fix extension detection with 2 .'s
Allow a string other than '*' as condition for DatabaseBase::delete()
Purge upstream caches when deleting file assets.
jquery.tablesorter: Add missing dependency on jquery.mwExtension

#597: Default owner for tickets

chris — Tue, 17 Sep 2013 08:04:58 GMT

Ben reported that the default owner for tickets is laura and that this should be changed.

#599: Server time drift

chris — Sun, 29 Sep 2013 21:57:59 GMT

The servers are not keeping good time at the moment.

#600: Numbers missing from Initiatives search page

ed — Mon, 30 Sep 2013 11:46:20 GMT

When users do a search on /initiatives we are missing a number which pertains to the results of the report:

It should say "XXX initiatives match your search criteria"

The XXX bit is missing

please check and resolve

#601: TTech Meeting 3rd October 2013

chris — Thu, 03 Oct 2013 09:03:42 GMT

This is a ticket for notes and times for the online meeting held on 3rd October 2013.

The ticket for the last meeting is ticket:585

#611: Adding Trac Account for Alan

chris — Thu, 17 Oct 2013 12:55:21 GMT

Alan, the other sysadmin at Webarchitects is in a better position than me to help with some of the more complex sysadmin issues so I hope it's OK that I have added an account for him here following the wiki:TracUserAdmin notes.

He is looking at helping with these tickets:

ticket:599 Server time drift
ticket:555 Load spikes causing the TN site to be stopped for 15 min at a time
ticket:593 Migrating Puffin to a ZFS file server

I don't expect he will be recording much time, I hope that is OK, he already has sudo access to all the servers.

Transition Technology: Ticket Query

#483: Nginx 502 Bad Gateway Errors with BOA

#487: robots.txt files for development sites

#545: Registration page: 502

#548: All Admin functions broken on TN.org

#553: Invalid response from server ERROR message

#562: Users being subscribed to widgets newsletter on registratoin

#566: Profile photo does not work

#568: TC blog: can't *not* send notifications

#569: 403s served to editors, admin very slow

#575: User system is blocking *everyone* - needs fixing urgently

#576: Site down

#583: tmp files on parrot exceeded inode limit

#584: News CTs and featured images: not handling as expected

#592: Many panes on homepage are in italics

#602: RSS problems

#608: User profile pictures are too big

#609: Videos not showing if added via video link

#610: Aegir database intensive (migrate, clone, restore) tasks hang for larger sites

#622: Maps not working on TN.org

#654: TIs appearing in 'TIs by list' view without a number

#657: Homepage slideshow: can't order the images

#658: Check caching on Social Reporters views

#659: Featured story on homepage not updating

#679: Filter Initiative by Country not working

#695: File upload problem with TTT WordPress site

#698: intransitionmovie.com returns 405 on submit

#699: Update Core & Plugins on transitionculture.org

#702: Attachments not being deleted from Trustees page

#703: Image not scaling on project

#704: Image not scaling on project

#708: Map not showing Indian initiative

#718: REconomy site showing adverts randomly

#724: Subscription emails from Rob's blog not arriving.

#748: Urgent: Fiona having problems sending email

#751: Email alert changes

#755: Site administrator's file upload capacity: not responding to changes

#771: cannot access puffin over sftp again

#777: Comments to blog post only showing up when logged in

#780: parrot

#837: Iframe in a panel page

#862: Puffin locked

#888: Adverts on Transition Network Front Page loaded via flickrit.com embedded content

#895: HTTPS wildcard *.transitionnnetwork.org expires on 22nd January 2016

#908: Please enable Google Analytics

#911: Disk space for /home on Parrot is running out

#502: patterns sub-domain for transitionresearchnetwork.org, was: Setting up specific pointer for URL

#560: Install drupal-based project management system onto our servers

#571: Force HTTP for anonymous, HTTPS for logged in users

#579: Special email alert for Rob Hopkins blog posts on TN.org

#614: Subscription option for Social Reporters CT

#621: Check and speed up updates to homepage sections: TC and slideshow

#643: Add paginator for /films view

#744: Add CSS Injector module to the D6 mix

#207: Logwatch Issues

#208: Dblog Issues

#218: Debian upgrades and updates

#371: Piwik Hosting

#382: Measurement and tracking requirements for TN and PSE

#470: Penguin install and configuration

#494: Email account for TRAC

#529: New Barracuda BOA-2.0.7 Edition available

#530: New Barracuda BOA-2.0.8 Edition available

#531: Disk usage on puffin

#532: Upgrade to Mediawiki 1.19.5

#534: Add accounts for Ben

#535: Upgrade Puffin, Penguin and Parrot from Debian Squeeze to Wheezy

#536: Upgrade Mediawiki to 1.19.6

#538: intransitionmovie.com checks and updates

#539: REconomy site migration and updates

#542: Parrot RAM

#543: Puffin Load Spike

#544: CSF / LDF false positive blocks on Puffin

#546: Trac documentation

#547: New Barracuda BOA-2.0.9 Edition available

#550: Transition Town Totnes migration

#551: Mediawiki 1.19.7 upgrade

#552: Puffin Downtime 23rd May 2013

#554: Site slow down and MySQL load increase

#555: Load spikes causing the TN site to be stopped for 15 min at a time

#568: TC blog: can't not send notifications

#575: User system is blocking everyone - needs fixing urgently