Transition Technology: Ticket #224: Add Varnish cache

hours, totalhours changed

jim — Thu, 27 Jan 2011 19:36:49 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.0 to 0.1

Update:

PressFlow - a minor fork of Drupal 6 with big improvements in scalablilty and other nice enhancements - is required for Varnish or any other reverse proxy to work.

Installed on the Sharing Engine with no side effects apart from needing PHP's error level to be dropped to error_reporting = E_ALL & ~E_NOTICE from just E_ALL.

Some modules (e.g. LoginToboggan?) might have issues with PressFlow?... Will need careful testing in addition to careful configuration.

jim — Thu, 27 Jan 2011 19:48:36 GMT

Ticket tiding: See also now closed #161

Varnish + Pressflow in 'Project Mercury': http://library.linode.com/development/frameworks/php/project-mercury/debian-5-lenny

owner, status changed

jim — Fri, 17 Jun 2011 16:24:46 GMT

owner changed from jim to chris
status changed from new to assigned

OK...

Pressflow lives here: https://launchpad.net/pressflow

I've downloaded it and installed it locally on the TN project - no issues, which is good since it's supposed to be a drop-in replacement.

It'll go to DEV once the other file-level bits are completed...

Handing this ticket to Chris to continue the love...

status changed

chris — Mon, 20 Jun 2011 14:00:05 GMT

status changed from assigned to accepted

What exactly do I need to do for this?

Follow the steps here?

http://library.linode.com/application-stacks/project-mercury/debian-5-lenny

jim — Mon, 20 Jun 2011 17:19:11 GMT

That seems like a very good start... Minus the Solr stuff.

We need two more modules: Varnish and Memcache

Varnish is here http://drupal.org/project/varnish - some more good links there Chris, note it supports Varnish 2.0.x and 2.1.x, not 3.x
Memcache is here http://drupal.org/project/memcache - seems to integrate with Cacherouter, which is already up and running.

I've added these two to SVN too.

jim — Mon, 20 Jun 2011 17:20:27 GMT

Are you gonna want Drush, too? Might be a good idea to have it up there anyway...

hours, totalhours changed

chris — Tue, 21 Jun 2011 09:54:59 GMT

hours changed from 0.0 to 1.0
totalhours changed from 0.1 to 1.1

OK, the live server has apc wiki:NewLiveServer#apc and memcache wiki:NewLiveServer#memcache already, the dev server isn't running memcache for some reason.

The version of varnish in debian lenny is 1.1.2 and the module, http://drupal.org/project/varnish requires Varnish 2.0.x and Varnish 2.1.x.

So I'll follow the instructions here: http://www.varnish-cache.org/installation/debian the other option is using the backports version: http://packages.debian.org/lenny-backports/varnish but I think we might as well use the vendor version.

I'll document it here: wiki:DevelopmentServer#Varnish

I have realised that varnish sits in front of apache, this complicates things as we are not just serving one site off these servers, I'm going to need to read some documentation and think about this for a while...

hours, totalhours changed

chris — Tue, 21 Jun 2011 11:45:17 GMT

hours changed from 0.0 to 1.4
totalhours changed from 1.1 to 2.5

If we use Varnish to proxy authenticated requests we are going to have to be extra careful with it security wise to make sure there are no leaks -- for example a authenticated page being cached and then served to another user.

I'd rather have all authenticated sessions for all web applications use HTTPS only and if this needs to be accelerated use a server which can do HTTPS like http://nginx.org/ and use that for static content and proxy dynamic content to apache.

Having Varnish only serving anon content would make things simpler, especially as we have to set it up for two drupal instances, www.transitionnetwork.org and news.transitionnetwork.org one mediawiki site, wiki.transitionnetwork.org the static archive, static.transitionnetwork.org and then all the domain redirects and also the other applications running on the server like munin and phpMyAdmin etc.

But the HTTPS for all authenticated sessions discussion is due for the next Skype chat, so I don't really want to pre-empt that but also want to get on with sorting this issue out... for now I'll do some more reading and thinking!

hours, totalhours changed

chris — Tue, 21 Jun 2011 14:14:16 GMT

hours changed from 0.0 to 1.0
totalhours changed from 2.5 to 3.5

Only allowing HTTPS authenticated sessions using the 443 Session module http://drupal.org/project/session443 and this patch to allow anon HTTPS http://drupal.org/node/1154618 is my suggestion and not using http://nginx.org/ at this stage for HTTPS.

Currently Apache is using on average 1.23Gb of the live servers 2Gb of RAM and this bursts past 2Gb at times (this is when the server starts to swap), see: https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net-multimemory.html

Varnish will need some dedicated RAM and this will have to come from the amount Apache uses.

hours, totalhours changed

jim — Tue, 21 Jun 2011 14:32:02 GMT

hours changed from 0.0 to 0.15
totalhours changed from 3.5 to 3.65

Session443 now in SVN - that plus other modules for this ticket waiting on #265
Patch manually applied, committed.
I'd wager these memory spikes are due to developer activity - I've been adding Content Types and fields today which would require lots of database work and caches to be dumped.
I'd also wager that less Apache & PHP thread usage due to caching will gain us some memory back/
I assume there will be a period of tuning where you keep an eye on the stats and tweak accordingly?

hours, totalhours changed

chris — Wed, 22 Jun 2011 09:40:34 GMT

hours changed from 0.0 to 0.25
totalhours changed from 3.65 to 3.9

OK, can we test varnish on the dev server -- is Session 443 etc ready to roll on the dev site?

hours, totalhours changed

jim — Wed, 22 Jun 2011 10:43:09 GMT

hours changed from 0.0 to 0.25
totalhours changed from 3.9 to 4.15

Session 443 on DEV and enabled, have copied in the exceptions from Secure Pages on LIVE to ensure AJAX etc works fine - will test that aspect another time.

It's set to force https for all users and login pages, and normal http for non-authenticated users (except login/registration of course).

FYI settings here: https://dev.transitionnetwork.org.webarch.net/admin/settings/session443

hours, totalhours changed

chris — Wed, 22 Jun 2011 10:59:21 GMT

hours changed from 0.0 to 0.1
totalhours changed from 4.15 to 4.25

It's set to force https for all users and login pages, and normal http for non-authenticated users (except login/registration of course).

I think we should allow non authenticated users to use HTTPS if they wish so I think "Force HTTP for anonymous users:" should be disabled, I'll change this setting and have a play with it on the dev server at the same time as sorting ticket:166

jim — Wed, 22 Jun 2011 11:01:35 GMT

OK...

FYI Varnish and Memcache modules now are on DEV, but not yet enabled.

jim — Thu, 23 Jun 2011 10:53:17 GMT

Chris, a thought:

Having Varnish only serving anon content would make things simpler, especially as we have to set it up for two drupal instances, www.transitionnetwork.org and news.transitionnetwork.org one mediawiki site, wiki.transitionnetwork.org the static archive, static.transitionnetwork.org and then all the domain redirects and also the other applications running on the server like munin and phpMyAdmin etc.

Do we really need wiki and other non-Drupal stuff passing through Varnish?

chris — Thu, 23 Jun 2011 11:01:42 GMT

Do we really need wiki and other non-Drupal stuff passing through Varnish?

Yes if these things are running on the same server on port 80 -- we can have either apache on port 80 or varnish, not both.

hours, totalhours changed

chris — Thu, 23 Jun 2011 11:45:38 GMT

hours changed from 0.0 to 0.5
totalhours changed from 4.25 to 4.75

Simple Varnish/Debian? install notes:

http://www.howtoforge.com/putting-varnish-in-front-of-apache-on-ubuntu-debian

Plus Drupal:

One key question: are cookies used for anything for anon users? It would be nice to simply remove them all.

Also we will need to make sure that we don't break Piwik tracking.

ed — Thu, 23 Jun 2011 11:50:21 GMT

on the cookie front Laura's got a ticket to keep us roughly in line with recent EC/UK changes: #258

jim — Thu, 23 Jun 2011 11:51:00 GMT

As per my other note ticket:276#comment:2, I cannot think of any off the top of my head.

Here's the readme from that module:

Description

The no_anon module eliminates the need for sessions for anonymous users

This has the potential to reduce the server load and/or database contention on busy sites, since crawlers do not create entries in the session table for every page view.

Side effects

Some modules use the $_SESSION superglobal variable to store info for anonymous users will be affected by this module. If your modules use this only for logged in users, then it will not be an issue. Before you install this module, make sure that your modules do not depend on that.

This approach is known to cause the following features not to work:

The core Who's Online block
The core Throttle module
The anonymous user count in the Administration Menu module.

Installation

To install this module, do the following:

Extract the tarball into your sites/all/modules directory.
Enable the module from admin/build/modules
You then must change your settings.php file to add the following line towards the end of the file:

$confsession_inc? = './sites/all/modules/no_anon/session-no-anon.inc';

Adjust the above to point to where you installed the module, e.g. './sites/example.com/modules/no_anon/session-no-anon.inc' if that is where you installed it.

FAQ --- Q: Help! I can no longer log in to my web site!

A: Most likely, you've forgotten to enable the No Anonymous module. It is not

enough to have the session_inc change above. Without the no_anon module, cookies will remain disabled at all times. The result is that while you can log in, your session ID will not be stored in your browser (because it couldn't store it in a cookie), which makes it impossible to remain logged in.

hours, totalhours changed

chris — Thu, 23 Jun 2011 11:57:27 GMT

hours changed from 0.0 to 0.1
totalhours changed from 4.75 to 4.85

This approach is known to cause the following features not to work:

The core Who's Online block

Do we need this since we have Piwik tracking which can give this info.

The core Throttle module

Won't be an issue since Varnish should be serving cached content and the back end apache/php/mysql won't be hit for anon users.

The anonymous user count in the Administration Menu module.

Again we can get this into from Piwik.

Notes on how to debug Varnish caching:

http://www.stewsnooze.com/content/what-stopping-varnish-and-drupal-pressflow-caching-anonymous-users-page-views

jim — Thu, 23 Jun 2011 12:05:21 GMT

We don't need the Who's online stuff or the counter or the throttle mod. And Piwik adds JS to the page that calls an external source (DEV) so the those stats will be unaffected, just the counter on the top-right of the black Admin Menu.

I suppose we just need to tread carefully. There are A LOT of modules running and it's quite possible some of those use the anonymous session.

So I suppose the best approach is to hope we can drop the anon sessions, but have a backup plan for Varnish if it turns out that's not possible. Is there such a plan, extra config aside, or is this a potential show-stopper?

hours, totalhours changed

chris — Thu, 23 Jun 2011 12:10:38 GMT

hours changed from 0.0 to 0.05
totalhours changed from 4.85 to 4.9

I suppose we just need to tread carefully. There are A LOT of modules running and it's quite possible some of those use the anonymous session.

Right, agreed.

So I suppose the best approach is to hope we can drop the anon sessions, but have a backup plan for Varnish if it turns out that's not possible. Is there such a plan, extra config aside, or is this a potential show-stopper?

No that's fine, I suggest we start by testing if the no_anon http://drupal.org/project/no_anon module can be used without any bad side effects. If it can then we won't need to do any cookie munging at a Varnish level.

If the no_anon module causes some problems with things not working for anon users then we can remove it and look at cookie munging at a Varnish level -- strip them from parts of the site where they are not needed to enable things to be cached.

Does that sound like an OK plan?

jim — Thu, 23 Jun 2011 12:12:02 GMT

I've pushed the module through SVN to DEV, and enabled it.

jim — Thu, 23 Jun 2011 12:19:30 GMT

And updated settings.php... Should be working now.

hours, totalhours changed

chris — Thu, 23 Jun 2011 12:43:24 GMT

hours changed from 0.0 to 0.2
totalhours changed from 4.9 to 5.1

Still isn't working -- I'm seeing session cookies for anon users.

Some varnish cookie examples:

A handy looking script:

https://kristianlyng.wordpress.com/2010/08/13/stripping-cookies-with-vcl/

hours, totalhours changed

chris — Fri, 24 Jun 2011 10:14:52 GMT

hours changed from 0.0 to 0.5
totalhours changed from 5.1 to 5.6

Varnish with it's default configuration is now running on port 80 on the dev server and apache is now on port 8080 and 443, so these pages come via Varnish:

Now for some more testing and experiments -- the Drupal Varnish module can be enabled.

hours, totalhours changed

chris — Fri, 24 Jun 2011 12:22:47 GMT

hours changed from 0.0 to 0.7
totalhours changed from 5.6 to 6.3

More Varnish notes here:

Enabling Munin starts, https://github.com/jnstq/munin-varnish

aptitude install git-core
cd /usr/share/munin/plugins
git clone git://github.com/basiszwo/munin-varnish.git
chmod a+x /usr/share/munin/plugins/munin-varnish/varnish_*
ln -s /usr/share/munin/plugins/munin-varnish/varnish_* /etc/munin/plugins/

Add the following to the end of /etc/munin/plugin-conf.d/munin-node

[varnish*]
user root

Restart the node, /etc/init.d/munin-node restart and stats should start to appear here:

https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net.html#Varnish

hours, totalhours changed

chris — Fri, 24 Jun 2011 13:00:59 GMT

hours changed from 0.0 to 0.5
totalhours changed from 6.3 to 6.8

Cookies, mediawiki at http://wiki.transitionnetwork.org/ were being set and also sent to the drupal site, these have been stopped by adding this to LocalSettings.php on the live and dev server:

$wgCookieSecure = true;

I wonder if we should also set only secure cookies with Drupal, according to a comment, http://www.turnkeylinux.org/blog/we-dont-need-no-stinking-ssl this in settings.php should do it:

ini_set('session.cookie_secure',  1);

Which parts of the site *require* cookies for non-authenticated users? }}}

hours, totalhours changed

chris — Fri, 24 Jun 2011 13:12:18 GMT

hours changed from 0.0 to 0.1
totalhours changed from 6.8 to 6.9

For the moment I have set Varnish to remove *all* cookies on http://dev.transitionnetwork.org/ via these rules in /etc/varnish/default.vcl:

sub vcl_recv {
    unset req.http.Cookie;
}
sub vcl_fetch {
    unset beresp.http.set-cookie;
}

The idea being we can now check to see what, if anything has broken...

hours, totalhours changed

chris — Fri, 24 Jun 2011 14:43:10 GMT

hours changed from 0.0 to 1.0
totalhours changed from 6.9 to 7.9

I did add this to settings.php for a while to ensure only secure cookies are set but it appears to me at the moment that this isn't needed:

if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off' || $_SERVER['SERVER_PORT'] == 443) {
  ini_set('session.cookie_secure', 1);
}

This is what the cookie that anon users get when they visit a HTTPS page looks like:

Set-Cookie: SESS86fe0bba91cd08c27fa111a4ec66ede1=8381d5e2b6f01d56c57804072bffaa5d; expires=Sun, 17 Jul 2011 18:13:09 GMT; path=/; domain=.dev.transitionnetwork.org; secure; HttpOnly

One thing I can't work out is what browser I can use on Linux which will give me mixed http and https content warnings, there doesn't seem to be a way to enable this with Firefox -- any suggestions?

I haven't been able to find anything that breaks with all HTTP cookies removed by Varnish, but I'm probably not looking in the right place -- anyone have any hints?

jim — Fri, 24 Jun 2011 15:46:17 GMT

I use Chromium - makes the padlock logo have a warning you can click if there's such a problem.

I also run IE 6 and 7 inside a virtualbox windows image if I want to feel really dirty!

hours, totalhours changed

chris — Mon, 27 Jun 2011 08:53:10 GMT

hours changed from 0.0 to 0.3
totalhours changed from 7.9 to 8.2

All cookies generated by the https version of the site should have the secure flag set -- if they don't then they will be sent to the server when pages are accessed using http and will potentially stop varnish caching pages.

I'm seeing non-secure cookies on the https version of the site on images and js etc, eg:

Cookie: NO_CACHE=Y; SESS311518397b2e496470fa7d208995f24c=c20665c6c4d8218a634946f2ea653db9; DRUPAL_UID=1

This cookie is set on login, I have tried added this to /etc/php5/apache2/php.ini:

session.cookie_secure = 1

But there is still this insecure cookie being set, anyone know if there is anything we can do about this and what code adds this (I assume it's a core Drupal thing?). And this seems to have solved this and I don't think there are any PHP applications we are running that this will break?

chris — Mon, 27 Jun 2011 08:57:41 GMT

Oops, Cookie line above isn't the right one, this is the HTTP header which sets an insecure cookie:

Set-Cookie: NO_CACHE=Y; expires=Mon, 27-Jun-2011 09:05:10 GMT; path=/; domain=.dev.transitionnetwork.org

hours, totalhours changed

chris — Mon, 27 Jun 2011 10:00:54 GMT

hours changed from 0.0 to 0.4
totalhours changed from 8.2 to 8.6

To further sort varnish out I think we need to enable the drupal varnish module -- Jim can you do that or tell me how to do it?

At the moment I'm seeing varnish cache css and images but not web pages, the headers for pages contain:

X-Drupal-Cache: HIT
X-Varnish-Cache: MISS

So varnish is currently just acting as a proxy for web pages and perhaps the varnish module will enable us to solve this?

hours, totalhours changed

chris — Mon, 27 Jun 2011 10:36:29 GMT

hours changed from 0.0 to 0.2
totalhours changed from 8.6 to 8.8

Are you gonna want Drush, too? Might be a good idea to have it up there anyway...

It's installed:

cd /usr/local
wget http://ftp.drupal.org/files/projects/drush-7.x-4.4.tar.gz
tar -zxvf drush-7.x-4.4.tar.gz
cd /usr/local/bin
ln -s ../drush/drush

hours, totalhours changed

chris — Mon, 27 Jun 2011 11:06:58 GMT

hours changed from 0.0 to 0.2
totalhours changed from 8.8 to 9.0

Looking at https://dev.transitionnetwork.org/admin/settings/performance

I'm playing with:

Caching mode:
  Disabled
  External (experts only, possible side effects)

With it set to External there is no [X-Drupal-Cache?] header but still a X-Varnish-Cache: MISS header, however I don't see hits to the back end apache, so I'm wondering of the code to generate that X-Varnish-Cache actually works, it was taken from here: http://www.stewsnooze.com/content/what-stopping-varnish-and-drupal-pressflow-caching-anonymous-users-page-views

However if Caching mode: is set to Disabled then we have X-Varnish-Cache: HIT and still no back end apache hits.

jim — Mon, 27 Jun 2011 11:16:39 GMT

Varnish module already (since Fri/Sat?) enabled on DEV: https://dev.transitionnetwork.org/admin/build/modules

No idea what is creating that cookie presently... Hmm.

And does Varnish (or CacheRouter? etc) need chagnes to the lines in Settings.php?

More info here you might find helpful: http://groups.drupal.org/pantheon/documentation

hours, totalhours changed

chris — Mon, 27 Jun 2011 11:34:09 GMT

hours changed from 0.0 to 0.1
totalhours changed from 9.0 to 9.1

Ah OK, so the varnish module is one which has no interface? Is there a settings file for it or anything I need to look at?

chris — Mon, 27 Jun 2011 12:10:40 GMT

We were adding headers at an apache level, I have commented this out:

  <FilesMatch "(\.html|\.html\.gz|\.xml|\.xml\.gz)$">
    <IfModule mod_headers.c>
      Header set Expires "Sun, 19 Nov 1978 05:00:00 GMT"
      Header set Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
    </IfModule>
  </FilesMatch>

hours, totalhours changed

chris — Mon, 27 Jun 2011 13:13:38 GMT

hours changed from 0.0 to 1.0
totalhours changed from 9.1 to 10.1

OK, once Varnish has cached a page it's *really* fast!

For example 100 requests fro the front page (no css js or images) from another server in the same colo:

ab -v 4 -n 1000 -H "Accept-Encoding: gzip" http://dev.transitionnetwork.org/
Time taken for tests:   2.077 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      7782000 bytes
HTML transferred:       7341000 bytes
Requests per second:    481.36 [#/sec] (mean)

480 requests per second isn't bad :-)

One thing I have noticed, the CSS and JS has generated file names, for example:

/sites/default/files/css/css_77d1f598b6b8d242bb38e945590a27e6.css
/sites/default/files/js/js_32ddb321767cd603e1552f85fad41dce.js

And these change quite often which reduces their cacheability -- Jim, is something that can be controlled?

I have found that we get more cache hits with Caching mode: Disabled here: https://dev.transitionnetwork.org/admin/settings/performance

I have also enabled Page compression: -- is this the best place to do it? It seems we have perhaps 3 levels we can produced gzipped content at:

Apache
PHP
Drupal

Any idea which might be best, the dev server is doing it at a Drupal level at the moment and varnish is caching the gziped content.

hours, totalhours changed

chris — Mon, 27 Jun 2011 13:47:27 GMT

hours changed from 0.0 to 0.1
totalhours changed from 10.1 to 10.2

Varnish does have a settings page according to this http://me.andering.com/2010/12/17/drupal-varnish-and-pressflow-fast-delivery-to-the-customer-with-some-speed-for-users-too/ it should be here:

https://dev.transitionnetwork.org/admin/settings/varnish

But I get access denied for it, should I be able to access it?

https://dev.transitionnetwork.org/admin/reports/event/1465547

hours, totalhours changed

chris — Mon, 27 Jun 2011 14:05:34 GMT

hours changed from 0.0 to 0.3
totalhours changed from 10.2 to 10.5

with https://dev.transitionnetwork.org/admin/settings/performance

If we set a Minimum cache lifetime: 10 min and Caching mode: external then varnish does cache content.

I also set this in settings.php, not sure if it's needed, suggested here http://drupal.org/node/941788

  'reverse_proxy' => TRUE,
  'reverse_proxy_addresses' => array('127.0.0.1'),

jim — Mon, 27 Jun 2011 14:20:48 GMT

I've updated Drupal's perms to allow 'developer' role (of which you're a member) to access Varnish stuff... All here: https://dev.transitionnetwork.org/admin/user/permissions - look under 'Varnish'... Didn't realise it had perms, sorry.

https://dev.transitionnetwork.org/admin/settings/varnish has lots of nice things... And an error that it can't speak to Varnish server.

hours, totalhours changed

chris — Mon, 27 Jun 2011 17:35:53 GMT

hours changed from 0.0 to 0.1
totalhours changed from 10.5 to 10.6

an error that it can't speak to Varnish server.

The port and secret key needed setting, it's working now:

https://dev.transitionnetwork.org/admin/reports/varnish

hours, totalhours changed

chris — Mon, 27 Jun 2011 18:57:10 GMT

hours changed from 0.0 to 0.2
totalhours changed from 10.6 to 10.8

OK Varnish and Drupal seem to be working OK, TODO:

Tune Varnish
Search for anything that might have broken from all http cookies being removed
Try to track down what is setting the NO_CACHE=Y cookie
Document all the settings
Possibly look at setting up Mediawiki to work with Varnish (non-urgent) http://www.mediawiki.org/wiki/Manual:Varnish_caching

I should also do some testing of the speed of the live server currently so we can compare it when Varnish is running.

The dev server apache is now very slow compared with Varnish as it's doing no caching, 10 requests for the front page takes around 30 seconds and with Varnish about 0.05 seconds!

ed — Tue, 28 Jun 2011 08:48:23 GMT

"I should also do some testing of the speed of the live server currently so we can compare it when Varnish is running."

Yes definitely - and the documentation - this is quite a big job and we need to represent the work to all users

hours, totalhours changed

chris — Tue, 28 Jun 2011 11:34:09 GMT

hours changed from 0.0 to 0.1
totalhours changed from 10.8 to 10.9

Try to track down what is setting the NO_CACHE=Y cookie

I think I have found it, here is this file, /web/dev.transitionnetwork.org.webarch.net/www/modules/cookie_cache_bypass/cookie_cache_bypass.module which contains:

<?php
function cookie_cache_bypass_form_alter(&$form, $form_state, $form_id) {
  $form['#submit'][] = 'cookie_cache_bypass_submit';
}
function cookie_cache_bypass_submit() {
  global $cookie_domain;
  $lifetime = variable_get('cache_lifetime', 300);
  setcookie('NO_CACHE', 'Y', $_SERVER['REQUEST_TIME'] + ($lifetime + 300), '/', $cookie_domain);
}

Jim -- any thoughts on this? Are there any times this is needed?

jim — Tue, 28 Jun 2011 11:39:28 GMT

I mentioned I had enabled it here: ticket:276#comment:6 - it's job is to set a cookie to temporarily tell Varnish et al NOT to return cached content in certain circumstances, like the page after submitting a contact form etc, so the user gets the message rendered into the page.

I reckon it's needed, but that doesn't mean it's behaving as we'd like... Is the cookie always there, or just occasionally? seems like the cookie lasts 5 minutes?

Have you done a find/grep on the /sites/all/modules folder for files containing "setcookie("? Might be nice to know exactly what modules use cookies...

hours, totalhours changed

chris — Tue, 28 Jun 2011 12:37:56 GMT

hours changed from 0.0 to 0.1
totalhours changed from 10.9 to 11.0

I mentioned I had enabled it here: ticket:276#comment:6 - it's job is to set a cookie to temporarily tell Varnish et al NOT to return cached content in certain circumstances, like the page after submitting a contact form etc, so the user gets the message rendered into the page.

Ah, OK, but I'm not sure it's needed, which forms are not redirected to HTTPS and which shouldn't have their results cached? I'm not sure what problem it's solving?

Is the cookie always there, or just occasionally? seems like the cookie lasts 5 minutes?

It is set as soon as you login:

Set-Cookie: NO_CACHE=Y; expires=Tue, 28-Jun-2011 12:33:25 GMT; path=/; domain=.dev.transitionnetwork.org

The live server version will have domain=.transitionnetwork.org and since it doesn't have secure it means it will be sent to all *.transitionnetwork.org sites for all requests, port 80 and 443 for 5 mins.

I wonder if we shouldn't be setting the cookie domain to www.transitionnetwork.org rather than .transitionnetwork.org for the Drupal session cookies -- with the current set up the session cookies could be potentially hi-jacked by another site (well another of our sites, not any site) -- would it cause any bad effects to tighten up on this? We could redirect requests from http://transitionnetwork.org/ to http://www.transitionnetwork.org/ The only problem would be if there were other sub-domains in use that people needed to be authenticated on after login to www.transitionnetwork.org -- is this the case?

You can test the above by logging in at https://dev.transitionnetwork.org/ and then visiting https://wiki.dev.transitionnetwork.org/Main_Page -- the Mediawiki site is sent the Drupal session cookie.

jim — Tue, 28 Jun 2011 13:20:39 GMT

Your shout on the Cookie Bypass module... It solves a problem we're not hitting as you say because forms are via HTTPS and hence uncached by Varnish. Shall I disable?

As for the cookie domain... It's supposed to be like that, starting with the . because several domains (esp. workspaces.tn.org) need to share the cookie so logins move around between subdomains. This is best Drupal practice so no, we can't tighten it really.

hours, totalhours changed

chris — Tue, 28 Jun 2011 14:02:28 GMT

hours changed from 0.0 to 0.2
totalhours changed from 11.0 to 11.2

Your shout on the Cookie Bypass module... It solves a problem we're not hitting as you say because forms are via HTTPS and hence uncached by Varnish. Shall I disable?

I think we might as well, I don't think it's doing anything we need. If we find forms that have a problem we can look at these specific cases and work out how to solve them, but I'm not yet aware of any.

As for the cookie domain... we can't tighten it really.

OK, no problem, as the secure flag is set all it means is that the secure Piwik and Mediawiki sites will sent the Drupal auth cookie, and they will ignore it unless there is something happening on this sites which shouldn't be happening.

Have you done a find/grep on the /sites/all/modules folder for files containing "setcookie("? Might be nice to know exactly what modules use cookies...

Yes, here are the results:

cd /web/dev.transitionnetwork.org.webarch.net
grep -r "setcookie(" . | grep -v svn

modules/cookie_cache_bypass/cookie_cache_bypass.module: setcookie('NO_CACHE', 'Y', $_SERVER['REQUEST_TIME'] + ($lifetime + 300), '/', $cookie_domain);
modules/comment/comment.module: setcookie('comment_info_'. $field, $form_state['values']![$field], time() + 31536000, '/');
includes/session.inc: setcookie(session_name(), ' ', $_SERVER['REQUEST_TIME'] - 3600, $params['path'], $params['domain'], $params['secure'], $params['httponly']);
includes/session.inc: setcookie(session_name(), ' ', $_SERVER['REQUEST_TIME'] - 3600, $params['path'], $params['domain'], $params['secure']);
sites/all/modules/memcache/memcache-session.inc: setcookie(session_name(), ' ', $_SERVER['REQUEST_TIME'] - 42000, '/');
sites/all/modules/memcache/memcache-session.inc: setcookie(session_name(), ' ', $_SERVER['REQUEST_TIME'] - 3600, $params['path'], $params['domain'], $params['secure'], $params['httponly']);
sites/all/modules/memcache/memcache-session.inc: setcookie(session_name(), ' ', $_SERVER['REQUEST_TIME'] - 3600, $params['path'], $params['domain'], $params['secure']);
sites/all/modules/cacherouter/cacherouter.module: setcookie($cookie_name, TRUE, $_SERVER['REQUEST_TIME'] + (60 * 60 * 24 * 30), $params['path'], $params['domain'], $params['secure'], $params['httponly']);
sites/all/modules/cacherouter/cacherouter.module: setcookie($cookie_name, TRUE, $_SERVER['REQUEST_TIME'] + (60 * 60 * 24 * 30), $params['path'], $params['domain'], $params['secure']);
sites/all/modules/cacherouter/cacherouter.module: setcookie($cookie_name, TRUE, $_SERVER['REQUEST_TIME'] - 3600, $params['path'], $params['domain'], $params['secure'], $params['httponly']);
sites/all/modules/cacherouter/cacherouter.module: setcookie($cookie_name, TRUE, $_SERVER['REQUEST_TIME'] - 3600, $params['path'], $params['domain'], $params['secure']);
sites/all/modules/cas/cas_server.module: setcookie(CAS_LOGIN_COOKIE, "", -3600);
sites/all/modules/cas/cas_server.module: setcookie(CAS_LOGIN_COOKIE, $_REQUEST['service']);
sites/all/modules/cas/cas.module: setcookie('cas_login_checked', 'true');
sites/all/modules/securepages_prevent_hijack/securepages_prevent_hijack.module: setcookie(SECUREPAGES_SESSID, $tok, $lifetime,

jim — Tue, 28 Jun 2011 20:19:46 GMT

OK, just going back through some stuff and catch up...:

Cache Bypass now module
If we set a Minimum cache lifetime: 10 min and Caching mode: external then varnish does cache content - Good but I'd suggest the "Page cache maximum age" on the performance page could go up to an hour without too much side-effects... Only news, events and blogs will be delayed worst case an hour... Ed's call really but worth upping this to at least 30 mins I reckon.
I have also enabled Page compression: -- is this the best place to do it? It seems we have perhaps 3 levels we can produced gzipped content at... I believe the best place is at an Apache level, but in an ideal world Varnish will cache both normal and gzipped versions of the page and serve according to request headers - is that set up?
OK, once Varnish has cached a page it's *really* fast! - I know, really impressive... Good work. I'd like to see what caching we're losing on Drupal's side since non-cached pages now take a long time to generate... Is Memcache up and running, or is that for another time?
One thing I have noticed, the CSS and JS has generated file names, for example: /sites/default/files/css/css_77d1f598b6b8d242bb38e945590a27e6.css /sites/default/files/js/js_32ddb321767cd603e1552f85fad41dce.js -- And these change quite often which reduces their cacheability -- Jim, is something that can be controlled? - Not really... I mean there are often a several versions of the collated, compressed CSS and JS files because some pages use more modules that need extra stuff, or because users use more modules and hence code an css. Plus, whenever the caches are cleared these files are recompiled and cached again, which is necessary if edits or theme updates have been done.

A quick recap of modules that set cookies according to your grep, for future reference:

Drupal core's session handling (obviously)
Comment (core, required mod)
Cacherouter (required mod)
Memcache (required mod, probably)
CAS (single sign on, hoping removing Secure Pages will fix this and make it required)
Cookie Cache Bypass (Pressflow, disabled)
Secure Pages Prevent Hijack (disabled, obsolete)

hours, totalhours changed

chris — Wed, 29 Jun 2011 09:51:55 GMT

hours changed from 0.0 to 0.8
totalhours changed from 11.2 to 12.0

I'd suggest the "Page cache maximum age" on the performance page could go up to an hour without too much side-effects

Sounds good to me, I have set "Minimum cache lifetime" and "Page cache maximum age" to one hour https://dev.transitionnetwork.org/admin/settings/performance I also set this on the live server by mistake -- was the live server set to one hour already or does it need changing to something less?

Page compression... believe the best place is at an Apache level

OK, we did have these rules for one of the back-end caching modules (boost?):

  <FilesMatch "(\.html|\.html\.gz|\.xml|\.xml\.gz)$">
    <IfModule mod_headers.c>
      Header set Expires "Sun, 19 Nov 1978 05:00:00 GMT"
      Header set Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
    </IfModule>
  </FilesMatch>
  <FilesMatch "(\.html|\.html\.gz)$">
    ForceType text/html
  </FilesMatch>
  <FilesMatch "(\.xml|\.xml\.gz)$">
    ForceType text/xml
  </FilesMatch>
  <FilesMatch "(\.js|\.js\.gz)$">
    ForceType text/javascript
  </FilesMatch>
  <FilesMatch "(\.css|\.css\.gz)$">
    ForceType text/css
  </FilesMatch>

I have commented the above out and added some extra mime types to the /etc/apache2//mods-available/deflate.conf:

<IfModule mod_deflate.c>
          AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/rss+xml
</IfModule>

Any other mime types need adding?

Jim -- what about all the boost rewrite rules, should these now be commented out?

I'm also not sure about this ETag setting:

FileETag MTime Size

And also there is this expires date, 1984:

Last-Modified: Wed, 29 Jun 2011 09:32:37 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT

Set via includes/bootstrap.inc:

  // HTTP/1.0 proxies do not support the Vary header, so prevent any caching
  // by sending an Expires date in the past. HTTP/1.1 clients ignores the
  // Expires header if a Cache-Control: max-age= directive is specified (see RFC
  // 2616, section 14.9.3).
  drupal_set_header('Expires', 'Sun, 11 Mar 1984 12:00:00 GMT');

It looks to me that we might need to take a close look at all the HTTTP header settings in that file -- Jim did these settings also come with boost?

And according to redbot we have an invalid Last-Modified date, so I'll look into that at some point.

Is Memcache up and running, or is that for another time?

I have installed memcached php5-memcache on the dev server, the config file is /etc/php5/conf.d/memcache.ini and it's now running -- where do we configure it for Drupal does the module need enabling?

jim — Wed, 29 Jun 2011 10:51:04 GMT

Jim -- what about all the boost rewrite rules, should these now be commented out?

Yes, kill 'em - boost hasn't been there for over a year!

Set via includes/bootstrap.inc... Jim did these settings also come with boost?

Can't hack Drupal core! Hopefully that's not acutally sent anywhere, or it might be a catch-all for non-cachable items... Hopefully Pressflow + Varnish overrides these.

I have installed memcached php5-memcache on the dev server -- where do we configure it for Drupal does the module need enabling?

I've just enabled the Memcache module, plus it's admin interface -- then had to patch it -- and updated the permissions, so you can access its stats here: https://dev.transitionnetwork.org/admin/reports/memcache and the pitiful admin screen here: https://dev.transitionnetwork.org/admin/settings/memcache.

Looks like there's more installation needed though, from /sites/all/modules/memcache/README.TXT':

These are the broad steps you need to take in order to use this software. Order
is important.
1. Install the memcached binaries on your server. See
http://www.lullabot.com/articles/how_install_memcache_debian_etch
2. Install the PECL memcache extension for PHP. This must be version 2.2.1 or
   higher or you will experience errors.
3. Put your site into offline mode.
4. Download and install the memcache module.
5. If you have previously been running the memcache module, run update.php.
6. Start at least one instance of memcached on your server.
7. Edit settings.php to configure the servers, clusters and bins that memcache
   is supposed to use.
8. Edit settings.php to include either memcache.inc. For
   example, $conf['cache_inc'] ='sites/all/modules/memcache/memcache.inc';
9. Bring your site back online.
For instructions on 1 and 2 above, please see the INSTALLATION.txt file that
comes with the memcache module download.
The memcache.inc file is intended to be used instead of cache.inc, utilizing
Drupal's pluggable cache system.
memcache.db.inc IS DEPRECATED AND IS NOT RECOMMENDED. It is still distributed
with the 6.x-1.x branch, but will not be included in any further versions and
may be removed in future 6.x releases.
Update $conf in settings.php to tell Drupal where the cache_inc file is:
 $conf = array(
   // The path to wherever memcache.inc is. The easiest is to simply point it
   // to the copy in your module's directory.
   'cache_inc' => './sites/all/modules/memcache/memcache.inc',
 );

hours, totalhours changed

chris — Wed, 29 Jun 2011 11:57:57 GMT

hours changed from 0.0 to 0.1
totalhours changed from 12.0 to 12.1

boost hasn't been there for over a year!

Yikes, what is the minimal set of mod_rewrite rules we need -- it might be easier to start again are some pressflow suggest one or a pressflow .htaccess file I can look at?

includes/bootstrap.inc ... Can't hack Drupal core! Hopefully Pressflow + Varnish overrides these.

It's not overwriting them at the moment, I'll do some more investigation...

And I'll look some more at memcache...

jim — Wed, 29 Jun 2011 12:01:22 GMT

The .htaccess file in the root of the site should be the basic version from Pressflow - and it should be the same as this:

https://github.com/pressflow/6/blob/master/.htaccess

hours, totalhours changed

chris — Wed, 29 Jun 2011 13:09:54 GMT

hours changed from 0.0 to 1.0
totalhours changed from 12.1 to 13.1

We have had memcache running on the live server before: wiki:NewLiveServer#memcache not sure why it wasn't on the dev server, it is now: wiki/DevelopmentServer#memcache and we have some stats here: https://dev.transitionnetwork.org/admin/reports/memcache

The .htaccess file in the root of the site should be the basic version from Pressflow - and it should be the same as this:

https://github.com/pressflow/6/blob/master/.htaccess

Great, I have applied those and commented out all the older ones, it's a lot simpler now :-)

I have also reduced the number of apache processes to reduce apache memory usage as the server is getting slow as it's swapping a lot, (see https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net.html), from /etc/apache2/apache2.conf:

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       2
    MaxSpareServers       4
    MaxClients           10
    MaxRequestsPerChild   2000
</IfModule>

jim — Thu, 30 Jun 2011 09:27:46 GMT

It's worth noting that the file caching engine of Cache Router is most likely the cause of #286 (Data loss) issues.

Therefore pressing on with Varnish on LIVE is certainly beneficial now CR has been disabled.

jim — Fri, 01 Jul 2011 12:00:32 GMT

May be of interest, Chris - purging items in Varnish from Drupal when the content is updated, hence allowing longer caching times:

ed — Fri, 01 Jul 2011 14:35:14 GMT

Chris doing research and tweaking settings Ed and Jim to torture DEV and try all functionality

hours, totalhours changed

chris — Tue, 05 Jul 2011 11:04:07 GMT

hours changed from 0.0 to 0.26
totalhours changed from 13.1 to 13.36

http://drupal.org/project/purge http://drupal.org/node/1054886

Thanks I have copied over the key things from there, some options are not needed as we currently have Varnish to remove all incoming and outgoing cookies.

hours, totalhours changed

chris — Tue, 05 Jul 2011 13:24:07 GMT

hours changed from 0.0 to 1.5
totalhours changed from 13.36 to 14.86

I have done some page load time tests, see https://tech.transitionnetwork.org/trac/wiki/LoadTimes

As expected the dev server is very fast for port 80 and very slow for 443, there is a summary of some other key points on the wiki page and also detail about what I did and we now have some numbers to compare with later on.

I'll look at fixing the gzip and etag issues for js on the dev server now.

Not sure what to look at next for speeding up HTTPS requests, Jim -- any suggestions?

hours, totalhours changed

chris — Wed, 06 Jul 2011 11:19:24 GMT

hours changed from 0.0 to 1.56
totalhours changed from 14.86 to 16.42

Not sure what to look at next for speeding up HTTPS requests

One suggestion is to use Edge Side Includes, and this is possibly worth looking at but the problem is that varnish doesn't do HTTPS...

Another suggestion is to use a HTTPS accelerator like pound, so we could then have:

pound -> varnish -> apache

It is possible to set up pound to cope with multiple HTTPS sites with one cert and one IP, see http://www.apsis.ch/pound/

I could set this up on the dev server and test it, it'll also require changes to the Varnish set up.

If it all works the end result should be that css/js/images are all served by varnish for HTTPS requests and all requests for web pages would get passed though, we also also consider using ESI module for additional performance.

Or another option would be to use Nginx for HTTPS, though this would also have some complications, for example with upload progress module.

And one thing we could also look at is using fcgid rather than mod_php.

I'll continue doing some research...

hours, totalhours changed

chris — Wed, 06 Jul 2011 13:15:48 GMT

hours changed from 0.0 to 1.5
totalhours changed from 16.42 to 17.92

I'm still unsure if we should try pound or nginx for HTTPS, will read more...

nginx

On the dev server at least we can't just use nginx rather than apache as it would break subversion:

The Nginx WebDAV module supports only a limited set of the protocol so in this case you need to reverse proxy to Apache and let it handle it. http://blog.martinfjordvald.com/2011/02/nginx-primer-2-from-apache-to-nginx/

More info on this here:

Apache and Nginx Together

fcgid

If pure speed is what you are after, then stay with mod_php.

However, for better resource usage and efficiency, consider moving to fcgid.

http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html

We do have an issue with memory usage, so fcgid is probably worth considering.

YSlow Issues

I have fixed the gzipping of js files and the etags issue -- now the dev server has A's on YSlow for everything we can have A's for.

hours, totalhours changed

chris — Thu, 07 Jul 2011 11:16:43 GMT

hours changed from 0.0 to 0.39
totalhours changed from 17.92 to 18.31

Regarding the pound/varnish vs nginx for https, it appears that nginx might be faster, but then we would can drupal/apache talkig to two reverse proxies -- is this possible / would it make things far too complicated?

https req --  nginx  \
                        apache
http req  -- varnish /

I'm thinking that using pound might be simpler as then there is only one reverse proxy to configure.

https req --  pound  \
                        varnish -- apache
http req  -----------/

With this set up the existing varnish config will need a rewriting and will be more complicated as it'll be doing authenticated and non-authenticated requests, also it'll be doing reverse proxy for the other sites on the server, mediawiki etc.

And for the dev server there is the added complication that we have subversion on port 443.

So, I'm tempted to have a play with pound and see if this is going to work -- does that sound OK?

jim — Thu, 07 Jul 2011 12:32:30 GMT

Sorry Chris, been meaning to help out on this ticket but other stuff keeps cropping up.

My gut says: keep it simple.

Varnish is quick, Drupal behind it is slow. This tells me that the Drupal-level caching is borked, and that this is a Drupal issue.

Sure, adding another layer of server-level stuff or an alternate server might improve things, but really it's down to Drupal not having the speed we need when creating pages for logged in users.

I would be interested to see what LIVE makes of the existing DEV configuration - after all, there's double as much memory and 4x as many processors.

Would it be possible to try Varnish out on LIVE and then disable it after a few hours? I know Ed and I haven't tested DEV, nor have I had chance to look into the Drupal-level speed issues I allude to above - but if we can turn it on/off easily enough (without the Conference exploding as a result), then we should test it out.

You see logged in user's pages are always slow compared, because they must be generated for each person's settings. However, if memcache is there aiding these requests then I cannot see why we shouldn't have the same speed or better than with the (broken) Cache Router file cache. If not, I'd say something is misconfiguration, probably at a Drupal/memcache level.

I'll hopefully have more time next week to help on this ticket, and will do testing on DEV ASAP.

What do you reckon?

hours, totalhours changed

chris — Thu, 07 Jul 2011 13:58:38 GMT

hours changed from 0.0 to 0.2
totalhours changed from 18.31 to 18.51

Would it be possible to try Varnish out on LIVE and then disable it after a few hours?

There are several ways we could do this, but I'm not sure about deploying exactly the set up we have on on the dev server on the live -- it's *so* slow for logged in users... I'll have a think about this, I guess we could try it...

I'll start by setting up Varnish running on another port like 81.

logged in user's pages are always slow compared, because they must be generated for each person's settings.

Right, this is where ESI is worth considering at some point http://drupal.org/project/esi -- the dynamic bits of the pages, eg the bit with the username in it, are dynamically generated and inserted into the rest of the page which is the same for all users.

if memcache is there aiding these requests

I think we need to test this, it's using hardly any memory, I'm not convinced it is doing anything much at the moment, it's using 27Mb of RAM on the dev server:

https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net-multimemory.html

And 1.4Mb RAM on the live server:

https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net-multimemory.html

ed — Thu, 07 Jul 2011 14:05:35 GMT

NO tinkering with this sort of stuff during the conference please.

Chris and Ed will be dashing around like blue arsed flies, so not able to concentrate on this work, and there may be multiple bloggers, etc. etc.

Happy to trial on Tuesday when we're together, calm, in front of computers with skype etc...

jim — Thu, 07 Jul 2011 18:02:46 GMT

Interesting...

I've just put RC1 of Cache Router onto DEV - is it me or is it faster? Or is that your work, Chris?

Bear in mind we're using memcache/varnish not file cache... Is it really faster? Seems so - pages are on average about 1 sec, and once caches are primed it's faster still.

Chris?!

jim — Thu, 07 Jul 2011 18:20:37 GMT

OK. It's plain fast now. Well, fast enough anyway.

Could Cache Router RC2 have been the problem here all along, or am I merely seeing a mirage of a fast website reflected off hot air rising from my new cup of tea??

In short: logged in users run at a very usable pace. Un-logged-in users get a delay for the first request (Varnish coming out of page files or warming up?), then it's like a greased cougar on speed with a red-hot poker in its ass.

My hunch: Cache Router RC2 is buggered.

jim — Thu, 07 Jul 2011 20:40:13 GMT

So I pushed RC1 to LIVE but we still have the 'have to delete the cache files manually because certain pages get obsolete data' bug is still present.

So though DEV seems better, and LIVE seems ok (as before) one major bug is still here.

I'll check for patches, or possibly pull the older version of CR out of SVN because I *think* I patched it at some point... will look back though the commits, though it could have been on the old repo.

Note normally I would never roll back a module like this, but CR doesn't have its own settings/database schema - code only - so it's safe to do so.

hours, totalhours changed

chris — Thu, 07 Jul 2011 23:22:09 GMT

hours changed from 0.0 to 1.0
totalhours changed from 18.51 to 19.51

It took a while to work out what was going on here!

What I think has happened is that the Session 443 module has changed to an older version which doesn't have the option to disable this feature:

Force all authenticated users to use HTTPS. Also force non-authenticated users on a login or registration form to use HTTPS. Otherwise force HTTP.

So we were testing https://dev.transtionnetwork.org/ but getting the results from http and therefore varnish.

So I have disabled Session 443 on the dev site and now it's back to being really slow...

I put some speed test results up here wiki:LoadTimes but they aren't very interesting.

I think the next thing to do is experiment with pound doing https in front of Apache.

jim — Fri, 08 Jul 2011 07:01:58 GMT

Hmmm. My bad, saw the update and downloaded the wrong one *sigh* The latest, Beta 2 is up, settings re-applied - you may want to check.

Site still seems pretty quick once the caches (memcache) is primed.

Getting a 'line through padlock' warning in Chromium that some elements are non-https. These need tracking down.

So, Chris, is it working? If so, the site is as fast as it's ever been on DEV.

hours, totalhours changed

chris — Fri, 08 Jul 2011 11:50:21 GMT

hours changed from 0.0 to 0.3
totalhours changed from 19.51 to 19.81

The latest, Beta 2 is up, settings re-applied - you may want to check.

Ah, nice, it's been improved a lot, looks fine now :-)

Site still seems pretty quick once the caches (memcache) is primed.

Yeah it doesn't seem too bad :-)

Also the live server will be faster...

Getting a 'line through padlock' warning in Chromium that some elements are non-https. These need tracking down.

I have loked at the HTML of the front page and there is no media embedded with http, also none of the CSS files link to any images via http, and the js is loaded via https so I'm at a loss to explain this, unless some js files dynamically generate links to http things and this is the cause?

Also I just installed chromium-browser and I didn't get a line through the padlock, so I'm extra confused about why you do!

jim — Fri, 08 Jul 2011 11:53:33 GMT

I thinks the padlock status is cached... Once I closed/opened a tab the padlock went green.

SO I reckon varnish + memcache + session443 is ok, at least as fast as before - DEV was never the speediest.

So that means we're almost ready to do this on LIVE - once the conference is done. Before we make the switch I'll try once more with RC2 of cacherouter because it does fix a number of bugs.

Will put it in SVN and on DEV shortly and see if anything changes...

hours, totalhours changed

chris — Fri, 08 Jul 2011 12:15:46 GMT

hours changed from 0.0 to 0.05
totalhours changed from 19.81 to 19.86

OK, I'm happy with that, if we find it's too slow then at that point we can revisit the pound / nginx options, I'd also be happy if we didn't need to because they would introduce a lot more complexity.

ed — Tue, 12 Jul 2011 14:20:44 GMT

Right then fellas - are we ready to try this on Wednesday?

Shall we organise a skype mid-wednesday to discuss it while we're doing it? Taht sounds like sense to me.

I don't want to go on hols wondering if the logged in experience is good enough...!

jim — Tue, 12 Jul 2011 14:23:02 GMT

I'm going to put my neck on the block and say: logged in will be fine.

DEV is slow because it gets no traffic, therefore the caches are always cold and need re-filling.

Can chat any time mid-wed, let me know when.

hours, totalhours changed

chris — Tue, 12 Jul 2011 20:30:37 GMT

hours changed from 0.0 to 0.04
totalhours changed from 19.86 to 19.9

Can chat any time mid-wed, let me know when.

OK, I'll get everything in place in the morning and then we could do the switch perhaps around noon, via skype -- I have a school sports day to go to from about 1:45pm onwards.

ed — Tue, 12 Jul 2011 21:08:23 GMT

ok then, standing by for further information - and be prepared to roll it back if it does'nt work out, too!

hours, totalhours changed

chris — Wed, 13 Jul 2011 13:09:10 GMT

hours changed from 0.0 to 3.0
totalhours changed from 19.9 to 22.9

Varnish is now live but for some reason HTML pages are not being cached, everything else seems fine so far....

There should be a HIT in the HTTP headers not a MISS:

X-Varnish-Cache: MISS

hours, totalhours changed

chris — Wed, 13 Jul 2011 18:41:31 GMT

hours changed from 0.0 to 0.3
totalhours changed from 22.9 to 23.2

The varnish settings for the dev server and the live server are exactly the same. However the dev server is serving pages via varnish:

lynx -head -dump http://dev.transitionnetwork.org/
HTTP/1.1 200 OK
Server: Apache
Cache-Control: public, max-age=3600
Last-Modified: Wed, 13 Jul 2011 18:10:42 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT
Vary: Cookie,Accept-Encoding
ETag: "1310580642"
Content-Type: text/html; charset=utf-8
Content-Length: 30814
Date: Wed, 13 Jul 2011 18:25:15 GMT
X-Varnish: 1553515735 1553515688
Age: 872
Via: 1.1 varnish
Connection: close
X-Varnish-Cache: HIT

And the live server is not:

lynx -head -dump http://www.transitionnetwork.org/
HTTP/1.1 200 OK
Server: Apache
X-Drupal-Cache: HIT
Etag: "1310580100-0"
Cache-Control: public, max-age=0
Last-Modified: Wed, 13 Jul 2011 18:01:40 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT
Vary: Cookie,Accept-Encoding
Content-Type: text/html; charset=utf-8
Content-Length: 30681
Date: Wed, 13 Jul 2011 18:25:46 GMT
X-Varnish: 201272008
Age: 0
Via: 1.1 varnish
Connection: close
X-Varnish-Cache: MISS

Note the X-Varnish-Cache: HIT header from the dev server and the X-Drupal-Cache: HIT from the live server.

I suspect this is because the dev server has:

Caching mode:

External (experts only, possible side effects)

And the live server has:

Caching mode:

Normal (recommended for production sites, no side effects)

Ideally we would be able to have Caching mode on Normal for authenticated requests and on External for non-authenticated requests, however I don't think this is an option.

So I suggest when we are all together we try switching the live server to External and see how it does.

I'm noting some load times here https://tech.transitionnetwork.org/trac/wiki/LoadTimes#a2011-07-13

hours, totalhours changed

chris — Thu, 14 Jul 2011 09:46:47 GMT

hours changed from 0.0 to 0.34
totalhours changed from 23.2 to 23.54

I have just written the following to explain the speed improvement:

Yesterday the configuration of the site was changed in a couple of key ways. Varnish, a HTTP caching reverse proxy server, was set up to serve requests on port 80, in other words all http://www.transitionnetwork.org/ requests. This has resulted in a dramatic improvement in speeds for people who are not logged in to the site. For example the speed test application at tools.pingdom.com would previously fail to load the front page and all the associated images, etc in 30 seconds (the maximum time it'll run for) -- it typically managed to fully download about half the files in this time. For example see the result from just before Varnish was deployed:

http://tools.pingdom.com/default.asp?url=http%3a%2f%2fwww.transitionnetwork.org%2f&id=5124069
And compare that the a typical result after Varnish was deployed:

http://tools.pingdom.com/default.asp?url=http%3a%2f%2fwww.transitionnetwork.org%2f&id=5127828
After Varnish was deployed the front page and all it's associated files now typically loads in about 1.5 seconds! The other key change that has been made to the site is that now authenticated sessions are only available via https -- while you are logged into the site your connection is encrypted. This makes the site a lot more secure, especially for people using unencrypted wifi access points, though it might take a little bit of getting used to for some users (if a logged in user visits the http version of a page they will appear to have been logged out).

ed — Thu, 14 Jul 2011 11:52:32 GMT

couple of things:

the /news and /blogs views lead users to a http link. if they are logged in, they get access denied. can we upgrade the views for them to show their https status, or perhaps we need to tinker with the access denied page, hmm?

the email alerts show the http link - see problem #1 - hmm - maybe i'm answering my own question here.

THEREFORE: let's edit the 403 page - what exact instructions should we give to punters seeing the access denied message because of this loop?

jim — Thu, 14 Jul 2011 12:07:31 GMT

the /news and /blogs views lead users to a http link. if they are logged in, they get access denied. can we upgrade the views for them to show their https status, or perhaps we need to tinker with the access denied page, hmm?

Still not seeing this - please post links/instructions.

This could *POSSIBLY* be resolved by ensuring all /user* links are pushed to SSL - Chris? Do you have an example message email to post, Ed?

Good idea about the 403 error page Ed. Though I think we can also improve things by changing the header toolbar links to go to /user rather than /user/login, that way people will see their page if they're logged in, rather than the access denied error. I'll do that now...

In the mean time, there are two paths:

User logged in but arrives at /user/login etc - in this case we just either say "You're already logged in, please use secure version", OR we just execute a drupal_goto("https://www.transitionnetwork.org/user") to get them to avoid the error all together
User NOT logged in but wants a auth-only page - show them the error message and page they get now but ensure the links from to secure pages are HTTPS.

(settings at: https://www.transitionnetwork.org/admin/settings/customerror)

Again, I'll try some of these now and post more soon.

jim — Thu, 14 Jul 2011 12:28:39 GMT

OK, have changed 'Log in' to "My Account & Log In" which points to /user (thus giving either the user's home if they're logged in, or the login form)

And the 403 page has a special case for logged people accessing /user/login or /user/register that looks like:

You're already logged in, Transition Admin!
For better security, we now require people to use HTTPS to access their accounts and perform logged-in actions.
Please continue securely to your account to continue. Thank you.

See it (when logged in) here: https://www.transitionnetwork.org/user/login

Non-logged in users get the usual login form instead.

Does this fix the issues, Ed? (Still can't see the http(s) issue with news or blogs.

ed — Thu, 14 Jul 2011 15:51:19 GMT

nice one. another question leads from this -

can it take them back to the page they were wanting to get to (as it used to when people logged in)?

hours, totalhours changed

chris — Thu, 14 Jul 2011 16:29:28 GMT

hours changed from 0.0 to 0.05
totalhours changed from 23.54 to 23.59

This could *POSSIBLY* be resolved by ensuring all /user* links are pushed to SSL - Chris?

We have that already:

RedirectPermanent /user https://www.transitionnetwork.org/user

jim — Thu, 14 Jul 2011 17:27:11 GMT

I've added some code to handle users logged in looking for a HTTPS logged-in page from HTTP (e.g. clicking 'add comment'): https://www.transitionnetwork.org/user/login?destination=node%2F17514%23comment-form -- complete with nice friendly green button

I reckon we've covered most of the cases here... I'm done, can't think of much more to do...

status changed; resolution set

ed — Tue, 13 Sep 2011 15:22:27 GMT

status changed from accepted to closed
resolution set to fixed