Transition Technology: Ticket #357: nginx research

milestone set

chris — Wed, 12 Oct 2011 13:33:49 GMT

milestone set to Phase 5

jim — Wed, 12 Oct 2011 13:59:34 GMT

As I mentioned I've set myself up a low-end Linode so I can play and consolidate my hosting away from a shared environment.

In my travels I've found 'Barracuda' which is a meaty script that took my empty Debian 6 server and got NGINX, PHP-FPM, memcache, a fire wall and bunch of other handy stuff... Now most of that is pointless for here but thought you might want to take a gander at the script that set it up so you can see what's what with regards to NGINX and PHP-FPM config: http://drupal.org/project/barracuda

Also, as mentioned, there's http://groups.drupal.org/nginx with lots of people and information.

hours, status, totalhours changed

chris — Thu, 20 Oct 2011 10:18:31 GMT

hours changed from 0.0 to 0.5
status changed from new to accepted
totalhours changed from 0.0 to 0.5

The pages are now available via nginx:

I'll next look at setting up munin before I look at making the (far more compicated) other virtualhosts work with nginx.

hours, totalhours changed

chris — Thu, 20 Oct 2011 16:30:59 GMT

hours changed from 0.0 to 0.2
totalhours changed from 0.5 to 0.7

I have added some munin monitoring for nginx and we should start to get some stats here:

https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html

hours, totalhours changed

chris — Fri, 21 Oct 2011 12:06:24 GMT

hours changed from 0.0 to 1.6
totalhours changed from 0.7 to 2.3

PHP-FPM, following http://fak3r.com/geek/howto/howto-install-php5-fpm-on-debian-squeeze/

Add this to /etc/apt/sources.list:

deb http://packages.dotdeb.org squeeze all
deb-src http://packages.dotdeb.org squeeze all

cd
gpg --keyserver keys.gnupg.net --recv-key 89DF5277
gpg -a --export 89DF5277 | sudo apt-key add -
aptitude update
aptitude safe-upgrade
 The following NEW packages will be installed:
   nginx-common{a} nginx-light{a}
 The following packages will be upgraded:
   libapache2-mod-php5 libmysqlclient16 mysql-client-5.1 mysql-common mysql-server mysql-server-5.1
   mysql-server-core-5.1 nginx php-pear php5 php5-cli php5-common php5-curl php5-dev php5-gd
   php5-imagick php5-mcrypt php5-memcache php5-mysql php5-suhosin

With a new LAMP stack all the config files need updating, the old ones were backed up, new ones installed and diffs were done, these were the changes to /etc/php5/apache2/php.ini:

expose_php = Off
memory_limit = 256M
post_max_size = 128M
default_charset = "utf-8"
upload_max_filesize = 100M
max_file_uploads = 50
default_socket_timeout = 120
extension=uploadprogress.so
session.cookie_secure = 1
mbstring.http_input = pass
mbstring.http_output = pass

Things that we had in the old php.ini which I'm note sure if we need in the new one:

output_buffering = Off
serialize_precision = 100
variables_order = "EGPCS"
register_long_arrays = On
register_argc_argv = On
cgi.nph = 1
cgi.fix_pathinfo=0

nginx and php still need more work to get them working...

priority changed

chris — Fri, 21 Oct 2011 12:08:16 GMT

priority changed from major to blocker

Oh, I see that the PHP upgrade has broken http://dev.transitionnetwork.org/

Parse error: syntax error, unexpected T_SL in /web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/imageapi/imageapi_imagemagick.module on line 152

I'll look at fixing this before looking at ticket:364

hours, totalhours changed

chris — Fri, 21 Oct 2011 12:14:49 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.3 to 2.4

Actually it's a diff in the code in /web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/imageapi/imageapi_imagemagick.module:

<<<<<<< .mine
  setlocale(LC_CTYPE, "en_GB.UTF-8");
=======
  // LANG added by chris for UTF-8 filenames
  setlocale(LC_CTYPE, "en_GB.UTF-8");
>>>>>>> .r224

I have fixed the for the dev server but not committed it.

hours, totalhours changed

chris — Mon, 16 Jan 2012 11:31:35 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.4 to 2.5

Replying to chris:

The pages are now available via nginx:

http://kiwi.transitionnetwork.org:8000/
https://kiwi.transitionnetwork.org:4430/

The new certs and debian upgrades produced some error messages:

Nginx 'soft' update failed, doing restart
Starting nginx: nginx: [emerg] SSL_CTX_use_certificate_chain_file("/etc/ssl/transitionnetwork.org/transitionnetwork.org.crt") failed (SSL: error:0906D066:PEM routines:PEM_read_bio:bad end line error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib)
nginx: configuration file /etc/nginx/nginx.conf test failed
invoke-rc.d: initscript nginx, action "start" failed.
dpkg: error processing nginx-light (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of nginx:
 nginx depends on nginx-full | nginx-light; however:
  Package nginx-full is not installed.
  Package nginx-light is not configured yet.
dpkg: error processing nginx (--configure):
 dependency problems - leaving unconfigured

So I'm surprised that nginx is still running on the dev server, I have put the errors here so they can be investigated next time this ticket is worked on.

hours, totalhours changed

chris — Fri, 09 Mar 2012 14:37:28 GMT

hours changed from 0.0 to 0.3
totalhours changed from 2.5 to 2.8

Following the last certificate update nginx on the dev server has been broken:

/etc/init.d/nginx start
Starting nginx: nginx: [emerg] SSL_CTX_use_certificate_chain_file("/etc/ssl/transitionnetwork.org/transitionnetwork.org.crt") failed (SSL: error:0906D066:PEM routines:PEM_read_bio:bad end line error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib)
nginx: configuration file /etc/nginx/nginx.conf test failed

This was fixed thus:

cd /etc/ssl/transitionnetwork.org/
wget http://crt.gandi.net/GandiStandardSSLCA.crt
wget http://crt.usertrust.com/UTNAddTrustServer_CA.crt
wget http://crt.usertrust.com/AddTrustExternalCARoot.crt
openssl x509 -inform DER -in GandiStandardSSLCA.crt -out GandiStandardSSLCA.pem
openssl x509 -inform DER -in AddTrustExternalCARoot.crt -out AddTrustExternalCARoot.pem
openssl x509 -inform DER -in UTNAddTrustServer_CA.crt -out UTNAddTrustServer_CA.pem
cat transitionnetwork.org.crt > gandi.pem
cat GandiStandardSSLCA.pem >> gandi.pem
cat AddTrustExternalCARoot.pem >> gandi.pem
cat UTNAddTrustServer_CA.pem >> gandi.pem
cat transitionnetwork.org.crt gandi.pem > transitionnetwork.org.chained.pem

And /etc/nginx/sites-available/kiwi was edited:

#ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.crt;
ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;

So now Nginx is working again for this site:

And the next task is to enable all the other sites on the server, one at a time, till they are all running and then we can switch the ports around and turn apache off.

hours, totalhours changed

chris — Thu, 15 Mar 2012 12:24:36 GMT

hours changed from 0.0 to 0.15
totalhours changed from 2.8 to 2.95

Working through each domain name at a time enabling all the services, last year kiwi.transitionnetwork.org was set up to work via nginx:

And Munin is available via nginx:

https://kiwi.transitionnetwork.org:4430/munin/

However /info/ and /phpmyadmin and /apc_info.php don't yet work:

So I'm going to work on enabling them to start with.

hours, totalhours changed

chris — Thu, 15 Mar 2012 13:46:55 GMT

hours changed from 0.0 to 1.2
totalhours changed from 2.95 to 4.15

One issue I have just realised that we had with the current setup with varnish, these pages (used by Munin to generate graphs), the second of which displays client's IP addresses (when clients use HTTPS) are set to only be available to the localhost via apache:

However they were available via varnish to anyone:

I have fixed this on the live and dev server by adding these rules to the varnish config at /etc/varnish/default.vcl

acl local {
  "localhost";         // myself
  "127.0.0.1";         // myself
  "81.95.52.78";       // this machines main ip address
  "81.95.52.79";       // this machines 2nd ip address
  "81.95.52.80";       // this machines 3rd ip address
}
    ## Pass cron jobs and server-status
    if (req.url ~ "cron.php") {
      if (client.ip ~ local) {
        return (pass);
      }
      else {
        error 403 "Access Denied";
      }
    }
    if (req.url ~ "/server-status$") {
      if (client.ip ~ local) {
        return (pass);
      }
      else {
        error 403 "Access Denied";
      }
    }
    if (req.url ~ "apc_info.php") {
      if (client.ip ~ local) {
        return (pass);
      }
      else {
        error 403 "Access Denied";
      }
    }

The varnish documentation has been updated to match the current configuration, wiki:DevelopmentServer#Varnish and wiki:NewLiveServer#varnish

hours, totalhours changed

chris — Thu, 15 Mar 2012 16:37:55 GMT

hours changed from 0.0 to 1.5
totalhours changed from 4.15 to 5.65

Enabling PHP and nginx, the notes previously followed in ticket:357#comment:5 have moved to http://fak3r.com/2011/09/27/howto-install-php5-fpm-on-debian-squeeze/ also following the notes here, http://www.webhostingtalk.com/showthread.php?t=1025286

aptitude install php5-fpm
  The following NEW packages will be installed:
    php5-fpm
  Creating config file /etc/php5/fpm/php.ini with new version

Start it:

/etc/init.d/php5-fpm start

Check it's running:

netstat -plunt|grep php
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      31616/php-fpm.conf)

Add this to /etc/nginx/sites-available/kiwi:

location ~ \.php$ {
      fastcgi_pass 127.0.0.1:9090;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME /web/kiwi.webarch.net/www$fastcgi_script_name;
      include fastcgi_params;
}

But the php files result in:

502 Bad Gateway

And this in /var/log/nginx/error.log:

2012/03/15 15:56:05 [error] 2763#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: XX.XX.XX.XX, server: kiwi.transitionnetwork.org, request: "GET /info/php-info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9090", host: "kiwi.transitionnetwork.org:4430"

I tried editing the list of allowed hosts, /etc/php5/fpm/pool.d/www.conf:

listen.allowed_clients = 127.0.0.1,81.95.52.78,kiwi.transitionnetwork.org,kiwi.webarch.net

I have read a lot of google results but have yet to find an answer to this problem, will pick this up again tomorrow...

hours, totalhours changed

chris — Fri, 16 Mar 2012 11:19:09 GMT

hours changed from 0.0 to 0.25
totalhours changed from 5.65 to 5.9

My bad, /etc/php5/fpm/pool.d/www.conf had port 9000 in it not port 9090, fixed:

listen = 127.0.0.1:9090

Added a root directive to the .php section as suggested here to get php scripts working:

location ~ \.php$ {
      fastcgi_pass 127.0.0.1:9090;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME /web/kiwi.webarch.net/www$fastcgi_script_name;
      include fastcgi_params;
      root   "/web/kiwi.webarch.net/www";
}

Now these work:

And I think that wraps it up for nginx doing everything apache does for the kiwi.transitionnetwork.org domain, 1 down, quite a few to go!

hours, totalhours changed

chris — Fri, 16 Mar 2012 12:30:19 GMT

hours changed from 0.0 to 0.34
totalhours changed from 5.9 to 6.24

Getting Piwik running with nginx, apache config here /etc/apache2/sites-enabled/03-piwik.transitionnetwork.org, nginx config here /etc/nginx/sites-available/piwiw, initially created as a copy of /etc/nginx/sites-available/default.dpkg-dist, referencing http://wiki.nginx.org/MediaWiki.

These lines were edited, initially to get it working without HTTPS:

        listen    8080;
        root /web/piwik.transitionnetwork.org/piwik;
        index index.php;
        server_name piwik.transitionnetwork.org piwik.transitionnetwork.org.webarch.net;
        root /web/piwik.transitionnetwork.org/piwik;
        index index.php;
        # Make site accessible from http://localhost/
        server_name piwik.transitionnetwork.org piwik.transitionnetwork.org.webarch.net;
        client_max_body_size 5m;
        client_body_timeout 60;
        # see http://wiki.nginx.org/MediaWiki
        location / {
                try_files $uri $uri/ @rewrite;
        }
        location @rewrite {
                rewrite ^/(.*)$ /index.php?title=$1&$args;
        }
        location ^~ /maintenance/ {
                return 403;
        }
        location ~ \.php$ {
                include fastcgi_params;
                fastcgi_pass unix:/tmp/phpfpm.sock;
        }
        location ~ \.php$ {
                fastcgi_pass 127.0.0.1:9090;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /web/piwik.transitionnetwork.org/piwik$fastcgi_script_name;
                include fastcgi_params;
                root   "/web/kiwi.webarch.net/www";
        }
        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                try_files $uri /index.php;
                expires max;
                log_not_found off;
        }
        location = /_.gif {
                expires max;
                empty_gif;
        }
        location ^~ /cache/ {
                deny all;
        }

Make the config available and restart

cd /etc/nginx/sites-enabled
ln -s ../sites-available/piwik 02-piwik
/etc/init.d/nginx restart
  Restarting nginx: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] still could not bind()
  nginx.

So, something isn't right, I'll come back to the later after looking at ticket:401

hours, totalhours changed

chris — Fri, 16 Mar 2012 14:15:47 GMT

hours changed from 0.0 to 0.17
totalhours changed from 6.24 to 6.41

Several mistakes in the server config were fixed and now the mediawiki site is available (unencrypted) via nginx at http://wiki.dev.transitionnetwork.org:8080/Main_Page

server {
        listen    8000;
        #listen   80; ## listen for ipv4; this line is default and implied
        #listen   [::]:80 default_server ipv6only=on; ## listen for ipv6
        root /web/wiki.dev.transitionnetwork.org/www;
        index index.php;
        # Make site accessible from http://localhost/
        server_name wiki.dev.transitionnetwork.org wiki.dev.transitionnetwork.org.webarch.net;
        client_max_body_size 5m;
        client_body_timeout 60;
        # see http://wiki.nginx.org/MediaWiki
        location / {
                try_files $uri $uri/ @rewrite;
        }
        location @rewrite {
                rewrite ^/(.*)$ /index.php?title=$1&$args;
        }
        location ^~ /maintenance/ {
                return 403;
        }
        location ~ \.php$ {
                include fastcgi_params;
                fastcgi_pass unix:/tmp/phpfpm.sock;
        }
        location ~ \.php$ {
                fastcgi_pass 127.0.0.1:9090;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /web/wiki.dev.transitionnetwork.org/www$fastcgi_script_name;
                include fastcgi_params;
                root   "/web/wiki.dev.transitionnetwork.org/www";
        }
        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                try_files $uri /index.php;
                expires max;
                log_not_found off;
        }
        location = /_.gif {
                expires max;
                empty_gif;
        }
        location ^~ /cache/ {
                deny all;
        }
}

Next task to make it available via https.

hours, totalhours changed

chris — Tue, 20 Mar 2012 14:39:33 GMT

hours changed from 0.0 to 4.0
totalhours changed from 6.41 to 10.41

All the nginx config is being revisited, been reading http://michael.lustfield.net/content/dummies-guide-nginx http://blog.martinfjordvald.com/2010/07/nginx-primer/ http://blog.martinfjordvald.com/2011/02/nginx-primer-2-from-apache-to-nginx/ https://nealpoole.com/blog/2011/04/setting-up-php-fastcgi-and-nginx-dont-trust-the-tutorials-check-your-configuration/ and http://wiki.nginx.org/Pitfalls

Changing php5-fpm to use a socket rather than tcp, create a dir for the sock file:

mkdir /var/run/php5-fpm
chown www-data:www-data /var/run/php5-fpm

Edit /etc/php5/fpm/pool.d/www.conf:

[www]
user = www-data
group = www-data
listen = /var/run/php5-fpm/phpfpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0666
listen.allowed_clients = 127.0.0.1,81.95.52.78,kiwi.transitionnetwork.org,kiwi.webarch.net
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /

Security fix, see https://nealpoole.com/blog/2011/04/setting-up-php-fastcgi-and-nginx-dont-trust-the-tutorials-check-your-configuration/ edit /etc/php5/fpm/php.ini:

cgi.fix_pathinfo=0

These are working:

This needs more work:

https://kiwi.transitionnetwork.org:4430/phpmyadmin/

hours, totalhours changed

chris — Wed, 21 Mar 2012 13:13:17 GMT

hours changed from 0.0 to 1.81
totalhours changed from 10.41 to 12.22

kiwi.transitionnetwork.org is now fully working with nginx (config file /etc/nginx/sites-available/kiwi) and I'm fairly confident that it's been set up to be as secure as possible, the configuration still needs documenting on this wiki, but the config file itself has lots of comments, these things are available on this domain:

http://kiwi.transitionnetwork.org:8000/ - server front page / documentation
http://kiwi.transitionnetwork.org:8000/apc_info.php - script for apc munin stats, only available from localhost
https://kiwi.transitionnetwork.org:4430/munin/ - munin stats
https://kiwi.transitionnetwork.org:4430/info/php-info.php - php info (password protected)
http://kiwi.transitionnetwork.org:8000/info/apc.php - apc info (password protected)
https://kiwi.transitionnetwork.org:4430/phpmyadmin/ - phpmyadmin (password protected)

Next to sort these out:

hours, totalhours changed

chris — Wed, 21 Mar 2012 14:37:05 GMT

hours changed from 0.0 to 0.4
totalhours changed from 12.22 to 12.62

Replying to chris:

Next to sort these out:

http://static.dev.transitionnetwork.org/

Done, config file /etc/nginx/sites-available/static:

http://static.dev.transitionnetwork.org:8000/
http://static.dev.transitionnetwork.org:8000/ttcon2010/ - 2010 conference static files
https://static.dev.transitionnetwork.org:4430/ttcon2010/private - private files from the conference

More to follow tomorrow...

hours, totalhours changed

chris — Thu, 22 Mar 2012 13:35:11 GMT

hours changed from 0.0 to 0.1
totalhours changed from 12.62 to 12.72

These two sites are going to be very time consuming to migrate due to the number of rewrite rules (98 lines of apache config containing the word Rewrite), I wonder if we shouldn't keep apache just for these and use nginx as a reverse proxy for these domains.

hours, totalhours changed

chris — Thu, 22 Mar 2012 17:00:48 GMT

hours changed from 0.0 to 3.3
totalhours changed from 12.72 to 16.02

I have done a load of work on a secure Mediawiki nginx config, but it's still not totally working and I'm not happy with it yet, hopefully I'll complete this tomorrow.

References:

hours, totalhours changed

chris — Fri, 23 Mar 2012 14:47:55 GMT

hours changed from 0.0 to 1.56
totalhours changed from 16.02 to 17.58

Did a lot of work on the wiki config, got things that didn't work that were suggested here http://blog.bigdinosaur.org/mediawiki-on-nginx/ working but then a combination of a really poor ADSL connection and god knows what resulted in the config file I was working on getting deleted, so I'll have to start again next week, really sorry about this :-(

I did lean more about nginx in the process though...

hours, totalhours changed

chris — Fri, 23 Mar 2012 21:38:10 GMT

hours changed from 0.0 to 0.1
totalhours changed from 17.58 to 17.68

Replying to chris:

a combination of a really poor ADSL connection and god knows what

The high load on kiwi making vim (the text editor I use) really unresponsive, which I mistook for connection problems (which we have been having at work) and my fustration at not being able to get stuff done as a result is probably what...

The thing which took ages to figure out was this code from http://blog.bigdinosaur.org/mediawiki-on-nginx/ which doesn't work:

#    Force potentially-malicious files in the /images directory to be served
#    with a text/plain mime type, to prevent them from being executed by
#    the PHP handler
  location ~* ^/images/.*.(html|htm|shtml|php)$ {
      types { }
      default_type text/plain;
  }

It's a very good idea to do this for any directory where users can upload files so I was keen to get it sorted so it could be used for other applications.

The code above simply does nothing, I'm not exactly sure why as the regex would be OK for apache, I haven't got to the bottom of why the above doesn't work.

The solution I finally got working was, in essence, this:

  location ^~ /images/ {
      types {
        text/plain    htm html shtml php php5;
         image/gif    gif;
         # ...
         # whitelist here of all file types allowed to be uploaded,
         # see LocalSettings.php for a list
      }
      try_files $uri /index.php;
  }

It won't take long on Monday to recreate the file that got deleted.

hours, totalhours changed

chris — Mon, 26 Mar 2012 12:18:10 GMT

hours changed from 0.0 to 2.0
totalhours changed from 17.68 to 19.68

Replying to chris:

It won't take long on Monday to recreate the file that got deleted.

I spent a couple of hours on this today, before I was interupted by a power outage in the office, I should get mediawiki sorted tomorrow.

hours, totalhours changed

chris — Tue, 27 Mar 2012 12:55:48 GMT

hours changed from 0.0 to 4.0
totalhours changed from 19.68 to 23.68

I have finally wrapped up Mediawiki on nginx:

All logins are redirected to HTTPS and I have tested everything I can think of and everything is working.

I have discovered an apache to nginx config convertor which should speed up progress:

http://www.anilcetin.com/convert-apache-htaccess-to-nginx/

For reference if you get this error message:

==> /var/log/nginx/wiki.ssl_error.log <==
2012/03/27 10:57:15 [info] 26009#0: *18 client sent plain HTTP request to HTTPS port while reading client request headers, client: XX.XX.XX.XX, server: wiki.dev.transitionnetwork.org, request: "OPTIONS /index.php?action=ajax HTTP/1.1", host: "wiki.dev.transitionnetwork.org:4430"
==> /var/log/nginx/wiki.ssl_access.log <==
81.95.52.29 - - [27/Mar/2012:10:57:15 +0100] "OPTIONS /index.php?action=ajax HTTP/1.1" 400 271 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0 Iceweasel/11.0"

It can be fixed by adding this to the location section of the config for the php scripts:

fastcgi_param HTTPS on;

hours, totalhours changed

chris — Tue, 27 Mar 2012 13:46:57 GMT

hours changed from 0.0 to 0.3
totalhours changed from 23.68 to 23.98

I have made a start on the regular expressions for:

hours, totalhours changed

chris — Wed, 28 Mar 2012 13:47:50 GMT

hours changed from 0.0 to 3.0
totalhours changed from 23.98 to 26.98

These sites are now available via nginx on port 8000:

But there is a problem with the regular expressions for the forum archive, I had it working earlier but it's now broken and needs fixing:

http://2011.archive.transitionnetwork.org:8000/forum/

hours, totalhours changed

chris — Thu, 29 Mar 2012 10:25:00 GMT

hours changed from 0.0 to 0.1
totalhours changed from 26.98 to 27.08

The fix for the old forum was easy:

http://2011.archive.transitionnetwork.org:8000/forum/

Now onto Pwiki, Trac and then Drupal...

hours, totalhours changed

chris — Thu, 29 Mar 2012 12:32:34 GMT

hours changed from 0.0 to 1.46
totalhours changed from 27.08 to 28.54

Piwki has been setup, following http://wiki.nginx.org/Piwik which is based on https://github.com/perusio/piwik-nginx

A directory for the cache was needed:

mkdir /var/cache/nginx/fcgicache -p
chown www-data:www-data /var/cache/nginx/fcgicache

I added a web bug to http://kiwi.transitionnetwork.org/ to test it:

https://piwik.transitionnetwork.org:4430/

hours, totalhours changed

chris — Thu, 29 Mar 2012 15:10:50 GMT

hours changed from 0.0 to 1.42
totalhours changed from 28.54 to 29.96

Setting up Trac on Nginx isn't going to be simple, some resources on this:

There is also the old subversion repo here https://tech.transitionnetwork.org/svn/ I guess this isn't needed any more?

Python is a bit of mess on the development server because (a) a source version was installed in order to get the GA Piwik import working and (b) a unstable version was installed in order to install a unstable version of Trac in order to get it working with git...

https://tech.transitionnetwork.org/trac/ticket/364#comment:36

I'm also concerned that although trac isn't working perfectly at the moment at least it's working...

Do we need trac to be integrated with github?

For now I think I'll look at setting up nginx as a reverse proxy for apache for trac and then fixing trac can be done after the switch to Nginx...

hours, totalhours changed

chris — Thu, 29 Mar 2012 16:29:31 GMT

hours changed from 0.0 to 0.4
totalhours changed from 29.96 to 30.36

Trac with Nginx acting as a reverse proxy to apache:

https://tech.transitionnetwork.org:4430/trac/

hours, totalhours changed

chris — Thu, 29 Mar 2012 16:58:33 GMT

hours changed from 0.0 to 0.4
totalhours changed from 30.36 to 30.76

I have made a start on the Drupal sites:

Nginx: http://dev.transitionnetwork.org:8000/

This still needs lots of work to replicate the varnish caching etc and setting it up so that authenticated sessions use HTTPS, at the moment it doesn't seem any quicker than apache / varnish, but it should be when it's done...

Apache: http://dev.transitionnetwork.org/

hours, totalhours changed

chris — Thu, 05 Apr 2012 11:27:23 GMT

hours changed from 0.0 to 1.0
totalhours changed from 30.76 to 31.76

The Drupal sites are now available via nginx (but HTTPS only authenticated sessions haven’t been configured yet):

I'm now going to switch all the ports around so that Nginx is on port 80 and 443 and apache will just be used for Trac, further configuration and all the php-fpm and nginx cache tuning can be done after this switch -- at the moment the server is really hard to work on due to the lack of memory, this should improve after the switch, see https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/memory.html (almost 1.5G is being used but there is only 1G -- it's swapping a lot).

hours, totalhours changed

chris — Thu, 05 Apr 2012 13:00:34 GMT

hours changed from 0.0 to 0.5
totalhours changed from 31.76 to 32.26

All references to port 8000 changed to 80 and all references to 4430 changed to 443 in /etc/nginx/sites-available.

For apache all 443 VirtualServers wrapped in <IfModule mod_ssl.c> and mod-ssl disabled (at a later date all the symlinks in /etc/apache2/sites-enabled apart from the trac one can be deleted).

cd /etc/apache2/mods-enabled
rm ssl.conf ssl.load

Varnish stopped, apache restarted, nginx restarted.

TODO:

HTTPS only for authenticated sessions for Drupal sites
Security checking
Nginx cache configuration
php-fpm configuration
Testing
Documentation
Live migration

hours, totalhours changed

chris — Thu, 05 Apr 2012 15:25:55 GMT

hours changed from 0.0 to 0.1
totalhours changed from 32.26 to 32.36

Do we still need the subversion repo?

It's currently not set up to work with nginx, I could set up nginx to proxy requests to apache to get it working again if needs be:

hours, totalhours changed

chris — Thu, 05 Apr 2012 15:30:37 GMT

hours changed from 0.0 to 0.03
totalhours changed from 32.36 to 32.39

Munin added to TODO list:

Sort out Munin stats
HTTPS only for authenticated sessions for Drupal sites
Security checking
Nginx cache configuration
php-fpm configuration
Testing
Documentation
Live migration

hours, totalhours changed

chris — Thu, 05 Apr 2012 16:41:21 GMT

hours changed from 0.0 to 0.43
totalhours changed from 32.39 to 32.82

The forum archive had no CSS:

http://2011.archive.transitionnetwork.org/forum/

And the tag pages and rss feeds were 404's, these have all been fixed in /etc/nginx/archive-shared.

hours, totalhours changed

chris — Thu, 05 Apr 2012 17:12:01 GMT

hours changed from 0.0 to 0.1
totalhours changed from 32.82 to 32.92

Cron added to TODO list:

Fix cron jobs: lynx -connect_timeout=60 -dump http://dev.transitionnetwork.org/cron.php > /dev/null (failed)
Sort out Munin stats
HTTPS only for authenticated sessions for Drupal sites
Security checking
Nginx cache configuration
php-fpm configuration
Testing
Documentation
Live migration

hours, totalhours changed

chris — Tue, 17 Apr 2012 10:42:42 GMT

hours changed from 0.0 to 0.23
totalhours changed from 32.92 to 33.15

Replying to chris:

Fix cron jobs: lynx -connect_timeout=60 -dump http://dev.transitionnetwork.org/cron.php > /dev/null (failed)

This has been fixed, only connections from the server itself are allowed to the cron.php script, in /etc/nginx/drupal-shared:

        location ~ /cron.php$ {
                allow 127.0.1.1;
                allow 81.95.52.78;
                deny all;
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
                fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;
        }

hours, totalhours changed

chris — Tue, 17 Apr 2012 11:43:25 GMT

hours changed from 0.0 to 0.5
totalhours changed from 33.15 to 33.65

Replying to chris:

Sort out Munin stats

Apache stats fixed by changing the port in /etc/munin/plugin-conf.d/munin-node:

[apache*]
user root
env.url   http://kiwi.webarch.net:%d/server-status?auto
env.ports 8080

See: https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#apache

The problem with the Munin stats, https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#munin was because of permissions, I'm not sure what changed to cause this, this was the problem:

munin-run munin_stats
update.extinfo Can't open /var/log/munin/munin-update.log for reading
update.value U
graph.extinfo Can't open /var/log/munin/munin-graph.log for reading
graph.value U
html.extinfo Can't open /var/log/munin/munin-html.log for reading
html.value U
limits.extinfo Can't open /var/log/munin/munin-limits.log for reading
limits.value U

This was fixed by adding the following to /etc/munin/plugin-conf.d/munin-node:

[munin_stats]
user root

Now the test returns some values:

munin-run munin_stats
update.value U
graph.value 39.53
html.value 6.65
limits.value U

Nginx stats fixed by changing the port in /etc/munin/plugin-conf.d/munin-node:

[nginx*]
env.url http://kiwi.webarch.net:80/nginx_status

See https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#nginx

hours, totalhours changed

chris — Tue, 17 Apr 2012 12:30:19 GMT

hours changed from 0.0 to 0.5
totalhours changed from 33.65 to 34.15

Replying to chris:

HTTPS only for authenticated sessions for Drupal sites

With apache we had these rules to redirect port 80 connections:

  # redirect for encrypted logins etc
  RedirectPermanent /user https://dev.transitionnetwork.org/user
  RedirectPermanent /admin https://dev.transitionnetwork.org/admin
  RedirectPermanent /contact https://dev.transitionnetwork.org/contact
  RedirectPermanent /install.php https://dev.transitionnetwork.org/install.php
  RedirectPermanent /update.php https://dev.transitionnetwork.org/update.php
  RedirectMatch /(.*)\/contact$ https://dev.transitionnetwork.org/$1/contact

These can be replaced with these nginx rules in /etc/nginx/sites-available/dev and /etc/nginx/sites-available/test:

        # Redirects to HTTPS
        location ^~ /user               { return 301 https://$host$request_uri; }
        location ^~ /admin              { return 301 https://$host$request_uri; }
        location ^~ /contact            { return 301 https://$host$request_uri; }
        location ^~ /install.php        { return 301 https://$host$request_uri; }
        location ^~ /update.php         { return 301 https://$host$request_uri; }
        location ^~ /(.*)\/contact$     { return 301 https://$host$request_uri; }

hours, totalhours changed

chris — Tue, 17 Apr 2012 13:22:57 GMT

hours changed from 0.0 to 0.2
totalhours changed from 34.15 to 34.35

Remaining TODO items:

Security checking
Nginx cache configuration
php-fpm configuration
Testing
Documentation
Live migration

Also I note that at the moment the dev site appears to be slower with nginx than it was with varnish and apache, for example the front page of the dev site (just the HTML not any associated files) takes a minimum of about 3 seconds and at times a lot longer:

ab -n 1 -v 4 http://dev.transitionnetwork.org/
Time taken for tests:   4.152 seconds
Time taken for tests:   2.617 seconds
Time taken for tests:   8.725 seconds
Time taken for tests:   4.231 seconds
Time taken for tests:   7.692 seconds
Time taken for tests:   14.641 seconds
Time taken for tests:   22.364 seconds
Time taken for tests:   6.614 seconds
Time taken for tests:   4.599 seconds
Time taken for tests:   4.237 seconds

I think a lot of optimisation is going to be needed before nginx is as fast as the current setup with apache / varnish.

chris — Tue, 17 Apr 2012 19:23:12 GMT

Some Drupal resources to consider from Jim:

http://drupalcode.org/project/barracuda.git/tree/HEAD:/aegir/conf

hours, totalhours changed

chris — Tue, 01 May 2012 12:15:32 GMT

hours changed from 0.0 to 1.0
totalhours changed from 34.35 to 35.35

Some related tickets which have been now closed:

HTTPS Security Issues ticket:409
Pressflow problem on Dev site ticket:406

Back to nginx and php-fpm configuration...

Error generated due to lack of memory for PHP when accessing http://dev.transitionnetwork.org/admin/reports/status

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 64 bytes) in /web/dev.transitionnetwork.org.webarch.net/www/includes/cache.inc on line 32

In /etc/php5/fpm/php.ini the memory limit was increased:

#memory_limit = 128M
memory_limit = 256M

Things that needed adding back into /web/dev.transitionnetwork.org.webarch.net/www/sites/default/settings.php:

$conf = array(
'site_name' => 'This is the Transition Network DEV server',
);
$cookie_domain = '.dev.transitionnetwork.org';
# JK - added according to no anon session module readme
# chris - commented as it doesn't appear to be installed any more?
#$confsession_inc = './sites/all/modules/no_anon/session-no-anon.inc';
# memcache
$conf = array(
  'cache_inc' => './sites/all/modules/memcache/memcache.inc',
  'memcache_servers' => array('127.0.0.1:11211' => 'default'),
  'memcache_bins' => array(
    'cache' => 'default',
    'cache_content' => 'database',
    'cache_form' => 'database',
    'cache_views' => 'database'
  ),
);
/**
 * Reroute Email 6.x-1.x-dev variable to send emails to a different address for TEST
 *
 * JK - this is TEST so am rerouting emails!
 */
$conf['reroute_email_enable'] = 1;
# chris - following variable added
$conf['reroute_email_address'] = "transition-dev@email-lists.org";

Note that the site_name settings doesn't appear to be working -- the dev site name was still "Transition Network", so it was fixed at a mysql level:

mysql> select * from variable where name="site_name";
+-----------+----------------------------+
| name      | value                      |
+-----------+----------------------------+
| site_name | s:18:"Transition Network"; |
+-----------+----------------------------+
1 row in set (0.00 sec)
mysql> update variable set value='s:27:"Transition Network DEV Site";' where name="site_name";
Query OK, 1 row affected (0.09 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> select * from variable where name="site_name";
+-----------+-------------------------------------+
| name      | value                               |
+-----------+-------------------------------------+
| site_name | s:27:"Transition Network DEV Site"; |
+-----------+-------------------------------------+
1 row in set (0.00 sec)

The S:27 being 1 less than the character count.

This error is appearing at the top of all pages in admin:

Notice: Undefined index: icon in /web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/ctools/includes/content.inc on line 73

Also HTTPS logins still don't work, they redirect to the Pressflow installer so perhaps ticket:406 was closed too fast :-|

hours, totalhours changed

chris — Tue, 01 May 2012 12:57:07 GMT

hours changed from 0.0 to 0.7
totalhours changed from 35.35 to 36.05

On the Performance admin page, http://dev.transitionnetwork.org/admin/settings/performance the Page cache, Caching mode settings were changed from "External (experts only, possible side effects)" to "Normal (recommended for production sites, no side effects)".

The dev site isn't serving data gzipped (apache was), this can be tested using apache bench:

ab -v 4 -n 1 -H "Accept-Encoding: gzip" "http://dev.transitionnetwork.org/"

The nginx gzip config was tried but wasn't working so compression was enabled at a drupal level on the performance page, "Page compression", "Enabled", however this didn't result in gzip'ed pages being served either.

Updated TODO list:

Sort out nginx HTTPS and Drupal, this isn't working at all for the dev site, try to login and you get the install page.
Sort out no-cookies and caching for port 80, authenticated only sessions on port 443.
Sort out compression for HTML and CSS.
Test the speed of the site and tune as needed.
Document the nginx setup.
When we are happy migrate the live server from apache to nginx.

hours, totalhours changed

chris — Fri, 04 May 2012 13:43:20 GMT

hours changed from 0.0 to 1.5
totalhours changed from 36.05 to 37.55

Finally getting somewhere with this there was a mistake in the nginx root path for HTTPS (do'h!). There was also a problem setting the HTTPS env var for php-fpm, the following in a location block (shared for the HTTP and HTTPS settings) didn't work but perhaps there is a way to get this to work, will look at it again next week:

if ($ssl = on) {
       fastcgi_param HTTPS on;
}

You can now login to the dev site using https, but the session cookies set wasn't secure, Session 443 has been re-enabled:

drush en session443
The following extensions will be enabled: session443
Do you really want to continue? (y/n): y
WD php: Notice: Undefined index: icon in ctools_content_process() (line 73 of                 [error]
/web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/ctools/includes/content.inc).
session443 was enabled successfully.                                                          [ok]
Redirects need to be enabled at admin/settings/session443                                     [status]

And the settings at https://dev.transitionnetwork.org/admin/settings/session443 were set to:

Enabled - Redirection may happen according to the rules below.
Redirect authenticated users to HTTPS and redirect anonymous users on login/registration pages to HTTPS. Anonymous users visiting other pages may use HTTP or HTTPS.
Force all pages with the login block to use HTTPS.

This config was also added to /web/dev.transitionnetwork.org.webarch.net/www/sites/default/settings.php:

      if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] != 'off') {
        ini_set('session.cookie_secure', 1);
      }

And now authenticated session cookies to have the secure flag set.

Also an insecure cookie is set:

LOGGED_IN=1

Which results in all HTTP requests being redirected to HTTPS -- this cookie was being destroyed with our live varnish set up to enable more content to be cached, but it's a handy thing to have so I think we should perhaps make sure it's kept for php pages even if we destroy it for css and images, something for next week...

A ticket has been opened regarding the http link to the favicon in the admin menu, ticket:411

hours, totalhours changed

chris — Sat, 05 May 2012 11:33:50 GMT

hours changed from 0.0 to 1.4
totalhours changed from 37.55 to 38.95

I have read thought all the config suggested by Jim, ticket:357#comment:42 and there are lots of things that doesn't apply to us, I think we need to have a discussion about what the caching strategy is going to be with nginx because I'm not sure what to do next -- the current nginx setup on the dev server is slower than the apache/varnish setup on the live server, for example testing 20 concurrent requests for 600 copies of the front page from another server on the local network, first the dev server:

ab -v 4 -n 600 -c 20 -H "Accept-Encoding: gzip, deflate" http://dev.transitionnetwork.org/
Concurrency Level:      20
Time taken for tests:   10.100 seconds
Complete requests:      600
Failed requests:        7

Live server:

ab -v 4 -n 600 -c 20 -H "Accept-Encoding: gzip, deflate" http://www.transitionnetwork.org/
Concurrency Level:      20
Time taken for tests:   0.641 seconds
Complete requests:      600
Failed requests:        0

Less than 1 second with the live server and 10 seconds with the dev server.

Regading the gzipping issue mentioned in ticket:357#comment:44 it turns out that is is working when checked using the Firefox Live HTTP Headers plugin, I don't understand why gziped content isn't served to apache bench or when this online test http://www.gidnetwork.com/tools/gzip-test.php is used.

This doesn't result in gzipped content:

ab -v 4 -n 1 -H "Accept-Encoding: gzip, deflate" http://kiwi.transitionnetwork.org/

However using curl we do get gzipped content:

curl -I --compressed  http://kiwi.transitionnetwork.org/

So gzipping is working for some clients and not others, with apache all these clients get gzipped content.

hours, totalhours changed

chris — Wed, 09 May 2012 13:31:23 GMT

hours changed from 0.0 to 2.0
totalhours changed from 38.95 to 40.95

I have spent a couple of hours reading various nginx documentation, including https://calomel.org/nginx.html

I think we basically need to cache everything requested on port 80 and everything we can on port 443.

I have updated the Drush setup on the dev server based on the suggestions here, https://github.com/perusio/drupal-with-nginx

mkdir /etc/drush
cp /usr/local/drush/examples/example.aliases.drushrc.php \
/etc/drush/aliases.drushrc.php

Then the following was added to that file:

$aliases['dev'] = array(
    'uri' => 'dev.transitionetwork.org',
    'root' => '/web/dev.transitionnetwork.org.webarch.net/www',
  );
$aliases['test'] = array(
    'uri' => 'test.transitionetwork.org',
    'root' => '/web/test.transitionnetwork.org.webarch.net/www',
  );

And the cronjob for the dev site was changed to:

DRUSH=/usr/local/bin/drush
*/50 * * * * $DRUSH @dev cron -q

Further drush configuration could be done to allow interaction with the live server if needs be.

chris — Thu, 10 May 2012 16:14:27 GMT

php-fpm is not working at the moment, see ticket:218#comment:38

hours, totalhours changed

chris — Mon, 28 May 2012 13:22:06 GMT

hours changed from 0.0 to 1.09
totalhours changed from 40.95 to 42.04

Before caching is sorted out we need to get session 443 working properly, see comment:45 -- at the moment it isn't, the secure flag is being set on the LOGGED_IN cookie and this means that requests to the http version of the site are not being redirected, I'm not sure why as it was working before. I have checked the environment and with https the PHP variable _SERVER["HTTPS"] is set to on and with http it isn't set -- this is how it should be.

There is also this issue with the lifetime of the LOGGED_IN cookie which hasn't been fixed in the current version of session443, https://drupal.org/node/1338266 -- we should probably apply it.

hours, totalhours changed

chris — Tue, 29 May 2012 10:25:52 GMT

hours changed from 0.0 to 0.2
totalhours changed from 42.04 to 42.24

I have done some more testing with the Firefox Live HTTP Headers and Advanced Cookie Manager and Session 443 does appear to be working correctly.

When one is on the login page at https://dev.transitionnetwork.org/user/login and the form is POST'ed a 302 is returned to redirect people to their user page and this Set-Cookie header is sent with the 302:

Set-Cookie: SESS456789012b2e466470fa7d2012345f24c=27f9cdaa43c6c0987f0f290d3456789b; expires=Thu, 21-Jun-2012 13:46:35 GMT; path=/; domain=.dev.transitionnetwork.org; secure; HttpOnly

And then the user page is requested and with that page the LOGGED_IN cookie is set (correctly not secure):

Set-Cookie: LOGGED_IN=1; path=/; domain=.dev.transitionnetwork.org

Note that this should have the same expired date as the session cookie, see comment:49

If one then visits the http version of the site the LOGGED_IN cookie is sent and a 302 redirect the user back to HTTPS.

I don't know why this wasn't workijng last night, since then I have restarted Firefox and cleared all the *.transitionnetwork.org cookies out -- this seems to have fixed it.

status changed; resolution set

chris — Fri, 25 Jan 2013 16:35:02 GMT

status changed from accepted to closed
resolution set to fixed

The new wiki:PenguinServer is now up and running and all the sites on it are using Nginx (Trac is using Nginx as a reverse proxy to tracd) for more details of the Nginx configuration see ticket:470.

The main transition network site is also due to be running on Nginx as soon as wiki:PuffinServer goes live, see ticket:466.

So this ticket can finally be closed.