nginx research

[chris]

Looking into replacing apache with nginx by testing it on the dev server.

[jim]

As I mentioned I've set myself up a low-end Linode so I can play and consolidate my hosting away from a shared environment.

In my travels I've found 'Barracuda' which is a meaty script that took my empty Debian 6 server and got NGINX, PHP-FPM, memcache, a fire wall and bunch of other handy stuff... Now most of that is pointless for here but thought you might want to take a gander at the script that set it up so you can see what's what with regards to NGINX and PHP-FPM config: ​http://drupal.org/project/barracuda

Also, as mentioned, there's ​http://groups.drupal.org/nginx with lots of people and information.

[chris]

The pages are now available via nginx:

• ​http://kiwi.transitionnetwork.org:8000/
• ​https://kiwi.transitionnetwork.org:4430/

I'll next look at setting up munin before I look at making the (far more compicated) other virtualhosts work with nginx.

[chris]

I have added some munin monitoring for nginx and we should start to get some stats here:

​https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html

[chris]

PHP-FPM, following ​http://fak3r.com/geek/howto/howto-install-php5-fpm-on-debian-squeeze/

Add this to /etc/apt/sources.list:

deb http://packages.dotdeb.org squeeze all
deb-src http://packages.dotdeb.org squeeze all

cd
gpg --keyserver keys.gnupg.net --recv-key 89DF5277
gpg -a --export 89DF5277 | sudo apt-key add -
aptitude update
aptitude safe-upgrade
 The following NEW packages will be installed:
   nginx-common{a} nginx-light{a} 
 The following packages will be upgraded:
   libapache2-mod-php5 libmysqlclient16 mysql-client-5.1 mysql-common mysql-server mysql-server-5.1 
   mysql-server-core-5.1 nginx php-pear php5 php5-cli php5-common php5-curl php5-dev php5-gd 
   php5-imagick php5-mcrypt php5-memcache php5-mysql php5-suhosin 

With a new LAMP stack all the config files need updating, the old ones were backed up, new ones installed and diffs were done, these were the changes to /etc/php5/apache2/php.ini:

expose_php = Off
memory_limit = 256M
post_max_size = 128M
default_charset = "utf-8"
upload_max_filesize = 100M
max_file_uploads = 50
default_socket_timeout = 120
extension=uploadprogress.so
session.cookie_secure = 1
mbstring.http_input = pass
mbstring.http_output = pass

Things that we had in the old php.ini which I'm note sure if we need in the new one:

output_buffering = Off
serialize_precision = 100
variables_order = "EGPCS"
register_long_arrays = On
register_argc_argv = On
cgi.nph = 1
cgi.fix_pathinfo=0

nginx and php still need more work to get them working...

[chris]

Oh, I see that the PHP upgrade has broken ​http://dev.transitionnetwork.org/

Parse error: syntax error, unexpected T_SL in /web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/imageapi/imageapi_imagemagick.module on line 152 

I'll look at fixing this before looking at ticket:364

[chris]

Actually it's a diff in the code in /web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/imageapi/imageapi_imagemagick.module:

<<<<<<< .mine
  setlocale(LC_CTYPE, "en_GB.UTF-8");
=======
  // LANG added by chris for UTF-8 filenames
  setlocale(LC_CTYPE, "en_GB.UTF-8");
>>>>>>> .r224

I have fixed the for the dev server but not committed it.

[chris]

Replying to chris:

The pages are now available via nginx:

• ​http://kiwi.transitionnetwork.org:8000/
• ​https://kiwi.transitionnetwork.org:4430/

The new certs and debian upgrades produced some error messages:

Nginx 'soft' update failed, doing restart
Starting nginx: nginx: [emerg] SSL_CTX_use_certificate_chain_file("/etc/ssl/transitionnetwork.org/transitionnetwork.org.crt") failed (SSL: error:0906D066:PEM routines:PEM_read_bio:bad end line error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib)
nginx: configuration file /etc/nginx/nginx.conf test failed
invoke-rc.d: initscript nginx, action "start" failed.
dpkg: error processing nginx-light (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of nginx:
 nginx depends on nginx-full | nginx-light; however:
  Package nginx-full is not installed.
  Package nginx-light is not configured yet.
dpkg: error processing nginx (--configure):
 dependency problems - leaving unconfigured

So I'm surprised that nginx is still running on the dev server, I have put the errors here so they can be investigated next time this ticket is worked on.

[chris]

Following the last certificate update nginx on the dev server has been broken:

/etc/init.d/nginx start
Starting nginx: nginx: [emerg] SSL_CTX_use_certificate_chain_file("/etc/ssl/transitionnetwork.org/transitionnetwork.org.crt") failed (SSL: error:0906D066:PEM routines:PEM_read_bio:bad end line error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib)
nginx: configuration file /etc/nginx/nginx.conf test failed

This was fixed thus:

cd /etc/ssl/transitionnetwork.org/
wget http://crt.gandi.net/GandiStandardSSLCA.crt
wget http://crt.usertrust.com/UTNAddTrustServer_CA.crt
wget http://crt.usertrust.com/AddTrustExternalCARoot.crt
openssl x509 -inform DER -in GandiStandardSSLCA.crt -out GandiStandardSSLCA.pem
openssl x509 -inform DER -in AddTrustExternalCARoot.crt -out AddTrustExternalCARoot.pem
openssl x509 -inform DER -in UTNAddTrustServer_CA.crt -out UTNAddTrustServer_CA.pem
cat transitionnetwork.org.crt > gandi.pem
cat GandiStandardSSLCA.pem >> gandi.pem
cat AddTrustExternalCARoot.pem >> gandi.pem
cat UTNAddTrustServer_CA.pem >> gandi.pem
cat transitionnetwork.org.crt gandi.pem > transitionnetwork.org.chained.pem

And /etc/nginx/sites-available/kiwi was edited:

#ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.crt;
ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;

So now Nginx is working again for this site:

• ​http://kiwi.transitionnetwork.org:8000/
• ​https://kiwi.transitionnetwork.org:4430/

And the next task is to enable all the other sites on the server, one at a time, till they are all running and then we can switch the ports around and turn apache off.

[chris]

Working through each domain name at a time enabling all the services, last year kiwi.transitionnetwork.org was set up to work via nginx:

• ​http://kiwi.transitionnetwork.org:8000/
• ​https://kiwi.transitionnetwork.org:4430/

And Munin is available via nginx:

• ​https://kiwi.transitionnetwork.org:4430/munin/

However /info/ and /phpmyadmin and /apc_info.php don't yet work:

• ​https://kiwi.transitionnetwork.org:4430/info/
• ​https://kiwi.transitionnetwork.org:4430/phpmyadmin
• ​https://kiwi.transitionnetwork.org:4430/apc_info.php

So I'm going to work on enabling them to start with.

[chris]

One issue I have just realised that we had with the current setup with varnish, these pages (used by Munin to generate graphs), the second of which displays client's IP addresses (when clients use HTTPS) are set to only be available to the localhost via apache:

• ​http://kiwi.transitionnetwork.org:8080/apc_info.php
• ​http://kiwi.transitionnetwork.org:8080/server-status

However they were available via varnish to anyone:

• ​http://kiwi.transitionnetwork.org/apc_info.php
• ​http://kiwi.transitionnetwork.org/server-status

I have fixed this on the live and dev server by adding these rules to the varnish config at /etc/varnish/default.vcl

acl local {
  "localhost";         // myself
  "127.0.0.1";         // myself
  "81.95.52.78";       // this machines main ip address
  "81.95.52.79";       // this machines 2nd ip address
  "81.95.52.80";       // this machines 3rd ip address
}

    ## Pass cron jobs and server-status
    if (req.url ~ "cron.php") {
      if (client.ip ~ local) {
        return (pass);
      }
      else {
        error 403 "Access Denied";
      }
    }
    if (req.url ~ "/server-status$") {
      if (client.ip ~ local) {
        return (pass);
      }
      else {
        error 403 "Access Denied";
      }
    }
    if (req.url ~ "apc_info.php") {
      if (client.ip ~ local) {
        return (pass);
      }
      else {
        error 403 "Access Denied";
      }
    }

The varnish documentation has been updated to match the current configuration, wiki:DevelopmentServer#Varnish and wiki:NewLiveServer#varnish

[chris]

Enabling PHP and nginx, the notes previously followed in ticket:357#comment:5 have moved to ​http://fak3r.com/2011/09/27/howto-install-php5-fpm-on-debian-squeeze/ also following the notes here, ​http://www.webhostingtalk.com/showthread.php?t=1025286

aptitude install php5-fpm
  The following NEW packages will be installed:
    php5-fpm 
  Creating config file /etc/php5/fpm/php.ini with new version

Start it:

/etc/init.d/php5-fpm start

Check it's running:

netstat -plunt|grep php
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      31616/php-fpm.conf)

Add this to /etc/nginx/sites-available/kiwi:

location ~ \.php$ {
      fastcgi_pass 127.0.0.1:9090;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME /web/kiwi.webarch.net/www$fastcgi_script_name;
      include fastcgi_params;
}

But the php files result in:

502 Bad Gateway

And this in /var/log/nginx/error.log:

2012/03/15 15:56:05 [error] 2763#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: XX.XX.XX.XX, server: kiwi.transitionnetwork.org, request: "GET /info/php-info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9090", host: "kiwi.transitionnetwork.org:4430"

I tried editing the list of allowed hosts, /etc/php5/fpm/pool.d/www.conf:

listen.allowed_clients = 127.0.0.1,81.95.52.78,kiwi.transitionnetwork.org,kiwi.webarch.net

I have read a lot of google results but have yet to find an answer to this problem, will pick this up again tomorrow...

[chris]

My bad, /etc/php5/fpm/pool.d/www.conf had port 9000 in it not port 9090, fixed:

listen = 127.0.0.1:9090

Added a root directive to the .php section as ​suggested here to get php scripts working:

location ~ \.php$ {
      fastcgi_pass 127.0.0.1:9090;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME /web/kiwi.webarch.net/www$fastcgi_script_name;
      include fastcgi_params;
      root   "/web/kiwi.webarch.net/www";
}

Now these work:

• ​https://kiwi.transitionnetwork.org:8080/apc_info.php
• ​https://kiwi.transitionnetwork.org:4430/apc_info.php
• ​https://kiwi.transitionnetwork.org:4430/info/php-info.php
• ​https://kiwi.transitionnetwork.org:4430/info/apc.php

And I think that wraps it up for nginx doing everything apache does for the kiwi.transitionnetwork.org domain, 1 down, quite a few to go!

[chris]

Getting Piwik running with nginx, apache config here /etc/apache2/sites-enabled/03-piwik.transitionnetwork.org, nginx config here /etc/nginx/sites-available/piwiw, initially created as a copy of /etc/nginx/sites-available/default.dpkg-dist, referencing ​http://wiki.nginx.org/MediaWiki.

These lines were edited, initially to get it working without HTTPS:

        listen    8080;
        root /web/piwik.transitionnetwork.org/piwik;
        index index.php;
        server_name piwik.transitionnetwork.org piwik.transitionnetwork.org.webarch.net;
        root /web/piwik.transitionnetwork.org/piwik;
        index index.php;

        # Make site accessible from http://localhost/
        server_name piwik.transitionnetwork.org piwik.transitionnetwork.org.webarch.net;
        client_max_body_size 5m;
        client_body_timeout 60;
                
        # see http://wiki.nginx.org/MediaWiki
        location / {
                try_files $uri $uri/ @rewrite;
        }       
        location @rewrite {
                rewrite ^/(.*)$ /index.php?title=$1&$args;
        }
        location ^~ /maintenance/ {
                return 403;
        }
        location ~ \.php$ {
                include fastcgi_params;
                fastcgi_pass unix:/tmp/phpfpm.sock;
        }
        location ~ \.php$ {
                fastcgi_pass 127.0.0.1:9090;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /web/piwik.transitionnetwork.org/piwik$fastcgi_script_name;
                include fastcgi_params;
                root   "/web/kiwi.webarch.net/www";
        }
        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                try_files $uri /index.php;
                expires max;
                log_not_found off;
        }       
        location = /_.gif {
                expires max;
                empty_gif;
        }
        location ^~ /cache/ {
                deny all;
        }

Make the config available and restart

cd /etc/nginx/sites-enabled
ln -s ../sites-available/piwik 02-piwik
/etc/init.d/nginx restart
  Restarting nginx: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
  nginx: [emerg] still could not bind()
  nginx.

So, something isn't right, I'll come back to the later after looking at ticket:401

[chris]

Several mistakes in the server config were fixed and now the mediawiki site is available (unencrypted) via nginx at ​http://wiki.dev.transitionnetwork.org:8080/Main_Page

server {
        listen    8000;
        #listen   80; ## listen for ipv4; this line is default and implied
        #listen   [::]:80 default_server ipv6only=on; ## listen for ipv6


        root /web/wiki.dev.transitionnetwork.org/www;
        index index.php;

        # Make site accessible from http://localhost/
        server_name wiki.dev.transitionnetwork.org wiki.dev.transitionnetwork.org.webarch.net;
        client_max_body_size 5m;
        client_body_timeout 60;

        # see http://wiki.nginx.org/MediaWiki
        location / {
                try_files $uri $uri/ @rewrite;
        }
        location @rewrite {
                rewrite ^/(.*)$ /index.php?title=$1&$args;
        }
        location ^~ /maintenance/ {
                return 403;
        }
        location ~ \.php$ {
                include fastcgi_params;
                fastcgi_pass unix:/tmp/phpfpm.sock;
        }
        location ~ \.php$ {
                fastcgi_pass 127.0.0.1:9090;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /web/wiki.dev.transitionnetwork.org/www$fastcgi_script_name;
                include fastcgi_params;
                root   "/web/wiki.dev.transitionnetwork.org/www";
        }
        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                try_files $uri /index.php;
                expires max;
                log_not_found off;
        }
        location = /_.gif {
                expires max;
                empty_gif;
        }
        location ^~ /cache/ {
                deny all;
        }

}

Next task to make it available via https.

[chris]

All the nginx config is being revisited, been reading ​http://michael.lustfield.net/content/dummies-guide-nginx ​http://blog.martinfjordvald.com/2010/07/nginx-primer/ ​http://blog.martinfjordvald.com/2011/02/nginx-primer-2-from-apache-to-nginx/ ​https://nealpoole.com/blog/2011/04/setting-up-php-fastcgi-and-nginx-dont-trust-the-tutorials-check-your-configuration/ and ​http://wiki.nginx.org/Pitfalls

Changing php5-fpm to use a socket rather than tcp, create a dir for the sock file:

mkdir /var/run/php5-fpm
chown www-data:www-data /var/run/php5-fpm

Edit /etc/php5/fpm/pool.d/www.conf:

[www]
user = www-data
group = www-data
listen = /var/run/php5-fpm/phpfpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0666
listen.allowed_clients = 127.0.0.1,81.95.52.78,kiwi.transitionnetwork.org,kiwi.webarch.net
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /

Security fix, see ​https://nealpoole.com/blog/2011/04/setting-up-php-fastcgi-and-nginx-dont-trust-the-tutorials-check-your-configuration/ edit /etc/php5/fpm/php.ini:

cgi.fix_pathinfo=0

These are working:

• ​https://kiwi.transitionnetwork.org:4430/
• ​https://kiwi.transitionnetwork.org:4430/munin/
• ​https://kiwi.transitionnetwork.org:4430/info/
• ​https://kiwi.transitionnetwork.org:4430/info/apc.php
• ​https://kiwi.transitionnetwork.org:4430/info/php-info.php

This needs more work:

• ​https://kiwi.transitionnetwork.org:4430/phpmyadmin/

[chris]

kiwi.transitionnetwork.org is now fully working with nginx (config file /etc/nginx/sites-available/kiwi) and I'm fairly confident that it's been set up to be as secure as possible, the configuration still needs documenting on this wiki, but the config file itself has lots of comments, these things are available on this domain:

• ​http://kiwi.transitionnetwork.org:8000/ - server front page / documentation
• ​http://kiwi.transitionnetwork.org:8000/apc_info.php - script for apc munin stats, only available from localhost
• ​https://kiwi.transitionnetwork.org:4430/munin/ - munin stats
• ​https://kiwi.transitionnetwork.org:4430/info/php-info.php - php info (password protected)
• ​http://kiwi.transitionnetwork.org:8000/info/apc.php - apc info (password protected)
• ​https://kiwi.transitionnetwork.org:4430/phpmyadmin/ - phpmyadmin (password protected)

Next to sort these out:

• ​http://static.dev.transitionnetwork.org/
• ​http://2011.archive.transitionnetwork.org/
• ​http://piwik.transitionnetwork.org/
• ​http://wiki.dev.transitionnetwork.org/
• ​http://dev.transitionnetwork.org/
• ​http://test.transitionnetwork.org/
• ​http://news.dev.transitionnetwork.org/

[chris]

Replying to chris:

Next to sort these out:

• ​http://static.dev.transitionnetwork.org/

Done, config file /etc/nginx/sites-available/static:

• ​http://static.dev.transitionnetwork.org:8000/
• ​http://static.dev.transitionnetwork.org:8000/ttcon2010/ - 2010 conference static files
• ​https://static.dev.transitionnetwork.org:4430/ttcon2010/private - private files from the conference

More to follow tomorrow...

[chris]

These two sites are going to be very time consuming to migrate due to the number of rewrite rules (98 lines of apache config containing the word Rewrite), I wonder if we shouldn't keep apache just for these and use nginx as a reverse proxy for these domains.

• ​http://2011.archive.transitionnetwork.org/
• ​http://2010.archive.transitionnetwork.org/

[chris]

I have done a load of work on a secure Mediawiki nginx config, but it's still not totally working and I'm not happy with it yet, hopefully I'll complete this tomorrow.

References:

• ​http://wiki.nginx.org/MediaWiki
• ​http://blog.bigdinosaur.org/mediawiki-on-nginx/

[chris]

Did a lot of work on the wiki config, got things that didn't work that were suggested here ​http://blog.bigdinosaur.org/mediawiki-on-nginx/ working but then a combination of a really poor ADSL connection and god knows what resulted in the config file I was working on getting deleted, so I'll have to start again next week, really sorry about this :-(

I did lean more about nginx in the process though...

[chris]

Replying to chris:

a combination of a really poor ADSL connection and god knows what

The high load on kiwi making vim (the text editor I use) really unresponsive, which I mistook for connection problems (which we have been having at work) and my fustration at not being able to get stuff done as a result is probably what...

The thing which took ages to figure out was this code from ​http://blog.bigdinosaur.org/mediawiki-on-nginx/ which doesn't work:

#    Force potentially-malicious files in the /images directory to be served
#    with a text/plain mime type, to prevent them from being executed by
#    the PHP handler
  location ~* ^/images/.*.(html|htm|shtml|php)$ {
      types { }
      default_type text/plain;
  }

It's a very good idea to do this for any directory where users can upload files so I was keen to get it sorted so it could be used for other applications.

The code above simply does nothing, I'm not exactly sure why as the regex would be OK for apache, I haven't got to the bottom of why the above doesn't work.

The solution I finally got working was, in essence, this:

  location ^~ /images/ {
      types { 
        text/plain    htm html shtml php php5;
         image/gif    gif;
         # ...
         # whitelist here of all file types allowed to be uploaded, 
         # see LocalSettings.php for a list
      }
      try_files $uri /index.php;
  }

It won't take long on Monday to recreate the file that got deleted.

[chris]

Replying to chris:

It won't take long on Monday to recreate the file that got deleted.

I spent a couple of hours on this today, before I was interupted by a power outage in the office, I should get mediawiki sorted tomorrow.

[chris]

I have finally wrapped up Mediawiki on nginx:

• ​http://wiki.dev.transitionnetwork.org:8000/Main_Page
• ​https://wiki.dev.transitionnetwork.org:4430/Main_Page

All logins are redirected to HTTPS and I have tested everything I can think of and everything is working.

I have discovered an apache to nginx config convertor which should speed up progress:

• ​http://www.anilcetin.com/convert-apache-htaccess-to-nginx/

For reference if you get this error message:

==> /var/log/nginx/wiki.ssl_error.log <==
2012/03/27 10:57:15 [info] 26009#0: *18 client sent plain HTTP request to HTTPS port while reading client request headers, client: XX.XX.XX.XX, server: wiki.dev.transitionnetwork.org, request: "OPTIONS /index.php?action=ajax HTTP/1.1", host: "wiki.dev.transitionnetwork.org:4430"

==> /var/log/nginx/wiki.ssl_access.log <==
81.95.52.29 - - [27/Mar/2012:10:57:15 +0100] "OPTIONS /index.php?action=ajax HTTP/1.1" 400 271 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0 Iceweasel/11.0"

It can be fixed by adding this to the location section of the config for the php scripts:

fastcgi_param HTTPS on;

[chris]

I have made a start on the regular expressions for:

• ​http://2010.archive.transitionnetwork.org/
• ​http://2011.archive.transitionnetwork.org/

[chris]

These sites are now available via nginx on port 8000:

• ​http://2010.archive.transitionnetwork.org:8000/
• ​http://2011.archive.transitionnetwork.org:8000/

But there is a problem with the regular expressions for the forum archive, I had it working earlier but it's now broken and needs fixing:

• ​http://2011.archive.transitionnetwork.org:8000/forum/

See also wiki:DevelopmentServer#bbPress

I have also discovered the files from a old Wordpress site which was at ​http://transitiontowns.org/webprojectblog -- I assume this is safe to ignore -- I have set nginx up not to allow access to it.

[chris]

The fix for the old forum was easy:

• ​http://2011.archive.transitionnetwork.org:8000/forum/

Now onto Pwiki, Trac and then Drupal...

[chris]

Piwki has been setup, following ​http://wiki.nginx.org/Piwik which is based on ​https://github.com/perusio/piwik-nginx

A directory for the cache was needed:

mkdir /var/cache/nginx/fcgicache -p
chown www-data:www-data /var/cache/nginx/fcgicache

I added a web bug to ​http://kiwi.transitionnetwork.org/ to test it:

• ​https://piwik.transitionnetwork.org:4430/

[chris]

Setting up Trac on Nginx isn't going to be simple, some resources on this:

• ​http://wiki.nginx.org/NonRootWebPathApacheStyle
• ​http://trac.edgewall.org/wiki/TracNginxRecipe
• ​http://trac.edgewall.org/wiki/TracFastCgi#SimpleNginxConfiguration
• ​https://coderanger.net/~coderanger/tracdoc/install/fastcgi.html#nginx
• ​http://wiki.nginx.org/NginxHttpFcgiModule

See also wiki:DevelopmentServer#Trac ticket:1 and ticket:364

There is also the old subversion repo here ​https://tech.transitionnetwork.org/svn/ I guess this isn't needed any more?

Python is a bit of mess on the development server because (a) a source version was installed in order to get the GA Piwik import working and (b) a unstable version was installed in order to install a unstable version of Trac in order to get it working with git...

​https://tech.transitionnetwork.org/trac/ticket/364#comment:36

I'm also concerned that although trac isn't working perfectly at the moment at least it's working...

Do we need trac to be integrated with github?

For now I think I'll look at setting up nginx as a reverse proxy for apache for trac and then fixing trac can be done after the switch to Nginx...

[chris]

Trac with Nginx acting as a reverse proxy to apache:

​https://tech.transitionnetwork.org:4430/trac/

[chris]

I have made a start on the Drupal sites:

• Nginx: ​http://dev.transitionnetwork.org:8000/

This still needs lots of work to replicate the varnish caching etc and setting it up so that authenticated sessions use HTTPS, at the moment it doesn't seem any quicker than apache / varnish, but it should be when it's done...

• Apache: ​http://dev.transitionnetwork.org/

[chris]

The Drupal sites are now available via nginx (but HTTPS only authenticated sessions haven’t been configured yet):

• ​http://dev.transitionnetwork.org:8000/
• ​https://dev.transitionnetwork.org:4430/

• ​http://test.transitionnetwork.org:8000/
• ​https://test.transitionnetwork.org:4430/

• ​http://news.dev.transitionnetwork.org:8000/
• ​https://news.dev.transitionnetwork.org:4430/

I'm now going to switch all the ports around so that Nginx is on port 80 and 443 and apache will just be used for Trac, further configuration and all the php-fpm and nginx cache tuning can be done after this switch -- at the moment the server is really hard to work on due to the lack of memory, this should improve after the switch, see ​https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/memory.html (almost 1.5G is being used but there is only 1G -- it's swapping a lot).

[chris]

All references to port 8000 changed to 80 and all references to 4430 changed to 443 in /etc/nginx/sites-available.

For apache all 443 VirtualServers wrapped in <IfModule mod_ssl.c> and mod-ssl disabled (at a later date all the symlinks in /etc/apache2/sites-enabled apart from the trac one can be deleted).

cd /etc/apache2/mods-enabled
rm ssl.conf ssl.load

Varnish stopped, apache restarted, nginx restarted.

TODO:

• HTTPS only for authenticated sessions for Drupal sites
• Security checking
• Nginx cache configuration
• php-fpm configuration
• Testing
• Documentation
• Live migration

[chris]

Do we still need the subversion repo?

It's currently not set up to work with nginx, I could set up nginx to proxy requests to apache to get it working again if needs be:

• ​http://mailman.nginx.org/pipermail/nginx/2007-January/000504.html
• ​http://stackoverflow.com/questions/2479346/502-bad-gateway-with-nginx-apache-subversion-ssl-svn-copy

[chris]

Munin added to TODO list:

• Sort out ​Munin stats
• HTTPS only for authenticated sessions for Drupal sites
• Security checking
• Nginx cache configuration
• php-fpm configuration
• Testing
• Documentation
• Live migration

[chris]

The forum archive had no CSS:

• ​http://2011.archive.transitionnetwork.org/forum/

And the tag pages and rss feeds were 404's, these have all been fixed in /etc/nginx/archive-shared.

[chris]

Cron added to TODO list:

• Fix cron jobs: lynx -connect_timeout=60 -dump http://dev.transitionnetwork.org/cron.php > /dev/null (failed)
• Sort out ​Munin stats
• HTTPS only for authenticated sessions for Drupal sites
• Security checking
• Nginx cache configuration
• php-fpm configuration
• Testing
• Documentation
• Live migration

[chris]

Replying to chris:

• Fix cron jobs: lynx -connect_timeout=60 -dump http://dev.transitionnetwork.org/cron.php > /dev/null (failed)

This has been fixed, only connections from the server itself are allowed to the cron.php script, in /etc/nginx/drupal-shared:

        location ~ /cron.php$ {
                allow 127.0.1.1;
                allow 81.95.52.78;
                deny all;
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
                fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;
        }

[chris]

Replying to chris:

• Sort out ​Munin stats

Apache stats fixed by changing the port in /etc/munin/plugin-conf.d/munin-node:

[apache*]
user root
env.url   http://kiwi.webarch.net:%d/server-status?auto
env.ports 8080

See: ​https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#apache

The problem with the Munin stats, ​https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#munin was because of permissions, I'm not sure what changed to cause this, this was the problem:

munin-run munin_stats
update.extinfo Can't open /var/log/munin/munin-update.log for reading
update.value U
graph.extinfo Can't open /var/log/munin/munin-graph.log for reading
graph.value U
html.extinfo Can't open /var/log/munin/munin-html.log for reading
html.value U
limits.extinfo Can't open /var/log/munin/munin-limits.log for reading
limits.value U

This was fixed by adding the following to /etc/munin/plugin-conf.d/munin-node:

[munin_stats]
user root

Now the test returns some values:

munin-run munin_stats
update.value U
graph.value 39.53
html.value 6.65
limits.value U

Nginx stats fixed by changing the port in /etc/munin/plugin-conf.d/munin-node:

[nginx*]
env.url http://kiwi.webarch.net:80/nginx_status

See ​https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#nginx

[chris]

Replying to chris:

• HTTPS only for authenticated sessions for Drupal sites

With apache we had these rules to redirect port 80 connections:

  # redirect for encrypted logins etc
  RedirectPermanent /user https://dev.transitionnetwork.org/user
  RedirectPermanent /admin https://dev.transitionnetwork.org/admin
  RedirectPermanent /contact https://dev.transitionnetwork.org/contact
  RedirectPermanent /install.php https://dev.transitionnetwork.org/install.php
  RedirectPermanent /update.php https://dev.transitionnetwork.org/update.php
  RedirectMatch /(.*)\/contact$ https://dev.transitionnetwork.org/$1/contact

These can be replaced with these nginx rules in /etc/nginx/sites-available/dev and /etc/nginx/sites-available/test:

        # Redirects to HTTPS
        location ^~ /user               { return 301 https://$host$request_uri; }
        location ^~ /admin              { return 301 https://$host$request_uri; }
        location ^~ /contact            { return 301 https://$host$request_uri; }
        location ^~ /install.php        { return 301 https://$host$request_uri; }
        location ^~ /update.php         { return 301 https://$host$request_uri; }
        location ^~ /(.*)\/contact$     { return 301 https://$host$request_uri; }

[chris]

Remaining TODO items:

• Security checking
• Nginx cache configuration
• php-fpm configuration
• Testing
• Documentation
• Live migration

Also I note that at the moment the dev site appears to be slower with nginx than it was with varnish and apache, for example the front page of the dev site (just the HTML not any associated files) takes a minimum of about 3 seconds and at times a lot longer:

ab -n 1 -v 4 http://dev.transitionnetwork.org/

Time taken for tests:   4.152 seconds
Time taken for tests:   2.617 seconds
Time taken for tests:   8.725 seconds
Time taken for tests:   4.231 seconds
Time taken for tests:   7.692 seconds
Time taken for tests:   14.641 seconds
Time taken for tests:   22.364 seconds
Time taken for tests:   6.614 seconds
Time taken for tests:   4.599 seconds
Time taken for tests:   4.237 seconds

I think a lot of optimisation is going to be needed before nginx is as fast as the current setup with apache / varnish.

[chris]

Some Drupal resources to consider from Jim:

​http://drupalcode.org/project/barracuda.git/tree/HEAD:/aegir/conf

[chris]

Some related tickets which have been now closed:

• HTTPS Security Issues ticket:409
• Pressflow problem on Dev site ticket:406

Back to nginx and php-fpm configuration...

Error generated due to lack of memory for PHP when accessing ​http://dev.transitionnetwork.org/admin/reports/status

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 64 bytes) in /web/dev.transitionnetwork.org.webarch.net/www/includes/cache.inc on line 32

In /etc/php5/fpm/php.ini the memory limit was increased:

#memory_limit = 128M
memory_limit = 256M

Things that needed adding back into /web/dev.transitionnetwork.org.webarch.net/www/sites/default/settings.php:

$conf = array(
'site_name' => 'This is the Transition Network DEV server',
);

$cookie_domain = '.dev.transitionnetwork.org';

# JK - added according to no anon session module readme
# chris - commented as it doesn't appear to be installed any more?
#$confsession_inc = './sites/all/modules/no_anon/session-no-anon.inc';

# memcache
$conf = array(
  'cache_inc' => './sites/all/modules/memcache/memcache.inc',
  'memcache_servers' => array('127.0.0.1:11211' => 'default'),
  'memcache_bins' => array(
    'cache' => 'default',
    'cache_content' => 'database',
    'cache_form' => 'database',
    'cache_views' => 'database'
  ),
);

/**
 * Reroute Email 6.x-1.x-dev variable to send emails to a different address for TEST
 * 
 * JK - this is TEST so am rerouting emails!
 */
$conf['reroute_email_enable'] = 1;
# chris - following variable added
$conf['reroute_email_address'] = "transition-dev@email-lists.org";

Note that the site_name settings doesn't appear to be working -- the dev site name was still "Transition Network", so it was fixed at a mysql level:

mysql> select * from variable where name="site_name";
+-----------+----------------------------+
| name      | value                      |
+-----------+----------------------------+
| site_name | s:18:"Transition Network"; |
+-----------+----------------------------+
1 row in set (0.00 sec)

mysql> update variable set value='s:27:"Transition Network DEV Site";' where name="site_name";
Query OK, 1 row affected (0.09 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from variable where name="site_name";
+-----------+-------------------------------------+
| name      | value                               |
+-----------+-------------------------------------+
| site_name | s:27:"Transition Network DEV Site"; |
+-----------+-------------------------------------+
1 row in set (0.00 sec)

The S:27 being 1 less than the character count.

This error is appearing at the top of all pages in admin:

Notice: Undefined index: icon in /web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/ctools/includes/content.inc on line 73

Also HTTPS logins still don't work, they redirect to the Pressflow installer so perhaps ticket:406 was closed too fast :-|

[chris]

On the Performance admin page, ​http://dev.transitionnetwork.org/admin/settings/performance the Page cache, Caching mode settings were changed from "External (experts only, possible side effects)" to "Normal (recommended for production sites, no side effects)".

The dev site isn't serving data gzipped (apache was), this can be tested using apache bench:

ab -v 4 -n 1 -H "Accept-Encoding: gzip" "http://dev.transitionnetwork.org/"

The nginx gzip config was tried but wasn't working so compression was enabled at a drupal level on the performance page, "Page compression", "Enabled", however this didn't result in gzip'ed pages being served either.

Updated TODO list:

• Sort out nginx HTTPS and Drupal, this isn't working at all for the dev site, try to login and you get the install page.
• Sort out no-cookies and caching for port 80, authenticated only sessions on port 443.
• Sort out compression for HTML and CSS.
• Test the speed of the site and tune as needed.
• Document the nginx setup.
• When we are happy migrate the live server from apache to nginx.

[chris]

Finally getting somewhere with this there was a mistake in the nginx root path for HTTPS (do'h!). There was also a problem setting the HTTPS env var for php-fpm, the following in a location block (shared for the HTTP and HTTPS settings) didn't work but perhaps there is a way to get this to work, will look at it again next week:

if ($ssl = on) { 
       fastcgi_param HTTPS on;
}

You can now login to the dev site using https, but the session cookies set wasn't secure, Session 443 has been re-enabled:

drush en session443
The following extensions will be enabled: session443
Do you really want to continue? (y/n): y
WD php: Notice: Undefined index: icon in ctools_content_process() (line 73 of                 [error]
/web/dev.transitionnetwork.org.webarch.net/www/sites/all/modules/ctools/includes/content.inc).
session443 was enabled successfully.                                                          [ok]
Redirects need to be enabled at admin/settings/session443                                     [status]

And the settings at ​https://dev.transitionnetwork.org/admin/settings/session443 were set to:

• Enabled - Redirection may happen according to the rules below.
• Redirect authenticated users to HTTPS and redirect anonymous users on login/registration pages to HTTPS. Anonymous users visiting other pages may use HTTP or HTTPS.
• Force all pages with the login block to use HTTPS.

This config was also added to /web/dev.transitionnetwork.org.webarch.net/www/sites/default/settings.php:

      if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] != 'off') {
        ini_set('session.cookie_secure', 1);
      }

And now authenticated session cookies to have the secure flag set.

Also an insecure cookie is set:

LOGGED_IN=1

Which results in all HTTP requests being redirected to HTTPS -- this cookie was being destroyed with our live varnish set up to enable more content to be cached, but it's a handy thing to have so I think we should perhaps make sure it's kept for php pages even if we destroy it for css and images, something for next week...

A ticket has been opened regarding the http link to the favicon in the admin menu, ticket:411

[chris]

I have read thought all the config suggested by Jim, ticket:357#comment:42 and there are lots of things that doesn't apply to us, I think we need to have a discussion about what the caching strategy is going to be with nginx because I'm not sure what to do next -- the current nginx setup on the dev server is slower than the apache/varnish setup on the live server, for example testing 20 concurrent requests for 600 copies of the front page from another server on the local network, first the dev server:

ab -v 4 -n 600 -c 20 -H "Accept-Encoding: gzip, deflate" http://dev.transitionnetwork.org/

Concurrency Level:      20
Time taken for tests:   10.100 seconds
Complete requests:      600
Failed requests:        7

Live server:

ab -v 4 -n 600 -c 20 -H "Accept-Encoding: gzip, deflate" http://www.transitionnetwork.org/

Concurrency Level:      20
Time taken for tests:   0.641 seconds
Complete requests:      600
Failed requests:        0

Less than 1 second with the live server and 10 seconds with the dev server.

Regading the gzipping issue mentioned in ticket:357#comment:44 it turns out that is is working when checked using the Firefox Live HTTP Headers plugin, I don't understand why gziped content isn't served to apache bench or when this online test ​http://www.gidnetwork.com/tools/gzip-test.php is used.

This doesn't result in gzipped content:

ab -v 4 -n 1 -H "Accept-Encoding: gzip, deflate" http://kiwi.transitionnetwork.org/

However using curl we do get gzipped content:

curl -I --compressed  http://kiwi.transitionnetwork.org/

So gzipping is working for some clients and not others, with apache all these clients get gzipped content.

[chris]

I have spent a couple of hours reading various nginx documentation, including ​https://calomel.org/nginx.html

I think we basically need to cache everything requested on port 80 and everything we can on port 443.

I have updated the Drush setup on the dev server based on the suggestions here, ​https://github.com/perusio/drupal-with-nginx

mkdir /etc/drush
cp /usr/local/drush/examples/example.aliases.drushrc.php \
/etc/drush/aliases.drushrc.php

Then the following was added to that file:

$aliases['dev'] = array(
    'uri' => 'dev.transitionetwork.org',
    'root' => '/web/dev.transitionnetwork.org.webarch.net/www',
  );
$aliases['test'] = array(
    'uri' => 'test.transitionetwork.org',
    'root' => '/web/test.transitionnetwork.org.webarch.net/www',
  );

And the cronjob for the dev site was changed to:

DRUSH=/usr/local/bin/drush
*/50 * * * * $DRUSH @dev cron -q

Further drush configuration could be done to allow interaction with the live server if needs be.

[chris]

php-fpm is not working at the moment, see ticket:218#comment:38

[chris]

Before caching is sorted out we need to get session 443 working properly, see comment:45 -- at the moment it isn't, the secure flag is being set on the LOGGED_IN cookie and this means that requests to the http version of the site are not being redirected, I'm not sure why as it was working before. I have checked the environment and with https the PHP variable _SERVER["HTTPS"] is set to on and with http it isn't set -- this is how it should be.

There is also this issue with the lifetime of the LOGGED_IN cookie which hasn't been fixed in the current version of session443, ​https://drupal.org/node/1338266 -- we should probably apply it.

[chris]

I have done some more testing with the Firefox ​Live HTTP Headers and ​Advanced Cookie Manager and Session 443 does appear to be working correctly.

When one is on the login page at ​https://dev.transitionnetwork.org/user/login and the form is POST'ed a 302 is returned to redirect people to their user page and this Set-Cookie header is sent with the 302:

Set-Cookie: SESS456789012b2e466470fa7d2012345f24c=27f9cdaa43c6c0987f0f290d3456789b; expires=Thu, 21-Jun-2012 13:46:35 GMT; path=/; domain=.dev.transitionnetwork.org; secure; HttpOnly

And then the user page is requested and with that page the LOGGED_IN cookie is set (correctly not secure):

Set-Cookie: LOGGED_IN=1; path=/; domain=.dev.transitionnetwork.org

Note that this should have the same expired date as the session cookie, see comment:49

If one then visits the http version of the site the LOGGED_IN cookie is sent and a 302 redirect the user back to HTTPS.

I don't know why this wasn't workijng last night, since then I have restarted Firefox and cleared all the *.transitionnetwork.org cookies out -- this seems to have fixed it.

[chris]

The new wiki:PenguinServer is now up and running and all the sites on it are using Nginx (Trac is using Nginx as a reverse proxy to tracd) for more details of the Nginx configuration see ticket:470.

The main transition network site is also due to be running on Nginx as soon as wiki:PuffinServer goes live, see ticket:466.

So this ticket can finally be closed.