[[PageOutline(2-9, Table of Contents)]]
= kiwi.webarch.net / kiwi.transitionnetwork.org =

This is the '''development server''' for '''[http://www.transitionnetwork.org/ The Transition Network]''', there is a [http://kiwi.transitionnetwork.org/ static html front page for this server] which lists the sites and applications which are running on it. The live server, quince.transitionnetwork.org is documented on the wiki:NewLiveServer page.

The development server has a development, http://dev.transitionnetwork.org/, and a testing, http://test.transitionnetwork.org/, version of the live server set-up. The test, dev and live code bases were kept in [https://svn.webarch.net/transition/ this subversion repository] but have now been switched to the Trac svn repo, browser:www via ticket:92. To mirror LIVE's set up, there are Drupal multi-sites for the workspaces sub-site at http://workspaces.dev.transitionnetwork.org/ and http://workspaces.test.transitionnetwork.org/.

The a development (and for now, testing) copy of the SharingEngine at http://news.dev.transitionnetwork.org/. It is a separate install of Drupal 6 with its own Trac svn repo at browser:sharing-engine.

The development server is [http://kiwi.transitionnetwork.org/ kiwi.transitionnetwork.org] also available at [http://kiwi.webarch.net/ kiwi.webarch.net], it's a encrypted 40G debian virtual machine with 1Gb of dedicated RAM. The encryption key can be shared with others if needs be, there is a ssh account on the physical machine that can be used to reboot and decrypt the machine. The physical server that virtual machine is running on has quad Intel Xeon 2.00GHz 64bit processors, 16Gb RAM, the host OS debian is running on a pair of mirrored 73Gb SAS disks and the virtual machines on a pair of mirrored 2Tb SATA disks.
/
There are [http://nsa.rat.burntout.org/munin/webarch.net/kiwi.webarch.net.html remote munin stats] for the server available. The stats are also [https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net.html available from the server itself] and the server hosts [https://kiwi.transitionnetwork.org/munin/gaiahost.coop/transitiontowns.gaiahost.coop.html munin stats for the live server], transitiontowns.gaiahost.coop.

The original documentation for this server was at http://atrium.transitiontoday.org/dev/development-server the parts that still seem relevant have been copied here and updated as necessary.

In September 2011 the server was upgraded from LennyToSqueeze and tracked on ticket:301

== Network and DNS ==

The server has 3 IP addresses, but only one is currently needed, '''81.95.52.78''', 81.95.52.79, 81.95.52.80

== SSH access ==

Which ever developers need ssh access can have it, contact chris@webarchitects.co.uk. When adding a new account don't forget to add the user to the sudo group -- ssh access is limited to only users in the sudo group.

Users can be added and also put in the sudo group with one command:

{{{
kiwi:~# adduser -G sudo newusername
}}}

/etc/sudoers is set up so that people don't need to type their passwords when sudo'ing. 

If you want to create a user and allow them to have ssh access but '''not''' sudo then they need to be in the sshaccess group:

{{{
kiwi:~# adduser -G sshaccess newusername
}}}

If users set up ssh keys then this means that people can use very long, essentially uncrackable, passwords but never actually need to remember or use them unless they lose their local ssh key or they need to login via the consol, these can be generated using:

{{{
kiwi:~# pwgen -sn 25
}}}


== live2dev ==

This is a script, run as root, which will update either the http://dev.transitionnetwork.org.webarch.net/ or the http://test.transitionnetwork.org.webarch.net/ sites with the last MySQL dump from the live site and also copy across backups of the uploaded files.

The script can be run interactively or by telling it which site to update on the command line (dev or test), for example to update the dev site:

{{{
kiwi:~# live2dev dev
Have you run /usr/local/bin/backup2kiwi on quince.webarch.net? (y):
}}}

The script will update using the latest data synced from the live server, this sync is done using the wiki:NewLiveServer#backup2kiwi script on kiwi.

After the script has run you need to '''Flush all caches''' to avoid error messages, ideally the script would drop the appropriate tables to make this unnecessary.

This script was sorted out via ticket:124 and the email settings via ticket:136 and there is still the outstanding matter of the https cert / domain name mismatch, see ticket:166.

A copy of the live2dev script is attached attachment:live2dev

== Varnish ==

See ticket:224.

Install Varnish 2.1 via the [http://www.varnish-cache.org/installation/debian varnish-cache.org repository]:

{{{
curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -
aptitude install lsb-release
echo "deb http://repo.varnish-cache.org/debian/ $(lsb_release -s -c) varnish-2.1" >> /etc/apt/sources.list.d/varnish.list
aptitude update
aptitude install varnish
}}}

Edit these things in the main config file, /etc/default/varnish :

{{{
#DAEMON_OPTS="-a :6081 \
#             -T localhost:6082 \
#             -f /etc/varnish/default.vcl \
#             -S /etc/varnish/secret \
#             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

DAEMON_OPTS="-a :80 \
             -T localhost:81 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,256M"
}}}

Make it listen on port 80 and have 256M RAM and also have the management interface on port 81, apache need it's configuration changing so it listens on port 8080.

Varnish 3.0 configuration:

{{{
backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
}
acl purge {
  "localhost";
  "127.0.0.1";
}

sub vcl_recv {

    # remove all cookies
    unset req.http.Cookie;

    ## Pass cron jobs and server-status
    if (req.url ~ "cron.php") {
      return (pass);
    }
    if (req.url ~ ".*/server-status$") {
      return (pass);
    }

    # Normalize the Accept-Encoding header
    # as per: http://varnish-cache.org/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
      if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
        # No point in compressing these
        remove req.http.Accept-Encoding;
      } elsif (req.http.Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
     } elsif (req.http.Accept-Encoding ~ "deflate") {
        set req.http.Accept-Encoding = "deflate";
      } else {
        # unkown algorithm
        remove req.http.Accept-Encoding;
      }
    }

    ## Default request checks
    if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE" &&
    req.request != "PURGE") {
    # Non-RFC2616 or CONNECT which is weird.
    return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD" && req.request != "PURGE") {
    # We only deal with GET, PURGE and HEAD by default
    return (pass);
    }

    if (req.http.Authorization) {
          # Not cacheable by default
          return (pass);
    }

    # Check the incoming request type is "PURGE", not "GET" or "POST"
    if (req.request == "PURGE") {
      # Check if the ip coresponds with the acl purge
      if (!client.ip ~ purge) {
      # Return error code 405 (Forbidden) when not
        error 405 "Not allowed.";
      }
      # purge all objects from cache that match the incoming url and host
      #ban("req.url == " + req.url " && req.http.host == " + req.http.host);
      # purge all objects from cache that match the incoming url
      #ban("req.url = " + req.url);
      ban("req.http.host == " + req.http.host + "&& req.url == " + req.url);
      # Return a http error code 200 (Ok)
      error 200 "Purged.";
      }

    # Grace to allow varnish to serve content if backend is lagged
    set req.grace = 5m;

    return (lookup);
}

sub vcl_fetch {
    unset beresp.http.set-cookie;
    return (deliver);
}

sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-Varnish-Cache = "HIT";
  }
  else {
    set resp.http.X-Varnish-Cache = "MISS";
  }
  return (deliver);
}
}}}

== Piwik ==

The [http://piwik.org/ Piwik] server is documented on a seperate PiwikServer page.


== memcache ==

Already installed on live server, wiki:NewLiveServer#memcache, following [http://www.iwebsource.net/blog/learning-center/install-memcache-in-php-on-debianubuntu.html this]:

{{{
pecl install memcache
 1. Enable memcache session handler support? : yes
 
 1-1, 'all', 'abort', or Enter to continue: 1
 Enable memcache session handler support? [yes] : yes
echo "extension=memcache.so" > /etc/php5/conf.d/memcache.ini
}}}

The main config file is {{{/etc/memcached.conf}}} and the following was added to {{{/web/dev.transitionnetwork.org.webarch.net/www/sites/default/settings.php}}}:

{{{
$conf['cache_inc'] = './sites/all/modules/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
        'default' => array(
        'engine'  => 'memcache',
        'server'  => array('127.0.0.1:11211'),
        'shared'  => TRUE,
),
);
}}}


== Mediawiki ==

There is a development copy of the [http://wiki.transitionnetwork.org/ wiki.transitionnetwork.org] site at http://wiki.dev.transitionnetwork.org/Main_Page which is running on kiwi.transitionnetwork.org it is installed in /web/wiki.dev.transitionnetwork.org/www and the apache !VirtualHost is configured in /etc/apache2/sites-available/wiki.dev.transitionnetwork.org

See also the NewLiveServer#mediawiki documentation.

To upgrade the site to the latest version of Mediawiki, from http://www.mediawiki.org/wiki/Download you could follow the instructions from http://www.mediawiki.org/wiki/Upgrade or use the '''mediawiki-upgrade''' script which takes the latest version of Mediawiki as an argument on the command line and then does everything for you:

{{{
kiwi:~# mediawiki-upgrade 1.16.0
}}}

=== wiki-live2dev ===

To sync the dev version of the wiki with a database dump and the latest uploaded files from wiki:NewLiveServer#Mediawiki run /usr/local/bin/wiki-live2dev on kiwi.transitionnetwork.org after you have run wiki:NewLiveServer#backup2kiwi on quince.transitionnetwork.org:

{{{
kiwi:~# wiki-live2dev
Have you run /usr/local/bin/backup2kiwi on quince.webarch.net? (y):
}}}

A copy of the wiki-live2dev script is attached attachment:wiki-live2dev

=== extensions ===

You can view the installed list of extensions [http://wiki.dev.transitionnetwork.org/Special:Version#Installed_extensions on the wiki], these are also update using the mediawiki-upgrade, using subversion.

The [http://www.mediawiki.org/wiki/Extension:FCKeditor_%28Official%29 FCKeditor extension] was installed via ticket:129.

== Packages ==

The following packages and their dependencies have been installed using aptitude:

    * rsync
    * libapache2-mod-fastcgi
    * mysql-server
    * php5-imagick
    * php5-mysql
    * phpmyadmin
    * php-pear
    * subversion
    * php5-suhosin
    * php-apc
    * php5-mcrypt
    * php5-dev
    * libmagic-dev
    * make
    * dnsutils
    * whois
    * munin
    * libwww-perl
    * lynx
    * imagemagick
    * trac
    * git-core

== Apache ==

After making any changes to the Apache configuration best do a configtest first to make sure the configuration is OK:
{{{
sudo /usr/sbin/apache2ctl configtest
}}}
And then to restart the apache server:
{{{
sudo /usr/sbin/apache2ctl restart
}}}

== phpMyAdmin ==

The phymyadmin debian package was installed and directives from /etc/phpmyadmin/apache.conf and /etc/apache2/sites-available/default-ssl were copied into /etc/apache2/sites-available/phpmyadmin and then the !VirtualServer was enabled, a2ensite phpmyadmin and apache restarted.

phpMyAdmin is available at: https://kiwi.transitionnetwork.org/phpmyadmin/

== MySQL ==

There is a copy of the root passwd for MySQL in /root/.

The MySQL settings have been tweaked to increase memory usage, see /etc/mysql/my.cnf

MySQL users and databases were set up using the phpMyAdmin interface.

== Postfix ==

Postfix has been installed for outgoing SMPT email, the only changes from the default configuration are related to enabling TLS and these changes are in /etc/postfix/main.cf.

Email to root is forwarded to chris@webarchitects.co.uk via /root/.forward -- if anyone else needs to get a copy of the root emails then please add yourself to this file.

== Backupninja ==

[https://labs.riseup.net/code/projects/show/backupninja/ Backupninja] has been installed and set up -- it's set to backup files to another server in the same rack. The main configuration file is /etc/backupninja.conf and the files containing the list of things to be backed up are in /etc/backup.d/. 60 days worth of backups are saved. It is set to backup MySQL and the following directories:
{{{
include = /etc
include = /home
include = /var
include = /home
include = /usr
include = /srv
include = /web
 
exclude = /var/run
}}}

== Mysql Backup ==

A MySQL Backup script from http://worldcommunitypress.com/opensource/mysql-backup is installed in /usr/local/bin and it's set to create backups in /var/backups/mysql/

It needs the libmime-lite-perl debian package to be installed.

To run it:

{{{
/usr/local/bin/mysql-backup
}}}

These lines have been changed from the original at http://worldcommunitypress.com/assets/files/opensource/utilities/mysql_backup.txt :

{{{
$admin_email_to              = 'chris@webarchitects.co.uk';
$admin_email_from            = 'root@kiwi.webarch.net';
$cnf_file                    = '/root/.my.cnf';
$site_name                   = 'kiwi.webarch.net';
$mysql_backup_dir            = '/var/backups/mysql';
}}}


== !DenyHosts ==

To prevent SSH brute force attacks [http://denyhosts.sourceforge.net/ DenyHosts] has been installed -- if you make too many mistakes with your password you will be locked out! To whitelist IP addresses add them to /etc/hosts.allow -- blacklisted IP's are written to /etc/hosts.deny. 

== Munin ==

There are [http://nsa.rat.burntout.org/munin/webarch.net/kiwi.webarch.net.html remote munin stats] for the server available, they are also [https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net.html available from the server itself].

The munin client has been configured via /etc/munin/munin-node.conf

Note the [https://kiwi.transitionnetwork.org/munin/localdomain/localhost.localdomain-iostat.html missing iostat graph] is caused by [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=552034 a debian munin xen bug].

Varnish stats: ticket:224#comment:27

== Trac ==

This Trac site was installed via ticket:1 and it resides in /web/tech.transitionnetwork.org/trac and the linked Subversion repo is in /web/tech.transitionnetwork.org/svn and the XSLT to style web access to https://tech.transitionnetwork.org/svn/ is in /web/tech.transitionnetwork.org/www.

The install followed notes from [http://svn.debian.org/viewsvn/python-apps/packages/trac/trunk/debian/README.Debian?view=markup Configuring Trac on Debian] and [http://trac.edgewall.org/wiki/TracOnDebian Trac on Debian].

Further information and links to help pages can be found here: wiki:WikiStart#Trac

Trac was upgraded to 1.12 on ticket:364

== bbPress ==

There is an archived copy of the old forum at http://2011.archive.transitionnetwork.org/forum/ this was upgraded to the latest version of bbPress when it was installed, the admin email was reset so the SQL upgrade could be run and the apache configuration generated.

The following things were done to lock the forum down as read-only, template edit, bb-templates/kakumei/header.php removed line 32:

{{{
<?php if ( !in_array( bb_get_location(), array( 'login-page', 'register-page' ) ) ) login_form(); ?>
}}}

In ./bb-templates/kakumei/front-page.php line 86 was edited to remove the Add Topic link:

{{{
<div class="bbcrumb"><a href="<?php bb_uri(); ?>"><?php bb_option('name'); ?></a> &raquo; <?php _e('Add New Topic'); ?></div>
}}}

In ./bb-includes/functions.bb-template.php a section around line 278 was commented out:

{{{
/*
        if ( empty( $h2 ) && false !== $h2 ) {
                if ( bb_is_topic() ) {
                        $h2 = __( 'Reply' );
                } elseif ( bb_is_forum() ) {
                        $h2 = __( 'New Topic in this Forum' );
                } elseif ( bb_is_tag() || bb_is_front() ) {
                        $h2 = __( 'Add New Topic' );
                }
        }
*/
}}}

And around line 310 another section was commented:

{{{
        } /* elseif ( !bb_is_user_logged_in() ) {
                echo '<p>';
                printf(
                        __('You must <a href="%s">log in</a> to post.'),
                        esc_attr( bb_get_uri( 'bb-login.php', null, BB_URI_CONTEXT_A_HREF + BB_URI_CONTEXT_BB_USER_FORMS ) )
                );
                echo '</p>';
        } */
}}}

And in bb-templates/kakumei/forum.php line 9:

{{{
        <th><?php _e('Topic'); ?> &#8212; <?php bb_new_topic_link(); ?></th>
}}}

And in bb-templates/kakumei/front-page.php line 17:

{{{
        <th><?php _e('Topic'); ?> &#8212; <?php bb_new_topic_link(); ?></th>
}}}

And in bb-templates/kakumei/tag-single.php line 11:

{{{
        <th><?php _e('Topic'); ?> &#8212; <?php bb_new_topic_link(); ?></th>
}}}

In bb-templates/kakumei the following files, forum.php front-page.php tags.php tag-single.php topic.php login.php register.php were edited to add:

{{{
<h2>These forums are now closed</h2>                                                  
<p>You cannot add any posts but all threads are readable. Please visit the                                                       
<a href="http://www.transitionnetwork.org/forum">Transition Network                                                              
forums</a> for current Transition-hosted discussions. Thanks.</p>
}}}

Apache configuration:

{{{
      RewriteRule ^bb-admin.*$ [G,L]
      RewriteRule ^bb-config.*$ [G,L]
      RewriteRule ^bb-edit.*$ [G,L]
      RewriteRule ^bb-cron.*$ [G,L]
      RewriteRule ^bb-login.*$ [G,L]
      RewriteRule ^bb-post.*$ [G,L]
      RewriteRule ^bb-reset-password.*$ [G,L]
      RewriteRule ^bb-settings.*$ [G,L]
      RewriteRule ^profile-edit.*$ [G,L]
      RewriteRule ^edit.*$ [G,L]
      RewriteRule ^register.*$ [G,L]
}}}

{{{
  <Directory /web/2011.archive.transitionnetwork.org/www/forum/bb-admin>
    Order deny,allow
    deny from all
  </Directory>
}}}

See ticket:234 for further information.