Table of Contents
Puffin
The server was deleted on 20th September 2016 on ticket:924.
puffin.webarch.net is a 18GB RAM, 4 CPU core (AMD Opteron(tm) Processor 6128), with 80GB root and 1GB swap (currently disabled) disk partitions which are BSD ZFS partitions, network mounted via NFS, with a 480GB monthly data allowance, running Debian Wheezy LTS, Xen virtual server, supplied by Webarchitects Co-operative (VPS 4 + 4GB RAM) which replaced NewLiveServer and DevelopmentServer for running the Transition Network Drupal sites. It went live in early 2013.
This server was migrated to run off a ZFS server in October 2013, see ticket:593 and it was upgraded from Squeeze to Wheezy on 17th November 2013, see ticket:535.
The RAM allocation was increased from 8GB to 9GB on 6th December 2014 (ticket:846#comment:67) in an attempt to reduce the ongoing issues with load spikes, see PuffinServer#LoadSpikes. The RAM allocation was increased from 9GB to 18GB on ticket:846#comment:82 and the CPUs were reduced from 14 to 4 on ticket:846#comment:88 on 23rd December 2015.
It was agreed to call this server puffin at the ttech meeting on 22nd November 2012, see ticket:463. The install and initial configuration of this server was tracked on ticket:466, see also the other PuffinServer#migrationtickets. Other services from the old server were migrated to PenguinServer.
System updates were recorded on ticket:218 and are currently recorded on ticket:692.
The server is running an old version of BOA, the last BOA update was done on 1st May 2015, subsequent updates have not been applied (see ticket:889, ticket:872, ticket:864, ticket:863 and ticket:854), because of the version of PHP needed by Drupal 6. All the BOA cron jobs were disabled in December 2015 ticket:893 due to the ongoing load spike issue ticket:846. Drupal 6 security support ended on February 24th 2016, but the Drupal 6 Long Term Support continues to support it, see ticket:701 for Drupal security updates.
This server was due to be retired in March 2014, October 2015 and in May 2016 when the Transition Network site has been migrated to a new content management system, however it is not known when this will take place, Debian Wheezy LTS is supported till 31st May 2018.
Barracuda Octopus Ageir
The server is using Octopus to manage Ageir and also the updates to the Transition Network Drupal site, this system is installed and upgraded using Barracuda, the Barracuda Octopus Aegir combination is documented on the BOA wiki and the omega8.cc site.
BOA no longer sends out emails when new versions are available so the Milestones page and Changelog of the BOA Github project need to be manually checked. New versions are scheduled to be released monthly, however we stopped updating BOA in early 2015, the last update was to BOA 2.4.2 on ticket:844, see PuffinServer#Upgradetickets.
The initial BOA install script output has been saved on ticket:466#comment:22 and the updates are now documented on tickets listed at PuffinServer#Upgradetickets.
All BoaCronJobs were stopped on ticket:846#comment:88 on 23rd December 2015 due to the persistent problems with load spikes and server suicide that they caused, there is a ticket to consider re-enabling some of these, ticket:893.
Upgrading BOA
The steps are documented in UPGRADE.txt, to upgrade everything run these commands, this process can take around 30 mins:
sudo -i screen cd;wget -q -U iCab http://files.aegir.cc/BOA.sh.txt;bash BOA.sh.txt barracuda up-stable barracuda up-stable system octopus up-stable all both bash /var/xdrago/manage_ltd_users.sh bash /var/xdrago/daily.sh
Useful links:
- BOA Changelog (HEAD) on Github
- BOA repository on Github
- Barracuda open issues
- Octopus open issues
Note also the new hotfix tool (around line 102 of CHANGELOG.txt at time of writing) that allows post release fixes and system tweaks to be applied between full stable releases - i.e. without doing a full update to HEAD.
Upgrade tickets
These updates were never applied, the tickets have been closed as wontfix as we are not updating PHP, see ticket:754 and the plan is to switch to WordPress around April 2016, see ticket:846#comment:86
- BOA 2.4.7 ticket:889
- BOA 2.4.6 ticket:872
- BOA 2.4.5 ticket:864
- BOA 2.4.4 ticket:863
- BOA 2.4.3 ticket:854
These updates were applied:
- BOA 2.4.2 ticket:844
- BOA-2.4.1 ticket:839
- BOA-2.4.0 ticket:827
- BOA-2.3.5 ticket:798
- BOA-2.3.3 ticket:788
The time each upgrade takes has been collected here due to concerns about how long the upgrades were taking, see ticket:629#comment:11
- BOA-2.3.0 ticket:784 (Total Hours: 48m)
- BOA-2.2.9 ticket:775 (Total Hours: 45m)
- BOA-2.2.8 ticket:765 (Total Hours: 30m)
- BOA-2.2.7 ticket:760 (Total Hours: 30m)
- BOA-2.2.6 ticket:745 (Total Hours: 1h 30m)
- BOA-2.2.5 ticket:725 (Total Hours: 1h 45m)
- BOA-2.2.3 ticket:721 (Total Hours: 1h 23m)
- BOA-2.2.2 ticket:707 (Total Hours: 10h 52m) and also ticket:670 (Total Hours: 6h 9m and counting...)
- BOA-2.1.3 ticket:629 (Total Hours: 8h 11m)
- BOA-2.1.1 ticket:612 (Total Hours: 1h 45m)
- BOA-2.0.9 ticket:547 (Total Hours: 1h 6m)
- BOA-2.0.8 ticket:530 (Total Hours: 1h 6m)
- BOA-2.0.7 ticket:529 (Total Hours: 45m)
- BOA-2.0.5 ticket:466#comment:26
Upgrade log
Following is the contents of /var/log/barracuda_log.txt:
Sat Dec 15 16:16:55 GMT 2012 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.4 / Barracuda BOA-2.0.4 / Nginx 1.3.8 / PHP 5.2.17 and 5.3.18 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.28a localhost / Wildcard YES Tue Jan 8 12:43:48 GMT 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.5 / Barracuda BOA-2.0.5 / Nginx 1.3.9 / PHP 5.2.17 and 5.3.20 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.28a localhost / Wildcard YES Wed Jan 23 22:12:22 GMT 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.5 / Barracuda BOA-2.0.5 / Nginx 1.3.9 / PHP 5.2.17 and 5.3.20 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.28a localhost / Wildcard YES Thu Jan 24 09:46:29 GMT 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.5 / Barracuda BOA-2.0.5 / Nginx 1.3.9 / PHP 5.2.17 and 5.3.20 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.28a localhost / Wildcard YES Mon Jan 28 00:08:36 GMT 2013 / Debian.squeeze x86_64 XEN / Aegir HEAD / Barracuda BOA-2.0.6-dev / Nginx 1.3.11 / PHP 5.2.17 and 5.3.21 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.28a localhost / Wildcard YES Sun Apr 7 20:48:52 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.7 / Barracuda BOA-2.0.7 / Nginx 1.3.15 / PHP 5.2.17 and 5.3.23 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.30 localhost / Wildcard YES Mon Apr 8 21:52:56 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.8 / Nginx 1.3.15 / PHP 5.2.17 and 5.3.23 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.30 localhost / Wildcard YES Sun May 12 22:23:52 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.0 / PHP 5.2.17 and 5.3.25 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.30 localhost / Wildcard YES Fri May 24 10:43:33 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.0 / PHP 5.2.17 and 5.3.25 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES Sun Jul 14 21:56:44 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES Sun Sep 15 09:26:49 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.32 localhost / Wildcard YES Mon Sep 30 02:05:54 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.33a localhost / Wildcard YES Thu Oct 3 21:16:47 BST 2013 / Debian.squeeze x86_64 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.33a localhost / Wildcard YES Mon Nov 18 00:12:08 GMT 2013 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.2 / Barracuda BOA-2.1.2 / Nginx 1.5.6 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.33a localhost / Wildcard YES Sat Nov 30 21:10:09 GMT 2013 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.34 localhost / Wildcard YES Sat Dec 14 13:32:00 GMT 2013 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.34 localhost / Wildcard YES Wed Mar 26 21:15:11 GMT 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.36 localhost / Wildcard YES Wed Mar 26 21:42:17 GMT 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.36 localhost / Wildcard YES Wed Mar 26 22:14:52 GMT 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.36 localhost / Wildcard YES Thu Mar 27 00:06:49 GMT 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.36 localhost / Wildcard YES Thu Mar 27 20:39:40 GMT 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.1.3 / Barracuda BOA-2.1.3 / Nginx 1.5.7 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.36 localhost / Wildcard YES Fri Apr 11 22:26:01 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.2.2 / Barracuda BOA-2.2.2 / Nginx 1.5.13 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.36 localhost / Wildcard YES Thu May 1 00:48:45 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.2.3 / Barracuda BOA-2.2.3 / Nginx 1.5.13 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.37 localhost / Wildcard YES Thu May 8 22:26:29 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.2.5 / Barracuda BOA-2.2.5 / Nginx 1.7.0 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.37 localhost / Wildcard YES Mon Jun 30 22:30:53 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.2.6 / Barracuda BOA-2.2.6 / Nginx 1.7.2 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.38 localhost / Wildcard YES Sun Jul 20 20:23:21 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.2.7 / Barracuda BOA-2.2.7 / Nginx 1.7.3 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.38 localhost / Wildcard YES Sun Jul 27 19:52:47 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.2.8 / Barracuda BOA-2.2.8 / Nginx 1.7.3 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.38 localhost / Wildcard YES Thu Aug 7 23:08:39 BST 2014 / Debian.wheezy x86_64 / Aegir BOA-2.2.9 / Octopus BOA-2.2.9 / FPM 5.3 / CLI 5.3 Wed Sep 10 22:07:10 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.3.0 / Barracuda BOA-2.3.0 / Nginx 1.7.4 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.39 localhost Mon Sep 29 20:30:32 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.3.3 / Barracuda BOA-2.3.3 / Nginx 1.7.5 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.39 localhost Fri Oct 17 00:02:19 BST 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.3.5 / Barracuda BOA-2.3.5 / Nginx 1.7.6 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.40 localhost Mon Nov 24 18:41:52 GMT 2014 / Debian.wheezy x86_64 XEN / Aegir BOA-2.3.6 / Barracuda BOA-2.3.6 / Nginx 1.7.7 / PHP 5.3 / FPM 5.3 / CLI 5.3 / MariaDB-5.5.40 localhost Thu Feb 19 15:22:09 GMT 2015 / Debian.wheezy x86_64 XEN / Aegir BOA-2.4.0 / Barracuda BOA-2.4.0 / Nginx 1.7.9 / PHP-MI 5.3 / PHP-SE / FPM 5.3 / CLI 5.3 / MariaDB-5.5.42 localhost Thu Feb 19 15:33:46 GMT 2015 / Debian.wheezy x86_64 XEN / Aegir BOA-2.4.0 / Barracuda BOA-2.4.0 / Nginx 1.7.9 / PHP-MI 5.3 / PHP-SE / FPM 5.3 / CLI 5.3 / MariaDB-5.5.42 localhost Tue Mar 24 18:56:38 GMT 2015 / Debian.wheezy x86_64 XEN / Aegir BOA-2.4.1 / Barracuda BOA-2.4.1 / Nginx 1.7.10 / PHP-MI 5.3 / PHP-SE / FPM 5.3 / CLI 5.3 / MariaDB-5.5.42 localhost Tue Mar 24 19:06:23 GMT 2015 / Debian.wheezy x86_64 XEN / Aegir BOA-2.4.1 / Barracuda BOA-2.4.1 / Nginx 1.7.10 / PHP-MI 5.3 / PHP-SE / FPM 5.3 / CLI 5.3 / MariaDB-5.5.42 localhost Fri May 1 22:31:24 BST 2015 / Debian.wheezy x86_64 XEN / Aegir BOA-2.4.2 / Barracuda BOA-2.4.2 / Nginx 1.8.0 / PHP-MI 5.3 / PHP-SE / FPM 5.3 / CLI 5.3 / MariaDB-5.5.43 localhost Fri May 1 22:50:58 BST 2015 / Debian.wheezy x86_64 XEN / Aegir BOA-2.4.2 / Barracuda BOA-2.4.2 / Nginx 1.8.0 / PHP-MI 5.3 / PHP-SE / FPM 5.3 / CLI 5.3 / MariaDB-5.5.43 localhost
Munin config changes
BOA resets the Redis password on some upgrades, so it needs copying from /etc/redis/redis.conf to /etc/munin/plugin-conf.d/munin-node and munin-node needed restarting, see ticket:730.
nginx config changes
To get the php-fpm munin stats working the following code starting with the comment needs adding to /var/aegir/config/server_master/nginx.conf in the nginx default server section:
#######################################################
### nginx default server
#######################################################
server {
limit_conn limreq 32; # like mod_evasive - this allows max 32 simultaneous connections from one IP address
listen *:80;
server_name _;
location / {
expires 60s;
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
add_header Access-Control-Allow-Origin *;
root /var/www/nginx-default;
index index.html index.htm;
}
}
server {
listen *:80;
server_name 127.0.0.1;
location /nginx_status {
stub_status on;
access_log off;
allow 127.0.0.1;
deny all;
}
# chris 2014-04-14
location ~ ^/fpm-(status|ping)$ {
fastcgi_pass 127.0.0.1:9090;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_intercept_errors on;
include fastcgi_params;
access_log off;
allow 127.0.0.1;
allow 81.95.52.103;
deny all;
}
}
Logs for analysis on penguin, see wiki:WebServerLogs can be generated via the following being added to the http section of the /etc/nginx/nginx.conf file:
# log for awstats
log_format apache '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent"';
access_log /var/log/nginx/awstats.log apache;
The above was removed following an problem, see ticket:644.
pure-ftpd
We don't need a FTP server so it was removed on ticket:845, if an upgrade reinstalls it then do this:
rm -f /usr/local/sbin/pure-config.pl killall -9 pure-ftpd
mysql config changes
Settings in /etc/mysql/my.cnf are changed from the default, following an experiment with running for a while with the BOA default settings, see ticket:670. The MySQL tuning has been documented on ticket:587 and for the recent changes to the BOA defaults see ticket:587#comment:16 onwards.
MariaDB
The MySQL root password is available in /root/.my.cnf.
Tuning of the MySQL server is being tracked on ticket:587.
We have set MySQL to use a RAM disk for temp tables, see ticket:591.
BOA installs MariaDB as the MySQL server using the debs from the MariaDB site, see /etc/apt/sources.list.d/mariadb.list, these are the current (2013-01-13) packages which are installed (note the config files only remain for php5-mysql as PHP in now installed from source code by BOA):
dpkg -l | grep -i mysql ii libdbd-mysql-perl 4.021-1+b1 amd64 Perl5 database interface to the MySQL database ii libmysqlclient16 5.1.72-2 amd64 MySQL database client library ii libmysqlclient18 5.5.34+maria-1~wheezy amd64 Virtual package to satisfy external depends ii mariadb-common 5.5.34+maria-1~wheezy all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf) ii mysql-common 5.5.34+maria-1~wheezy all MariaDB database common files (e.g. /etc/mysql/my.cnf) ii mytop 1.6-6 all top like query monitor for MySQL rc php5-mysql 5.3.27-1~dotdeb.0 amd64 MySQL module for php5 ii python-mysqldb 1.2.3-2 amd64 Python interface to MySQL
Nginx
BOA did use Nginx from dotdeb but now it compiles it from source, the dotdeb config files remain:
dpkg -l | grep -i nginx rc nginx-common 1.4.1-1~dotdeb.0 all small, powerful, scalable web/proxy server - common files
The only changes made to the default nginx configuration during the initial install was to move the key and cert it was using out of the way and symlink to the *.transitionnetwork.org ones, see ticket:466#comment:25 and also ticket:707#comment:21.
The other change made from the default BOA config are to enable Munin graphs, see wiki:PuffinServer#nginxconfigchanges
php-fpm
Please note that the version of php-fpm that the http://transitionnetwork.org/ site needs to be running to work properly is:
/etc/init.d/php53-fpm
The config file for it is /opt/local/etc/php53-fpm.conf and when it is running it is listed in top and ps as php-fpm:
ps -lA | grep php 1 S 0 29482 1 0 80 0 - 188067 - ? 00:00:00 php-fpm 5 S 33 29483 29482 2 80 0 - 205351 - ? 00:01:32 php-fpm 5 S 33 29484 29482 2 80 0 - 199726 - ? 00:01:28 php-fpm ...
Please note the settings that we changed from the default BOA ones in /opt/local/etc/php53-fpm.conf below.
When the server boots another version of php-fpm was also started, which is listed in top and ps as php5-fpm, this one:
/etc/init.d/php5-fpm
Which is configured via files in /etc/php5/fpm/. This version should be stopped if it is found to be running:
/etc/init.d/php5-fpm stop
It was stopped from running at runlevel 2 by deleting this symlink (see ticket:560#comment:17):
/etc/rc2.d/S01php5-fpm -> ../init.d/php5-fpm
But that didn't solve the problem, see ticket:580.
Redis
Tickets related to Redis issues:
- ticket:730 Redist Munin stats stop working after BOA upgrade
- ticket:554 Site slow down and MySQL load increase
- ticket:677 Spike in MyISAM (search) database activity, Redis unable to cache such requests
Redis Munin graphs:
Munin Stats
There are munin stats for the server available here
See ticket:555#comment:13 for the notes regarding the installation of the MySQL munin stats package. See ticket:677#comment:3 for the Redis plugin install notes.
Sometimes the IO State graph stops, this can be fixed by deleting the lock files, see ticket:555#IOstategraph.
Some BOA upgrades change the Redis password and then it needed to be copied from /etc/redis/redis.conf to /etc/munin/plugin-conf.d/munin-node and munin-node needed restarting, see ticket:730 and PuffinServer#Muninconfigchanges.
We did have a trial with New Relic in 2013, see ticket:586 but this isn't on-going.
HTTP Stats
The wiki:PiwikServer generates stats from the humans visiting the server and some of these stats have been made public on wiki:WebStats.
There are some notes on analysing the raw Nginx stats on wiki:WebServerLogs and Webalizer stats for Puffin are available using the same username/password as this Trac site.
There is a wiki:ErrorCodeCheck script which emails the total number of HTTP errors each day, see ticket:483#comment:63 for a list of the total for August, September and October 2013.
Load Spikes
The documentation of the load spike suicides that the server suffered from in 2013 has been archived to wiki:PuffinServerBoaLoadSpikes as that documentation is now out dated.
When the server was updated to BOA-2.2.3 on ticket:721 the scripts in /var/xdrago/ were changed, however the load spike issue wasn't resolved, see ticket:670#comment:22.
The ongoing problems with BOA load spikes were most recently documented on ticket:846, however the problem was solved by commenting out all the BOA root cron jobs, in late December 2015, see ticket:893 -- it appears that all the problems caused over the years with load spikes was caused by BOA itself.
Tickets
Most the "live server" tickets relate to puffin, but the older ones, prior to ticket number #466, are for previous servers.
Current live server tickets
Closed live server tickets
System Updates
We don't use the BOA tool for updating packages:
barracuda up-stable system
As it's very slow and after running the above command to update the system you also need to follow the steps documented above at PuffinServer#UpgradingBOA for php-fpm to get the Munin stats working again.
Nginx and PHP are complied from source code so the above command should be run when these need updating, for other updates use the wiki:AptitudeUpdateScript script and document the updates on ticket:692.
See also ticket:548#comment:33 for the steps that need to be followed after this to get BOA to work with the Session443 plugin.
CSF / LDF
To restart the firewall script:
csf -r
We have set the following variable in /root/.barracuda.cnf to ensure that the CSF / LDF changes are not clobbered by BOA:
_CUSTOM_CONFIG_CSF=YES
We could do with a link here to the ticket on which the CSF / LDF configuration had a lot of work done. Some changed to the load level alerting was made on ticket:707#comment:37
False positives
BOA installs CSF / LDF and automatically blocks IP addresses after too many failed SSH login attempts, if someone is blocked who shouldn't be then they can be unblocked like this:
csf -dr 81.95.52.66
To check if a IP address is blocked:
csf -g 81.95.52.66
See this ticket for problems caused by CSF / LDF blocking the monitoring server: ticket:544
Blocklists
Blocklists are configured in /etc/csf/csf.blocklists and some were enabled on ticket:589
Console and SSH Access
There is a Xen shell available for console access, see wiki:XenShell.
For developers and sysadmins there is SSH access, contact chris@… if you need an account creating.
The server is also running Mosh : the mobile shell which is very handy when you internet connection is poor, for example on a train. Mosh was installed on ticket:673.
Cron
BOA controls the root crontab and any changes made there will be overwritten, so things that would normally be in the root crontab need to go into users ones and use sudo, these are the ones in chris' crontab:
# delete metche backups which are more than a day old # see https://tech.transitionnetwork.org/trac/ticket/531 28 11 * * * sudo /usr/local/bin/metche-clean -d # set the clock after a reboot # see /trac/ticket/599 @reboot sudo rdate -s ntp.demon.co.uk # create a tmp dir on the ram disk for mysql # see /trac/ticket/591 @reboot sudo mkdir /run/shm/mysql ; sudo chown mysql:mysql /run/shm/mysql # ssl cert check 32 09 * * * sudo ssl-cert-check -qac "/etc/ssl/transitionnetwork.org/transitionnetwork.org.crt" -e "chris@webarchitects.co.uk"
To edit chris' crontab after logging in as another user:
sudo -i export EDITOR=vim crontab -e -u chris
Backups
backupninja has been installed and two backup tasks have been configured in /etc/backup.d/, 10.sys which does backups of system settings and 20.mysql which dumps all the mysql databases into /var/backups/mysql and uses /etc/mysql/debian.cnf for authentication.
In October 2013 we switched the servers filesystem to a ZFS server on the network, see ticket:593#comment:5 and now filesystem backups are done via ZFS snapshots so the rsync backup was disabled, see ticket:535#comment:22 however these backups are not available to anyone apart from the Webarchitects sysadmin so on 23rd July 2014 on ticket:763 additional backups were set up, these are done via /usr/local/bin/agile-backup, see AgileBackup and people who have had their ssh public keys added can access these backups via SFTP:
sftp tn-puffin@store1.webarch.net
The latest backups are in puffin.webarch.net and 60 days worth of snapshots are in ~/.zfs/, you can mount these backups locally, for example on Debian:
aptitude install sshfs mkdir -p /media/tn-puffin chmod 700 /media/tn-puffin/ mkdir /media/tn-puffin/latest mkdir /media/tn-puffin/archive echo "sshfs#tn-puffin@store1.webarch.net:puffin.webarch.net /media/tn-puffin/latest fuse ro,nobootwait 0 0" >> /etc/fstab echo "sshfs#tn-puffin@store1.webarch.net::.zfs/snapshot /media/tn-puffin/archive fuse ro 0,nobootwait 0" >> /etc/fstab mount -a
Postfix
Two changes were made the the default postfix install, it was set to send root emails out, see ticket:466#comment:23 and it was configured to use TLS with the transition network cert, see ticket:466#comment:25.
Handy commands
There are some Bash aliases to quickly get around the system added by JK...
For root:
alias cdtn='cd /data/disk/tn/' # cd to tn directory
alias totn='su -s /bin/bash tn' # log into the tn user
# show file usages
alias duf='du -sk * | sort -n | perl -ne '\''($s,$f)=split(m{\t});for (qw(K M G)) {if($s<1024) {printf("%.1f",$s);print "$_\t$f"; last};$s=$s/1024}'\'
For tn
alias la='ls -Al --color=auto' alias lc='ls -ltcr --color=auto' alias lk='ls -lSr --color=auto' alias ll='ls -la --group-directories-first --color=auto' alias lr='ls -lR --color=auto' alias ls='ls -hF --color=auto' alias lt='ls -ltr --color=auto' alias lu='ls -ltur --color=auto' alias lx='ls -lXB --color=auto'
Vim config
To make vim the default editor for root the following was added to /root/.bashrc:
export EDITOR="vim"
To make config files nicer to read in vim the following was added to /root/.vimrc:
syntax on
And a /root/.vim/filetype.vim files was created with the following in it:
au BufRead,BufNewFile /etc/mysql/my.cnf, set ft=mycnf autocmd BufRead,BufNewFile /etc/php5/fpm/* set syntax=dosini autocmd BufRead,BufNewFile /opt/local/etc/php53-fpm.conf set syntax=dosini au BufRead,BufNewFile /etc/nginx/*,/etc/nginx/conf.d/*,/var/aegir/config/server_master/nginx/*/* set ft=nginx au BufRead,BufNewFile /data/disk/tn/config/server_master/nginx/vhost.d/* set ft=nginx
And a /root/.vim/syntax/ directory was created and mycnf.vim was created in it by downloading it from http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/vim-syntax-mycnf/ and nginx.vim was downloaded from http://www.vim.org/scripts/script.php?script_id=1886
Migration Tickets
Tickets created during the migration of the http://www.transitionnetwork.org/ site from NewLiveServer to this server:
- ticket:466 Puffin install and configuration
- ticket:472 Script to copy files from NewLiveServer to puffin
- ticket:479 Transfer live transitionnetwork.org site to puffin
- ticket:480 Transfer news.transitionnetwork.org to puffin
- ticket:483 Nginx 502 Bad Gateway Errors with BOA see the summary on ticket:483#comment:46
- ticket:487 robots.txt files for development sites
