[[PageOutline(2-5, Table of Contents, floated)]]

= Puffin =

puffin.webarch.net is a 8GB RAM, 14 CPU core Debian Squeeze virtual server which replaced NewLiveServer and DevelopmentServer for running the [http://www.transitionnetwork.org/ Transition Network] Drupal sites. It went live in early 2013. 

This server is due to be upgraded from LennyToSqueeze on ticket:535 in May 2013.

It was agreed to call this server {{{puffin}}} at the ttech meeting on 22nd November 2012, see ticket:463. The install and initial configuration of this server was tracked on ticket:466, see also the other PuffinServer#migrationtickets.  Other services from the old server were migrated to PenguinServer.

== Munin Stats ==

There are munin stats for the server available here

* https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/

See ticket:555#comment:13 for the notes regarding the installation of the mysql munin stats package.

== Tickets ==

Most the "live server" tickets relate to puffin, but the older ones are for previous servers.

=== Current live server tickets ===

[[TicketQuery(status=accepted|new|assigned|reopened&component=Live server,order=id,desc=1,format=table,col=summary|owner|reporter)]]

=== Closed live server tickets ===

[[TicketQuery(status=closed&component=Live server,order=id,desc=1,format=table,col=summary|owner|reporter)]]

== Console Access ==

There is a Xen shell available for console access, see wiki:XenShell.

== Barracuda Octopus Ageir ==

The server is using [https://drupal.org/project/octopus Octopus] to manage [http://community.aegirproject.org/ Ageir] and also the updates to the Transition Network Drupal site, this system is installed and upgraded using [https://drupal.org/project/barracuda Barracuda], the Barracuda Octopus Aegir combination is documented on the [http://groups.drupal.org/node/163784 BOA wiki].

The BOA install script output has been saved on ticket:466#comment:22

=== php-fpm ===

Please note that the version of php-fpm that the http://transitionnetwork.org/ site needs to be running to work properly is:

{{{
/etc/init.d/php53-fpm 
}}}

The config file for it is {{{/opt/local/etc/php53-fpm.conf}}} and when it is running it is listed in top and ps as php-fpm:

{{{
ps -lA | grep php
1 S     0 29482     1  0  80   0 - 188067 -     ?        00:00:00 php-fpm
5 S    33 29483 29482  2  80   0 - 205351 -     ?        00:01:32 php-fpm
5 S    33 29484 29482  2  80   0 - 199726 -     ?        00:01:28 php-fpm
...
}}}

Please note the settings that we changed from the default BOA ones in {{{/opt/local/etc/php53-fpm.conf}}} below.

When the server boots another version of php-fpm was also started, which is listed in top and ps as php5-fpm, this one:

{{{
/etc/init.d/php5-fpm
}}}

Which is configured via files in {{{/etc/php5/fpm/}}}. This version should be stopped if it is found to be running:

{{{
/etc/init.d/php5-fpm stop
}}}

It has been stopped from running at runlevel 2 by deleting this symlink (see ticket:560#comment:17):

{{{
/etc/rc2.d/S01php5-fpm -> ../init.d/php5-fpm
}}}


=== Upgrading BOA ===

The steps are documented in [http://drupalcode.org/project/barracuda.git/blob/HEAD:/docs/UPGRADE.txt UPGRADE.txt], to upgrade everything run these commands, this process can take around 30 mins:

{{{
sudo -i
cd
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
barracuda up-stable
octopus up-stable all
}}}

==== nginx config changes ====

To get the nginx and php-fpm munin stats working the following code starting with the comment needs adding to {{{/var/aegir/config/server_master/nginx.conf}}} in the nginx default server section:

{{{
#######################################################
###  nginx default server
#######################################################

server {
  limit_conn   gulag 32; # like mod_evasive - this allows max 32 simultaneous connections from one IP address
  listen       *:80;
  server_name  _;
  location / {
     root   /var/www/nginx-default;
     index  index.html index.htm;
  }
## chris
  location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    allow 81.95.52.103;
    deny all;
  }
  location ~ ^/(status|ping)$ {
    fastcgi_pass 127.0.0.1:9090;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    include fastcgi_params;
    access_log off;
    allow 127.0.0.1;
    deny all;
  }
}
}}}

Logs for analysis on penguin, see wiki:WebServerLogs are generated via the following being added to the {{{http}}} section of the {{{/etc/nginx/nginx.conf}}} file:

{{{
  # log for awstats
  log_format apache '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent"';
  access_log         /var/log/nginx/awstats.log apache;
}}}

==== php-fpm config changes ====

And the following lines need uncommenting in {{{/opt/local/etc/php53-fpm.conf}}}:

{{{
pm.status_path = /status
ping.path = /ping
}}}

Also the following lines are changed from the default BOA settings (see ticket:555 for the notes on this):

{{{
emergency_restart_threshold = 0

emergency_restart_interval = 0

pm.max_children = 90

pm.start_servers = 20

pm.min_spare_servers = 10

pm.max_spare_servers = 20

pm.max_requests = 0
}}}

After the edits above have been made nginx and php-fpm need restarting:

{{{
/etc/init.d/php53-fpm reload
/etc/init.d/nginx restart
}}}

Best check the error log:

{{{
tail -f /var/log/php/php53-fpm-error.log
}}}

These fixes can be tested like this:

{{{
cd /etc/munin/plugins
munin-run phpfpm_connections
munin-run phpfpm_status
munin-run nginx_status 
munin-run nginx_request
}}}

==== mysql config changes ====

These settings in {{{/etc/mysql/my.cnf}}} are changed from the default and need checking after each upgrade:

{{{
max_connections         = 75
max_user_connections    = 75
}}}

==== xdrago shell script changes ====

To disable the clobbering of log files two shell scripts need editing and some lines commenting out (see ticket:555#comment:22):

{{{
vim /var/xdrago/graceful.sh /var/xdrago/clear.sh
:1,$s/echo rotate/# echo rotate/gc
}}}

To adjust the restarting of nginx and the killing of nginx and php-fpm under heave loads edit {{{/var/xdrago/second.sh}}} changing these values (see ticket:563#comment:9 and ticket:555#comment:52):

{{{
CTL_ONEX_SPIDER_LOAD=1940
CTL_FIVX_SPIDER_LOAD=1940
CTL_ONEX_LOAD=7220
CTL_FIVX_LOAD=4440
CTL_ONEX_LOAD_CRIT=9440
CTL_FIVX_LOAD_CRIT=7775
}}}

==== Upgrade tickets ====

* BOA-2.0.9 ticket:547
* BOA-2.0.8 ticket:530
* BOA-2.0.7 ticket:529
* ticket:466#comment:26

=== System Updates ===

Don't use the regular debian tools for updating packages, do this:

{{{
barracuda up-stable system
}}}

After running the above command to update the system you also need to follow the steps documented above at PuffinServer#UpgradingBOA for php-fpm to get the Munin stats working again.

See also ticket:548#comment:33 for the steps that need to be followed after this to get BOA to work with the Session443 plugin.

=== CSF / LDF ===

BOA installs [http://configserver.com/cp/csf.html CSF / LDF] and automatically blocks IP addresses after too many failed SSH login attempts, if someone is blocked who shouldn't be then they can be unblocked like this:

{{{
csf -dr 81.95.52.66
}}}

To check if a IP address is blocked:

{{{
csf -g 81.95.52.66
}}}

See this ticket for problems caused by CSF / LDF blocking the monitoring server: ticket:544

== Backupninja ==

backupninja has been installed and configured to backup to another server in the Sheffield colo, three backup tasks have been configured in {{{/etc/backup.d/}}}, {{{10.sys}}} which does backups of system settings, like all the packages installed, {{{20.mysql}}} which dumps all the mysql databases into {{{/var/backups/mysql}}} and uses {{{/etc/mysql/debian.cnf}}} for authentication and finally {{{90.rdiff}}} which is set to backup all these directories:

{{{
include = /var/spool/cron/crontabs
include = /var/backups
include = /var/aegir
include = /etc
include = /root
include = /home
include = /usr/local/
include = /var/lib/dpkg/status*
include = /opt
include = /srv
include = /data
exclude = /home/*/.gnupg
exclude = /home/*/.local/share/Trash
exclude = /home/*/.Trash
exclude = /home/*/.thumbnails
exclude = /home/*/.beagle
exclude = /home/*/.aMule
exclude = /home/*/gtk-gnutella-downloads
exclude = /var/cache/backupninja/duplicity
}}}

== Postfix ==

Two changes were made the the default postfix install, it was set to send root emails out, see ticket:466#comment:23 and it was configured to use TLS with the transition network cert, see ticket:466#comment:25.

== Nginx ==

The only changes made to the default nginx configuration was to move the key and cert it was using out of the way and symlink to the *.transitionnetwork.org ones, see ticket:466#comment:25.

== Handy commands ==

There are some Bash aliases to quickly get around the system added by JK...

For {{{root}}}:
{{{
alias cdtn='cd /data/disk/tn/' # cd to tn directory
alias totn='su -s /bin/bash tn' # log into the tn user

# show file usages
alias duf='du -sk * | sort -n | perl -ne '\''($s,$f)=split(m{\t});for (qw(K M G)) {if($s<1024) {printf("%.1f",$s);print "$_\t$f"; last};$s=$s/1024}'\'
}}}

For {{{tn}}}
{{{
alias la='ls -Al --color=auto'
alias lc='ls -ltcr --color=auto'
alias lk='ls -lSr --color=auto'
alias ll='ls -la --group-directories-first --color=auto'
alias lr='ls -lR --color=auto'
alias ls='ls -hF --color=auto'
alias lt='ls -ltr --color=auto'
alias lu='ls -ltur --color=auto'
alias lx='ls -lXB --color=auto'
}}}

=== Vim config ===

To make vim the default editor for root the following was added to {{{/root/.bashrc}}}:

{{{
export EDITOR="vim"
}}}

To make config files nicer to read in vim the following was added to {{{/root/.vimrc}}}:

{{{
syntax on
}}}

And a {{{/root/.vim/filetype.vim}}} files was created with the following in it:

{{{
au BufRead,BufNewFile /etc/mysql/my.cnf, set ft=mycnf
autocmd BufRead,BufNewFile /etc/php5/fpm/* set syntax=dosini
autocmd BufRead,BufNewFile /opt/local/etc/php53-fpm.conf set syntax=dosini
au BufRead,BufNewFile /etc/nginx/*,/etc/nginx/conf.d/*,/var/aegir/config/server_master/nginx/*/* set ft=nginx
au BufRead,BufNewFile /data/disk/tn/config/server_master/nginx/vhost.d/* set ft=nginx

}}}

And a {{{/root/.vim/syntax/}}} directory was created and {{{mycnf.vim}}} was created in it by downloading it from http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/vim-syntax-mycnf/ and {{{nginx.vim}}} was downloaded from http://www.vim.org/scripts/script.php?script_id=1886

== Migration Tickets ==

Tickets created during the migration of the http://www.transitionnetwork.org/ site from NewLiveServer to this server:

* ticket:466 Puffin install and configuration
* ticket:472 Script to copy files from NewLiveServer to puffin 
* ticket:479 Transfer live transitionnetwork.org site to puffin 
* ticket:480 Transfer news.transitionnetwork.org to puffin 
* ticket:483 Nginx 502 Bad Gateway Errors with BOA see the summary on ticket:483#comment:46
* ticket:487 robots.txt files for development sites