wiki:PuffinServer

Version 50 (modified by chris, 3 years ago) (diff)

--

Table of Contents

  1. Munin Stats
  2. Console Access
  3. Barracuda Octopus Ageir
    1. php-fpm
    2. Upgrading BOA
    3. System Updates
    4. CSF / LDF
  4. Backupninja
  5. Postfix
  6. Nginx
  7. Handy commands
    1. Vim config
  8. Migration Tickets

Puffin

puffin.webarch.net is a 4GB RAM, 14 CPU core Debian Squeeze virtual server which replaced NewLiveServer and DevelopmentServer for running the ​Transition Network Drupal sites. It went live in early 2013.

This server is due to be upgraded from LennyToSqueeze on ticket:535 in May 2013.

It was agreed to call this server puffin at the ttech meeting on 22nd November 2012, see ticket:463. The install and initial configuration of this server was tracked on ticket:466, see also the other PuffinServer#migrationtickets. Other services from the old server were migrated to PenguinServer.

Munin Stats

There are munin stats for the server available here

See ticket:555#comment:13 for the notes regarding the installation of the mysql munin stats package.

Console Access

There is a Xen shell available for console access, see wiki:XenShell.

Barracuda Octopus Ageir

The server is using ​Octopus to manage ​Ageir and also the updates to the Transition Network Drupal site, this system is installed and upgraded using ​Barracuda, the Barracuda Octopus Aegir combination is documented on the ​BOA wiki.

The BOA install script output has been saved on ticket:466#comment:22

php-fpm

Please note that the version of php-fpm that the ​http://transitionnetwork.org/ site needs to be running to work properly is: 

/etc/init.d/php53-fpm

The config file for it is /opt/local/etc/php53-fpm.conf and when it is running it is listed in top and ps as php-fpm: 

ps -lA | grep php
1 S     0 29482     1  0  80   0 - 188067 -     ?        00:00:00 php-fpm
5 S    33 29483 29482  2  80   0 - 205351 -     ?        00:01:32 php-fpm
5 S    33 29484 29482  2  80   0 - 199726 -     ?        00:01:28 php-fpm
...

Please note the settings that we changed from the default BOA ones in /opt/local/etc/php53-fpm.conf below.

When the server boots another version of php-fpm was also started, which is listed in top and ps as php5-fpm, this one: 

/etc/init.d/php5-fpm

Which is configured via files in /etc/php5/fpm/. This version should be stopped if it is found to be running: 

/etc/init.d/php5-fpm stop

It has been stopped from running at runlevel 2 by deleting this symlink (see ticket:560#comment:17): 

/etc/rc2.d/S01php5-fpm -> ../init.d/php5-fpm

Upgrading BOA

The steps are documented in ​UPGRADE.txt, to upgrade everything run these commands, this process can take around 30 mins: 

sudo -i
cd
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
barracuda up-stable
octopus up-stable all

To get the nginx and php-fpm munin stats working the following code starting with the comment needs adding to /var/aegir/config/server_master/nginx.conf in the nginx default server section: 

#######################################################
###  nginx default server
#######################################################

server {
  limit_conn   gulag 32; # like mod_evasive - this allows max 32 simultaneous connections from one IP address
  listen       *:80;
  server_name  _;
  location / {
     root   /var/www/nginx-default;
     index  index.html index.htm;
  }
## chris
  location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    allow 81.95.52.103;
    deny all;
  }
  location ~ ^/(status|ping)$ {
    fastcgi_pass 127.0.0.1:9090;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    include fastcgi_params;
    access_log off;
    allow 127.0.0.1;
    deny all;
  }
}

And the following lines need uncommenting in /opt/local/etc/php53-fpm.conf: 

pm.status_path = /status
ping.path = /ping

Also the following lines are changed from the default BOA settings (see ticket:555 for the notes on this): 

emergency_restart_threshold = 0

emergency_restart_interval = 0

pm.max_children = 90

pm.start_servers = 20

pm.min_spare_servers = 10

pm.max_spare_servers = 20

pm.max_requests = 0

After the edits above have been made nginx and php-fpm need restarting: 

/etc/init.d/php53-fpm reload
/etc/init.d/nginx restart

Best check the error log: 

tail -f /var/log/php/php53-fpm-error.log

These fixes can be tested like this: 

cd /etc/munin/plugins
munin-run phpfpm_connections
munin-run phpfpm_status
munin-run nginx_status 
munin-run nginx_request

These settings in /etc/mysql/my.cnf are changed from the default and need checking after each upgrade: 

max_connections         = 75
max_user_connections    = 75

To disable the clobbering of log files two shell scripts need editing and some lines commenting out (see ticket:555#comment:22): 

vim /var/xdrago/graceful.sh /var/xdrago/clear.sh
:1,$s/echo rotate/# echo rotate/gc

To adjust the restarting of nginx and the killing of nginx and php-fpm under heave loads edit /var/xdrago/second.sh changing these values (see ticket:563#comment:9): 

CTL_ONEX_SPIDER_LOAD=1552
CTL_FIVX_SPIDER_LOAD=1552
CTL_ONEX_LOAD=5776
CTL_FIVX_LOAD=3552
CTL_ONEX_LOAD_CRIT=7552
CTL_FIVX_LOAD_CRIT=6220

Upgrade tickets:

System Updates

Don't use the regular debian tools for updating packages, do this: 

barracuda up-stable system

After running the above command to update the system you also need to follow the steps documented above at PuffinServer#UpgradingBOA for php-fpm to get the Munin stats working again.

See also ticket:548#comment:33 for the steps that need to be followed after this to get BOA to work with the Session443 plugin.

CSF / LDF

BOA installs ​CSF / LDF and automatically blocks IP addresses after too many failed SSH login attempts, if someone is blocked who shouldn't be then they can be unblocked like this: 

csf -dr 12.34.56.78

To check if a IP address is blocked: 

csf -g 12.34.56.78

See this ticket for problems caused by CSF / LDF blocking the monitoring server: ticket:544

Backupninja

backupninja has been installed and configured to backup to another server in the Sheffield colo, three backup tasks have been configured in /etc/backup.d/, 10.sys which does backups of system settings, like all the packages installed, 20.mysql which dumps all the mysql databases into /var/backups/mysql and uses /etc/mysql/debian.cnf for authentication and finally 90.rdiff which is set to backup all these directories: 

include = /var/spool/cron/crontabs
include = /var/backups
include = /var/aegir
include = /etc
include = /root
include = /home
include = /usr/local/
include = /var/lib/dpkg/status*
include = /opt
include = /srv
include = /data
exclude = /home/*/.gnupg
exclude = /home/*/.local/share/Trash
exclude = /home/*/.Trash
exclude = /home/*/.thumbnails
exclude = /home/*/.beagle
exclude = /home/*/.aMule
exclude = /home/*/gtk-gnutella-downloads
exclude = /var/cache/backupninja/duplicity

Postfix

Two changes were made the the default postfix install, it was set to send root emails out, see ticket:466#comment:23 and it was configured to use TLS with the transition network cert, see ticket:466#comment:25.

Nginx

The only changes made to the default nginx configuration was to move the key and cert it was using out of the way and symlink to the *.transitionnetwork.org ones, see ticket:466#comment:25.

Handy commands

There are some Bash aliases to quickly get around the system added by JK...

For root: 

alias cdtn='cd /data/disk/tn/' # cd to tn directory
alias totn='su -s /bin/bash tn' # log into the tn user

# show file usages
alias duf='du -sk * | sort -n | perl -ne '\''($s,$f)=split(m{\t});for (qw(K M G)) {if($s<1024) {printf("%.1f",$s);print "$_\t$f"; last};$s=$s/1024}'\'

For tn 

alias la='ls -Al --color=auto'
alias lc='ls -ltcr --color=auto'
alias lk='ls -lSr --color=auto'
alias ll='ls -la --group-directories-first --color=auto'
alias lr='ls -lR --color=auto'
alias ls='ls -hF --color=auto'
alias lt='ls -ltr --color=auto'
alias lu='ls -ltur --color=auto'
alias lx='ls -lXB --color=auto'

Vim config

To make vim the default editor for root the following was added to /root/.bashrc: 

export EDITOR="vim"

To make config files nicer to read in vim the following was added to /root/.vimrc: 

syntax on

And a /root/.vim/filetype.vim files was created with the following in it: 

au BufRead,BufNewFile /etc/mysql/my.cnf, set ft=mycnf
autocmd BufRead,BufNewFile /etc/php5/fpm/* set syntax=dosini
autocmd BufRead,BufNewFile /opt/local/etc/php53-fpm.conf set syntax=dosini
au BufRead,BufNewFile /etc/nginx/*,/etc/nginx/conf.d/*,/var/aegir/config/server_master/nginx/*/* set ft=nginx
au BufRead,BufNewFile /data/disk/tn/config/server_master/nginx/vhost.d/* set ft=nginx

And a /root/.vim/syntax/ directory was created and mycnf.vim was created in it by downloading it from ​http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/vim-syntax-mycnf/ and nginx.vim was downloaded from ​http://www.vim.org/scripts/script.php?script_id=1886

Migration Tickets

Tickets created during the migration of the ​http://www.transitionnetwork.org/ site from NewLiveServer to this server: