Changes between Version 170 and Version 171 of PuffinServer


Ignore:
Timestamp:
04/10/15 11:45:36 (20 months ago)
Author:
chris
Comment:

Updates and most importent info moved to the top

Legend:

Unmodified
Added
Removed
Modified

  • PuffinServer

    v170 v171  
    1111System updates were recorded on ticket:218 and are currently recorded on ticket:692. BOA update tickets are listed at PuffinServer#Upgradetickets. 
    1212 
    13 == Munin Stats == 
    14  
    15 There are munin stats for the server available here 
    16  
    17 * https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/ 
    18  
    19 See ticket:555#comment:13 for the notes regarding the installation of the MySQL munin stats package. See ticket:677#comment:3 for the Redis plugin install notes. 
    20  
    21 Sometimes the IO State graph stops, this can be fixed by deleting the lock files, see ticket:555#IOstategraph. 
    22  
    23 Some BOA upgrades change the Redis password and then it needed to be copied from {{{/etc/redis/redis.conf}}} to {{{/etc/munin/plugin-conf.d/munin-node}}} and munin-node needed restarting, see ticket:730 and PuffinServer#Muninconfigchanges. 
    24  
    25 We did have a trial with New Relic in 2013, see ticket:586 but this isn't on-going.  
    26  
    27 == HTTP Stats == 
    28  
    29 The wiki:PiwikServer generates stats from the humans visiting the server and some of these stats have been made public on wiki:WebStats. 
    30  
    31 There are some notes on analysing the raw Nginx stats on wiki:WebServerLogs and [https://penguin.transitionnetwork.org/webalizer/puffin/ Webalizer stats for Puffin] are available using the same username/password as this Trac site.  
    32  
    33 There is a wiki:ErrorCodeCheck script which emails the total number of HTTP errors each day, see ticket:483#comment:63 for a list of the total for August, September and October 2013. 
    34  
    35 == Load Spikes == 
    36  
    37 The documentation of the load spike suicides that the server suffered from in 2013 has been archived to wiki:PuffinServerBoaLoadSpikes as that documentation is now out dated. 
    38  
    39 When the server was updated to BOA-2.2.3 on ticket:721 the scripts in {{{/var/xdrago/}}} were changed, however the load spike issue hasn't been finally resolved, see ticket:670#comment:22. 
    40  
    41 == Tickets == 
    42  
    43 Most the "live server" tickets relate to puffin, but the older ones, prior to ticket number #466, are for previous servers. 
    44  
    45 === Current live server tickets === 
    46  
    47 [[TicketQuery(status=accepted|new|assigned|reopened&component=Live server,order=id,desc=1,format=table,col=summary|owner|reporter)]] 
    48  
    49 === Closed live server tickets === 
    50  
    51 [[TicketQuery(status=closed&component=Live server,order=id,desc=1,format=table,col=summary|owner|reporter)]] 
    52  
    5313== Barracuda Octopus Ageir == 
    5414 
    55 The server is using [https://drupal.org/project/octopus Octopus] to manage [http://community.aegirproject.org/ Ageir] and also the updates to the Transition Network Drupal site, this system is installed and upgraded using [https://drupal.org/project/barracuda Barracuda], the Barracuda Octopus Aegir combination is documented on the [http://groups.drupal.org/node/163784 BOA wiki]. 
     15The server is using [https://drupal.org/project/octopus Octopus] to manage [http://community.aegirproject.org/ Ageir] and also the updates to the Transition Network Drupal site, this system is installed and upgraded using [https://drupal.org/project/barracuda Barracuda], the Barracuda Octopus Aegir combination is documented on the [http://groups.drupal.org/node/163784 BOA wiki] and the [https://omega8.cc/options omega8.cc site]. 
     16 
     17BOA no longer sends out email when new versions are available so the [https://github.com/omega8cc/boa/milestones Milestones page] and [https://github.com/omega8cc/boa/blob/master/CHANGELOG.txt Changelog] of the [https://github.com/omega8cc/boa BOA Githut project] need to be manually checked. New versions are scheduled to be released monthly. 
    5618 
    5719The initial BOA install script output has been saved on ticket:466#comment:22 and the updates are now documented on tickets listed at  PuffinServer#Upgradetickets. 
    58  
    59 === MariaDB === 
    60  
    61 The MySQL root password is available in {{{/root/.my.cnf}}}. 
    62  
    63 Tuning of the MySQL server is being tracked on ticket:587. 
    64  
    65 We have set MySQL to use a RAM disk for temp tables, see ticket:591. 
    66  
    67 BOA installs MariaDB as the MySQL server using the debs from the MariaDB site, see {{{/etc/apt/sources.list.d/mariadb.list}}}, these are the current (2013-01-13) packages which are installed (note the config files only remain for php5-mysql as PHP in now installed from source code by BOA): 
    68  
    69 {{{ 
    70 dpkg -l | grep -i mysql 
    71 ii  libdbd-mysql-perl                       4.021-1+b1                    amd64        Perl5 database interface to the MySQL database 
    72 ii  libmysqlclient16                        5.1.72-2                      amd64        MySQL database client library 
    73 ii  libmysqlclient18                        5.5.34+maria-1~wheezy         amd64        Virtual package to satisfy external depends 
    74 ii  mariadb-common                          5.5.34+maria-1~wheezy         all          MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf) 
    75 ii  mysql-common                            5.5.34+maria-1~wheezy         all          MariaDB database common files (e.g. /etc/mysql/my.cnf) 
    76 ii  mytop                                   1.6-6                         all          top like query monitor for MySQL 
    77 rc  php5-mysql                              5.3.27-1~dotdeb.0             amd64        MySQL module for php5 
    78 ii  python-mysqldb                          1.2.3-2                       amd64        Python interface to MySQL 
    79 }}} 
    80  
    81 === Nginx === 
    82  
    83 BOA did use Nginx from dotdeb but now it compiles it from source, the dotdeb config files remain: 
    84  
    85 {{{ 
    86 dpkg -l | grep -i nginx 
    87 rc  nginx-common                            1.4.1-1~dotdeb.0              all          small, powerful, scalable web/proxy server - common files 
    88 }}} 
    89  
    90 The only changes made to the default nginx configuration during the initial install was to move the key and cert it was using out of the way and symlink to the *.transitionnetwork.org ones, see ticket:466#comment:25 and also ticket:707#comment:21. 
    91  
    92 The other change made from the default BOA config are to enable Munin graphs, see wiki:PuffinServer#nginxconfigchanges 
    93  
    94 === php-fpm === 
    95  
    96 Please note that the version of php-fpm that the http://transitionnetwork.org/ site needs to be running to work properly is: 
    97  
    98 {{{ 
    99 /etc/init.d/php53-fpm  
    100 }}} 
    101  
    102 The config file for it is {{{/opt/local/etc/php53-fpm.conf}}} and when it is running it is listed in top and ps as php-fpm: 
    103  
    104 {{{ 
    105 ps -lA | grep php 
    106 1 S     0 29482     1  0  80   0 - 188067 -     ?        00:00:00 php-fpm 
    107 5 S    33 29483 29482  2  80   0 - 205351 -     ?        00:01:32 php-fpm 
    108 5 S    33 29484 29482  2  80   0 - 199726 -     ?        00:01:28 php-fpm 
    109 ... 
    110 }}} 
    111  
    112 Please note the settings that we changed from the default BOA ones in {{{/opt/local/etc/php53-fpm.conf}}} below. 
    113  
    114 When the server boots another version of php-fpm was also started, which is listed in top and ps as php5-fpm, this one: 
    115  
    116 {{{ 
    117 /etc/init.d/php5-fpm 
    118 }}} 
    119  
    120 Which is configured via files in {{{/etc/php5/fpm/}}}. This version should be stopped if it is found to be running: 
    121  
    122 {{{ 
    123 /etc/init.d/php5-fpm stop 
    124 }}} 
    125  
    126 It was stopped from running at runlevel 2 by deleting this symlink (see ticket:560#comment:17): 
    127  
    128 {{{ 
    129 /etc/rc2.d/S01php5-fpm -> ../init.d/php5-fpm 
    130 }}} 
    131  
    132 But that didn't solve the problem, see ticket:580. 
    133  
    134 === Redis === 
    135  
    136 Tickets related to Redis issues: 
    137  
    138 * ticket:730 Redist Munin stats stop working after BOA upgrade 
    139 * ticket:554 Site slow down and MySQL load increase 
    140 * ticket:677 Spike in MyISAM (search) database activity, Redis unable to cache such requests 
    141  
    142 Redis Munin graphs: 
    143  
    144 * https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/index.html#redis 
    14520 
    14621=== Upgrading BOA === 
    … …  
    299174Settings in {{{/etc/mysql/my.cnf}}} are changed from the default, following an experiment with running for a while with the BOA default settings, see ticket:670. The MySQL tuning has been documented on ticket:587 and for the recent changes to the BOA defaults see ticket:587#comment:16 onwards. 
    300175 
     176=== MariaDB === 
     177 
     178The MySQL root password is available in {{{/root/.my.cnf}}}. 
     179 
     180Tuning of the MySQL server is being tracked on ticket:587. 
     181 
     182We have set MySQL to use a RAM disk for temp tables, see ticket:591. 
     183 
     184BOA installs MariaDB as the MySQL server using the debs from the MariaDB site, see {{{/etc/apt/sources.list.d/mariadb.list}}}, these are the current (2013-01-13) packages which are installed (note the config files only remain for php5-mysql as PHP in now installed from source code by BOA): 
     185 
     186{{{ 
     187dpkg -l | grep -i mysql 
     188ii  libdbd-mysql-perl                       4.021-1+b1                    amd64        Perl5 database interface to the MySQL database 
     189ii  libmysqlclient16                        5.1.72-2                      amd64        MySQL database client library 
     190ii  libmysqlclient18                        5.5.34+maria-1~wheezy         amd64        Virtual package to satisfy external depends 
     191ii  mariadb-common                          5.5.34+maria-1~wheezy         all          MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf) 
     192ii  mysql-common                            5.5.34+maria-1~wheezy         all          MariaDB database common files (e.g. /etc/mysql/my.cnf) 
     193ii  mytop                                   1.6-6                         all          top like query monitor for MySQL 
     194rc  php5-mysql                              5.3.27-1~dotdeb.0             amd64        MySQL module for php5 
     195ii  python-mysqldb                          1.2.3-2                       amd64        Python interface to MySQL 
     196}}} 
     197 
     198=== Nginx === 
     199 
     200BOA did use Nginx from dotdeb but now it compiles it from source, the dotdeb config files remain: 
     201 
     202{{{ 
     203dpkg -l | grep -i nginx 
     204rc  nginx-common                            1.4.1-1~dotdeb.0              all          small, powerful, scalable web/proxy server - common files 
     205}}} 
     206 
     207The only changes made to the default nginx configuration during the initial install was to move the key and cert it was using out of the way and symlink to the *.transitionnetwork.org ones, see ticket:466#comment:25 and also ticket:707#comment:21. 
     208 
     209The other change made from the default BOA config are to enable Munin graphs, see wiki:PuffinServer#nginxconfigchanges 
     210 
     211=== php-fpm === 
     212 
     213Please note that the version of php-fpm that the http://transitionnetwork.org/ site needs to be running to work properly is: 
     214 
     215{{{ 
     216/etc/init.d/php53-fpm  
     217}}} 
     218 
     219The config file for it is {{{/opt/local/etc/php53-fpm.conf}}} and when it is running it is listed in top and ps as php-fpm: 
     220 
     221{{{ 
     222ps -lA | grep php 
     2231 S     0 29482     1  0  80   0 - 188067 -     ?        00:00:00 php-fpm 
     2245 S    33 29483 29482  2  80   0 - 205351 -     ?        00:01:32 php-fpm 
     2255 S    33 29484 29482  2  80   0 - 199726 -     ?        00:01:28 php-fpm 
     226... 
     227}}} 
     228 
     229Please note the settings that we changed from the default BOA ones in {{{/opt/local/etc/php53-fpm.conf}}} below. 
     230 
     231When the server boots another version of php-fpm was also started, which is listed in top and ps as php5-fpm, this one: 
     232 
     233{{{ 
     234/etc/init.d/php5-fpm 
     235}}} 
     236 
     237Which is configured via files in {{{/etc/php5/fpm/}}}. This version should be stopped if it is found to be running: 
     238 
     239{{{ 
     240/etc/init.d/php5-fpm stop 
     241}}} 
     242 
     243It was stopped from running at runlevel 2 by deleting this symlink (see ticket:560#comment:17): 
     244 
     245{{{ 
     246/etc/rc2.d/S01php5-fpm -> ../init.d/php5-fpm 
     247}}} 
     248 
     249But that didn't solve the problem, see ticket:580. 
     250 
     251=== Redis === 
     252 
     253Tickets related to Redis issues: 
     254 
     255* ticket:730 Redist Munin stats stop working after BOA upgrade 
     256* ticket:554 Site slow down and MySQL load increase 
     257* ticket:677 Spike in MyISAM (search) database activity, Redis unable to cache such requests 
     258 
     259Redis Munin graphs: 
     260 
     261* https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/index.html#redis 
     262 
     263== Munin Stats == 
     264 
     265There are munin stats for the server available here 
     266 
     267* https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/ 
     268 
     269See ticket:555#comment:13 for the notes regarding the installation of the MySQL munin stats package. See ticket:677#comment:3 for the Redis plugin install notes. 
     270 
     271Sometimes the IO State graph stops, this can be fixed by deleting the lock files, see ticket:555#IOstategraph. 
     272 
     273Some BOA upgrades change the Redis password and then it needed to be copied from {{{/etc/redis/redis.conf}}} to {{{/etc/munin/plugin-conf.d/munin-node}}} and munin-node needed restarting, see ticket:730 and PuffinServer#Muninconfigchanges. 
     274 
     275We did have a trial with New Relic in 2013, see ticket:586 but this isn't on-going.  
     276 
     277== HTTP Stats == 
     278 
     279The wiki:PiwikServer generates stats from the humans visiting the server and some of these stats have been made public on wiki:WebStats. 
     280 
     281There are some notes on analysing the raw Nginx stats on wiki:WebServerLogs and [https://penguin.transitionnetwork.org/webalizer/puffin/ Webalizer stats for Puffin] are available using the same username/password as this Trac site.  
     282 
     283There is a wiki:ErrorCodeCheck script which emails the total number of HTTP errors each day, see ticket:483#comment:63 for a list of the total for August, September and October 2013. 
     284 
     285== Load Spikes == 
     286 
     287The documentation of the load spike suicides that the server suffered from in 2013 has been archived to wiki:PuffinServerBoaLoadSpikes as that documentation is now out dated. 
     288 
     289When the server was updated to BOA-2.2.3 on ticket:721 the scripts in {{{/var/xdrago/}}} were changed, however the load spike issue hasn't been finally resolved, see ticket:670#comment:22. 
     290 
     291== Tickets == 
     292 
     293Most the "live server" tickets relate to puffin, but the older ones, prior to ticket number #466, are for previous servers. 
     294 
     295=== Current live server tickets === 
     296 
     297[[TicketQuery(status=accepted|new|assigned|reopened&component=Live server,order=id,desc=1,format=table,col=summary|owner|reporter)]] 
     298 
     299=== Closed live server tickets === 
     300 
     301[[TicketQuery(status=closed&component=Live server,order=id,desc=1,format=table,col=summary|owner|reporter)]] 
     302 
     303 
    301304=== System Updates === 
    302305