Ticket #567 (closed maintenance: fixed)

Opened 3 years ago

Last modified 3 years ago

Update BOA for new Redis 2.6.14

Reported by: chris Owned by: chris
Priority: critical Milestone: Maintenance
Component: Live server Keywords:
Cc: ed, jim Estimated Number of Hours: 1.0
Add Hours to Ticket: 0 Billable?: yes
Total Hours: 1.0

Description

A suggestion from Jim:

BOA now includes Redis
2.6.14 <https://raw.github.com/antirez/redis/2.6/00-RELEASENOTES> if
you do a 'barracda up-stable system'... What interests me about this is
these lines from the changelog:

UPGRADE URGENCY: HIGH because of the following two issues:

  • Lua scripting + Replication + AOF in slaves problem (see Issue #1164).
  • AOF + expires possible race condition (see Issue #1079).

It's a long shot, but that could maybe be part of the issue we've seen
recently.

I'm not sure if this is best done now or later tonight when the site is less busy?

Change History

comment:1 Changed 3 years ago by jim

It could wait for the next OS-related updates I guess... I wouldn't rush,
this is just a head-up that Redis race conditions could be the cause of
these lockups we see...

comment:2 Changed 3 years ago by chris

I'm going to do this update now, the site might be down for a few mins, I have checked in admin and nobody else is logged in and Sunday is the lowest traffic time of the week.

comment:3 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 1.0
  • Total Hours changed from 0.0 to 1.0

Following the notes here wiki:PuffinServer#UpgradingBOA

cd
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
  BOA Meta Installer setup completed
  Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
  Bye
barracuda up-stable system
  waiting 8 sec
  REPORT: Successful Barracuda upgrade on puffin.webarch.net sent to chris@webarchitects.co.uk
  BARRACUDA upgrade completed
  Bye

The email sent contains:

Barracuda [Sun Jul 14 21:45:07 BST 2013] ==> BOA Skynet welcomes you aboard!

Barracuda [Sun Jul 14 21:45:11 BST 2013] ==> INFO: UPGRADE
Barracuda [Sun Jul 14 21:45:11 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Sun Jul 14 21:45:12 BST 2013] ==> NOTE! Please review all config options displayed below
Barracuda [Sun Jul 14 21:45:12 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script

###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
#_MY_OWNIP=""
_MY_HOSTN="puffin.webarch.net"
#_MY_HOSTN=""
_MY_FRONT="master.puffin.webarch.net"
_THIS_DB_HOST=localhost
#_THIS_DB_HOST=FQDN
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV"
_AUTOPILOT=YES
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
#_LOAD_LIMIT_ONE=1444
#_LOAD_LIMIT_TWO=888
_LOAD_LIMIT_ONE==7220
_LOAD_LIMIT_TWO=4440
_CUSTOM_CONFIG_CSF=NO
#_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_SQL=YES
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
#_CUSTOM_CONFIG_PHP_5_3=NO
_CUSTOM_CONFIG_PHP_5_3=YES
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=YES
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
##
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### JK reinstall PHP
_EXTRA_PACKAGES=
_PHP_EXTRA_CONF=""
_STRONG_PASSWORDS=NO
_DB_BINARY_LOG=NO
_DB_ENGINE=InnoDB
_NGINX_LDAP=NO
_PHP_GEOS=NO
_PHP_MONGODB=NO
_AEGIR_UPGRADE_ONLY=NO

Barracuda [Sun Jul 14 21:45:14 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Sun Jul 14 21:45:16 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Sun Jul 14 21:45:16 BST 2013] ==> INFO: Downloading little helpers, please wait...
Barracuda [Sun Jul 14 21:45:17 BST 2013] ==> INFO: Checking BARRACUDA version...
Barracuda [Sun Jul 14 21:45:17 BST 2013] ==> INFO: Version test result: OK
Barracuda [Sun Jul 14 21:45:17 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...

Barracuda [Sun Jul 14 21:45:20 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9


Barracuda [Sun Jul 14 21:45:24 BST 2013] ==> UPGRADE START -> checkpoint:

  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your server hostname is puffin.webarch.net.
  * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.


Barracuda [Sun Jul 14 21:45:24 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Sun Jul 14 21:45:26 BST 2013] ==> INFO: Updating apt sources
Barracuda [Sun Jul 14 21:45:28 BST 2013] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Sun Jul 14 21:45:32 BST 2013] ==> INFO: Running aptitude update, please wait...
Barracuda [Sun Jul 14 21:45:40 BST 2013] ==> INFO: Upgrading required libraries and tools
Barracuda [Sun Jul 14 21:45:40 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Sun Jul 14 21:46:15 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Sun Jul 14 21:46:18 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.0, upgrade required
Barracuda [Sun Jul 14 21:46:20 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Sun Jul 14 21:46:23 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Sun Jul 14 21:46:23 BST 2013] ==> INFO: Upgrading Nginx, please wait...
Barracuda [Sun Jul 14 21:47:48 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Sun Jul 14 21:48:55 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Sun Jul 14 21:48:57 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Sun Jul 14 21:49:00 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Sun Jul 14 21:49:02 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Sun Jul 14 21:49:02 BST 2013] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Sun Jul 14 21:49:05 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
Barracuda [Sun Jul 14 21:49:13 BST 2013] ==> INFO: Checking if PHP upgrade is available
Barracuda [Sun Jul 14 21:49:17 BST 2013] ==> INFO: Installed PHP version 5.3.26-1~dotdeb.0, no upgrade required
Barracuda [Sun Jul 14 21:49:17 BST 2013] ==> INFO: Installing PhpRedis upgrade for PHP-FPM 5.3.27, please wait...
Barracuda [Sun Jul 14 21:50:40 BST 2013] ==> INFO: Installing UploadProgress upgrade for PHP-FPM 5.3.27, please wait...
Barracuda [Sun Jul 14 21:51:14 BST 2013] ==> INFO: Installing JSMin upgrade for PHP-FPM 5.3.27, please wait...
Barracuda [Sun Jul 14 21:52:12 BST 2013] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
Barracuda [Sun Jul 14 21:52:15 BST 2013] ==> INFO: Upgrading Limited Shell, please wait...
Barracuda [Sun Jul 14 21:52:55 BST 2013] ==> INFO: Installed Redis version 2.6.13, upgrade/rebuild required
Barracuda [Sun Jul 14 21:52:58 BST 2013] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
cp: cannot create regular file `/usr/bin/redis-server': Text file busy
Barracuda [Sun Jul 14 21:55:09 BST 2013] ==> INFO: Generating random password for Redis server
Barracuda [Sun Jul 14 21:55:13 BST 2013] ==> INFO: OS and services upgrade completed

Barracuda [Sun Jul 14 21:55:15 BST 2013] ==> INFO: Aegir Master Instance upgrade skipped


Barracuda [Sun Jul 14 21:55:16 BST 2013] ==> INFO: Installing extra Drush versions
Barracuda [Sun Jul 14 21:55:26 BST 2013] ==> INFO: Drush 4 installation complete
Barracuda [Sun Jul 14 21:55:27 BST 2013] ==> INFO: Drush 5 installation complete
Barracuda [Sun Jul 14 21:55:29 BST 2013] ==> INFO: Drush 6 installation complete
Barracuda [Sun Jul 14 21:55:36 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
Barracuda [Sun Jul 14 21:55:59 BST 2013] ==> INFO: Restarting MariaDB server

Barracuda [Sun Jul 14 21:56:41 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
Barracuda [Sun Jul 14 21:56:44 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Sun Jul 14 21:56:51 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
Barracuda [Sun Jul 14 21:57:18 BST 2013] ==> INFO: csf/lfd firewall upgrade completed

Barracuda [Sun Jul 14 21:57:21 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
Barracuda [Sun Jul 14 21:57:27 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)

Barracuda [Sun Jul 14 21:57:32 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Sun Jul 14 21:58:31 BST 2013] ==> BYE!

No updates as documented here were needed wiki:PuffinServer#nginxconfigchanges

The updates here were manually applied, wiki:PuffinServer#php-fpmconfigchanges

No updates as documented (docs also need updating here) were needed wiki:PuffinServer#mysqlconfigchanges

The updates here were manually applied, wiki:PuffinServer#xdragoshellscriptchanges

The backup copy of second.sh was copied across:

cd /var/xdrago/
mv second.sh second.sh.bak
cp /root/second.sh .

And the diff was checked:

diff second.sh.bak second.sh | vim -
7a8,22
>   # start additions
>   echo "====================" >> /var/log/high-load.log
>   echo "php-fpm and nginx about to be killed" >> /var/log/high-load.log
>   echo "ONEX_LOAD = $ONEX_LOAD" >> /var/log/high-load.log
>   echo "FIVX_LOAD = $FIVX_LOAD" >> /var/log/high-load.log
>   echo "uptime : " >> /var/log/high-load.log
>   uptime >> /var/log/high-load.log
>   echo "top : " >> /var/log/high-load.log
>   top -n 1 -b >> /var/log/high-load.log 
>   #echo "processes : " >> /var/log/high-load.log
>   #ps -lA >> /var/log/high-load.log 
>   #echo "cat /proc/interrupts : " >> /var/log/high-load.log
>   #cat /proc/interrupts >> /var/log/high-load.log
>   echo "====================" >> /var/log/high-load.log
>   # end additions
15a31
> 
28a45,59
>   # start additions
>   echo "====================" >> /var/log/high-load.log
>   echo "nginx high load on" >> /var/log/high-load.log
>   echo "ONEX_LOAD = $ONEX_LOAD" >> /var/log/high-load.log
>   echo "FIVX_LOAD = $FIVX_LOAD" >> /var/log/high-load.log
>   echo "uptime : " >> /var/log/high-load.log
>   uptime >> /var/log/high-load.log
>   echo "top : " >> /var/log/high-load.log
>   top -n 1 -b >> /var/log/high-load.log 
>   #echo "processes : " >> /var/log/high-load.log
>   #ps -lA >> /var/log/high-load.log 
>   #echo "cat /proc/interrupts : " >> /var/log/high-load.log
>   #cat /proc/interrupts >> /var/log/high-load.log
>   echo "====================" >> /var/log/high-load.log
>   # end additions
34a66,78
>   # start additions
>   echo "====================" >> /var/log/high-load.log
>   echo "nginx high load off" >> /var/log/high-load.log
>   echo "ONEX_LOAD = $ONEX_LOAD" >> /var/log/high-load.log
>   echo "FIVX_LOAD = $FIVX_LOAD" >> /var/log/high-load.log
>   echo "uptime : " >> /var/log/high-load.log
>   uptime >> /var/log/high-load.log
>   echo "top : " >> /var/log/high-load.log
>   top -n 1 -b >> /var/log/high-load.log 
>   #echo "processes : " >> /var/log/high-load.log
>   #ps -lA >> /var/log/high-load.log 
>   echo "====================" >> /var/log/high-load.log
>   # end additions
36a81
>   echo "nginx_high_load_off" >> /var/log/high-load.log
43,45c88,105
< CTL_ONEX_SPIDER_LOAD=388
< CTL_FIVX_SPIDER_LOAD=388
< CTL_ONEX_LOAD==7220
---
> # Original values:
> #CTL_ONEX_SPIDER_LOAD=388
> #CTL_FIVX_SPIDER_LOAD=388
> #CTL_ONEX_LOAD=1444
> #CTL_FIVX_LOAD=888
> #CTL_ONEX_LOAD_CRIT=1888
> #CTL_FIVX_LOAD_CRIT=1555
> # x4 of original:
> #CTL_ONEX_SPIDER_LOAD=1552
> #CTL_FIVX_SPIDER_LOAD=1552
> #CTL_ONEX_LOAD=5776
> #CTL_FIVX_LOAD=3552
> #CTL_ONEX_LOAD_CRIT=7552
> #CTL_FIVX_LOAD_CRIT=6220
> # 5x of original:
> CTL_ONEX_SPIDER_LOAD=1940
> CTL_FIVX_SPIDER_LOAD=1940
> CTL_ONEX_LOAD=7220
47,48c107,115
< CTL_ONEX_LOAD_CRIT=1888
< CTL_FIVX_LOAD_CRIT=1555
---
> CTL_ONEX_LOAD_CRIT=9440
> CTL_FIVX_LOAD_CRIT=7775
> # x6 of original:
> #CTL_ONEX_SPIDER_LOAD=2328
> #CTL_FIVX_SPIDER_LOAD=2328
> #CTL_ONEX_LOAD=8664
> #CTL_FIVX_LOAD=5328
> #CTL_ONEX_LOAD_CRIT=11328
> #CTL_FIVX_LOAD_CRIT=9330

Some of the extra logging in the above can probably be removed.

I think the site was down for around 10 to 20 mins during the upgrade, I only noticed after starting the upgrade that the site was suffering another sustained load increase, see the munin stats, this will have made the upgrade take longer that it should have taken, but it all seems OK now.

The firewall has blocked the webarchitects monitoring server, see wiki:PuffinServer#CSFLDF

 csf -g 81.95.52.66

Chain            num   pkts bytes target     prot opt in     out     source               destination         

DENYIN           101     74  5784 DROP       all  --  !lo    *       81.95.52.66          0.0.0.0/0

Temporary Blocks: IP:81.95.52.66 Port: Dir:in TTL:3600 (lfd - *Port Scan* detected from 81.95.52.66 (GB/United Kingdom/nsa.rat.burntout.org). 20 hits in the last 115 seconds)

But as it is not yet the permanent block the command I have been using to clear the block doesn't work:

csf -dr 81.95.52.66
csf: 81.95.52.66 not found in csf.deny

I'll check this again in a hour or so or perhaps tomorrow.

I have added 10 mins to this ticket to cover the reading the the two reports Ed sent to the ttech list and also for the email I sent there about looking at following up the issue with the 10x hits in the ngnix logs compared with the piwik stats on the piwik forum.

Last edited 3 years ago by chris (previous) (diff)

comment:4 Changed 3 years ago by chris

  • Status changed from new to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.