Ticket #567 (closed maintenance: fixed)
Update BOA for new Redis 2.6.14
| Reported by: | chris | Owned by: | chris |
|---|---|---|---|
| Priority: | critical | Milestone: | Maintenance |
| Component: | Live server | Keywords: | |
| Cc: | ed, jim | Estimated Number of Hours: | 1.0 |
| Add Hours to Ticket: | 0 | Billable?: | yes |
| Total Hours: | 1.0 |
Description
A suggestion from Jim:
BOA now includes Redis
2.6.14 <https://raw.github.com/antirez/redis/2.6/00-RELEASENOTES> if
you do a 'barracda up-stable system'... What interests me about this is
these lines from the changelog:
UPGRADE URGENCY: HIGH because of the following two issues:
- Lua scripting + Replication + AOF in slaves problem (see Issue #1164).
- AOF + expires possible race condition (see Issue #1079).
It's a long shot, but that could maybe be part of the issue we've seen
recently.
I'm not sure if this is best done now or later tonight when the site is less busy?
Change History
comment:2 Changed 3 years ago by chris
I'm going to do this update now, the site might be down for a few mins, I have checked in admin and nobody else is logged in and Sunday is the lowest traffic time of the week.
comment:3 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 1.0
- Total Hours changed from 0.0 to 1.0
Following the notes here wiki:PuffinServer#UpgradingBOA
cd wget -q -U iCab http://files.aegir.cc/BOA.sh.txt bash BOA.sh.txt BOA Meta Installer setup completed Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to Bye barracuda up-stable system waiting 8 sec REPORT: Successful Barracuda upgrade on puffin.webarch.net sent to chris@webarchitects.co.uk BARRACUDA upgrade completed Bye
The email sent contains:
Barracuda [Sun Jul 14 21:45:07 BST 2013] ==> BOA Skynet welcomes you aboard! Barracuda [Sun Jul 14 21:45:11 BST 2013] ==> INFO: UPGRADE Barracuda [Sun Jul 14 21:45:11 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file Barracuda [Sun Jul 14 21:45:12 BST 2013] ==> NOTE! Please review all config options displayed below Barracuda [Sun Jul 14 21:45:12 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script ### ### Configuration created on 121215-1545 ### with Barracuda version BOA-2.0.4 ### ### NOTE: the group of settings displayed bellow will *not* be overriden ### on upgrade by the Barracuda script nor by this configuration file. ### They can be defined only on initial Barracuda install. ### _HTTP_WILDCARD=YES _MY_OWNIP="81.95.52.103" #_MY_OWNIP="" _MY_HOSTN="puffin.webarch.net" #_MY_HOSTN="" _MY_FRONT="master.puffin.webarch.net" _THIS_DB_HOST=localhost #_THIS_DB_HOST=FQDN _SMTP_RELAY_TEST=YES _SMTP_RELAY_HOST="" _LOCAL_NETWORK_IP="" _LOCAL_NETWORK_HN="" ### ### NOTE: the group of settings displayed bellow ### will *override* all listed settings in the Barracuda script, ### both on initial install and upgrade. ### _MY_EMAIL="chris@webarchitects.co.uk" _XTRAS_LIST="PDS CSF CHV" _AUTOPILOT=YES _DEBUG_MODE=NO _DB_SERVER=MariaDB _SSH_PORT=22 _LOCAL_DEBIAN_MIRROR="ftp.debian.org" _LOCAL_UBUNTU_MIRROR="archive.ubuntu.com" _FORCE_GIT_MIRROR="" _DNS_SETUP_TEST=YES _NGINX_EXTRA_CONF="" _NGINX_WORKERS=AUTO _PHP_FPM_WORKERS=AUTO _BUILD_FROM_SRC=NO _PHP_MODERN_ONLY=YES _PHP_FPM_VERSION=5.3 _PHP_CLI_VERSION=5.3 #_LOAD_LIMIT_ONE=1444 #_LOAD_LIMIT_TWO=888 _LOAD_LIMIT_ONE==7220 _LOAD_LIMIT_TWO=4440 _CUSTOM_CONFIG_CSF=NO #_CUSTOM_CONFIG_SQL=NO _CUSTOM_CONFIG_SQL=YES _CUSTOM_CONFIG_REDIS=NO _CUSTOM_CONFIG_PHP_5_2=NO #_CUSTOM_CONFIG_PHP_5_3=NO _CUSTOM_CONFIG_PHP_5_3=YES _SPEED_VALID_MAX=3600 _NGINX_DOS_LIMIT=300 _SYSTEM_UPGRADE_ONLY=YES _USE_MEMCACHED=NO _NEWRELIC_KEY= _USE_STOCK=NO ## ### Configuration created on 121215-1545 ### with Barracuda version BOA-2.0.4 ### ### JK reinstall PHP _EXTRA_PACKAGES= _PHP_EXTRA_CONF="" _STRONG_PASSWORDS=NO _DB_BINARY_LOG=NO _DB_ENGINE=InnoDB _NGINX_LDAP=NO _PHP_GEOS=NO _PHP_MONGODB=NO _AEGIR_UPGRADE_ONLY=NO Barracuda [Sun Jul 14 21:45:14 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait... Barracuda [Sun Jul 14 21:45:16 BST 2013] ==> INFO: GitHub mirror repository will be used for this install Barracuda [Sun Jul 14 21:45:16 BST 2013] ==> INFO: Downloading little helpers, please wait... Barracuda [Sun Jul 14 21:45:17 BST 2013] ==> INFO: Checking BARRACUDA version... Barracuda [Sun Jul 14 21:45:17 BST 2013] ==> INFO: Version test result: OK Barracuda [Sun Jul 14 21:45:17 BST 2013] ==> INFO: Checking your Debian or Ubuntu version... Barracuda [Sun Jul 14 21:45:20 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9 Barracuda [Sun Jul 14 21:45:24 BST 2013] ==> UPGRADE START -> checkpoint: * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct? * Your server hostname is puffin.webarch.net. * Your Aegir control panel is/will be available at https://master.puffin.webarch.net. Barracuda [Sun Jul 14 21:45:24 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/ Barracuda [Sun Jul 14 21:45:26 BST 2013] ==> INFO: Updating apt sources Barracuda [Sun Jul 14 21:45:28 BST 2013] ==> INFO: We will use Debian mirror ftp.debian.org Barracuda [Sun Jul 14 21:45:32 BST 2013] ==> INFO: Running aptitude update, please wait... Barracuda [Sun Jul 14 21:45:40 BST 2013] ==> INFO: Upgrading required libraries and tools Barracuda [Sun Jul 14 21:45:40 BST 2013] ==> NOTE! This step may take a few minutes, please wait... Barracuda [Sun Jul 14 21:46:15 BST 2013] ==> INFO: Testing Nginx version... Barracuda [Sun Jul 14 21:46:18 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.0, upgrade required Barracuda [Sun Jul 14 21:46:20 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait... Barracuda [Sun Jul 14 21:46:23 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean Barracuda [Sun Jul 14 21:46:23 BST 2013] ==> INFO: Upgrading Nginx, please wait... Barracuda [Sun Jul 14 21:47:48 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait... Barracuda [Sun Jul 14 21:48:55 BST 2013] ==> INFO: Testing Nginx version... Barracuda [Sun Jul 14 21:48:57 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required Barracuda [Sun Jul 14 21:49:00 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait... Barracuda [Sun Jul 14 21:49:02 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean Barracuda [Sun Jul 14 21:49:02 BST 2013] ==> INFO: Checking SMTP connections, please wait... Barracuda [Sun Jul 14 21:49:05 BST 2013] ==> INFO: Upgrading a few more tools, please wait... Barracuda [Sun Jul 14 21:49:13 BST 2013] ==> INFO: Checking if PHP upgrade is available Barracuda [Sun Jul 14 21:49:17 BST 2013] ==> INFO: Installed PHP version 5.3.26-1~dotdeb.0, no upgrade required Barracuda [Sun Jul 14 21:49:17 BST 2013] ==> INFO: Installing PhpRedis upgrade for PHP-FPM 5.3.27, please wait... Barracuda [Sun Jul 14 21:50:40 BST 2013] ==> INFO: Installing UploadProgress upgrade for PHP-FPM 5.3.27, please wait... Barracuda [Sun Jul 14 21:51:14 BST 2013] ==> INFO: Installing JSMin upgrade for PHP-FPM 5.3.27, please wait... Barracuda [Sun Jul 14 21:52:12 BST 2013] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait... Barracuda [Sun Jul 14 21:52:15 BST 2013] ==> INFO: Upgrading Limited Shell, please wait... Barracuda [Sun Jul 14 21:52:55 BST 2013] ==> INFO: Installed Redis version 2.6.13, upgrade/rebuild required Barracuda [Sun Jul 14 21:52:58 BST 2013] ==> INFO: Installing Redis update for Debian/squeeze, please wait... cp: cannot create regular file `/usr/bin/redis-server': Text file busy Barracuda [Sun Jul 14 21:55:09 BST 2013] ==> INFO: Generating random password for Redis server Barracuda [Sun Jul 14 21:55:13 BST 2013] ==> INFO: OS and services upgrade completed Barracuda [Sun Jul 14 21:55:15 BST 2013] ==> INFO: Aegir Master Instance upgrade skipped Barracuda [Sun Jul 14 21:55:16 BST 2013] ==> INFO: Installing extra Drush versions Barracuda [Sun Jul 14 21:55:26 BST 2013] ==> INFO: Drush 4 installation complete Barracuda [Sun Jul 14 21:55:27 BST 2013] ==> INFO: Drush 5 installation complete Barracuda [Sun Jul 14 21:55:29 BST 2013] ==> INFO: Drush 6 installation complete Barracuda [Sun Jul 14 21:55:36 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx Barracuda [Sun Jul 14 21:55:59 BST 2013] ==> INFO: Restarting MariaDB server Barracuda [Sun Jul 14 21:56:41 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt Barracuda [Sun Jul 14 21:56:44 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt Barracuda [Sun Jul 14 21:56:51 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait... Barracuda [Sun Jul 14 21:57:18 BST 2013] ==> INFO: csf/lfd firewall upgrade completed Barracuda [Sun Jul 14 21:57:21 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service... Barracuda [Sun Jul 14 21:57:27 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :) Barracuda [Sun Jul 14 21:57:32 BST 2013] ==> Final post-upgrade cleaning, please wait a moment... Barracuda [Sun Jul 14 21:58:31 BST 2013] ==> BYE!
No updates as documented here were needed wiki:PuffinServer#nginxconfigchanges
The updates here were manually applied, wiki:PuffinServer#php-fpmconfigchanges
No updates as documented (docs also need updating here) were needed wiki:PuffinServer#mysqlconfigchanges
The updates here were manually applied, wiki:PuffinServer#xdragoshellscriptchanges
The backup copy of second.sh was copied across:
cd /var/xdrago/ mv second.sh second.sh.bak cp /root/second.sh .
And the diff was checked:
diff second.sh.bak second.sh | vim - 7a8,22 > # start additions > echo "====================" >> /var/log/high-load.log > echo "php-fpm and nginx about to be killed" >> /var/log/high-load.log > echo "ONEX_LOAD = $ONEX_LOAD" >> /var/log/high-load.log > echo "FIVX_LOAD = $FIVX_LOAD" >> /var/log/high-load.log > echo "uptime : " >> /var/log/high-load.log > uptime >> /var/log/high-load.log > echo "top : " >> /var/log/high-load.log > top -n 1 -b >> /var/log/high-load.log > #echo "processes : " >> /var/log/high-load.log > #ps -lA >> /var/log/high-load.log > #echo "cat /proc/interrupts : " >> /var/log/high-load.log > #cat /proc/interrupts >> /var/log/high-load.log > echo "====================" >> /var/log/high-load.log > # end additions 15a31 > 28a45,59 > # start additions > echo "====================" >> /var/log/high-load.log > echo "nginx high load on" >> /var/log/high-load.log > echo "ONEX_LOAD = $ONEX_LOAD" >> /var/log/high-load.log > echo "FIVX_LOAD = $FIVX_LOAD" >> /var/log/high-load.log > echo "uptime : " >> /var/log/high-load.log > uptime >> /var/log/high-load.log > echo "top : " >> /var/log/high-load.log > top -n 1 -b >> /var/log/high-load.log > #echo "processes : " >> /var/log/high-load.log > #ps -lA >> /var/log/high-load.log > #echo "cat /proc/interrupts : " >> /var/log/high-load.log > #cat /proc/interrupts >> /var/log/high-load.log > echo "====================" >> /var/log/high-load.log > # end additions 34a66,78 > # start additions > echo "====================" >> /var/log/high-load.log > echo "nginx high load off" >> /var/log/high-load.log > echo "ONEX_LOAD = $ONEX_LOAD" >> /var/log/high-load.log > echo "FIVX_LOAD = $FIVX_LOAD" >> /var/log/high-load.log > echo "uptime : " >> /var/log/high-load.log > uptime >> /var/log/high-load.log > echo "top : " >> /var/log/high-load.log > top -n 1 -b >> /var/log/high-load.log > #echo "processes : " >> /var/log/high-load.log > #ps -lA >> /var/log/high-load.log > echo "====================" >> /var/log/high-load.log > # end additions 36a81 > echo "nginx_high_load_off" >> /var/log/high-load.log 43,45c88,105 < CTL_ONEX_SPIDER_LOAD=388 < CTL_FIVX_SPIDER_LOAD=388 < CTL_ONEX_LOAD==7220 --- > # Original values: > #CTL_ONEX_SPIDER_LOAD=388 > #CTL_FIVX_SPIDER_LOAD=388 > #CTL_ONEX_LOAD=1444 > #CTL_FIVX_LOAD=888 > #CTL_ONEX_LOAD_CRIT=1888 > #CTL_FIVX_LOAD_CRIT=1555 > # x4 of original: > #CTL_ONEX_SPIDER_LOAD=1552 > #CTL_FIVX_SPIDER_LOAD=1552 > #CTL_ONEX_LOAD=5776 > #CTL_FIVX_LOAD=3552 > #CTL_ONEX_LOAD_CRIT=7552 > #CTL_FIVX_LOAD_CRIT=6220 > # 5x of original: > CTL_ONEX_SPIDER_LOAD=1940 > CTL_FIVX_SPIDER_LOAD=1940 > CTL_ONEX_LOAD=7220 47,48c107,115 < CTL_ONEX_LOAD_CRIT=1888 < CTL_FIVX_LOAD_CRIT=1555 --- > CTL_ONEX_LOAD_CRIT=9440 > CTL_FIVX_LOAD_CRIT=7775 > # x6 of original: > #CTL_ONEX_SPIDER_LOAD=2328 > #CTL_FIVX_SPIDER_LOAD=2328 > #CTL_ONEX_LOAD=8664 > #CTL_FIVX_LOAD=5328 > #CTL_ONEX_LOAD_CRIT=11328 > #CTL_FIVX_LOAD_CRIT=9330
Some of the extra logging in the above can probably be removed.
I think the site was down for around 10 to 20 mins during the upgrade, I only noticed after starting the upgrade that the site was suffering another sustained load increase, see the munin stats, this will have made the upgrade take longer that it should have taken, but it all seems OK now.
The firewall has blocked the webarchitects monitoring server, see wiki:PuffinServer#CSFLDF
csf -g 81.95.52.66 Chain num pkts bytes target prot opt in out source destination DENYIN 101 74 5784 DROP all -- !lo * 81.95.52.66 0.0.0.0/0 Temporary Blocks: IP:81.95.52.66 Port: Dir:in TTL:3600 (lfd - *Port Scan* detected from 81.95.52.66 (GB/United Kingdom/nsa.rat.burntout.org). 20 hits in the last 115 seconds)
But as it is not yet the permanent block the command I have been using to clear the block doesn't work:
csf -dr 81.95.52.66 csf: 81.95.52.66 not found in csf.deny
I'll check this again in a hour or so or perhaps tomorrow.
I have added 10 mins to this ticket to cover the reading the the two reports Ed sent to the ttech list and also for the email I sent there about looking at following up the issue with the 10x hits in the ngnix logs compared with the piwik stats on the piwik forum.
Note: See
TracTickets for help on using
tickets.
