New Relic Monitoring for BOA

[chris]

This is a ticket to document the consideration, and possible installation, of ​https://newrelic.com/ on the PuffinServer

[chris]

We currently have Munin stats:

• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/

And Piwik and Webalizer stats:

• ​https://stats.transitionnetwork.org/
• ​https://penguin.transitionnetwork.org/webalizer/puffin/

And the output of the mysql scripts, tuning-primer.sh and mysqltuner.pl as well as the output from things like top etc.

[chris]

Installation Implications

These are not insignificant, there are lots of required dependencies, it looks like a few hours work.

PHP

See ​https://newrelic.com/install

Add their GPG key, add their repo and install newrelic-php5 and then run newrelic-install install:

sudo -i
wget -O - http://download.newrelic.com/548C16BF.gpg | apt-key add -
echo "deb http://apt.newrelic.com/debian/ newrelic non-free" > /etc/apt/sources.list.d/newrelic.list
apt-get update
apt-get install newrelic-php5
newrelic-install install

Note: The New Relic installer will try to drop a newrelic.ini file in place. Some PHP installations are hard for our installer to find, or will not accept files. In these cases you will need to manually edit your php.ini file.

Nginx

See ​https://newrelic.com/plugins/nginx-inc/13

To use this plugin, you must have the following:

• Ruby (>= 1.8.7)
• bundler for Ruby (​https://github.com/carlhuda/bundler)

The server doesn't have ruby installed, the squeeze ruby version is 1.8.7, see ​http://packages.debian.org/squeeze/ruby1.8

More info on installing Bundler at ​http://bundler.io/

Plugin install instructions:

1. Download the NGINX web server plugin .tar.gz from Plugin Central.
2. Unpack the .tar.gz file in a new directory.
3. Run bundle install to install all needed prerequisites.
4. Copy config/newrelic_plugin.yml.in to config/newrelic_plugin.yml.
5. Insert your New Relic license key and NGINX status URL into config/newrelic_plugin.yml.
6. Run ./newrelic_nginx_agent.daemon start to start the agent in daemon mode.
7. More in-depth installation and configuration details are included in the README.txt file packaged with the plugin.

PHP-FPM

See ​https://newrelic.com/plugins/meetme-inc/123

Install via the Python Package Index or click the Download button below, then follow the directions in the README at ​https://github.com/MeetMe/newrelic-plugin-agent#installation-instructions

From there:

The agent requires Python 2.6 or 2.7 and pip for installation.

The server has python 2.6 installed.

The pip install instructions are here:

​http://www.pip-installer.org/en/latest/installing.html

MySQL

See ​https://newrelic.com/plugins/new-relic-inc/52

To use this plugin, you must have:

• a Java Runtime Environment (JRE) of 1.6 or higher

The server currently has these jre packages installed, I don't know if this is enough:

aptitude search jre | grep ^i
i A gcj-4.4-jre-lib                 - Java runtime library for use with gcj (jar
i A icedtea-6-jre-cacao             - Alternative JVM for OpenJDK, using Cacao  
i A openjdk-6-jre                   - OpenJDK Java runtime, using Hotspot JIT   
i A openjdk-6-jre-headless          - OpenJDK Java runtime, using Hotspot JIT (h
i A openjdk-6-jre-lib               - OpenJDK Java runtime (architecture indepen

Installation information

1. Download the MySQL plugin .tar.gz file from Plugin Central
2. Unpack the .tar.gz file in a new directory (typically tar zxvf newrelic_mysql_plugin-X.X.X.tar.gz)
3. Copy config/template_newrelic.properties to config/newrelic.properties
4. Open the config/newrelic.properties file and enter your license key (found in "Account Settings" menu at ​https://rpm.newrelic.com)
5. Copy config/template_mysql.instance.json to config/mysql.instance.json
6. Configure config/mysql.instance.json for each MySQL instance you want to monitor
7. Run java -jar <the plugin jar name> in your console
8. More in-depth installation and configuration details are included in the README file packaged with the plugin.

Redis

See ​https://newrelic.com/plugins/meetme-inc/28

This is the same plugin as for php-fpm.

Privacy and Security Concerns

New Relic is US based. Clearly we would be allowing far greater access to critical user and system data to a untrusted 3rd party than we currently do.

[jim]

It's no work... The install process for BOA is to add the key to the .barracuda.cnf file, then re-run barracuda up stable.

You should probably remove the 3/4 hour for this pointless research.

[chris]

Replying to jim:

It's no work... The install process for BOA is to add the key to the .barracuda.cnf file, then re-run barracuda up stable.

Does that install all these plugins?

• ​https://newrelic.com/plugins/nginx-inc/13
• ​https://newrelic.com/plugins/new-relic-inc/52
• ​https://newrelic.com/plugins/meetme-inc/28

[jim]

Replying to chris:

Does that install all these plugins?

• ​https://newrelic.com/plugins/nginx-inc/13
• ​https://newrelic.com/plugins/new-relic-inc/52
• ​https://newrelic.com/plugins/meetme-inc/28

Not at present -- it's the core install with Drupal/PHP monitoring... I'd suggest we go with the stock install for now, and if there's an area we want more visibility on we can add that/those particular plugin(s).

[chris]

So sorry this didn't happen this morning, have had a mad day with the server crashing, we replaced most the RAM last night and then more RAM and a CPU today plus we migrated a lot of things off it, I might get this done over the weekend, if not I will first thing on Monday.

[chris]

I have signed up for a New Relic account, I'd like it noted that I'm not happy with this:

You understand that the technical processing and transmission of the Service may be transferred unencrypted over a network, and assume all risks related thereto.

​https://newrelic.com/terms

This is what they say about the data they will collect:

We may share your Application Data with third parties to provide technical support or to provide specific services, such as hosting of your applications.

We may enter into agreements with Partner Companies such as hosting companies or other services providers. A Partner Company may want access to Application Data that we collect from its customers. As a result, we may provide your Application Data to a Partner Company on a confidential basis in order assist the Partner Company in providing customer support. Because we do not control the privacy practices of our Partner Companies, you should read and understand their privacy policies.

New Relic may disclose Application Data if it believes in good faith that such disclosure is necessary to (a) comply with relevant laws or to respond to subpoenas or warrants served on New Relic; or (b) protect or defend the rights or property of New Relic or users of the Products or related services.

If another company acquires our company or our assets, that company will possess the Application Data collected by it and us and will assume the rights and obligations regarding your Application Data as described in this Privacy Policy.

Except as otherwise stated in this policy, we do not disclose the Application Data collected from our services to third parties without consent from you. You expressly consent to the sharing of your Application Data as described in this policy.

​https://newrelic.com/privacy

I have added the license key to /root/.barracuda.cnf and following the steps documented at wiki:PuffinServer#UpgradingBOA

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
 BOA Meta Installer setup completed
 Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
 Bye
barracuda up-stable
  
 Barracuda [Sun Sep 15 09:18:00 BST 2013] ==> BOA Skynet welcomes you aboard!
  
 Barracuda [Sun Sep 15 09:18:04 BST 2013] ==> INFO: UPGRADE
 Barracuda [Sun Sep 15 09:18:04 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
 Barracuda [Sun Sep 15 09:18:05 BST 2013] ==> NOTE! Please review all config options displayed below
 Barracuda [Sun Sep 15 09:18:05 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script
  
 ###
 ### Configuration created on 121215-1545
 ### with Barracuda version BOA-2.0.4
 ###
 ### NOTE: the group of settings displayed bellow will *not* be overriden
 ### on upgrade by the Barracuda script nor by this configuration file.
 ### They can be defined only on initial Barracuda install.
 ###
 _HTTP_WILDCARD=YES
 _MY_OWNIP="81.95.52.103"
 #_MY_OWNIP=""
 _MY_HOSTN="puffin.webarch.net"
 #_MY_HOSTN=""
 _MY_FRONT="master.puffin.webarch.net"
 _THIS_DB_HOST=localhost
 #_THIS_DB_HOST=FQDN
 _SMTP_RELAY_TEST=YES
 _SMTP_RELAY_HOST=""
 _LOCAL_NETWORK_IP=""
 _LOCAL_NETWORK_HN=""
 ###
 ### NOTE: the group of settings displayed bellow
 ### will *override* all listed settings in the Barracuda script,
 ### both on initial install and upgrade.
 ###
 _MY_EMAIL="chris@webarchitects.co.uk"
 _XTRAS_LIST="PDS CSF CHV"
 _AUTOPILOT=NO
 _DEBUG_MODE=NO
 _DB_SERVER=MariaDB
 _SSH_PORT=22
 _LOCAL_DEBIAN_MIRROR="ftp.debian.org"
 _LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
 _FORCE_GIT_MIRROR=""
 _DNS_SETUP_TEST=YES
 _NGINX_EXTRA_CONF=""
 _NGINX_WORKERS=AUTO
 _PHP_FPM_WORKERS=AUTO
 _BUILD_FROM_SRC=NO
 _PHP_MODERN_ONLY=YES
 _PHP_FPM_VERSION=5.3
 _PHP_CLI_VERSION=5.3
 #_LOAD_LIMIT_ONE=1444
 #_LOAD_LIMIT_TWO=888
 _LOAD_LIMIT_ONE==7220
 _LOAD_LIMIT_TWO=4440
 _CUSTOM_CONFIG_CSF=NO
 #_CUSTOM_CONFIG_SQL=NO
 _CUSTOM_CONFIG_SQL=YES
 _CUSTOM_CONFIG_REDIS=NO
 _CUSTOM_CONFIG_PHP_5_2=NO
 #_CUSTOM_CONFIG_PHP_5_3=NO
 _CUSTOM_CONFIG_PHP_5_3=YES
 _SPEED_VALID_MAX=3600
 _NGINX_DOS_LIMIT=300
 _SYSTEM_UPGRADE_ONLY=NO
 _USE_MEMCACHED=NO
 _NEWRELIC_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 _USE_STOCK=NO
 ###
 ### Configuration created on 121215-1545
 ### with Barracuda version BOA-2.0.4
 ###
 ### JK reinstall PHP
 _EXTRA_PACKAGES=
 _PHP_EXTRA_CONF=""
 _STRONG_PASSWORDS=NO
 _DB_BINARY_LOG=NO
 _DB_ENGINE=InnoDB
 _NGINX_LDAP=NO
 _PHP_GEOS=NO
 _PHP_MONGODB=NO
 _AEGIR_UPGRADE_ONLY=NO
  
 Do you want to proceed with the upgrade? [Y/n] Y
 Barracuda [Sun Sep 15 09:19:00 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
 Barracuda [Sun Sep 15 09:19:02 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
 Barracuda [Sun Sep 15 09:19:02 BST 2013] ==> INFO: Downloading little helpers, please wait...
 Barracuda [Sun Sep 15 09:19:03 BST 2013] ==> INFO: Checking BARRACUDA version...
 Barracuda [Sun Sep 15 09:19:03 BST 2013] ==> INFO: Version test result: OK
 Barracuda [Sun Sep 15 09:19:03 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...
  
 Barracuda [Sun Sep 15 09:19:05 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9
  
 Barracuda [Sun Sep 15 09:19:10 BST 2013] ==> UPGRADE START -> checkpoint: 
 
   * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
   * Your server hostname is puffin.webarch.net.
   * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.
 
  
 Do you want to proceed with the upgrade? [Y/n] Y
 Barracuda [Sun Sep 15 09:19:19 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
 Barracuda [Sun Sep 15 09:19:21 BST 2013] ==> INFO: Updating apt sources
 Barracuda [Sun Sep 15 09:19:27 BST 2013] ==> INFO: Running aptitude update, please wait...
 Barracuda [Sun Sep 15 09:19:35 BST 2013] ==> INFO: Upgrading required libraries and tools
 Barracuda [Sun Sep 15 09:19:35 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
 Barracuda [Sun Sep 15 09:20:07 BST 2013] ==> INFO: Testing Nginx version...
 Barracuda [Sun Sep 15 09:20:09 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
 Barracuda [Sun Sep 15 09:20:11 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
 Barracuda [Sun Sep 15 09:20:14 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
 Barracuda [Sun Sep 15 09:20:14 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
 Barracuda [Sun Sep 15 09:21:05 BST 2013] ==> INFO: Testing Nginx version...
 Barracuda [Sun Sep 15 09:21:08 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
 Barracuda [Sun Sep 15 09:21:10 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
 Barracuda [Sun Sep 15 09:21:12 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
 Barracuda [Sun Sep 15 09:21:12 BST 2013] ==> INFO: Checking SMTP connections, please wait...
 Barracuda [Sun Sep 15 09:21:14 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
 Barracuda [Sun Sep 15 09:21:19 BST 2013] ==> INFO: Checking if PHP upgrade is available
 Barracuda [Sun Sep 15 09:21:21 BST 2013] ==> INFO: Installed PHP version 5.3.27-1~dotdeb.0, no upgrade required
 Barracuda [Sun Sep 15 09:21:26 BST 2013] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
 Barracuda [Sun Sep 15 09:21:29 BST 2013] ==> INFO: Installing New Relic Apps and Server Monitor, please wait...
 Barracuda [Sun Sep 15 09:22:26 BST 2013] ==> INFO: Upgrading Limited Shell to version 0.9.16.3-om8, please wait...
 Barracuda [Sun Sep 15 09:22:34 BST 2013] ==> INFO: Installed Redis version 2.6.14, upgrade/rebuild required
 Barracuda [Sun Sep 15 09:22:36 BST 2013] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
 Barracuda [Sun Sep 15 09:23:48 BST 2013] ==> INFO: Generating random password for Redis server
 Barracuda [Sun Sep 15 09:23:59 BST 2013] ==> INFO: Updating MariaDB and PHP configuration
 Barracuda [Sun Sep 15 09:24:01 BST 2013] ==> INFO: OS and services upgrade completed
  
 Barracuda [Sun Sep 15 09:24:03 BST 2013] ==> INFO: Restarting MariaDB server, please wait...
 Barracuda [Sun Sep 15 09:24:25 BST 2013] ==> INFO: Upgrading MariaDB tables if necessary, please wait a minute...
 Do you want to upgrade Aegir Master Instance? [Y/n] N
 Barracuda [Sun Sep 15 09:25:38 BST 2013] ==> INFO: Aegir Master Instance not upgraded this time
 Barracuda [Sun Sep 15 09:25:41 BST 2013] ==> INFO: Installing extra Drush versions
 Barracuda [Sun Sep 15 09:25:44 BST 2013] ==> INFO: Drush 4 installation complete
 Barracuda [Sun Sep 15 09:25:45 BST 2013] ==> INFO: Drush 5 installation complete
 Barracuda [Sun Sep 15 09:25:48 BST 2013] ==> INFO: Drush 6 installation complete
 Barracuda [Sun Sep 15 09:25:55 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
 Barracuda [Sun Sep 15 09:26:32 BST 2013] ==> INFO: Restarting MariaDB server
  
 Barracuda [Sun Sep 15 09:26:46 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
 Barracuda [Sun Sep 15 09:26:49 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
 Barracuda [Sun Sep 15 09:26:52 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
 Barracuda [Sun Sep 15 09:27:03 BST 2013] ==> INFO: csf/lfd firewall upgrade completed
  
 Barracuda [Sun Sep 15 09:27:05 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
 Barracuda [Sun Sep 15 09:27:11 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
  
 Barracuda [Sun Sep 15 09:27:15 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
 Barracuda [Sun Sep 15 09:27:27 BST 2013] ==> BYE!
 
 BARRACUDA upgrade completed
 Bye

Then these values were changed in /opt/local/etc/php53-fpm.conf:

pm.status_path = /status
ping.path = /ping

And the munin plugins were checked:

cd /etc/munin/plugins
munin-run phpfpm_connections
  accepted.value 4
munin-run phpfpm_status
  idle.value 17
  active.value 1
  total.value 18
munin-run nginx_status 
  total.value 4
  reading.value 0
  writing.value 1
  waiting.value 3
munin-run nginx_request
  request.value 272001

Accouding the the New Relic Drupal views monitoring page we need to restart:

/etc/init.d/newrelic-daemon restart
/etc/init.d/apache2 restart

So I did:

/etc/init.d/newrelic-daemon restart
/etc/init.d/nginx restart
/etc/init.d/php53-fpm restart

I had a bit of a look around in the web interface, it appears to be working apart from this error when I tried to load the data for www.transitionnetwork.org:

A script on this page may be busy, or it may have stopped responding. You can stop the script now, or you can continue to see if the script will complete.

Script: https://rpm.newrelic.com/maps/FusionCharts.js:49

Over to Jim now to get the data out and translate it into actions to make the site faster!

[chris]

New Relic DB Activity 2013-09-16

[chris]

New Relic Drupal Modules 2013-09-16

[chris]

New Relic Drupal Hooks 2013-09-16

[chris]

New Relic Drupal Views 2013-09-16

[chris]

I have just taken a look around the New Relic data and took some screen shots of data that we don't get from the Munin stats:

[jim]

Thanks Chris, I saw these last night and it's given me some good insight as to what to target next.

I'm going to let it carry on accumilating data for another day or so before I act, but when I do it'll be on:

• Context module's slowness
• Speeding up/cleaning up sessions and path_alias tables
• Revisiting key views caches and making use of Views Content Cache
• Other stuff I'm sure.

I'll do more on #590 on Wednesday then... Thanks, good data!

BTW, what browser are you using?, these screenshots look tortured! Perhaps an update is in order...!

[chris]

Replying to jim:

BTW, what browser are you using?, these screenshots look tortured! Perhaps an update is in order...!

No update needed I'm running tha latest Firefox 23.0.1.

The grey bar that is in the middle of the images is at the bottom of the browser window -- it's not getting moved down by the screenshop plugin, ​http://pearlcrescent.com/products/pagesaver/ -- is this what you were referring to?

[jim]

Aha, yes... makes sense!

I see the server is now stable again, so some of the doubts we had about performance/locking etc have been resolved too -- do you still think it's worth adding any other NR plugins you mentioned? TBH the Drupal/module/MySQL reporting is v handy for what I needed and Munin seems to cover the rest... Thoughts?

[chris]

Replying to jim:

I see the server is now stable again

Yes, it's not clear to me if this is because of the RAM/CPU that were replaced and/or if it is because several virtual servers were moved off it.

do you still think it's worth adding any other NR plugins you mentioned?

I don't know, they all have a lot of dependencies -- I'd rather not spend time sorting these out if they are not needed.

TBH the Drupal/module/MySQL reporting is v handy for what I needed and Munin seems to cover the rest... Thoughts?

My main thought is can we find some free tool(s) that produce data that corresponds to that in the screenshots above and then can we drop New Relic for cost / privacy / security reasons...?

[jim]

Well there's some profiling tools for PHP (e.g. xhprof‎), but they tend to absolutely destroy performance when they're running (2x memory, 3x slower). Then there's Devel in Drupal which I've already used to establish what queries were being called and if they were slow/duplicates etc.

It's one thing to get data, but another to store it somewhere and visualise it. NR does this very well, and hopefully we won't need if past the 2 weeks. Any alternative we ran ourselves should be similarly light-footed and cheap to implement/run.

Agreed, lets leave the plugins for now. I'll get back on this on Wednesday.

[jim]

Web transactions - page controllers stats

[jim]

Finally, ​attached is the CSV export from New Relic that shows the 'web transactions' for the page controllers (the code/callback responsible for generating a page) that are slowest.

With this and the past 3-4 posts on this ticket, I'm personally happy we have enough leads to do more work as needed and turn off New Relic for now.

A simple version of the attached file is:

Most time consuming
55.4% - /page_manager_node_view
17.4% - /drupal_get_form
8.66% - /views_page
7.51% - /taxonomy_term_page
3.66% - /page_manager_page_execute
2.09% - /index.php
1.71% - /user_page
1.51% - /advanced_forum_page
0.5% - /page_manager_search_view
0.44% - /page_manager_edit_operation

Slowest average response time
12.4 sec - /page_manager_edit_page_operation
7.31 sec - /block_admin_display
3320 ms - /page_manager_node_add
3180 ms - /page_manager_search_view
2600 ms - /page_manager_node_edit
1630 ms - /page_manager_page_execute
1470 ms - /page_manager_node_view
1460 ms - /content_profile_page_edit
1370 ms - /page_manager_user_view
1360 ms - /mailchimp_subscribe_page
1240 ms - /update.php
1230 ms - /taxonomy_term_page
1220 ms - /panels_admin_page
1220 ms - /upload_js
1210 ms - /drupal_get_form
1140 ms - /transition_pse_entry_form_steps
1110 ms - /og_views_users_faces
1080 ms - /page_manager_page_add_subtask
1060 ms - /system_admin_menu_block_page
964 ms - /filefield_js

Highest thoughput
9.96 rpm - /index.php
9.6 rpm - /page_manager_node_view
3.65 rpm - /drupal_get_form
2.35 rpm - /views_page
1.56 rpm - /taxonomy_term_page
0.81 rpm - /user_page
0.57 rpm - /page_manager_page_execute
0.52 rpm - /advanced_forum_page
0.05 rpm - /imagecache_cache
0.05 rpm - /notifications_anonymous_page_subscribe
0.04 rpm - /mailchimp_subscribe_page
0.04 rpm - /transition_pse_view_sidebar
0.04 rpm - /page_manager_search_view
0.02 rpm - /search_view
0.02 rpm - /page_manager_user_view
0.02 rpm - /node_feed
0.01 rpm - /page_manager_node_edit
0.01 rpm - /comment_reply
0.01 rpm - /user_edit
0.01 rpm - /page_manager_edit_page_operation

[jim]

I'm noting here the New Relic findings, based on 7 days of server running without (too much) interruption and reportedly good performance (based on charts, and me browsing, not other users):

Drupal

Drupal modules

Most time consuming
11300 sec - context
10400 sec - image_resize_filter
5920 sec - path_alias_cache
4740 sec - og
4170 sec - cck_private_fields
3550 sec - views_bulk_operations
2480 sec - content
2220 sec - views
2110 sec - notifications_content
1090 sec - insert
1010 sec - moopapi
1010 sec - php
912 sec - botcha
842 sec - node
689 sec - filefield
646 sec - date
627 sec - filter
590 sec - mollom
568 sec - nodehierarchy
533 sec - update

Slowest average call time
256 ms - panel_context
39.3 ms - context_contrib
31.5 ms - context
24.8 ms - image_resize_filter
10.5 ms - notifications_content
8.22 ms - colorbox
6.55 ms - views
6.44 ms - views_bulk_operations
6.39 ms - emvideo
5.37 ms - moopapi
5.33 ms - og
5.08 ms - botcha
4.69 ms - content
2.97 ms - update
2.96 ms - insert
2.8 ms - transition_events
2.01 ms - recaptcha
1.76 ms - menu_block
1.66 ms - transition_people
1.59 ms - php

Most function calls
31M - path_alias_cache
5.45M - node_content
3.87M - node
3.67M - cck_private_fields
3.48M - content_permissions
1.93M - features
1.57M - og_access
1.57M - content_access
1.57M - nodeaccess_userreference
1.49M - filter
1.29M - user
1.03M - og
973k - webform
949k - fivestar
927k - filefield
895k - ctools
816k - forum
772k - comment
772k - taxonomy
769k - calendar

Drupal hooks

Most time consuming
15900 sec - init
12400 sec - filter
10700 sec - context_page_condition
5850 sec - lookup_path
4800 sec - token_values
4230 sec - field_access
3000 sec - block
1870 sec - help
1150 sec - node_info
651 sec - elements
581 sec - link
572 sec - file_download
429 sec - captcha
380 sec - db_rewrite_sql
373 sec - forms
313 sec - search
228 sec - access
192 sec - footer
174 sec - emfield_rss
110 sec - form_all

Slowest average call time
1770 - file_delete
684 - context_conditions
297 - requirements
287 - panels_cache_get
172 - form_all
151 - search
61.3 - context_page_condition
41.5 - quicktabs_tabstyles
38.0 - context_reactions
21.8 - mailchimp_merge_values
20.3 - panels_cache_set
13.6 - mollom_form_info
12.2 - captcha
11.5 - insert_content
9.07 - block
7.39 - views_data
6.48 - emfield_rss
5.46 - file_references
4.91 - filter
4.79 - itweak_upload_preview

Most function calls
30.8 - lookup_path
13.8 - help
8.29 - init
7.66 - access
6.94 - field_access
5.87 - elements
4.7 - node_grants
4.63 - token_values
2.69 - for - 2.58 - link
2.58 - db_rewrite_sql
2.56 - filter
1.66 - node_info
1.43 - widget_info
1.08 - hook_info
1.02 - exit
569k - views_query_substitutions
501k - api_version
477k - cb_breadcrumb_info
367k - insert_widgets

[jim]

Continuing...

MySQL

Slowest average response time
1170 ms - content_field_logo - INSERT
1120 ms - node - INSERT
1010 ms - content_field_bios - INSERT
678 ms - url_alias - UPDATE
635 ms - term_node - DELETE
574 ms - content_type_profile - INSERT
546 ms - content_field_relatedprojects - INSERT
453 ms - content_field_user_types - INSERT
453 ms - content_field_image - INSERT
404 ms - cck_private_fields - DELETE
391 ms - node - UPDATE
377 ms - location_instance - INSERT
369 ms - location - INSERT
364 ms - content_field_roles_offered - INSERT
347 ms - content_field_initiative - INSERT
340 ms - url_alias - INSERT
283 ms - node_comment_statistics - INSERT
268 ms - content_field_editor_users - INSERT
255 ms - cache_page - INSERT
244 ms - content_field_type - INSERT

Most time consuming
19.2% - url_alias - SELECT
15.3% - node - SELECT
3.91% - users - SELECT
3.71% - system - SELECT
3.33% - files - SELECT
2.91% - menu_links - SELECT
2.48% - cck_private_fields - SELECT
1.9% - role - SELECT
1.89% - node_comment_statistics - SELECT
1.8% - term_node - SELECT
1.7% - nodehierarchy - SELECT
1.63% - menu_router - SELECT
1.53% - content_type_profile - SELECT
1.34% - node_access - SELECT
1.27% - feeds_node_item - SELECT
1.25% - vocabulary - SELECT
1.24% - captcha_sessions - UPDATE
1.22% - location_instance - SELECT
1.16% - other - SQL
1.09% - access - SELECT

Highest thoughput
974cpm - url_alias - SELECT
226 cpm - node - SELECT
87 cpm - system - SELECT
86.6 cpm - cck_private_fields - SELECT
85.6 cpm - role - SELECT
81.8 cpm - menu_links - SELECT
71.6 cpm - users - SELECT
66.3 cpm - other - SQL
53.2 cpm - files - SELECT
52.2 cpm - node_comment_statistics - SELECT
49.6 cpm - term_node - SELECT
47.8 cpm - nodehierarchy - SELECT
47.7 cpm - location_instance - SELECT
47.7 cpm - feeds_node_item - SELECT
40.8 cpm - filter_formats - SELECT
35.9 cpm - menu_router - SELECT
33.6 cpm - botcha_form - SELECT
32.8 cpm - captcha_points - SELECT
31.8 cpm - location - SELECT
31.6 cpm - content_group_fields - SELECT

Slowest queries by Web Transaction
417ms - term_data - SELECT (/tagadelic_page_chunk)
220 ms - menu_router - SELECT (/update_status)
161 ms - term_node - SELECT (/site_map_page)
135 ms - content_type_profile - SELECT (/page_manager_node_edit)
114 ms - locales_source - SELECT (/og_subscribe)
83.3 ms - (subquery) - SELECT (/page_manager_search_view)
79.4 ms - search_index - SELECT (/page_manager_search_view)
63.5 ms - (subquery) - SELECT (/advanced_forum_page)
45.9 ms - content_field_featured_image - SELECT (/content_profile_page_edit)
29.5 ms - cache_form - SELECT (/notifications_page_unsubscribe)
19.2 ms - imagecache_preset - SELECT (/advanced_forum_page)
18.9 ms - blocks - SELECT (/update_status)
13.5 ms - page_manager_weights - SELECT (/page_manager_user_view)
13.4 ms - cache_update - SELECT (/system_status)
12.0 ms - trigger_assignments - SELECT (/logintoboggan_validate_email)
11.9 ms - blocks - SELECT (/panels_admin_page)
11.4 ms - (subquery) - SELECT (/views_page)
11.3 ms - variable - SELECT (/index.php)
11.2 ms - node - SELECT (/ctools_content_autocomplete_node)
11.1 ms - variable - SELECT (/drupal_get_form)

[jim]

External Services

Slowest average response time
17.1 sec - images.bookworld.com.au -
4.61 sec - www.the-south-asian.com
4.0 sec - www.jamessamuel.co.nz
3930 ms - www.biglotteryfund.org.uk
3460 ms - www.transitiontowns.org.nz
2590 ms - www.llnsllc.com
2570 ms - www.transitionscotland.org
2180 ms - thelittlegymoftexas.com
2040 ms - www.thelivingcentre.com
1970 ms - www.pbs.org
1930 ms - jamessamuel.co.nz
1900 ms - www.thebiglunch.com
1890 ms - www.facebook.com
1840 ms - nu-project.org
1790 ms - fbcdn-profile-a.akamaihd.net
1770 ms - www.akpress.org
1630 ms - www.communitychef.org.uk
1570 ms - valeharvest.weebly.com
1510 ms - transitionitalia.files.wordpress.com
1510 ms - www.healthyfeetblog.com

Most time consuming
24.9% - news.transitionnetwork.org
15.2% - transitionculture.org
11.9% - transitionvoice.com
6.5% - www.resilience.org
3.09% - updates.drupal.org
2.85% - www.stopforumspam.com
2.6% - farm5.static.flickr.com
2.39% - transitionus.org
2.31% - 3.bp.blogspot.com
2.13% - 1.bp.blogspot.com
1.78% - markinflowers.files.wordpress.com
1.63% - 2.bp.blogspot.com
1.63% - 4.bp.blogspot.com
1.3% - images.bookworld.com.au
1.27% - feeds.feedburner.com
1.24% - weldingweb.com
0.72% - farm2.static.flickr.com
0.71% - www.transitiontowns.org.nz
0.65% - www.transitionnetwork.org
0.54% - feedads.g.doubleclick.net

Highest thoughput
0.92 cpm - news.transitionnetwork.org
0.89 cpm - transitionculture.org
0.69 cpm - www.stopforumspam.com
0.62 cpm - feeds.feedburner.com
0.37 cpm - www.transitionnetwork.org
0.21 cpm - feedads.g.doubleclick.net
0.2 cpm - transitionvoice.com
0.14 cpm - www.resilience.org
0.08 cpm - updates.drupal.org
0.07 cpm - 3.bp.blogspot.com
0.06 cpm - gdata.youtube.com
0.06 cpm - 4.bp.blogspot.com
0.06 cpm - youtube.com
0.06 cpm - www.youtube.com
0.06 cpm - youtube.googleapis.com
0.06 cpm - 1.bp.blogspot.com
0.06 cpm - transitiontowns.org
0.05 cpm - 2.bp.blogspot.com
0.04 cpm - transitionus.org
0.04 cpm - markinflowers.files.wordpress.com

[jim]

Some less relevant but still interesting stats...

Geography

Geography

Average page load time

10.3 sec - India
7.96 sec - New Zealand
6.89 sec - Australia
6.16 sec - Canada
5.39 sec - United States
5.18 sec - Italy
5.12 sec - Spain
5.12 sec - France
4.93 sec - Portugal
4.3 sec - Germany
3820 ms - United Kingdom
3140 ms - Belgium
3110 ms - Netherlands
1820 ms - Switzerland

'Apdex' most dissatisfying (% of total dissatisfied
23% - United Kingdom
21.5% - United States
8.16% - Australia
6.94% - Canada
4.11% - India
2.97% - France
2.74% - New Zealand
2.33% - Germany
2.23% - Spain
2.03% - Italy
1.74% - Portugal
1.41% - Philippines
1.27% - Mexico
1.13% - Brazil
1.08% - China
1% - Argentina
0.9% - Hong Kong
0.82% - Ireland
0.74% - Pakistan
0.72% - Malaysia

Countries by average sessions (logins per visit)
4.63 - United Kingdom
3.47 - Panama
2.59 - United States
2.3 - Luxembourg
2 - Zambia
2 - Cyprus
1.74 - Germany
1.72 - Netherlands
1.7 - Norway
1.69 - Canada
1.66 - Australia
1.58 - New Zealand
1.57 - France
1.53 - Brazil
1.52 - Senegal
1.5 - Slovenia
1.5 - Ireland
1.47 - Kuwait
1.47 - Sweden
1.46 - Korea - Republic of

[chris]

Uninstalling New Relic, these are the currently installed packages:

aptitude search relic
 i A newrelic-daemon                 - The New Relic proxy daemon                
 i   newrelic-php5                   - The New Relic agent for PHP               
 i A newrelic-php5-common            - Scripts and samples for the New Relic PHP 
 i   newrelic-sysmond                - The New Relic agent for server monitoring

The /root/.barracuda.cnf file was edited and the key commented out:

_NEWRELIC_KEY=

Updating BOA:

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
 BOA Meta Installer setup completed
 Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
 Bye
barracuda up-stable
 
Barracuda [Mon Sep 30 01:57:57 BST 2013] ==> BOA Skynet welcomes you aboard!
 
Barracuda [Mon Sep 30 01:58:01 BST 2013] ==> INFO: UPGRADE
Barracuda [Mon Sep 30 01:58:01 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Mon Sep 30 01:58:02 BST 2013] ==> NOTE! Please review all config options displayed below
Barracuda [Mon Sep 30 01:58:02 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script
 
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
#_MY_OWNIP=""
_MY_HOSTN="puffin.webarch.net"
#_MY_HOSTN=""
_MY_FRONT="master.puffin.webarch.net"
_THIS_DB_HOST=localhost
#_THIS_DB_HOST=FQDN
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV"
_AUTOPILOT=NO
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
#_LOAD_LIMIT_ONE=1444
#_LOAD_LIMIT_TWO=888
_LOAD_LIMIT_ONE==7220
_LOAD_LIMIT_TWO=4440
_CUSTOM_CONFIG_CSF=NO
#_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_SQL=YES
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
#_CUSTOM_CONFIG_PHP_5_3=NO
_CUSTOM_CONFIG_PHP_5_3=YES
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### JK reinstall PHP
_EXTRA_PACKAGES=
_PHP_EXTRA_CONF=""
_STRONG_PASSWORDS=NO
_DB_BINARY_LOG=NO
_DB_ENGINE=InnoDB
_NGINX_LDAP=NO
_PHP_GEOS=NO
_PHP_MONGODB=NO
_AEGIR_UPGRADE_ONLY=NO
 
Do you want to proceed with the upgrade? [Y/n] Y
Barracuda [Mon Sep 30 01:58:45 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Mon Sep 30 01:58:46 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Mon Sep 30 01:58:46 BST 2013] ==> INFO: Downloading little helpers, please wait...
Barracuda [Mon Sep 30 01:58:47 BST 2013] ==> INFO: Checking BARRACUDA version...
Barracuda [Mon Sep 30 01:58:47 BST 2013] ==> INFO: Version test result: OK
Barracuda [Mon Sep 30 01:58:47 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...
 
Barracuda [Mon Sep 30 01:58:50 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9
 
 
Barracuda [Mon Sep 30 01:58:54 BST 2013] ==> UPGRADE START -> checkpoint: 

  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your server hostname is puffin.webarch.net.
  * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.

 
Do you want to proceed with the upgrade? [Y/n] Y
Barracuda [Mon Sep 30 01:59:03 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Mon Sep 30 01:59:06 BST 2013] ==> INFO: Updating apt sources
Barracuda [Mon Sep 30 01:59:08 BST 2013] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Mon Sep 30 01:59:12 BST 2013] ==> INFO: Running aptitude update, please wait...
Barracuda [Mon Sep 30 01:59:19 BST 2013] ==> INFO: Upgrading required libraries and tools
Barracuda [Mon Sep 30 01:59:19 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Mon Sep 30 01:59:50 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Mon Sep 30 01:59:52 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Mon Sep 30 01:59:54 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Mon Sep 30 01:59:56 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Mon Sep 30 01:59:56 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...

Barracuda [Mon Sep 30 02:00:47 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Mon Sep 30 02:00:49 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Mon Sep 30 02:00:51 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Mon Sep 30 02:00:53 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Mon Sep 30 02:00:53 BST 2013] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Mon Sep 30 02:00:55 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
Barracuda [Mon Sep 30 02:00:59 BST 2013] ==> INFO: Checking if PHP upgrade is available
Barracuda [Mon Sep 30 02:01:01 BST 2013] ==> INFO: Installed PHP version 5.3.27-1~dotdeb.0, no upgrade required
Barracuda [Mon Sep 30 02:01:05 BST 2013] ==> INFO: Installed Redis version 2.6.16, no upgrade/rebuild required
Barracuda [Mon Sep 30 02:01:18 BST 2013] ==> INFO: Updating MariaDB and PHP configuration
Barracuda [Mon Sep 30 02:01:19 BST 2013] ==> INFO: OS and services upgrade completed
 
Barracuda [Mon Sep 30 02:01:21 BST 2013] ==> INFO: Restarting MariaDB server, please wait...
Barracuda [Mon Sep 30 02:01:39 BST 2013] ==> INFO: Upgrading MariaDB tables if necessary, please wait a minute...

Do you want to upgrade Aegir Master Instance? [Y/n]
Barracuda [Mon Sep 30 02:02:42 BST 2013] ==> INFO: Running Aegir Master Instance upgrade
Barracuda [Mon Sep 30 02:02:45 BST 2013] ==> INFO: Syncing provision backend db_passwd, please wait...
Barracuda [Mon Sep 30 02:02:49 BST 2013] ==> INFO: Running hosting-dispatch (1/3), please wait...
Barracuda [Mon Sep 30 02:02:57 BST 2013] ==> INFO: Running hosting-dispatch (2/3), please wait...
Barracuda [Mon Sep 30 02:03:04 BST 2013] ==> INFO: Running hosting-dispatch (3/3), please wait...
Barracuda [Mon Sep 30 02:03:04 BST 2013] ==> INFO: Syncing hostmaster frontend db_passwd, please wait...
Barracuda [Mon Sep 30 02:03:06 BST 2013] ==> INFO: Testing previous install...
Barracuda [Mon Sep 30 02:03:06 BST 2013] ==> INFO: Test OK, we can proceed with Hostmaster upgrade
Barracuda [Mon Sep 30 02:03:06 BST 2013] ==> INFO: Moving old directories
Barracuda [Mon Sep 30 02:03:07 BST 2013] ==> INFO: Downloading drush, please wait...
Barracuda [Mon Sep 30 02:03:08 BST 2013] ==> INFO: Drush seems to be functioning properly
Barracuda [Mon Sep 30 02:03:09 BST 2013] ==> INFO: Installing provision backend in /var/aegir/.drush
Barracuda [Mon Sep 30 02:03:09 BST 2013] ==> INFO: Downloading Drush and Provision extensions, please wait...
Barracuda [Mon Sep 30 02:03:09 BST 2013] ==> INFO: Running hostmaster-migrate, please wait...
Barracuda [Mon Sep 30 02:04:12 BST 2013] ==> INFO: Syncing hostmaster frontend db_passwd, please wait...
Barracuda [Mon Sep 30 02:04:15 BST 2013] ==> INFO: Generating random password for Redis server
Barracuda [Mon Sep 30 02:04:45 BST 2013] ==> INFO: Aegir Master Instance upgrade completed
Barracuda [Mon Sep 30 02:04:46 BST 2013] ==> INFO: Installing extra Drush versions
Barracuda [Mon Sep 30 02:04:49 BST 2013] ==> INFO: Drush 4 installation complete
Barracuda [Mon Sep 30 02:04:50 BST 2013] ==> INFO: Drush 5 installation complete
Barracuda [Mon Sep 30 02:04:52 BST 2013] ==> INFO: Drush 6 installation complete
Barracuda [Mon Sep 30 02:04:56 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
Barracuda [Mon Sep 30 02:05:41 BST 2013] ==> INFO: Restarting MariaDB server
 
Barracuda [Mon Sep 30 02:05:52 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
Barracuda [Mon Sep 30 02:05:54 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Mon Sep 30 02:06:00 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
Barracuda [Mon Sep 30 02:06:06 BST 2013] ==> INFO: csf/lfd firewall upgrade completed
 
Barracuda [Mon Sep 30 02:06:08 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
Barracuda [Mon Sep 30 02:06:14 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
 
Barracuda [Mon Sep 30 02:06:18 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Mon Sep 30 02:09:44 BST 2013] ==> BYE!

BARRACUDA upgrade completed
Bye

The New Relic packages are still installed:

aptitude search relic
i A newrelic-daemon      - The New Relic proxy daemon
i   newrelic-php5        - The New Relic agent for PHP
i A newrelic-php5-common - Scripts and samples for the New Relic PHP agent
i   newrelic-sysmond     - The New Relic agent for server monitoring

Removing them:

aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
The following packages will be REMOVED:  
  newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond 
0 packages upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 41.5 MB will be freed.
(Reading database ... 49988 files and directories currently installed.)
Removing newrelic-php5 ...
Removing newrelic-daemon ...
Removing newrelic-php5-common ...
Removing newrelic-sysmond ...
Stopping New Relic Server Monitor: newrelic-sysmond OK

Note the server time is off, see ticket:599

[chris]

The barracuda upgrade clobbered some tweaks that are needed to generate Munin stats:

• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/nginx_request.html
• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/nginx_status.html
• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/phpfpm_connections.html

Checking the wiki:PuffinServer#UpgradingBOA notes, the following were edited:

• /var/aegir/config/server_master/nginx.conf
• /opt/local/etc/php53-fpm.conf

And nginx and php-fpm restarted.

[jim]

It would be better if you didn't use /var/aegir/config/server_master/nginx.conf which is for Aegir, and created a new config file in /var/aegir/config/}}} somewhere... There are folders in there that include extra conf files.

[chris]

I think this is a coincidence, but it's worth nothing that while New Relic was running the load spikes dropped off and now they are back, see ticket:555#comment:93

Thanks for the suggestion Jim, I tried creating /var/aegir/config/includes/nginx_force_include_munin containing:

#######################################################
###  nginx default server munin
#######################################################

server {
  location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    allow 81.95.52.103;
    deny all;
  }
  location ~ ^/(status|ping)$ {
    fastcgi_pass 127.0.0.1:9090;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    include fastcgi_params;
    access_log off;
    allow 127.0.0.1;
    deny all;
  }
}

And I tried removing the code from /var/aegir/config/server_master/nginx.conf but when I tried to restart nginx I got:

Configuration test failed!
nginx: [emerg] "server" directive is not allowed here in /var/aegir/config/includes/nginx_force_include_munin:5
nginx: configuration file /etc/nginx/nginx.conf test failed

So I reverted the changed -- Jim if you can find a place where these changes can be made in a way that they don't get clobbered by BOA that would be great -- I haven't found a way to do this.

[jim]

OK, so I just ran Drush for a simple task and got this error:

tn@puffin:~/static/transition-network-d6-p005$ drush www.transitionnetwork.org en views_ui
PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0
The following extensions will be enabled: views_ui
Do you really want to continue? (y/n): n

So I wonder if the *LACK* of newrelic includes/modules is the issue for load spikes -- NR is not properly uninstalled in that PHP config is still referring to it.

Looking at config now, but if you have any thoughts Chris...

[jim]

It's in /opt/local/etc/php53.ini, still referrs to NR at end of file...

; Enable Extensions
extension=uploadprogress.so
extension=imagick.so
extension=jsmin.so
extension=redis.so
;extension=memcached.so
extension=newrelic.so
[newrelic]
newrelic.enabled=true
newrelic.license = "[snip]"
newrelic.logfile = "/var/log/newrelic/php_agent.log"
newrelic.loglevel = "error"
newrelic.appname = "Drush Tasks and Sites Cron"
newrelic.daemon.logfile = "/var/log/newrelic/newrelic-daemon.log"
newrelic.daemon.loglevel = "error"
newrelic.daemon.pidfile = "/var/run/newrelic-daemon.pid"
newrelic.daemon.dont_launch = 3
newrelic.framework.drupal.modules = 1

Am commenting out...

[chris]

Thanks Jim, I did notice that the daemon was still running in the top dump on ticket:555#comment:96 and was going to get to that...

41997 root      20   0  3956  608  492 S    0  0.0   0:00.00 newrelic-daemon

And it's still running:

 ps -lA | grep relic
4 S     0  8575  8466  3  80   0 -   989 -      ?        00:00:03 newrelic-daemon
1 S     0  8640  8575  0  80   0 -   989 -      ?        00:00:00 newrelic-daemon

So I have now killed it:

killall -9 newrelic-daemon

Thanks for finding the config in /opt/local/etc/php53.ini -- I had assumed that commenting the _NEWRELIC_KEY in /root/.barracuda.cnf and running barracuda up-stable would remove any New Relic config that had been added, sorry.

[jim]

Note this is commented out (as of about 14:30), and also in /usr/local/lib/php.ini is the file that Drush uses (I think).

I also noted that service php5-fpm was running. I killed this since when I restarted it (thinking it was what Drush used) it gave a few 502s. php53-fpm has been restarted and all looks ok..

However, the drush warning about missing /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so is still occurring, so I'm still confused.

According to this search in GitHub?, new relic is only mentioned in afew places: ​https://github.com/omega8cc/nginx-for-drupal/search?q=newrelic&ref=cmdform

Parking for now, going over to #602.

[jim]

Adding time and replying to chris:

So I have now killed it: killall -9 newrelic-daemon

Thanks for finding the config in /opt/local/etc/php53.ini -- I had assumed that commenting the _NEWRELIC_KEY in /root/.barracuda.cnf and running barracuda up-stable would remove any New Relic config that had been added, sorry.

No, I've had this in the past... It seems BOA is great at installing things and integrating them. but not really taking them back out again!

Drush still complains though, at a loss to explain that.

[jim]

In ​line 294 onwards of BOND.sh it lists where it puts the newrelic.ini include in to the php config files. They are:

• /opt/etc/php.ini
• /usr/local/lib/php.ini [DONE JK]
• /opt/local/etc/php53.ini [DONE JK]
• /opt/local/lib/php.ini

So I'll comment out the other ones now too...

[jim]

Done, and Drush is now complaint-free... Suggest we wait an hour on the offchance my and Chris' recent tweaks have any effect on the load spikes.

[chris]

Urm jim you made the New Relic license key public -- does this have security / privacy implications?

Reenabling New Relic, starting at Thu Oct 3 21:03:25 BST 2013, first the _NEWRELIC_KEY variable in /root/.barracuda.cnf was added back then:

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
 BOA Meta Installer setup completed
 Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
 Bye
barracuda up-stable
 
Barracuda [Thu Oct  3 21:06:18 BST 2013] ==> BOA Skynet welcomes you aboard!
 
Barracuda [Thu Oct  3 21:06:22 BST 2013] ==> INFO: UPGRADE
Barracuda [Thu Oct  3 21:06:22 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Thu Oct  3 21:06:23 BST 2013] ==> NOTE! Please review all config options displayed below
Barracuda [Thu Oct  3 21:06:23 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script
 
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
#_MY_OWNIP=""
_MY_HOSTN="puffin.webarch.net"
#_MY_HOSTN=""
_MY_FRONT="master.puffin.webarch.net"
_THIS_DB_HOST=localhost
#_THIS_DB_HOST=FQDN
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV"
_AUTOPILOT=NO
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
#_LOAD_LIMIT_ONE=1444
#_LOAD_LIMIT_TWO=888
_LOAD_LIMIT_ONE==7220
_LOAD_LIMIT_TWO=4440
_CUSTOM_CONFIG_CSF=NO
#_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_SQL=YES
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
#_CUSTOM_CONFIG_PHP_5_3=NO
_CUSTOM_CONFIG_PHP_5_3=YES
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### JK reinstall PHP
_EXTRA_PACKAGES=
_PHP_EXTRA_CONF=""
_STRONG_PASSWORDS=NO
_DB_BINARY_LOG=NO
_DB_ENGINE=InnoDB
_NGINX_LDAP=NO
_PHP_GEOS=NO
_PHP_MONGODB=NO
_AEGIR_UPGRADE_ONLY=NO
 
Do you want to proceed with the upgrade? [Y/n] Y

Barracuda [Thu Oct  3 21:06:56 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Thu Oct  3 21:06:58 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Thu Oct  3 21:06:58 BST 2013] ==> INFO: Downloading little helpers, please wait...
Barracuda [Thu Oct  3 21:06:59 BST 2013] ==> INFO: Checking BARRACUDA version...
Barracuda [Thu Oct  3 21:06:59 BST 2013] ==> INFO: Version test result: OK
Barracuda [Thu Oct  3 21:06:59 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...
 
Barracuda [Thu Oct  3 21:07:01 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9
 
 
Barracuda [Thu Oct  3 21:07:05 BST 2013] ==> UPGRADE START -> checkpoint: 

  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your server hostname is puffin.webarch.net.
  * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.

 
Do you want to proceed with the upgrade? [Y/n] Y

Barracuda [Thu Oct  3 21:07:19 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Thu Oct  3 21:07:21 BST 2013] ==> INFO: Updating apt sources
Barracuda [Thu Oct  3 21:07:23 BST 2013] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Thu Oct  3 21:07:27 BST 2013] ==> INFO: Running aptitude update, please wait...
Barracuda [Thu Oct  3 21:07:34 BST 2013] ==> INFO: Upgrading required libraries and tools
Barracuda [Thu Oct  3 21:07:34 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Thu Oct  3 21:08:05 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Thu Oct  3 21:08:07 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Thu Oct  3 21:08:09 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Thu Oct  3 21:08:11 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Thu Oct  3 21:08:11 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Thu Oct  3 21:09:03 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Thu Oct  3 21:09:06 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Thu Oct  3 21:09:08 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Thu Oct  3 21:09:10 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Thu Oct  3 21:09:10 BST 2013] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Thu Oct  3 21:09:12 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
Barracuda [Thu Oct  3 21:09:15 BST 2013] ==> INFO: Checking if PHP upgrade is available
Barracuda [Thu Oct  3 21:09:18 BST 2013] ==> INFO: Installed PHP version 5.3.27-1~dotdeb.0, no upgrade required
Barracuda [Thu Oct  3 21:09:18 BST 2013] ==> INFO: Uninstalling previous version of New Relic Apps and Server Monitor, please wait...
Barracuda [Thu Oct  3 21:09:24 BST 2013] ==> INFO: Installing latest version of New Relic Apps and Server Monitor, please wait...
Barracuda [Thu Oct  3 21:10:21 BST 2013] ==> INFO: Installed Redis version 2.6.16, no upgrade/rebuild required
Barracuda [Thu Oct  3 21:10:33 BST 2013] ==> INFO: Updating MariaDB and PHP configuration
Barracuda [Thu Oct  3 21:10:34 BST 2013] ==> INFO: OS and services upgrade completed
 
Barracuda [Thu Oct  3 21:10:36 BST 2013] ==> INFO: Restarting MariaDB server, please wait...
Barracuda [Thu Oct  3 21:10:57 BST 2013] ==> INFO: Upgrading MariaDB tables if necessary, please wait a minute...
Do you want to upgrade Aegir Master Instance? [Y/n] N
Barracuda [Thu Oct  3 21:15:28 BST 2013] ==> INFO: Aegir Master Instance not upgraded this time
Barracuda [Thu Oct  3 21:15:39 BST 2013] ==> INFO: Installing extra Drush versions
Barracuda [Thu Oct  3 21:15:41 BST 2013] ==> INFO: Drush 4 installation complete
Barracuda [Thu Oct  3 21:15:43 BST 2013] ==> INFO: Drush 5 installation complete
Barracuda [Thu Oct  3 21:15:45 BST 2013] ==> INFO: Drush 6 installation complete
Barracuda [Thu Oct  3 21:15:49 BST 2013] ==> INFO: Generating random password for Redis server
Barracuda [Thu Oct  3 21:15:50 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
Barracuda [Thu Oct  3 21:16:31 BST 2013] ==> INFO: Restarting MariaDB server
 
Barracuda [Thu Oct  3 21:16:45 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
Barracuda [Thu Oct  3 21:16:48 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Thu Oct  3 21:17:02 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
Barracuda [Thu Oct  3 21:17:07 BST 2013] ==> INFO: csf/lfd firewall upgrade completed
 
Barracuda [Thu Oct  3 21:17:09 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
Barracuda [Thu Oct  3 21:17:16 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
 
Barracuda [Thu Oct  3 21:17:20 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Thu Oct  3 21:17:16 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
 
Barracuda [Thu Oct  3 21:17:20 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Thu Oct  3 21:20:34 BST 2013] ==> BYE!

BARRACUDA upgrade completed
Bye

Check that New Relic is running:

ps -lA | grep relic
1 S     0 50464     1  0  80   0 -  3935 -      ?        00:00:00 newrelic-daemon
1 S     0 50466 50464  0  80   0 - 26507 -      ?        00:00:00 newrelic-daemon

Apply the edits for Munin:

• wiki:PuffinServer#nginxconfigchanges these edits were not needed this time
• wiki:PuffinServer#php-fpmconfigchanges these edits were done and munin tested

[chris]

When I try to login to New Relic I get:

We're sorry, you haven't verified your email address yet. If you've misplaced the verification email we sent, contact support for assistance.

As far as I'm aware no verification email has been sent, I have sent them a email about this and CC'd it to the ttech list.

[chris]

Jim - have you managed to login to New Relic since it was added back? I still haven't.

I was going to remove it again tonight but I think it's worth taking one last look, especially since it was added back at 8:30pm on 3rd Oct and the load spikes, once again, dramatically reduced:

Of course it was the weekend and the load has picked up since and it does correspond with the Piwik stats -- I fiund it hard to believe there is a direct correlation:

[chris]

I still can't login to New Relic, I requested a password reset but didn't get a email from them.

But the week extension is almost due to run out so, I'll remove it tonight.

Also the load spikes are back, see ticket:555#comment:100

[jim]

I've hadn't tried until a few minutes ago... I'm in though. I sent an email to newrelic@… a few days ago -- was that received? I'm wondering if that account is actually working or not.

I've added my gmail account as an admin to avoid being locked out -- I can add yours too Chris if you'd like.

---

However, there is NOTHING being reported to NR, so something is up...

So I checked and there services are running... looking at the php.ini files from comment:27, the NR settings were still commented out in /opt/local/etc/php53.ini and /opt/local/lib/php.ini.

I've uncommented them and restarted php53-fpm. Now NR is getting data.

[jim]

Note NR has reverted to the free tier, so Drupal and database detail is now unavailable.

[jim]

I've now removed NR, first I removed (not commented) the PHP.inis, then I uninstalled it:

puffin:~# nano /opt/etc/php.ini
puffin:~# nano /usr/local/lib/php.ini
puffin:~# nano  /opt/local/etc/php53.ini
puffin:~# nano /opt/local/lib/php.ini
puffin:~# service php53-fpm reload
Reload service php53-fpm.. done
puffin:~# aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
The following packages will be REMOVED:  
  newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond 
0 packages upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 50.1 MB will be freed.
(Reading database ... 49990 files and directories currently installed.)
Removing newrelic-php5 ...
Removing newrelic-daemon ...
Removing newrelic-php5-common ...
Removing newrelic-sysmond ...
Stopping New Relic Server Monitor: newrelic-sysmond OKpuffin:~# totn
tn@puffin:~$ drush status
 PHP configuration     :  /opt/local/lib/php.ini                                                               
 Drush version         :  4.6-dev                                                                              
 Drush configuration   :                                                                                       
 Drush alias files     :  /data/disk/tn/.drush/server_master.alias.drushrc.php                                 
                          /data/disk/tn/.drush/space.transitionnetwork.org.alias.drushrc.php                   
                          /data/disk/tn/.drush/platform_TransitionNetworkD611.alias.drushrc.php                
                          /data/disk/tn/.drush/platform_TransitionNetworkD6S002.alias.drushrc.php              
                          /data/disk/tn/.drush/platform_TransitionNetworkD6P005.alias.drushrc.php              
                          /data/disk/tn/.drush/platform_TransitionNetworkD6S006.alias.drushrc.php              
                          /data/disk/tn/.drush/news.transitionnetwork.org.alias.drushrc.php                    
                          /data/disk/tn/.drush/platform_OpenAtrium7x20alpha5.alias.drushrc.php                 
                          /data/disk/tn/.drush/hostmaster.alias.drushrc.php                                    
                          /data/disk/tn/.drush/stg.transitionnetwork.org.alias.drushrc.php                     
                          /data/disk/tn/.drush/platform_OpenAtriumOpenAtrium7x2xDEV13Oct2013.alias.drushrc.php 
                          /data/disk/tn/.drush/platform_005.alias.drushrc.php                                  
                          /data/disk/tn/.drush/platform_OpenAtrium2xAlpha5devJun18.alias.drushrc.php           
                          /data/disk/tn/.drush/server_localhost.alias.drushrc.php                              
                          /data/disk/tn/.drush/platform_TransitionNetworkD6S005.alias.drushrc.php              
                          /data/disk/tn/.drush/www.transitionnetwork.org.alias.drushrc.php                     
                          /data/disk/tn/.drush/platform_OpenAtrium2xGit.alias.drushrc.php   

The last command demonstrates no warning about a missing .so file.

I have also commented out the .barracuda.cnf NR key, plus uncommented the empty NR key line.

I reckon this is gone... Did I miss anything Chris?

[chris]

New Relic has been uninstalled, a link to this ticket has been added here wiki:PuffinServer#MuninStats now closing this ticket.

[jim]

Drush plagued by this:

 >> PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

So on further investigation NR was installed still and appeared to be running...

I've now:

• repeated the php.ini file changes per comment:27
• moved /etc/newrelic to /root/scratch just in case
• removed them aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
• purged them aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond

I'm re-opening this ticket as 'minor' so we can track this -- it seems to be coming back for some reason, perhaps the recent Debian or BOA updates reinstated it.

Aware there's no time this month so stopping here, but we need to track this down and kill it. Parking until next BOA update.

[chris]

Replying to jim:

Drush plagued by this:

 >> PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

... we need to track this down and kill it. Parking until next BOA update.

Have the drush errors stopped? If they have this ticket can be closed.

[jim]

Confirmed errors not happening since my fix, closing.