Transition Technology: Ticket #586: New Relic Monitoring for BOA

chris — Thu, 05 Sep 2013 08:40:15 GMT

We currently have Munin stats:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/

And Piwik and Webalizer stats:

And the output of the mysql scripts, tuning-primer.sh and mysqltuner.pl as well as the output from things like top etc.

hours, totalhours changed

chris — Thu, 05 Sep 2013 09:40:34 GMT

hours changed from 0.0 to 0.75
totalhours changed from 0.0 to 0.75

Installation Implications

These are not insignificant, there are lots of required dependencies, it looks like a few hours work.

PHP

See https://newrelic.com/install

Add their GPG key, add their repo and install newrelic-php5 and then run newrelic-install install:

sudo -i
wget -O - http://download.newrelic.com/548C16BF.gpg | apt-key add -
echo "deb http://apt.newrelic.com/debian/ newrelic non-free" > /etc/apt/sources.list.d/newrelic.list
apt-get update
apt-get install newrelic-php5
newrelic-install install

Note: The New Relic installer will try to drop a newrelic.ini file in place. Some PHP installations are hard for our installer to find, or will not accept files. In these cases you will need to manually edit your php.ini file.

Nginx

See https://newrelic.com/plugins/nginx-inc/13

To use this plugin, you must have the following:

Ruby (>= 1.8.7)
bundler for Ruby (https://github.com/carlhuda/bundler)

The server doesn't have ruby installed, the squeeze ruby version is 1.8.7, see http://packages.debian.org/squeeze/ruby1.8

More info on installing Bundler at http://bundler.io/

Plugin install instructions:

Download the NGINX web server plugin .tar.gz from Plugin Central.
Unpack the .tar.gz file in a new directory.
Run bundle install to install all needed prerequisites.
Copy config/newrelic_plugin.yml.in to config/newrelic_plugin.yml.
Insert your New Relic license key and NGINX status URL into config/newrelic_plugin.yml.
Run ./newrelic_nginx_agent.daemon start to start the agent in daemon mode.
More in-depth installation and configuration details are included in the README.txt file packaged with the plugin.

PHP-FPM

See https://newrelic.com/plugins/meetme-inc/123

Install via the Python Package Index or click the Download button below, then follow the directions in the README at https://github.com/MeetMe/newrelic-plugin-agent#installation-instructions

From there:

The agent requires Python 2.6 or 2.7 and pip for installation.

The server has python 2.6 installed.

The pip install instructions are here:

http://www.pip-installer.org/en/latest/installing.html

MySQL

See https://newrelic.com/plugins/new-relic-inc/52

To use this plugin, you must have:

a Java Runtime Environment (JRE) of 1.6 or higher

The server currently has these jre packages installed, I don't know if this is enough:

aptitude search jre | grep ^i
i A gcj-4.4-jre-lib                 - Java runtime library for use with gcj (jar
i A icedtea-6-jre-cacao             - Alternative JVM for OpenJDK, using Cacao
i A openjdk-6-jre                   - OpenJDK Java runtime, using Hotspot JIT
i A openjdk-6-jre-headless          - OpenJDK Java runtime, using Hotspot JIT (h
i A openjdk-6-jre-lib               - OpenJDK Java runtime (architecture indepen

Installation information

Download the MySQL plugin .tar.gz file from Plugin Central
Unpack the .tar.gz file in a new directory (typically tar zxvf newrelic_mysql_plugin-X.X.X.tar.gz)
Copy config/template_newrelic.properties to config/newrelic.properties
Open the config/newrelic.properties file and enter your license key (found in "Account Settings" menu at https://rpm.newrelic.com)
Copy config/template_mysql.instance.json to config/mysql.instance.json
Configure config/mysql.instance.json for each MySQL instance you want to monitor
Run java -jar <the plugin jar name> in your console
More in-depth installation and configuration details are included in the README file packaged with the plugin.

Redis

See https://newrelic.com/plugins/meetme-inc/28

This is the same plugin as for php-fpm.

Privacy and Security Concerns

New Relic is US based. Clearly we would be allowing far greater access to critical user and system data to a untrusted 3rd party than we currently do.

jim — Thu, 05 Sep 2013 10:08:14 GMT

It's no work... The install process for BOA is to add the key to the .barracuda.cnf file, then re-run barracuda up stable.

You should probably remove the 3/4 hour for this pointless research.

chris — Thu, 05 Sep 2013 10:15:17 GMT

Replying to jim:

It's no work... The install process for BOA is to add the key to the .barracuda.cnf file, then re-run barracuda up stable.

Does that install all these plugins?

jim — Mon, 09 Sep 2013 19:35:26 GMT

Replying to chris:

Does that install all these plugins?

https://newrelic.com/plugins/nginx-inc/13
https://newrelic.com/plugins/new-relic-inc/52
https://newrelic.com/plugins/meetme-inc/28

Not at present -- it's the core install with Drupal/PHP monitoring... I'd suggest we go with the stock install for now, and if there's an area we want more visibility on we can add that/those particular plugin(s).

chris — Fri, 13 Sep 2013 21:41:24 GMT

So sorry this didn't happen this morning, have had a mad day with the server crashing, we replaced most the RAM last night and then more RAM and a CPU today plus we migrated a lot of things off it, I might get this done over the weekend, if not I will first thing on Monday.

hours, status, totalhours changed

chris — Sun, 15 Sep 2013 08:53:24 GMT

hours changed from 0.0 to 1.0
status changed from new to accepted
totalhours changed from 0.75 to 1.75

I have signed up for a New Relic account, I'd like it noted that I'm not happy with this:

You understand that the technical processing and transmission of the Service may be transferred unencrypted over a network, and assume all risks related thereto.

https://newrelic.com/terms

This is what they say about the data they will collect:

We may share your Application Data with third parties to provide technical support or to provide specific services, such as hosting of your applications.

We may enter into agreements with Partner Companies such as hosting companies or other services providers. A Partner Company may want access to Application Data that we collect from its customers. As a result, we may provide your Application Data to a Partner Company on a confidential basis in order assist the Partner Company in providing customer support. Because we do not control the privacy practices of our Partner Companies, you should read and understand their privacy policies.

New Relic may disclose Application Data if it believes in good faith that such disclosure is necessary to (a) comply with relevant laws or to respond to subpoenas or warrants served on New Relic; or (b) protect or defend the rights or property of New Relic or users of the Products or related services.

If another company acquires our company or our assets, that company will possess the Application Data collected by it and us and will assume the rights and obligations regarding your Application Data as described in this Privacy Policy.

Except as otherwise stated in this policy, we do not disclose the Application Data collected from our services to third parties without consent from you. You expressly consent to the sharing of your Application Data as described in this policy.

https://newrelic.com/privacy

I have added the license key to /root/.barracuda.cnf and following the steps documented at wiki:PuffinServer#UpgradingBOA

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
 BOA Meta Installer setup completed
 Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
 Bye
barracuda up-stable
 Barracuda [Sun Sep 15 09:18:00 BST 2013] ==> BOA Skynet welcomes you aboard!
 Barracuda [Sun Sep 15 09:18:04 BST 2013] ==> INFO: UPGRADE
 Barracuda [Sun Sep 15 09:18:04 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
 Barracuda [Sun Sep 15 09:18:05 BST 2013] ==> NOTE! Please review all config options displayed below
 Barracuda [Sun Sep 15 09:18:05 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script
 ###
 ### Configuration created on 121215-1545
 ### with Barracuda version BOA-2.0.4
 ###
 ### NOTE: the group of settings displayed bellow will *not* be overriden
 ### on upgrade by the Barracuda script nor by this configuration file.
 ### They can be defined only on initial Barracuda install.
 ###
 _HTTP_WILDCARD=YES
 _MY_OWNIP="81.95.52.103"
 #_MY_OWNIP=""
 _MY_HOSTN="puffin.webarch.net"
 #_MY_HOSTN=""
 _MY_FRONT="master.puffin.webarch.net"
 _THIS_DB_HOST=localhost
 #_THIS_DB_HOST=FQDN
 _SMTP_RELAY_TEST=YES
 _SMTP_RELAY_HOST=""
 _LOCAL_NETWORK_IP=""
 _LOCAL_NETWORK_HN=""
 ###
 ### NOTE: the group of settings displayed bellow
 ### will *override* all listed settings in the Barracuda script,
 ### both on initial install and upgrade.
 ###
 _MY_EMAIL="chris@webarchitects.co.uk"
 _XTRAS_LIST="PDS CSF CHV"
 _AUTOPILOT=NO
 _DEBUG_MODE=NO
 _DB_SERVER=MariaDB
 _SSH_PORT=22
 _LOCAL_DEBIAN_MIRROR="ftp.debian.org"
 _LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
 _FORCE_GIT_MIRROR=""
 _DNS_SETUP_TEST=YES
 _NGINX_EXTRA_CONF=""
 _NGINX_WORKERS=AUTO
 _PHP_FPM_WORKERS=AUTO
 _BUILD_FROM_SRC=NO
 _PHP_MODERN_ONLY=YES
 _PHP_FPM_VERSION=5.3
 _PHP_CLI_VERSION=5.3
 #_LOAD_LIMIT_ONE=1444
 #_LOAD_LIMIT_TWO=888
 _LOAD_LIMIT_ONE==7220
 _LOAD_LIMIT_TWO=4440
 _CUSTOM_CONFIG_CSF=NO
 #_CUSTOM_CONFIG_SQL=NO
 _CUSTOM_CONFIG_SQL=YES
 _CUSTOM_CONFIG_REDIS=NO
 _CUSTOM_CONFIG_PHP_5_2=NO
 #_CUSTOM_CONFIG_PHP_5_3=NO
 _CUSTOM_CONFIG_PHP_5_3=YES
 _SPEED_VALID_MAX=3600
 _NGINX_DOS_LIMIT=300
 _SYSTEM_UPGRADE_ONLY=NO
 _USE_MEMCACHED=NO
 _NEWRELIC_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 _USE_STOCK=NO
 ###
 ### Configuration created on 121215-1545
 ### with Barracuda version BOA-2.0.4
 ###
 ### JK reinstall PHP
 _EXTRA_PACKAGES=
 _PHP_EXTRA_CONF=""
 _STRONG_PASSWORDS=NO
 _DB_BINARY_LOG=NO
 _DB_ENGINE=InnoDB
 _NGINX_LDAP=NO
 _PHP_GEOS=NO
 _PHP_MONGODB=NO
 _AEGIR_UPGRADE_ONLY=NO
 Do you want to proceed with the upgrade? [Y/n] Y
 Barracuda [Sun Sep 15 09:19:00 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
 Barracuda [Sun Sep 15 09:19:02 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
 Barracuda [Sun Sep 15 09:19:02 BST 2013] ==> INFO: Downloading little helpers, please wait...
 Barracuda [Sun Sep 15 09:19:03 BST 2013] ==> INFO: Checking BARRACUDA version...
 Barracuda [Sun Sep 15 09:19:03 BST 2013] ==> INFO: Version test result: OK
 Barracuda [Sun Sep 15 09:19:03 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...
 Barracuda [Sun Sep 15 09:19:05 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9
 Barracuda [Sun Sep 15 09:19:10 BST 2013] ==> UPGRADE START -> checkpoint:
   * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
   * Your server hostname is puffin.webarch.net.
   * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.
 Do you want to proceed with the upgrade? [Y/n] Y
 Barracuda [Sun Sep 15 09:19:19 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
 Barracuda [Sun Sep 15 09:19:21 BST 2013] ==> INFO: Updating apt sources
 Barracuda [Sun Sep 15 09:19:27 BST 2013] ==> INFO: Running aptitude update, please wait...
 Barracuda [Sun Sep 15 09:19:35 BST 2013] ==> INFO: Upgrading required libraries and tools
 Barracuda [Sun Sep 15 09:19:35 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
 Barracuda [Sun Sep 15 09:20:07 BST 2013] ==> INFO: Testing Nginx version...
 Barracuda [Sun Sep 15 09:20:09 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
 Barracuda [Sun Sep 15 09:20:11 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
 Barracuda [Sun Sep 15 09:20:14 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
 Barracuda [Sun Sep 15 09:20:14 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
 Barracuda [Sun Sep 15 09:21:05 BST 2013] ==> INFO: Testing Nginx version...
 Barracuda [Sun Sep 15 09:21:08 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
 Barracuda [Sun Sep 15 09:21:10 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
 Barracuda [Sun Sep 15 09:21:12 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
 Barracuda [Sun Sep 15 09:21:12 BST 2013] ==> INFO: Checking SMTP connections, please wait...
 Barracuda [Sun Sep 15 09:21:14 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
 Barracuda [Sun Sep 15 09:21:19 BST 2013] ==> INFO: Checking if PHP upgrade is available
 Barracuda [Sun Sep 15 09:21:21 BST 2013] ==> INFO: Installed PHP version 5.3.27-1~dotdeb.0, no upgrade required
 Barracuda [Sun Sep 15 09:21:26 BST 2013] ==> INFO: Installing IonCube x86_64 version for PHP-FPM, please wait...
 Barracuda [Sun Sep 15 09:21:29 BST 2013] ==> INFO: Installing New Relic Apps and Server Monitor, please wait...
 Barracuda [Sun Sep 15 09:22:26 BST 2013] ==> INFO: Upgrading Limited Shell to version 0.9.16.3-om8, please wait...
 Barracuda [Sun Sep 15 09:22:34 BST 2013] ==> INFO: Installed Redis version 2.6.14, upgrade/rebuild required
 Barracuda [Sun Sep 15 09:22:36 BST 2013] ==> INFO: Installing Redis update for Debian/squeeze, please wait...
 Barracuda [Sun Sep 15 09:23:48 BST 2013] ==> INFO: Generating random password for Redis server
 Barracuda [Sun Sep 15 09:23:59 BST 2013] ==> INFO: Updating MariaDB and PHP configuration
 Barracuda [Sun Sep 15 09:24:01 BST 2013] ==> INFO: OS and services upgrade completed
 Barracuda [Sun Sep 15 09:24:03 BST 2013] ==> INFO: Restarting MariaDB server, please wait...
 Barracuda [Sun Sep 15 09:24:25 BST 2013] ==> INFO: Upgrading MariaDB tables if necessary, please wait a minute...
 Do you want to upgrade Aegir Master Instance? [Y/n] N
 Barracuda [Sun Sep 15 09:25:38 BST 2013] ==> INFO: Aegir Master Instance not upgraded this time
 Barracuda [Sun Sep 15 09:25:41 BST 2013] ==> INFO: Installing extra Drush versions
 Barracuda [Sun Sep 15 09:25:44 BST 2013] ==> INFO: Drush 4 installation complete
 Barracuda [Sun Sep 15 09:25:45 BST 2013] ==> INFO: Drush 5 installation complete
 Barracuda [Sun Sep 15 09:25:48 BST 2013] ==> INFO: Drush 6 installation complete
 Barracuda [Sun Sep 15 09:25:55 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
 Barracuda [Sun Sep 15 09:26:32 BST 2013] ==> INFO: Restarting MariaDB server
 Barracuda [Sun Sep 15 09:26:46 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
 Barracuda [Sun Sep 15 09:26:49 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
 Barracuda [Sun Sep 15 09:26:52 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
 Barracuda [Sun Sep 15 09:27:03 BST 2013] ==> INFO: csf/lfd firewall upgrade completed
 Barracuda [Sun Sep 15 09:27:05 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
 Barracuda [Sun Sep 15 09:27:11 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
 Barracuda [Sun Sep 15 09:27:15 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
 Barracuda [Sun Sep 15 09:27:27 BST 2013] ==> BYE!
 BARRACUDA upgrade completed
 Bye

Then these values were changed in /opt/local/etc/php53-fpm.conf:

pm.status_path = /status
ping.path = /ping

And the munin plugins were checked:

cd /etc/munin/plugins
munin-run phpfpm_connections
  accepted.value 4
munin-run phpfpm_status
  idle.value 17
  active.value 1
  total.value 18
munin-run nginx_status
  total.value 4
  reading.value 0
  writing.value 1
  waiting.value 3
munin-run nginx_request
  request.value 272001

Accouding the the New Relic Drupal views monitoring page we need to restart:

/etc/init.d/newrelic-daemon restart
/etc/init.d/apache2 restart

So I did:

/etc/init.d/newrelic-daemon restart
/etc/init.d/nginx restart
/etc/init.d/php53-fpm restart

I had a bit of a look around in the web interface, it appears to be working apart from this error when I tried to load the data for www.transitionnetwork.org:

A script on this page may be busy, or it may have stopped responding. You can stop the script now, or you can continue to see if the script will complete.
Script: https://rpm.newrelic.com/maps/FusionCharts.js:49

Over to Jim now to get the data out and translate it into actions to make the site faster!

attachment set

chris — Mon, 16 Sep 2013 09:02:55 GMT

attachment set to Database_activity_Site_www.transitionnetwork.org _New_Relic_2013-09-16.png

New Relic DB Activity 2013-09-16

attachment set

chris — Mon, 16 Sep 2013 09:03:26 GMT

attachment set to Drupal_modules_Site_www.transitionnetwork.org_New_Relic_2013-09-16.png

New Relic Drupal Modules 2013-09-16

attachment set

chris — Mon, 16 Sep 2013 09:03:48 GMT

attachment set to Drupal_hooks_Site_www.transitionnetwork.org_New_Relic_2013-09-16.png

New Relic Drupal Hooks 2013-09-16

attachment set

chris — Mon, 16 Sep 2013 09:04:12 GMT

attachment set to Drupal_views_Site_www.transitionnetwork.org_New_Relic_2013-09-16.png

New Relic Drupal Views 2013-09-16

hours, totalhours changed

chris — Mon, 16 Sep 2013 09:09:26 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.75 to 2.0

I have just taken a look around the New Relic data and took some screen shots of data that we don't get from the Munin stats:

hours, totalhours changed

jim — Mon, 16 Sep 2013 10:33:52 GMT

hours changed from 0.0 to 0.15
totalhours changed from 2.0 to 2.15

Thanks Chris, I saw these last night and it's given me some good insight as to what to target next.

I'm going to let it carry on accumilating data for another day or so before I act, but when I do it'll be on:

Context module's slowness
Speeding up/cleaning up sessions and path_alias tables
Revisiting key views caches and making use of Views Content Cache
Other stuff I'm sure.

I'll do more on #590 on Wednesday then... Thanks, good data!

BTW, what browser are you using?, these screenshots look tortured! Perhaps an update is in order...!

chris — Mon, 16 Sep 2013 10:42:43 GMT

Replying to jim:

BTW, what browser are you using?, these screenshots look tortured! Perhaps an update is in order...!

No update needed I'm running tha latest Firefox 23.0.1.

The grey bar that is in the middle of the images is at the bottom of the browser window -- it's not getting moved down by the screenshop plugin, http://pearlcrescent.com/products/pagesaver/ -- is this what you were referring to?

jim — Mon, 16 Sep 2013 11:31:44 GMT

Aha, yes... makes sense!

I see the server is now stable again, so some of the doubts we had about performance/locking etc have been resolved too -- do you still think it's worth adding any other NR plugins you mentioned? TBH the Drupal/module/MySQL reporting is v handy for what I needed and Munin seems to cover the rest... Thoughts?

chris — Mon, 16 Sep 2013 11:42:31 GMT

Replying to jim:

I see the server is now stable again

Yes, it's not clear to me if this is because of the RAM/CPU that were replaced and/or if it is because several virtual servers were moved off it.

do you still think it's worth adding any other NR plugins you mentioned?

I don't know, they all have a lot of dependencies -- I'd rather not spend time sorting these out if they are not needed.

TBH the Drupal/module/MySQL reporting is v handy for what I needed and Munin seems to cover the rest... Thoughts?

My main thought is can we find some free tool(s) that produce data that corresponds to that in the screenshots above and then can we drop New Relic for cost / privacy / security reasons...?

jim — Mon, 16 Sep 2013 14:49:53 GMT

Well there's some profiling tools for PHP (e.g. xhprof‎), but they tend to absolutely destroy performance when they're running (2x memory, 3x slower). Then there's Devel in Drupal which I've already used to establish what queries were being called and if they were slow/duplicates etc.

It's one thing to get data, but another to store it somewhere and visualise it. NR does this very well, and hopefully we won't need if past the 2 weeks. Any alternative we ran ourselves should be similarly light-footed and cheap to implement/run.

Agreed, lets leave the plugins for now. I'll get back on this on Wednesday.

attachment set

jim — Thu, 26 Sep 2013 11:00:06 GMT

attachment set to controller_summary.csv

Web transactions - page controllers stats

hours changed

jim — Thu, 26 Sep 2013 11:08:37 GMT

hours changed from 0 to .1

Finally, attached is the CSV export from New Relic that shows the 'web transactions' for the page controllers (the code/callback responsible for generating a page) that are slowest.

With this and the past 3-4 posts on this ticket, I'm personally happy we have enough leads to do more work as needed and turn off New Relic for now.

A simple version of the attached file is:

Most time consuming 55.4% - /page_manager_node_view 17.4% - /drupal_get_form 8.66% - /views_page 7.51% - /taxonomy_term_page 3.66% - /page_manager_page_execute 2.09% - /index.php 1.71% - /user_page 1.51% - /advanced_forum_page 0.5% - /page_manager_search_view 0.44% - /page_manager_edit_operation

Slowest average response time 12.4 sec - /page_manager_edit_page_operation 7.31 sec - /block_admin_display 3320 ms - /page_manager_node_add 3180 ms - /page_manager_search_view 2600 ms - /page_manager_node_edit 1630 ms - /page_manager_page_execute 1470 ms - /page_manager_node_view 1460 ms - /content_profile_page_edit 1370 ms - /page_manager_user_view 1360 ms - /mailchimp_subscribe_page 1240 ms - /update.php 1230 ms - /taxonomy_term_page 1220 ms - /panels_admin_page 1220 ms - /upload_js 1210 ms - /drupal_get_form 1140 ms - /transition_pse_entry_form_steps 1110 ms - /og_views_users_faces 1080 ms - /page_manager_page_add_subtask 1060 ms - /system_admin_menu_block_page 964 ms - /filefield_js

Highest thoughput 9.96 rpm - /index.php 9.6 rpm - /page_manager_node_view 3.65 rpm - /drupal_get_form 2.35 rpm - /views_page 1.56 rpm - /taxonomy_term_page 0.81 rpm - /user_page 0.57 rpm - /page_manager_page_execute 0.52 rpm - /advanced_forum_page 0.05 rpm - /imagecache_cache 0.05 rpm - /notifications_anonymous_page_subscribe 0.04 rpm - /mailchimp_subscribe_page 0.04 rpm - /transition_pse_view_sidebar 0.04 rpm - /page_manager_search_view 0.02 rpm - /search_view 0.02 rpm - /page_manager_user_view 0.02 rpm - /node_feed 0.01 rpm - /page_manager_node_edit 0.01 rpm - /comment_reply 0.01 rpm - /user_edit 0.01 rpm - /page_manager_edit_page_operation

hours, totalhours changed

jim — Thu, 26 Sep 2013 15:30:49 GMT

hours changed from 0.0 to 0.2
totalhours changed from 2.15 to 2.35

I'm noting here the New Relic findings, based on 7 days of server running without (too much) interruption and reportedly good performance (based on charts, and me browsing, not other users):

Drupal

Drupal modules

Most time consuming 11300 sec - context 10400 sec - image_resize_filter 5920 sec - path_alias_cache 4740 sec - og 4170 sec - cck_private_fields 3550 sec - views_bulk_operations 2480 sec - content 2220 sec - views 2110 sec - notifications_content 1090 sec - insert 1010 sec - moopapi 1010 sec - php 912 sec - botcha 842 sec - node 689 sec - filefield 646 sec - date 627 sec - filter 590 sec - mollom 568 sec - nodehierarchy 533 sec - update

Slowest average call time 256 ms - panel_context 39.3 ms - context_contrib 31.5 ms - context 24.8 ms - image_resize_filter 10.5 ms - notifications_content 8.22 ms - colorbox 6.55 ms - views 6.44 ms - views_bulk_operations 6.39 ms - emvideo 5.37 ms - moopapi 5.33 ms - og 5.08 ms - botcha 4.69 ms - content 2.97 ms - update 2.96 ms - insert 2.8 ms - transition_events 2.01 ms - recaptcha 1.76 ms - menu_block 1.66 ms - transition_people 1.59 ms - php

Most function calls 31M - path_alias_cache 5.45M - node_content 3.87M - node 3.67M - cck_private_fields 3.48M - content_permissions 1.93M - features 1.57M - og_access 1.57M - content_access 1.57M - nodeaccess_userreference 1.49M - filter 1.29M - user 1.03M - og 973k - webform 949k - fivestar 927k - filefield 895k - ctools 816k - forum 772k - comment 772k - taxonomy 769k - calendar

Drupal hooks

Most time consuming 15900 sec - init 12400 sec - filter 10700 sec - context_page_condition 5850 sec - lookup_path 4800 sec - token_values 4230 sec - field_access 3000 sec - block 1870 sec - help 1150 sec - node_info 651 sec - elements 581 sec - link 572 sec - file_download 429 sec - captcha 380 sec - db_rewrite_sql 373 sec - forms 313 sec - search 228 sec - access 192 sec - footer 174 sec - emfield_rss 110 sec - form_all

Slowest average call time (ms) 1770 - file_delete 684 - context_conditions 297 - requirements 287 - panels_cache_get 172 - form_all 151 - search 61.3 - context_page_condition 41.5 - quicktabs_tabstyles 38.0 - context_reactions 21.8 - mailchimp_merge_values 20.3 - panels_cache_set 13.6 - mollom_form_info 12.2 - captcha 11.5 - insert_content 9.07 - block 7.39 - views_data 6.48 - emfield_rss 5.46 - file_references 4.91 - filter 4.79 - itweak_upload_preview

Most function calls 30.8M - lookup_path 13.8M - help 8.29M - init 7.66M - access 6.94M - field_access 5.87M - elements 4.7M - node_grants 4.63M - token_values 2.69M - for - 2.58 - link 2.58M - db_rewrite_sql 2.56M - filter 1.66M - node_info 1.43M - widget_info 1.08M - hook_info 1.02M - exit 569k - views_query_substitutions 501k - api_version 477k - cb_breadcrumb_info 367k - insert_widgets

jim — Thu, 26 Sep 2013 15:36:16 GMT

Continuing...

MySQL

Slowest average response time 1170 ms - content_field_logo - INSERT 1120 ms - node - INSERT 1010 ms - content_field_bios - INSERT 678 ms - url_alias - UPDATE 635 ms - term_node - DELETE 574 ms - content_type_profile - INSERT 546 ms - content_field_relatedprojects - INSERT 453 ms - content_field_user_types - INSERT 453 ms - content_field_image - INSERT 404 ms - cck_private_fields - DELETE 391 ms - node - UPDATE 377 ms - location_instance - INSERT 369 ms - location - INSERT 364 ms - content_field_roles_offered - INSERT 347 ms - content_field_initiative - INSERT 340 ms - url_alias - INSERT 283 ms - node_comment_statistics - INSERT 268 ms - content_field_editor_users - INSERT 255 ms - cache_page - INSERT 244 ms - content_field_type - INSERT

Most time consuming 19.2% - url_alias - SELECT 15.3% - node - SELECT 3.91% - users - SELECT 3.71% - system - SELECT 3.33% - files - SELECT 2.91% - menu_links - SELECT 2.48% - cck_private_fields - SELECT 1.9% - role - SELECT 1.89% - node_comment_statistics - SELECT 1.8% - term_node - SELECT 1.7% - nodehierarchy - SELECT 1.63% - menu_router - SELECT 1.53% - content_type_profile - SELECT 1.34% - node_access - SELECT 1.27% - feeds_node_item - SELECT 1.25% - vocabulary - SELECT 1.24% - captcha_sessions - UPDATE 1.22% - location_instance - SELECT 1.16% - other - SQL 1.09% - access - SELECT

Highest thoughput 974cpm - url_alias - SELECT 226 cpm - node - SELECT 87 cpm - system - SELECT 86.6 cpm - cck_private_fields - SELECT 85.6 cpm - role - SELECT 81.8 cpm - menu_links - SELECT 71.6 cpm - users - SELECT 66.3 cpm - other - SQL 53.2 cpm - files - SELECT 52.2 cpm - node_comment_statistics - SELECT 49.6 cpm - term_node - SELECT 47.8 cpm - nodehierarchy - SELECT 47.7 cpm - location_instance - SELECT 47.7 cpm - feeds_node_item - SELECT 40.8 cpm - filter_formats - SELECT 35.9 cpm - menu_router - SELECT 33.6 cpm - botcha_form - SELECT 32.8 cpm - captcha_points - SELECT 31.8 cpm - location - SELECT 31.6 cpm - content_group_fields - SELECT

Slowest queries by Web Transaction 417ms - term_data - SELECT (/tagadelic_page_chunk) 220 ms - menu_router - SELECT (/update_status) 161 ms - term_node - SELECT (/site_map_page) 135 ms - content_type_profile - SELECT (/page_manager_node_edit) 114 ms - locales_source - SELECT (/og_subscribe) 83.3 ms - (subquery) - SELECT (/page_manager_search_view) 79.4 ms - search_index - SELECT (/page_manager_search_view) 63.5 ms - (subquery) - SELECT (/advanced_forum_page) 45.9 ms - content_field_featured_image - SELECT (/content_profile_page_edit) 29.5 ms - cache_form - SELECT (/notifications_page_unsubscribe) 19.2 ms - imagecache_preset - SELECT (/advanced_forum_page) 18.9 ms - blocks - SELECT (/update_status) 13.5 ms - page_manager_weights - SELECT (/page_manager_user_view) 13.4 ms - cache_update - SELECT (/system_status) 12.0 ms - trigger_assignments - SELECT (/logintoboggan_validate_email) 11.9 ms - blocks - SELECT (/panels_admin_page) 11.4 ms - (subquery) - SELECT (/views_page) 11.3 ms - variable - SELECT (/index.php) 11.2 ms - node - SELECT (/ctools_content_autocomplete_node) 11.1 ms - variable - SELECT (/drupal_get_form)

hours, totalhours changed

jim — Thu, 26 Sep 2013 15:42:07 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.35 to 2.45

External Services

Slowest average response time 17.1 sec - images.bookworld.com.au - 4.61 sec - www.the-south-asian.com 4.0 sec - www.jamessamuel.co.nz 3930 ms - www.biglotteryfund.org.uk 3460 ms - www.transitiontowns.org.nz 2590 ms - www.llnsllc.com 2570 ms - www.transitionscotland.org 2180 ms - thelittlegymoftexas.com 2040 ms - www.thelivingcentre.com 1970 ms - www.pbs.org 1930 ms - jamessamuel.co.nz 1900 ms - www.thebiglunch.com 1890 ms - www.facebook.com 1840 ms - nu-project.org 1790 ms - fbcdn-profile-a.akamaihd.net 1770 ms - www.akpress.org 1630 ms - www.communitychef.org.uk 1570 ms - valeharvest.weebly.com 1510 ms - transitionitalia.files.wordpress.com 1510 ms - www.healthyfeetblog.com

Most time consuming 24.9% - news.transitionnetwork.org 15.2% - transitionculture.org 11.9% - transitionvoice.com 6.5% - www.resilience.org 3.09% - updates.drupal.org 2.85% - www.stopforumspam.com 2.6% - farm5.static.flickr.com 2.39% - transitionus.org 2.31% - 3.bp.blogspot.com 2.13% - 1.bp.blogspot.com 1.78% - markinflowers.files.wordpress.com 1.63% - 2.bp.blogspot.com 1.63% - 4.bp.blogspot.com 1.3% - images.bookworld.com.au 1.27% - feeds.feedburner.com 1.24% - weldingweb.com 0.72% - farm2.static.flickr.com 0.71% - www.transitiontowns.org.nz 0.65% - www.transitionnetwork.org 0.54% - feedads.g.doubleclick.net

Highest thoughput 0.92 cpm - news.transitionnetwork.org 0.89 cpm - transitionculture.org 0.69 cpm - www.stopforumspam.com 0.62 cpm - feeds.feedburner.com 0.37 cpm - www.transitionnetwork.org 0.21 cpm - feedads.g.doubleclick.net 0.2 cpm - transitionvoice.com 0.14 cpm - www.resilience.org 0.08 cpm - updates.drupal.org 0.07 cpm - 3.bp.blogspot.com 0.06 cpm - gdata.youtube.com 0.06 cpm - 4.bp.blogspot.com 0.06 cpm - youtube.com 0.06 cpm - www.youtube.com 0.06 cpm - youtube.googleapis.com 0.06 cpm - 1.bp.blogspot.com 0.06 cpm - transitiontowns.org 0.05 cpm - 2.bp.blogspot.com 0.04 cpm - transitionus.org 0.04 cpm - markinflowers.files.wordpress.com

hours, totalhours changed

jim — Thu, 26 Sep 2013 15:54:53 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.55 to 2.65

Some less relevant but still interesting stats...

Geography

Average page load time

10.3 sec - India 7.96 sec - New Zealand 6.89 sec - Australia 6.16 sec - Canada 5.39 sec - United States 5.18 sec - Italy 5.12 sec - Spain 5.12 sec - France 4.93 sec - Portugal 4.3 sec - Germany 3820 ms - United Kingdom 3140 ms - Belgium 3110 ms - Netherlands 1820 ms - Switzerland

'Apdex' most dissatisfying (% of total dissatisfied 23% - United Kingdom 21.5% - United States 8.16% - Australia 6.94% - Canada 4.11% - India 2.97% - France 2.74% - New Zealand 2.33% - Germany 2.23% - Spain 2.03% - Italy 1.74% - Portugal 1.41% - Philippines 1.27% - Mexico 1.13% - Brazil 1.08% - China 1% - Argentina 0.9% - Hong Kong 0.82% - Ireland 0.74% - Pakistan 0.72% - Malaysia

Countries by average sessions (logins per visit) 4.63 - United Kingdom 3.47 - Panama 2.59 - United States 2.3 - Luxembourg 2 - Zambia 2 - Cyprus 1.74 - Germany 1.72 - Netherlands 1.7 - Norway 1.69 - Canada 1.66 - Australia 1.58 - New Zealand 1.57 - France 1.53 - Brazil 1.52 - Senegal 1.5 - Slovenia 1.5 - Ireland 1.47 - Kuwait 1.47 - Sweden 1.46 - Korea - Republic of

hours, totalhours changed

chris — Sun, 29 Sep 2013 21:58:37 GMT

hours changed from 0.0 to 0.4
totalhours changed from 2.65 to 3.05

Uninstalling New Relic, these are the currently installed packages:

aptitude search relic
 i A newrelic-daemon                 - The New Relic proxy daemon
 i   newrelic-php5                   - The New Relic agent for PHP
 i A newrelic-php5-common            - Scripts and samples for the New Relic PHP
 i   newrelic-sysmond                - The New Relic agent for server monitoring

The /root/.barracuda.cnf file was edited and the key commented out:

_NEWRELIC_KEY=

Updating BOA:

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
 BOA Meta Installer setup completed
 Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
 Bye
barracuda up-stable
Barracuda [Mon Sep 30 01:57:57 BST 2013] ==> BOA Skynet welcomes you aboard!
Barracuda [Mon Sep 30 01:58:01 BST 2013] ==> INFO: UPGRADE
Barracuda [Mon Sep 30 01:58:01 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Mon Sep 30 01:58:02 BST 2013] ==> NOTE! Please review all config options displayed below
Barracuda [Mon Sep 30 01:58:02 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
#_MY_OWNIP=""
_MY_HOSTN="puffin.webarch.net"
#_MY_HOSTN=""
_MY_FRONT="master.puffin.webarch.net"
_THIS_DB_HOST=localhost
#_THIS_DB_HOST=FQDN
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV"
_AUTOPILOT=NO
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
#_LOAD_LIMIT_ONE=1444
#_LOAD_LIMIT_TWO=888
_LOAD_LIMIT_ONE==7220
_LOAD_LIMIT_TWO=4440
_CUSTOM_CONFIG_CSF=NO
#_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_SQL=YES
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
#_CUSTOM_CONFIG_PHP_5_3=NO
_CUSTOM_CONFIG_PHP_5_3=YES
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### JK reinstall PHP
_EXTRA_PACKAGES=
_PHP_EXTRA_CONF=""
_STRONG_PASSWORDS=NO
_DB_BINARY_LOG=NO
_DB_ENGINE=InnoDB
_NGINX_LDAP=NO
_PHP_GEOS=NO
_PHP_MONGODB=NO
_AEGIR_UPGRADE_ONLY=NO
Do you want to proceed with the upgrade? [Y/n] Y
Barracuda [Mon Sep 30 01:58:45 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Mon Sep 30 01:58:46 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Mon Sep 30 01:58:46 BST 2013] ==> INFO: Downloading little helpers, please wait...
Barracuda [Mon Sep 30 01:58:47 BST 2013] ==> INFO: Checking BARRACUDA version...
Barracuda [Mon Sep 30 01:58:47 BST 2013] ==> INFO: Version test result: OK
Barracuda [Mon Sep 30 01:58:47 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Mon Sep 30 01:58:50 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9
Barracuda [Mon Sep 30 01:58:54 BST 2013] ==> UPGRADE START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your server hostname is puffin.webarch.net.
  * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.
Do you want to proceed with the upgrade? [Y/n] Y
Barracuda [Mon Sep 30 01:59:03 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Mon Sep 30 01:59:06 BST 2013] ==> INFO: Updating apt sources
Barracuda [Mon Sep 30 01:59:08 BST 2013] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Mon Sep 30 01:59:12 BST 2013] ==> INFO: Running aptitude update, please wait...
Barracuda [Mon Sep 30 01:59:19 BST 2013] ==> INFO: Upgrading required libraries and tools
Barracuda [Mon Sep 30 01:59:19 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Mon Sep 30 01:59:50 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Mon Sep 30 01:59:52 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Mon Sep 30 01:59:54 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Mon Sep 30 01:59:56 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Mon Sep 30 01:59:56 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Mon Sep 30 02:00:47 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Mon Sep 30 02:00:49 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Mon Sep 30 02:00:51 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Mon Sep 30 02:00:53 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Mon Sep 30 02:00:53 BST 2013] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Mon Sep 30 02:00:55 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
Barracuda [Mon Sep 30 02:00:59 BST 2013] ==> INFO: Checking if PHP upgrade is available
Barracuda [Mon Sep 30 02:01:01 BST 2013] ==> INFO: Installed PHP version 5.3.27-1~dotdeb.0, no upgrade required
Barracuda [Mon Sep 30 02:01:05 BST 2013] ==> INFO: Installed Redis version 2.6.16, no upgrade/rebuild required
Barracuda [Mon Sep 30 02:01:18 BST 2013] ==> INFO: Updating MariaDB and PHP configuration
Barracuda [Mon Sep 30 02:01:19 BST 2013] ==> INFO: OS and services upgrade completed
Barracuda [Mon Sep 30 02:01:21 BST 2013] ==> INFO: Restarting MariaDB server, please wait...
Barracuda [Mon Sep 30 02:01:39 BST 2013] ==> INFO: Upgrading MariaDB tables if necessary, please wait a minute...
Do you want to upgrade Aegir Master Instance? [Y/n]
Barracuda [Mon Sep 30 02:02:42 BST 2013] ==> INFO: Running Aegir Master Instance upgrade
Barracuda [Mon Sep 30 02:02:45 BST 2013] ==> INFO: Syncing provision backend db_passwd, please wait...
Barracuda [Mon Sep 30 02:02:49 BST 2013] ==> INFO: Running hosting-dispatch (1/3), please wait...
Barracuda [Mon Sep 30 02:02:57 BST 2013] ==> INFO: Running hosting-dispatch (2/3), please wait...
Barracuda [Mon Sep 30 02:03:04 BST 2013] ==> INFO: Running hosting-dispatch (3/3), please wait...
Barracuda [Mon Sep 30 02:03:04 BST 2013] ==> INFO: Syncing hostmaster frontend db_passwd, please wait...
Barracuda [Mon Sep 30 02:03:06 BST 2013] ==> INFO: Testing previous install...
Barracuda [Mon Sep 30 02:03:06 BST 2013] ==> INFO: Test OK, we can proceed with Hostmaster upgrade
Barracuda [Mon Sep 30 02:03:06 BST 2013] ==> INFO: Moving old directories
Barracuda [Mon Sep 30 02:03:07 BST 2013] ==> INFO: Downloading drush, please wait...
Barracuda [Mon Sep 30 02:03:08 BST 2013] ==> INFO: Drush seems to be functioning properly
Barracuda [Mon Sep 30 02:03:09 BST 2013] ==> INFO: Installing provision backend in /var/aegir/.drush
Barracuda [Mon Sep 30 02:03:09 BST 2013] ==> INFO: Downloading Drush and Provision extensions, please wait...
Barracuda [Mon Sep 30 02:03:09 BST 2013] ==> INFO: Running hostmaster-migrate, please wait...
Barracuda [Mon Sep 30 02:04:12 BST 2013] ==> INFO: Syncing hostmaster frontend db_passwd, please wait...
Barracuda [Mon Sep 30 02:04:15 BST 2013] ==> INFO: Generating random password for Redis server
Barracuda [Mon Sep 30 02:04:45 BST 2013] ==> INFO: Aegir Master Instance upgrade completed
Barracuda [Mon Sep 30 02:04:46 BST 2013] ==> INFO: Installing extra Drush versions
Barracuda [Mon Sep 30 02:04:49 BST 2013] ==> INFO: Drush 4 installation complete
Barracuda [Mon Sep 30 02:04:50 BST 2013] ==> INFO: Drush 5 installation complete
Barracuda [Mon Sep 30 02:04:52 BST 2013] ==> INFO: Drush 6 installation complete
Barracuda [Mon Sep 30 02:04:56 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
Barracuda [Mon Sep 30 02:05:41 BST 2013] ==> INFO: Restarting MariaDB server
Barracuda [Mon Sep 30 02:05:52 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
Barracuda [Mon Sep 30 02:05:54 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Mon Sep 30 02:06:00 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
Barracuda [Mon Sep 30 02:06:06 BST 2013] ==> INFO: csf/lfd firewall upgrade completed
Barracuda [Mon Sep 30 02:06:08 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
Barracuda [Mon Sep 30 02:06:14 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Mon Sep 30 02:06:18 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Mon Sep 30 02:09:44 BST 2013] ==> BYE!
BARRACUDA upgrade completed
Bye

The New Relic packages are still installed:

aptitude search relic
i A newrelic-daemon      - The New Relic proxy daemon
i   newrelic-php5        - The New Relic agent for PHP
i A newrelic-php5-common - Scripts and samples for the New Relic PHP agent
i   newrelic-sysmond     - The New Relic agent for server monitoring

Removing them:

aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
The following packages will be REMOVED:
  newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
0 packages upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 41.5 MB will be freed.
(Reading database ... 49988 files and directories currently installed.)
Removing newrelic-php5 ...
Removing newrelic-daemon ...
Removing newrelic-php5-common ...
Removing newrelic-sysmond ...
Stopping New Relic Server Monitor: newrelic-sysmond OK

Note the server time is off, see ticket:599

hours, totalhours changed

chris — Wed, 02 Oct 2013 13:46:33 GMT

hours changed from 0.0 to 0.25
totalhours changed from 3.05 to 3.3

The barracuda upgrade clobbered some tweaks that are needed to generate Munin stats:

Checking the wiki:PuffinServer#UpgradingBOA notes, the following were edited:

/var/aegir/config/server_master/nginx.conf
/opt/local/etc/php53-fpm.conf

And nginx and php-fpm restarted.

jim — Wed, 02 Oct 2013 14:14:33 GMT

It would be better if you didn't use /var/aegir/config/server_master/nginx.conf which is for Aegir, and created a new config file in /var/aegir/config/}}} somewhere... There are folders in there that include extra conf files.

hours, totalhours changed

chris — Thu, 03 Oct 2013 08:56:44 GMT

hours changed from 0.0 to 0.25
totalhours changed from 3.3 to 3.55

I think this is a coincidence, but it's worth nothing that while New Relic was running the load spikes dropped off and now they are back, see ticket:555#comment:93

Thanks for the suggestion Jim, I tried creating /var/aegir/config/includes/nginx_force_include_munin containing:

#######################################################
###  nginx default server munin
#######################################################
server {
  location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    allow 81.95.52.103;
    deny all;
  }
  location ~ ^/(status|ping)$ {
    fastcgi_pass 127.0.0.1:9090;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    include fastcgi_params;
    access_log off;
    allow 127.0.0.1;
    deny all;
  }
}

And I tried removing the code from /var/aegir/config/server_master/nginx.conf but when I tried to restart nginx I got:

Configuration test failed!
nginx: [emerg] "server" directive is not allowed here in /var/aegir/config/includes/nginx_force_include_munin:5
nginx: configuration file /etc/nginx/nginx.conf test failed

So I reverted the changed -- Jim if you can find a place where these changes can be made in a way that they don't get clobbered by BOA that would be great -- I haven't found a way to do this.

hours, totalhours changed

jim — Thu, 03 Oct 2013 13:21:01 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.55 to 3.65

OK, so I just ran Drush for a simple task and got this error:

tn@puffin:~/static/transition-network-d6-p005$ drush www.transitionnetwork.org en views_ui
PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0
The following extensions will be enabled: views_ui
Do you really want to continue? (y/n): n

So I wonder if the *LACK* of newrelic includes/modules is the issue for load spikes -- NR is not properly uninstalled in that PHP config is still referring to it.

Looking at config now, but if you have any thoughts Chris...

hours, totalhours changed

jim — Thu, 03 Oct 2013 13:24:00 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.65 to 3.75

It's in /opt/local/etc/php53.ini, still referrs to NR at end of file...

; Enable Extensions
extension=uploadprogress.so
extension=imagick.so
extension=jsmin.so
extension=redis.so
;extension=memcached.so
extension=newrelic.so
[newrelic]
newrelic.enabled=true
newrelic.license = "[snip]"
newrelic.logfile = "/var/log/newrelic/php_agent.log"
newrelic.loglevel = "error"
newrelic.appname = "Drush Tasks and Sites Cron"
newrelic.daemon.logfile = "/var/log/newrelic/newrelic-daemon.log"
newrelic.daemon.loglevel = "error"
newrelic.daemon.pidfile = "/var/run/newrelic-daemon.pid"
newrelic.daemon.dont_launch = 3
newrelic.framework.drupal.modules = 1

Am commenting out...

hours, totalhours changed

chris — Thu, 03 Oct 2013 13:36:10 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.75 to 3.85

Thanks Jim, I did notice that the daemon was still running in the top dump on ticket:555#comment:96 and was going to get to that...

41997 root      20   0  3956  608  492 S    0  0.0   0:00.00 newrelic-daemon

And it's still running:

 ps -lA | grep relic
4 S     0  8575  8466  3  80   0 -   989 -      ?        00:00:03 newrelic-daemon
1 S     0  8640  8575  0  80   0 -   989 -      ?        00:00:00 newrelic-daemon

So I have now killed it:

killall -9 newrelic-daemon

Thanks for finding the config in /opt/local/etc/php53.ini -- I had assumed that commenting the _NEWRELIC_KEY in /root/.barracuda.cnf and running barracuda up-stable would remove any New Relic config that had been added, sorry.

jim — Thu, 03 Oct 2013 13:51:05 GMT

Note this is commented out (as of about 14:30), and also in /usr/local/lib/php.ini is the file that Drush uses (I think).

I also noted that service php5-fpm was running. I killed this since when I restarted it (thinking it was what Drush used) it gave a few 502s. php53-fpm has been restarted and all looks ok..

However, the drush warning about missing /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so is still occurring, so I'm still confused.

According to this search in GitHub?, new relic is only mentioned in afew places: https://github.com/omega8cc/nginx-for-drupal/search?q=newrelic&ref=cmdform

Parking for now, going over to #602.

hours, totalhours changed

jim — Thu, 03 Oct 2013 13:53:13 GMT

hours changed from 0.0 to 0.25
totalhours changed from 3.85 to 4.1

Adding time and replying to chris:

So I have now killed it: killall -9 newrelic-daemon

Thanks for finding the config in /opt/local/etc/php53.ini -- I had assumed that commenting the _NEWRELIC_KEY in /root/.barracuda.cnf and running barracuda up-stable would remove any New Relic config that had been added, sorry.

No, I've had this in the past... It seems BOA is great at installing things and integrating them. but not really taking them back out again!

Drush still complains though, at a loss to explain that.

hours, totalhours changed

jim — Thu, 03 Oct 2013 14:00:55 GMT

hours changed from 0.0 to 0.15
totalhours changed from 4.1 to 4.25

In line 294 onwards of BOND.sh it lists where it puts the newrelic.ini include in to the php config files. They are:

/opt/etc/php.ini
/usr/local/lib/php.ini [DONE JK]
/opt/local/etc/php53.ini [DONE JK]
/opt/local/lib/php.ini

So I'll comment out the other ones now too...

hours, totalhours changed

jim — Thu, 03 Oct 2013 14:02:01 GMT

hours changed from 0.0 to 0.05
totalhours changed from 4.25 to 4.3

Done, and Drush is now complaint-free... Suggest we wait an hour on the offchance my and Chris' recent tweaks have any effect on the load spikes.

hours, totalhours changed

chris — Thu, 03 Oct 2013 20:28:39 GMT

hours changed from 0.0 to 0.4
totalhours changed from 4.3 to 4.7

Urm jim you made the New Relic license key public -- does this have security / privacy implications?

Reenabling New Relic, starting at Thu Oct 3 21:03:25 BST 2013, first the _NEWRELIC_KEY variable in /root/.barracuda.cnf was added back then:

wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt
 BOA Meta Installer setup completed
 Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs for how-to
 Bye
barracuda up-stable
Barracuda [Thu Oct  3 21:06:18 BST 2013] ==> BOA Skynet welcomes you aboard!
Barracuda [Thu Oct  3 21:06:22 BST 2013] ==> INFO: UPGRADE
Barracuda [Thu Oct  3 21:06:22 BST 2013] ==> INFO: Reading your /root/.barracuda.cnf config file
Barracuda [Thu Oct  3 21:06:23 BST 2013] ==> NOTE! Please review all config options displayed below
Barracuda [Thu Oct  3 21:06:23 BST 2013] ==> NOTE! It will *override* all settings in the Barracuda script
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="81.95.52.103"
#_MY_OWNIP=""
_MY_HOSTN="puffin.webarch.net"
#_MY_HOSTN=""
_MY_FRONT="master.puffin.webarch.net"
_THIS_DB_HOST=localhost
#_THIS_DB_HOST=FQDN
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="chris@webarchitects.co.uk"
_XTRAS_LIST="PDS CSF CHV"
_AUTOPILOT=NO
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_EXTRA_CONF=""
_NGINX_WORKERS=AUTO
_PHP_FPM_WORKERS=AUTO
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
#_LOAD_LIMIT_ONE=1444
#_LOAD_LIMIT_TWO=888
_LOAD_LIMIT_ONE==7220
_LOAD_LIMIT_TWO=4440
_CUSTOM_CONFIG_CSF=NO
#_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_SQL=YES
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
#_CUSTOM_CONFIG_PHP_5_3=NO
_CUSTOM_CONFIG_PHP_5_3=YES
_SPEED_VALID_MAX=3600
_NGINX_DOS_LIMIT=300
_SYSTEM_UPGRADE_ONLY=NO
_USE_MEMCACHED=NO
_NEWRELIC_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#_NEWRELIC_KEY=
_USE_STOCK=NO
###
### Configuration created on 121215-1545
### with Barracuda version BOA-2.0.4
###
### JK reinstall PHP
_EXTRA_PACKAGES=
_PHP_EXTRA_CONF=""
_STRONG_PASSWORDS=NO
_DB_BINARY_LOG=NO
_DB_ENGINE=InnoDB
_NGINX_LDAP=NO
_PHP_GEOS=NO
_PHP_MONGODB=NO
_AEGIR_UPGRADE_ONLY=NO
Do you want to proceed with the upgrade? [Y/n] Y
Barracuda [Thu Oct  3 21:06:56 BST 2013] ==> INFO: Testing GitHub, Drupal and Gitorious servers availability, please wait...
Barracuda [Thu Oct  3 21:06:58 BST 2013] ==> INFO: GitHub mirror repository will be used for this install
Barracuda [Thu Oct  3 21:06:58 BST 2013] ==> INFO: Downloading little helpers, please wait...
Barracuda [Thu Oct  3 21:06:59 BST 2013] ==> INFO: Checking BARRACUDA version...
Barracuda [Thu Oct  3 21:06:59 BST 2013] ==> INFO: Version test result: OK
Barracuda [Thu Oct  3 21:06:59 BST 2013] ==> INFO: Checking your Debian or Ubuntu version...
Barracuda [Thu Oct  3 21:07:01 BST 2013] ==> Aegir with Nginx on Debian/squeeze - Skynet Agent v.BOA-2.0.9
Barracuda [Thu Oct  3 21:07:05 BST 2013] ==> UPGRADE START -> checkpoint:
  * Your e-mail address appears to be chris@webarchitects.co.uk - is that correct?
  * Your server hostname is puffin.webarch.net.
  * Your Aegir control panel is/will be available at https://master.puffin.webarch.net.
Do you want to proceed with the upgrade? [Y/n] Y
Barracuda [Thu Oct  3 21:07:19 BST 2013] ==> INFO: Cleaning up temp files in /var/opt/
Barracuda [Thu Oct  3 21:07:21 BST 2013] ==> INFO: Updating apt sources
Barracuda [Thu Oct  3 21:07:23 BST 2013] ==> INFO: We will use Debian mirror ftp.debian.org
Barracuda [Thu Oct  3 21:07:27 BST 2013] ==> INFO: Running aptitude update, please wait...
Barracuda [Thu Oct  3 21:07:34 BST 2013] ==> INFO: Upgrading required libraries and tools
Barracuda [Thu Oct  3 21:07:34 BST 2013] ==> NOTE! This step may take a few minutes, please wait...
Barracuda [Thu Oct  3 21:08:05 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Thu Oct  3 21:08:07 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Thu Oct  3 21:08:09 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Thu Oct  3 21:08:11 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Thu Oct  3 21:08:11 BST 2013] ==> INFO: Running aptitude full-upgrade again, please wait...
Barracuda [Thu Oct  3 21:09:03 BST 2013] ==> INFO: Testing Nginx version...
Barracuda [Thu Oct  3 21:09:06 BST 2013] ==> INFO: Installed Nginx version nginx/1.5.2, no upgrade required
Barracuda [Thu Oct  3 21:09:08 BST 2013] ==> INFO: Checking for Linux/Cdorked.A malware, please wait...
Barracuda [Thu Oct  3 21:09:10 BST 2013] ==> INFO: No Linux/Cdorked.A malware traces found - system clean
Barracuda [Thu Oct  3 21:09:10 BST 2013] ==> INFO: Checking SMTP connections, please wait...
Barracuda [Thu Oct  3 21:09:12 BST 2013] ==> INFO: Upgrading a few more tools, please wait...
Barracuda [Thu Oct  3 21:09:15 BST 2013] ==> INFO: Checking if PHP upgrade is available
Barracuda [Thu Oct  3 21:09:18 BST 2013] ==> INFO: Installed PHP version 5.3.27-1~dotdeb.0, no upgrade required
Barracuda [Thu Oct  3 21:09:18 BST 2013] ==> INFO: Uninstalling previous version of New Relic Apps and Server Monitor, please wait...
Barracuda [Thu Oct  3 21:09:24 BST 2013] ==> INFO: Installing latest version of New Relic Apps and Server Monitor, please wait...
Barracuda [Thu Oct  3 21:10:21 BST 2013] ==> INFO: Installed Redis version 2.6.16, no upgrade/rebuild required
Barracuda [Thu Oct  3 21:10:33 BST 2013] ==> INFO: Updating MariaDB and PHP configuration
Barracuda [Thu Oct  3 21:10:34 BST 2013] ==> INFO: OS and services upgrade completed
Barracuda [Thu Oct  3 21:10:36 BST 2013] ==> INFO: Restarting MariaDB server, please wait...
Barracuda [Thu Oct  3 21:10:57 BST 2013] ==> INFO: Upgrading MariaDB tables if necessary, please wait a minute...
Do you want to upgrade Aegir Master Instance? [Y/n] N
Barracuda [Thu Oct  3 21:15:28 BST 2013] ==> INFO: Aegir Master Instance not upgraded this time
Barracuda [Thu Oct  3 21:15:39 BST 2013] ==> INFO: Installing extra Drush versions
Barracuda [Thu Oct  3 21:15:41 BST 2013] ==> INFO: Drush 4 installation complete
Barracuda [Thu Oct  3 21:15:43 BST 2013] ==> INFO: Drush 5 installation complete
Barracuda [Thu Oct  3 21:15:45 BST 2013] ==> INFO: Drush 6 installation complete
Barracuda [Thu Oct  3 21:15:49 BST 2013] ==> INFO: Generating random password for Redis server
Barracuda [Thu Oct  3 21:15:50 BST 2013] ==> INFO: Restarting Redis and PHP-FPM, reloading Nginx
Barracuda [Thu Oct  3 21:16:31 BST 2013] ==> INFO: Restarting MariaDB server
Barracuda [Thu Oct  3 21:16:45 BST 2013] ==> INFO: New random password for MariaDB generated and stored in /root/.my.pass.txt
Barracuda [Thu Oct  3 21:16:48 BST 2013] ==> INFO: New entry added to /var/log/barracuda_log.txt
Barracuda [Thu Oct  3 21:17:02 BST 2013] ==> INFO: Upgrading csf/lfd firewall, please wait...
Barracuda [Thu Oct  3 21:17:07 BST 2013] ==> INFO: csf/lfd firewall upgrade completed
Barracuda [Thu Oct  3 21:17:09 BST 2013] ==> CARD: Now charging your credit card for this automated upgrade service...
Barracuda [Thu Oct  3 21:17:16 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Thu Oct  3 21:17:20 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Thu Oct  3 21:17:16 BST 2013] ==> JOKE: Just kidding! Enjoy your Aegir Hosting System :)
Barracuda [Thu Oct  3 21:17:20 BST 2013] ==> Final post-upgrade cleaning, please wait a moment...
Barracuda [Thu Oct  3 21:20:34 BST 2013] ==> BYE!
BARRACUDA upgrade completed
Bye

Check that New Relic is running:

ps -lA | grep relic
1 S     0 50464     1  0  80   0 -  3935 -      ?        00:00:00 newrelic-daemon
1 S     0 50466 50464  0  80   0 - 26507 -      ?        00:00:00 newrelic-daemon

Apply the edits for Munin:

wiki:PuffinServer#nginxconfigchanges these edits were not needed this time
wiki:PuffinServer#php-fpmconfigchanges these edits were done and munin tested

hours, totalhours, component changed

chris — Fri, 04 Oct 2013 08:52:28 GMT

hours changed from 0.0 to 0.25
totalhours changed from 4.7 to 4.95
component changed from Unassigned to Live server

When I try to login to New Relic I get:

We're sorry, you haven't verified your email address yet. If you've misplaced the verification email we sent, contact support for assistance.

As far as I'm aware no verification email has been sent, I have sent them a email about this and CC'd it to the ttech list.

attachment set

chris — Mon, 07 Oct 2013 19:43:00 GMT

attachment set to puffin-load-week-2013-10-07.png

attachment set

chris — Mon, 07 Oct 2013 19:49:44 GMT

attachment set to puffin-piwik-2013-10-07_hits_and_visits.png

hours, totalhours changed

chris — Mon, 07 Oct 2013 19:56:10 GMT

hours changed from 0.0 to 0.25
totalhours changed from 4.95 to 5.2

Jim - have you managed to login to New Relic since it was added back? I still haven't.

I was going to remove it again tonight but I think it's worth taking one last look, especially since it was added back at 8:30pm on 3rd Oct and the load spikes, once again, dramatically reduced:

Of course it was the weekend and the load has picked up since and it does correspond with the Piwik stats -- I fiund it hard to believe there is a direct correlation:

chris — Wed, 09 Oct 2013 08:35:27 GMT

I still can't login to New Relic, I requested a password reset but didn't get a email from them.

But the week extension is almost due to run out so, I'll remove it tonight.

Also the load spikes are back, see ticket:555#comment:100

hours, totalhours changed

jim — Wed, 09 Oct 2013 11:23:29 GMT

hours changed from 0.0 to 0.25
totalhours changed from 5.2 to 5.45

I've hadn't tried until a few minutes ago... I'm in though. I sent an email to newrelic@… a few days ago -- was that received? I'm wondering if that account is actually working or not.

I've added my gmail account as an admin to avoid being locked out -- I can add yours too Chris if you'd like.

However, there is NOTHING being reported to NR, so something is up...

So I checked and there services are running... looking at the php.ini files from comment:27, the NR settings were still commented out in /opt/local/etc/php53.ini and /opt/local/lib/php.ini.

I've uncommented them and restarted php53-fpm. Now NR is getting data.

jim — Fri, 11 Oct 2013 11:21:08 GMT

Note NR has reverted to the free tier, so Drupal and database detail is now unavailable.

hours, totalhours changed

jim — Tue, 15 Oct 2013 19:45:37 GMT

hours changed from 0.0 to 0.15
totalhours changed from 5.45 to 5.6

I've now removed NR, first I removed (not commented) the PHP.inis, then I uninstalled it:

puffin:~# nano /opt/etc/php.ini
puffin:~# nano /usr/local/lib/php.ini
puffin:~# nano  /opt/local/etc/php53.ini
puffin:~# nano /opt/local/lib/php.ini
puffin:~# service php53-fpm reload
Reload service php53-fpm.. done
puffin:~# aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
The following packages will be REMOVED:
  newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
0 packages upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 50.1 MB will be freed.
(Reading database ... 49990 files and directories currently installed.)
Removing newrelic-php5 ...
Removing newrelic-daemon ...
Removing newrelic-php5-common ...
Removing newrelic-sysmond ...
Stopping New Relic Server Monitor: newrelic-sysmond OKpuffin:~# totn
tn@puffin:~$ drush status
 PHP configuration     :  /opt/local/lib/php.ini
 Drush version         :  4.6-dev
 Drush configuration   :
 Drush alias files     :  /data/disk/tn/.drush/server_master.alias.drushrc.php
                          /data/disk/tn/.drush/space.transitionnetwork.org.alias.drushrc.php
                          /data/disk/tn/.drush/platform_TransitionNetworkD611.alias.drushrc.php
                          /data/disk/tn/.drush/platform_TransitionNetworkD6S002.alias.drushrc.php
                          /data/disk/tn/.drush/platform_TransitionNetworkD6P005.alias.drushrc.php
                          /data/disk/tn/.drush/platform_TransitionNetworkD6S006.alias.drushrc.php
                          /data/disk/tn/.drush/news.transitionnetwork.org.alias.drushrc.php
                          /data/disk/tn/.drush/platform_OpenAtrium7x20alpha5.alias.drushrc.php
                          /data/disk/tn/.drush/hostmaster.alias.drushrc.php
                          /data/disk/tn/.drush/stg.transitionnetwork.org.alias.drushrc.php
                          /data/disk/tn/.drush/platform_OpenAtriumOpenAtrium7x2xDEV13Oct2013.alias.drushrc.php
                          /data/disk/tn/.drush/platform_005.alias.drushrc.php
                          /data/disk/tn/.drush/platform_OpenAtrium2xAlpha5devJun18.alias.drushrc.php
                          /data/disk/tn/.drush/server_localhost.alias.drushrc.php
                          /data/disk/tn/.drush/platform_TransitionNetworkD6S005.alias.drushrc.php
                          /data/disk/tn/.drush/www.transitionnetwork.org.alias.drushrc.php
                          /data/disk/tn/.drush/platform_OpenAtrium2xGit.alias.drushrc.php

The last command demonstrates no warning about a missing .so file.

I have also commented out the .barracuda.cnf NR key, plus uncommented the empty NR key line.

I reckon this is gone... Did I miss anything Chris?

status changed; resolution set

chris — Fri, 15 Nov 2013 12:19:44 GMT

status changed from accepted to closed
resolution set to fixed

New Relic has been uninstalled, a link to this ticket has been added here wiki:PuffinServer#MuninStats now closing this ticket.

hours, priority, status, totalhours changed; resolution deleted

jim — Mon, 09 Dec 2013 16:24:27 GMT

hours changed from 0.0 to 0.25
priority changed from major to minor
status changed from closed to reopened
resolution fixed deleted
totalhours changed from 5.6 to 5.85

Drush plagued by this:

 >> PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

So on further investigation NR was installed still and appeared to be running...

I've now:

repeated the php.ini file changes per comment:27
moved /etc/newrelic to /root/scratch just in case
removed them aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond
purged them aptitude remove newrelic-daemon newrelic-php5 newrelic-php5-common newrelic-sysmond

I'm re-opening this ticket as 'minor' so we can track this -- it seems to be coming back for some reason, perhaps the recent Debian or BOA updates reinstated it.

Aware there's no time this month so stopping here, but we need to track this down and kill it. Parking until next BOA update.

chris — Sun, 15 Dec 2013 15:11:32 GMT

Replying to jim:

Drush plagued by this:

 >> PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

... we need to track this down and kill it. Parking until next BOA update.

Have the drush errors stopped? If they have this ticket can be closed.

hours, status, totalhours changed; resolution set

jim — Fri, 27 Dec 2013 17:10:11 GMT

hours changed from 0.0 to 0.05
status changed from reopened to closed
resolution set to fixed
totalhours changed from 5.85 to 5.9

Confirmed errors not happening since my fix, closing.