Transition Technology: Ticket #881: Site on ParrotServer with a memory leak?

attachment set

chris — Fri, 23 Oct 2015 11:19:42 GMT

attachment set to memory-pinpoint=1411038915,1445598915.png

attachment set

chris — Fri, 23 Oct 2015 11:21:51 GMT

attachment set to parrot-memory-pinpoint-1411038915-1445598915.png

status, description changed

chris — Fri, 23 Oct 2015 11:22:56 GMT

status changed from new to accepted
description modified (diff)

sam — Fri, 23 Oct 2015 11:35:08 GMT

Hi Chris
Are you able to pinpoint when it started? I guess it's some plugin I have
installed on the cop21 or dev sites.
I'll disable a bunch of plugins and see if that fixes it, then re-enable
them one by one.
I see that Apache use has risen dramatically today too:
https://penguin.transitionnetwork.org/munin/transitionnetwork.org/parrot.transitionnetwork.org/
Thanks
Sam
On 23 October 2015 at 12:22, Transition Technology Trac <
trac@tech.transitionnetwork.org> wrote:
> #881: Site on ParrotServer with a memory leak?
> -------------------------------------+-------------------------------------
>            Reporter:  chris          |                      Owner:  chris
>                Type:  maintenance    |                     Status:
>            Priority:  major          |  accepted
>           Component:  Parrot server  |                  Milestone:
>            Keywords:                 |  Maintenance
> Add Hours to Ticket:  0              |                 Resolution:
>         Total Hours:  0              |  Estimated Number of Hours:  0.0
>                                      |                  Billable?:  1
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * status:  new => accepted
>
>
> Old description:
>
> > It appears a site on ParrotServer might have a memory leak.
>
> New description:
>
>  It appears a site, or application, on ParrotServer might have a memory
>  leak.
>
>  [[Image(parrot-memory-pinpoint-1411038915-1445598915.png)]]
>
> --
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/881#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>

attachment set

chris — Fri, 23 Oct 2015 11:49:56 GMT

attachment set to multips_memory-pinpoint-1411040716-1445600716.png

attachment set

chris — Fri, 23 Oct 2015 11:52:14 GMT

attachment set to memory-pinpoint-1439680515-1446203715.png

hours, totalhours changed

chris — Fri, 23 Oct 2015 11:56:59 GMT

hours changed from 0.0 to 0.15
totalhours changed from 0.0 to 0.15

Looks like it started in August:

The first of the last two memory usage spikes started on Tuesday 1st September, and the last one on Tuesday 13th October, does that ring a bell?

attachment set

chris — Fri, 23 Oct 2015 12:02:24 GMT

attachment set to swap-pinpoint-1411041656-1445601656.png

hours, totalhours changed

chris — Fri, 23 Oct 2015 12:14:59 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.15 to 0.25

The memory usage is causing it to swap more:

These are the bandwidth stats:

 parrot  /  monthly
       month        rx      |     tx      |    total    |   avg. rate
    ------------------------+-------------+-------------+---------------
      Nov '14     59.22 GiB |    4.31 GiB |   63.54 GiB |  205.63 kbit/s
      Dec '14     50.64 GiB |    3.78 GiB |   54.42 GiB |  170.46 kbit/s
      Jan '15     55.43 GiB |    4.16 GiB |   59.59 GiB |  186.64 kbit/s
      Feb '15     50.01 GiB |    3.50 GiB |   53.51 GiB |  185.55 kbit/s
      Mar '15     59.65 GiB |    4.27 GiB |   63.92 GiB |  200.20 kbit/s
      Apr '15     54.71 GiB |    3.97 GiB |   58.68 GiB |  189.91 kbit/s
      May '15     94.76 GiB |   13.97 GiB |  108.73 GiB |  340.55 kbit/s
      Jun '15     86.88 GiB |   12.03 GiB |   98.92 GiB |  320.13 kbit/s
      Jul '15     57.50 GiB |    4.59 GiB |   62.09 GiB |  194.46 kbit/s
      Aug '15     62.12 GiB |    5.65 GiB |   67.77 GiB |  212.25 kbit/s
      Sep '15     72.12 GiB |    6.98 GiB |   79.10 GiB |  256.00 kbit/s
      Oct '15     57.61 GiB |    4.65 GiB |   62.25 GiB |  281.35 kbit/s
    ------------------------+-------------+-------------+---------------
    estimated     83.12 GiB |    6.70 GiB |   89.83 GiB |

Nothing unsual there though note it is over the data transfer/month allowance for the server of 60GB, last month it was 84.9GB, Dec 2014 and Feb 2015 have been the only two months in the last year when it has been under.

hours, totalhours changed

chris — Sat, 24 Oct 2015 12:34:18 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.25 to 0.75

The brute force attempts could well be related since the increase in memory usage appears to be directly related to the number of apache processes, see ticket:871, also on another server I have seen massive loads from huge number of POSTs to /xmlrpc.php, so I have written a little script to quantify this, /usr/local/bin/wp-xmlrpc-abuse:

#!/bin/bash

LOG_LINES="1000"
echo "IP addresses accessing xmlrpc.php more than twice for the last $LOG_LINES lines of each access.log:"
tail  -n $LOG_LINES /home/*/logs/access.log | grep "POST" | grep  "xmlrpc.php" | awk '{ print $1 }' | uniq -cd | rev | sort | rev

Running it and omitting the IP addresses that have only made 2 requests we have the following results (note each line represents on WordPress site)

wp-xmlrpc-abuse
IP addresses accessing xmlrpc.php more than twice for the last 1000 lines of each access.log:
      2 193.19.119.173
     13 193.19.119.173
    374 193.19.119.173
      6 193.19.119.173
     47 193.19.119.173

This is an IP address in Russia, I have blocked it using the ipdrop script:

ipdrop 193.19.119.173

Over the last month there have been 46k hits on /xmlrpc.php:

grep "xmlrpc.php" /home/*/logs/*access* | wc -l
46149

This plugin looks like a sensible option:

https://wordpress.org/plugins/stop-xmlrpc-attack/

I could install it on all the WP sites on the server?