Site on ParrotServer with a memory leak?

[chris]

It appears a site, or application, on ParrotServer might have a memory leak.

[sam]

Hi Chris

Are you able to pinpoint when it started? I guess it's some plugin I have
installed on the cop21 or dev sites.

I'll disable a bunch of plugins and see if that fixes it, then re-enable
them one by one.

I see that Apache use has risen dramatically today too:
https://penguin.transitionnetwork.org/munin/transitionnetwork.org/parrot.transitionnetwork.org/

Thanks

Sam

On 23 October 2015 at 12:22, Transition Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #881: Site on ParrotServer with a memory leak?
> -------------------------------------+-------------------------------------
>            Reporter:  chris          |                      Owner:  chris
>                Type:  maintenance    |                     Status:
>            Priority:  major          |  accepted
>           Component:  Parrot server  |                  Milestone:
>            Keywords:                 |  Maintenance
> Add Hours to Ticket:  0              |                 Resolution:
>         Total Hours:  0              |  Estimated Number of Hours:  0.0
>                                      |                  Billable?:  1
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * status:  new => accepted
>
>
> Old description:
>
> > It appears a site on ParrotServer might have a memory leak.
>
> New description:
>
>  It appears a site, or application, on ParrotServer might have a memory
>  leak.
>
>  [[Image(parrot-memory-pinpoint-1411038915-1445598915.png)]]
>
> --
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/881#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>

[chris]

Looks like it started in August:

The first of the last two memory usage spikes started on Tuesday 1st September, and the last one on Tuesday 13th October, does that ring a bell?

[chris]

The memory usage is causing it to swap more:

These are the bandwidth stats:

 parrot  /  monthly                                                                                                            
                                                                                                                               
       month        rx      |     tx      |    total    |   avg. rate                                                          
    ------------------------+-------------+-------------+---------------                                                       
      Nov '14     59.22 GiB |    4.31 GiB |   63.54 GiB |  205.63 kbit/s                                                       
      Dec '14     50.64 GiB |    3.78 GiB |   54.42 GiB |  170.46 kbit/s                                                       
      Jan '15     55.43 GiB |    4.16 GiB |   59.59 GiB |  186.64 kbit/s                                                       
      Feb '15     50.01 GiB |    3.50 GiB |   53.51 GiB |  185.55 kbit/s                                                       
      Mar '15     59.65 GiB |    4.27 GiB |   63.92 GiB |  200.20 kbit/s                                                       
      Apr '15     54.71 GiB |    3.97 GiB |   58.68 GiB |  189.91 kbit/s                                                       
      May '15     94.76 GiB |   13.97 GiB |  108.73 GiB |  340.55 kbit/s                                                       
      Jun '15     86.88 GiB |   12.03 GiB |   98.92 GiB |  320.13 kbit/s                                                       
      Jul '15     57.50 GiB |    4.59 GiB |   62.09 GiB |  194.46 kbit/s                                                       
      Aug '15     62.12 GiB |    5.65 GiB |   67.77 GiB |  212.25 kbit/s                                                       
      Sep '15     72.12 GiB |    6.98 GiB |   79.10 GiB |  256.00 kbit/s                                                       
      Oct '15     57.61 GiB |    4.65 GiB |   62.25 GiB |  281.35 kbit/s                                                       
    ------------------------+-------------+-------------+---------------                                                       
    estimated     83.12 GiB |    6.70 GiB |   89.83 GiB |  

Nothing unsual there though note it is over the data transfer/month allowance for the server of 60GB, last month it was ​84.9GB, Dec 2014 and Feb 2015 have been the only two months in the last year when it has been under.

[chris]

The brute force attempts could well be related since the increase in memory usage appears to be directly related to the number of apache processes, see ticket:871, also on another server I have seen massive loads from huge number of POSTs to /xmlrpc.php, so I have written a little script to quantify this, /usr/local/bin/wp-xmlrpc-abuse:

#!/bin/bash

LOG_LINES="1000"
echo "IP addresses accessing xmlrpc.php more than twice for the last $LOG_LINES lines of each access.log:"
tail  -n $LOG_LINES /home/*/logs/access.log | grep "POST" | grep  "xmlrpc.php" | awk '{ print $1 }' | uniq -cd | rev | sort | rev

Running it and omitting the IP addresses that have only made 2 requests we have the following results (note each line represents on WordPress site)

wp-xmlrpc-abuse 
IP addresses accessing xmlrpc.php more than twice for the last 1000 lines of each access.log:
      2 193.19.119.173
     13 193.19.119.173
    374 193.19.119.173
      6 193.19.119.173
     47 193.19.119.173

This is ​an IP address in Russia, I have blocked it using the ipdrop script:

ipdrop 193.19.119.173

Over the last month there have been 46k hits on /xmlrpc.php:

grep "xmlrpc.php" /home/*/logs/*access* | wc -l
46149

This plugin looks like a sensible option:

• ​https://wordpress.org/plugins/stop-xmlrpc-attack/

I could install it on all the WP sites on the server?