Ticket #881 (accepted maintenance)

Opened 13 months ago

Last modified 13 months ago

Site on ParrotServer with a memory leak?

Reported by: chris Owned by: chris
Priority: major Milestone: Maintenance
Component: Parrot server Keywords:
Cc: sam, ade Estimated Number of Hours: 0.0
Add Hours to Ticket: 0 Billable?: yes
Total Hours: 0.75

Description (last modified by chris) (diff)

It appears a site, or application, on ParrotServer might have a memory leak.


Change History

Changed 13 months ago by chris

Changed 13 months ago by chris

comment:1 Changed 13 months ago by chris

  • Status changed from new to accepted
  • Description modified (diff)

comment:4 Changed 13 months ago by sam

Hi Chris

Are you able to pinpoint when it started? I guess it's some plugin I have
installed on the cop21 or dev sites.

I'll disable a bunch of plugins and see if that fixes it, then re-enable
them one by one.

I see that Apache use has risen dramatically today too:
https://penguin.transitionnetwork.org/munin/transitionnetwork.org/parrot.transitionnetwork.org/

Thanks

Sam

On 23 October 2015 at 12:22, Transition Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #881: Site on ParrotServer with a memory leak?
> -------------------------------------+-------------------------------------
>            Reporter:  chris          |                      Owner:  chris
>                Type:  maintenance    |                     Status:
>            Priority:  major          |  accepted
>           Component:  Parrot server  |                  Milestone:
>            Keywords:                 |  Maintenance
> Add Hours to Ticket:  0              |                 Resolution:
>         Total Hours:  0              |  Estimated Number of Hours:  0.0
>                                      |                  Billable?:  1
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * status:  new => accepted
>
>
> Old description:
>
> > It appears a site on ParrotServer might have a memory leak.
>
> New description:
>
>  It appears a site, or application, on ParrotServer might have a memory
>  leak.
>
>  [[Image(parrot-memory-pinpoint-1411038915-1445598915.png)]]
>
> --
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/881#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>

Changed 13 months ago by chris

Changed 13 months ago by chris

comment:5 Changed 13 months ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.15
  • Total Hours changed from 0.0 to 0.15

Looks like it started in August:


The first of the last two memory usage spikes started on Tuesday 1st September, and the last one on Tuesday 13th October, does that ring a bell?


Changed 13 months ago by chris

comment:6 Changed 13 months ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.1
  • Total Hours changed from 0.15 to 0.25

The memory usage is causing it to swap more:


These are the bandwidth stats:

 parrot  /  monthly                                                                                                            
                                                                                                                               
       month        rx      |     tx      |    total    |   avg. rate                                                          
    ------------------------+-------------+-------------+---------------                                                       
      Nov '14     59.22 GiB |    4.31 GiB |   63.54 GiB |  205.63 kbit/s                                                       
      Dec '14     50.64 GiB |    3.78 GiB |   54.42 GiB |  170.46 kbit/s                                                       
      Jan '15     55.43 GiB |    4.16 GiB |   59.59 GiB |  186.64 kbit/s                                                       
      Feb '15     50.01 GiB |    3.50 GiB |   53.51 GiB |  185.55 kbit/s                                                       
      Mar '15     59.65 GiB |    4.27 GiB |   63.92 GiB |  200.20 kbit/s                                                       
      Apr '15     54.71 GiB |    3.97 GiB |   58.68 GiB |  189.91 kbit/s                                                       
      May '15     94.76 GiB |   13.97 GiB |  108.73 GiB |  340.55 kbit/s                                                       
      Jun '15     86.88 GiB |   12.03 GiB |   98.92 GiB |  320.13 kbit/s                                                       
      Jul '15     57.50 GiB |    4.59 GiB |   62.09 GiB |  194.46 kbit/s                                                       
      Aug '15     62.12 GiB |    5.65 GiB |   67.77 GiB |  212.25 kbit/s                                                       
      Sep '15     72.12 GiB |    6.98 GiB |   79.10 GiB |  256.00 kbit/s                                                       
      Oct '15     57.61 GiB |    4.65 GiB |   62.25 GiB |  281.35 kbit/s                                                       
    ------------------------+-------------+-------------+---------------                                                       
    estimated     83.12 GiB |    6.70 GiB |   89.83 GiB |  

Nothing unsual there though note it is over the data transfer/month allowance for the server of 60GB, last month it was 84.9GB, Dec 2014 and Feb 2015 have been the only two months in the last year when it has been under.

comment:7 Changed 13 months ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.5
  • Total Hours changed from 0.25 to 0.75

The brute force attempts could well be related since the increase in memory usage appears to be directly related to the number of apache processes, see ticket:871, also on another server I have seen massive loads from huge number of POSTs to /xmlrpc.php, so I have written a little script to quantify this, /usr/local/bin/wp-xmlrpc-abuse:

#!/bin/bash

LOG_LINES="1000"
echo "IP addresses accessing xmlrpc.php more than twice for the last $LOG_LINES lines of each access.log:"
tail  -n $LOG_LINES /home/*/logs/access.log | grep "POST" | grep  "xmlrpc.php" | awk '{ print $1 }' | uniq -cd | rev | sort | rev

Running it and omitting the IP addresses that have only made 2 requests we have the following results (note each line represents on WordPress site)

wp-xmlrpc-abuse 
IP addresses accessing xmlrpc.php more than twice for the last 1000 lines of each access.log:
      2 193.19.119.173
     13 193.19.119.173
    374 193.19.119.173
      6 193.19.119.173
     47 193.19.119.173

This is an IP address in Russia, I have blocked it using the ipdrop script:

ipdrop 193.19.119.173

Over the last month there have been 46k hits on /xmlrpc.php:

grep "xmlrpc.php" /home/*/logs/*access* | wc -l
46149

This plugin looks like a sensible option:

I could install it on all the WP sites on the server?

Last edited 13 months ago by chris (previous) (diff)
Note: See TracTickets for help on using tickets.