Large load spike on PuffinServer

[chris]

There was a large load spike this morning on PuffinServer, which appears to have been caused by 12k requests for pages (Nginx doesn't log requests for anything other than PHP generated pages) from one IP address, this IP address has been blocked and I'll post some details below.

[chris]

Between 7:55am and when I blocked the poneytelecom.eu IP address at 8:43am, 12,107 requests for PHP pages were made by this IP address, which reported multiple user agents. This fact alone, in my view, justifies blocking the IP address -- I very much doubt it was a transitioner mirroring the site if they had gone to the bother of using multiple random user agent strings -- it was abuse, a denial of service attack, that had the potential to seriously disrupt other users of the site.

I was sent 13 lfd load alerts, spiking at 23.03, following are some ​munin graphs of the load spike.

I installed a script, wiki:IpDrop to block the IP address, this records the blocked address in /root/Changelog.

[ade]

Awesome... many thanks for being on top of that Chris.

A

On 8 February 2016 at 09:08, Transition Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #903: Large load spike on PuffinServer
> -------------------------------------+-------------------------------------
>            Reporter:  chris          |                      Owner:  chris
>                Type:  maintenance    |                     Status:  new
>            Priority:  major          |                  Milestone:
>           Component:  Live server    |  Maintenance
>            Keywords:                 |                 Resolution:
> Add Hours to Ticket:  0.36           |  Estimated Number of Hours:  0.0
>         Total Hours:  0              |                  Billable?:  1
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.36
>  * totalhours:  0.0 => 0.36
>
>
> Comment:
>
>  Between 7:55am and when I blocked the poneytelecom.eu IP address at
>  8:43am, 12,107 requests for PHP pages were made by this IP address, which
>  reported multiple user agents. This fact alone, in my view, justifies
>  blocking the IP address -- I very much doubt it was a transitioner
>  mirroring the site if they had gone to the bother of using multiple random
>  user agent strings -- it was abuse, a denial of service attack, that had
>  the potential to seriously disrupt other users of the site.
>
>  I was sent 13 lfd load alerts, spiking at 23.03, following are some
>  [
> https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/index.html
>  munin graphs] of the load spike.
>
>  I installed a script, wiki:IpDrop to block the IP address, this records
>  the blocked address in `/root/Changelog`.
>
>  [[Image(puffin-2016-02-08_load-day.png)]]
>  [[Image(puffin-2016-02-08_cpu-day.png)]]
>  [[Image(puffin-2016-02-08_redis_commands-day.png)]]
>  [[Image(puffin-2016-02-08_multips_memory-day.png)]]
>  [[Image(puffin-2016-02-08_nginx_vhost_traffic-day.png)]]
>  [[Image(puffin-2016-02-08_nginx_request-day.png)]]
>  [[Image(puffin-2016-02-08_http_loadtime-day.png)]]
>  [[Image(puffin-2016-02-08_fw_conntrack-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_qcache-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_queries-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_innodb_rows-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_innodb_io-day.png)]]
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/903#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>



-- 
Ade Stuart
Web Manager - Transition network

07595 331877

The Transition Network is a registered charity
address: 43 Fore St, Totnes, Devon, TQ9 5HN, UK
website: www.transitionnetwork.org
TN company no: 6135675 TN charity no: 1128675