Transition Technology: Ticket #903: Large load spike on PuffinServer

attachment set

chris — Mon, 08 Feb 2016 08:58:13 GMT

attachment set to puffin-2016-02-08_load-day.png

attachment set

chris — Mon, 08 Feb 2016 08:58:28 GMT

attachment set to puffin-2016-02-08_cpu-day.png

attachment set

chris — Mon, 08 Feb 2016 08:58:37 GMT

attachment set to puffin-2016-02-08_redis_commands-day.png

attachment set

chris — Mon, 08 Feb 2016 08:58:49 GMT

attachment set to puffin-2016-02-08_multips_memory-day.png

attachment set

chris — Mon, 08 Feb 2016 08:59:00 GMT

attachment set to puffin-2016-02-08_nginx_vhost_traffic-day.png

attachment set

chris — Mon, 08 Feb 2016 08:59:12 GMT

attachment set to puffin-2016-02-08_nginx_request-day.png

attachment set

chris — Mon, 08 Feb 2016 08:59:27 GMT

attachment set to puffin-2016-02-08_http_loadtime-day.png

attachment set

chris — Mon, 08 Feb 2016 08:59:39 GMT

attachment set to puffin-2016-02-08_fw_conntrack-day.png

attachment set

chris — Mon, 08 Feb 2016 08:59:52 GMT

attachment set to puffin-2016-02-08_mysql_qcache-day.png

attachment set

chris — Mon, 08 Feb 2016 09:00:03 GMT

attachment set to puffin-2016-02-08_mysql_queries-day.png

attachment set

chris — Mon, 08 Feb 2016 09:00:13 GMT

attachment set to puffin-2016-02-08_mysql_innodb_rows-day.png

attachment set

chris — Mon, 08 Feb 2016 09:00:24 GMT

attachment set to puffin-2016-02-08_mysql_innodb_io-day.png

hours, totalhours changed

chris — Mon, 08 Feb 2016 09:08:06 GMT

hours changed from 0.0 to 0.36
totalhours changed from 0.0 to 0.36

Between 7:55am and when I blocked the poneytelecom.eu IP address at 8:43am, 12,107 requests for PHP pages were made by this IP address, which reported multiple user agents. This fact alone, in my view, justifies blocking the IP address -- I very much doubt it was a transitioner mirroring the site if they had gone to the bother of using multiple random user agent strings -- it was abuse, a denial of service attack, that had the potential to seriously disrupt other users of the site.

I was sent 13 lfd load alerts, spiking at 23.03, following are some munin graphs of the load spike.

I installed a script, wiki:IpDrop to block the IP address, this records the blocked address in /root/Changelog.

ade — Tue, 09 Feb 2016 11:10:10 GMT

Awesome... many thanks for being on top of that Chris.
A
On 8 February 2016 at 09:08, Transition Technology Trac <
trac@tech.transitionnetwork.org> wrote:
> #903: Large load spike on PuffinServer
> -------------------------------------+-------------------------------------
>            Reporter:  chris          |                      Owner:  chris
>                Type:  maintenance    |                     Status:  new
>            Priority:  major          |                  Milestone:
>           Component:  Live server    |  Maintenance
>            Keywords:                 |                 Resolution:
> Add Hours to Ticket:  0.36           |  Estimated Number of Hours:  0.0
>         Total Hours:  0              |                  Billable?:  1
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.36
>  * totalhours:  0.0 => 0.36
>
>
> Comment:
>
>  Between 7:55am and when I blocked the poneytelecom.eu IP address at
>  8:43am, 12,107 requests for PHP pages were made by this IP address, which
>  reported multiple user agents. This fact alone, in my view, justifies
>  blocking the IP address -- I very much doubt it was a transitioner
>  mirroring the site if they had gone to the bother of using multiple random
>  user agent strings -- it was abuse, a denial of service attack, that had
>  the potential to seriously disrupt other users of the site.
>
>  I was sent 13 lfd load alerts, spiking at 23.03, following are some
>  [
> https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/index.html
>  munin graphs] of the load spike.
>
>  I installed a script, wiki:IpDrop to block the IP address, this records
>  the blocked address in `/root/Changelog`.
>
>  [[Image(puffin-2016-02-08_load-day.png)]]
>  [[Image(puffin-2016-02-08_cpu-day.png)]]
>  [[Image(puffin-2016-02-08_redis_commands-day.png)]]
>  [[Image(puffin-2016-02-08_multips_memory-day.png)]]
>  [[Image(puffin-2016-02-08_nginx_vhost_traffic-day.png)]]
>  [[Image(puffin-2016-02-08_nginx_request-day.png)]]
>  [[Image(puffin-2016-02-08_http_loadtime-day.png)]]
>  [[Image(puffin-2016-02-08_fw_conntrack-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_qcache-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_queries-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_innodb_rows-day.png)]]
>  [[Image(puffin-2016-02-08_mysql_innodb_io-day.png)]]
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/903#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>
--
Ade Stuart
Web Manager - Transition network
07595 331877
The Transition Network is a registered charity
address: 43 Fore St, Totnes, Devon, TQ9 5HN, UK
website: www.transitionnetwork.org
TN company no: 6135675 TN charity no: 1128675