403s served to editors, admin very slow

[ed]

1. Rob is getting 403s when trying to submit his work. Report from 07:12am this morning (Tuesday)

2. Ed tried to add a blog post at node add:

​https://www.transitionnetwork.org/node/add/blog
It took nearly 15 seconds to get this published
​https://www.transitionnetwork.org/blogs/ed-mitchell/2013-07/eds-test-blog-item-check-403

3. Running admin functions takes ages.This request took well over 30 seconds:

​https://www.transitionnetwork.org/admin/content/node/overview

Please advise?

[chris]

Puffin MySQL Queries 2013-07-09

[chris]

Puffin Firewall Connections 2013-07-09

[chris]

Puffin Nginx Requests 2013-07-09

[chris]

Puffin php-fpm 2013-07-09

[chris]

Puffin Load 2013-07-09

[chris]

There was a very significant load spike this morning, and there was also a massive one yesterday, these graphs illustrate this:

Looking at the Nginx access log 59% of the requests since 6:30am today have been from one IP address in Belgium, which looks like a domestic ADSL IP, the reverse DNS ends in adsl-dyn.isp.belgacom.be and the User Agent string is rather unique, I'm not exactly sure what is going on here but I guess it's someone spidering the server:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36 Squider/0.01

I'll do some more work on analysing the nginx logs, following the work done on ticket:555.

Yesterday (6:30am yesterday to 6:30am today) there were 47 502 errors served and 3788 503 errors served. Since 6:30am today there have been 2 502 errors and 581 503 errors and 43 403 errors.

I'm in our office today if you want to ring to have a quick chat about this.

[ed]

Thanks Chris, I'm in Sheffield with you next week so let's talk then. Lots of catching up and budgets for me this week...

[chris]

35% of the hits to the server yesterday, 16433 in total, were from one domestic IP address in Belguim with "Squider" in their User-Agent string, it looks to me as if this was the primary cause of the load spike and the slow response times. I guess it's worth noting that it looks like a domestic internet connection is sufficient to mount a effective denial of service attack on the site (not that I have any idea for the reason for yesterdays spidering via "Squider", it might not have been malicious).

This ticket is probably safe to close, I have spent an hour looking at all the 403 errors from yesterday, see the detail below, I'm going to now do some more work on the web log analysing script via ticket:555.

The script which checks the Nginx logs for errors each day has had support for 403 errors added, see wiki:ErrorCodeCheck

The result summary from yesterday:

Date: Wed, 10 Jul 2013 06:25:28 +0100 
Subject: 770 403, 4219 404, 12 502, 2271 503 and 0 504 errors from puffin.webarch.net

Looking at the 770 403s there were 5 very large HEAD requests (requests for the HTTP header and not the body of a page) which look like they were trying to exploit a vulnerability or something:

"84.95.88.100" transitionnetwork.org [09/Jul/2013:06:50:59 +0100] "HEAD /..oaogbi.com.uasqueezeezy.co.ukwww.thaiebayuser.comwww.collegiogeometri.pg.itwww.cool1075.comwww.bmxp.netcrhmedical.co.ukgusetcie.canalblog.commuywww.xiyuit.commyonlystar.comwww.r-e-petersdorf.dewww.tyto.com.auwww.fbla.orgcofcircles.blogspot.comsearchposition.co.ukwww.cinerium.comwww.illamasqua.comcodify.co.ukukwide.co.ukxarizo.blogspot.co.ilm-c.lake.mn.uswww.stenametall.comadvancedmedia.co.ukwww.triadehotel.com.brwww.hodgkinsdisease.orgsoundpartners.co.ukwww.weinsheim.dewww.dieseldoctor.comfunction-design.co.ukfrankbarton.co.ukcasadacalli.blogspot.co.ilserver12.cyon.chwww.gfca-foot.commotherlovebone.netgirlsvideo.co.uksimon-barber.comzagham.comwww.sipreal.comwww.christianshavns-gymnasium.dkmipimliegeois.pingouin.globulebleu.comharveeriggs.comsmokefreeoregon.comwww.promopti.comrheumatism.co.ukwww.henhluyy.comweblognews.irciokorea.comdoddsie.comblackpear.co.ukthecoughlins.co.ukwww.antisectik.narod.ruwww.kingstoncounselling.co.ukwww.lab-13.dewww.hotheadstudios.comsupport.vaio.sony.euwww.edimond.rewebit.comwww.markmcdermottart.comsupersister.co.ukbmhh.med.sawww.fikirbaz.comprofytwebdesign.comcpms.ruwww.ifj.comtaichi18.comwww.project-audio.comcafe-the.co.ukwww.custer.com.augypframe.co.ukwww.star883.comksk-kusel.degiantrobot.com12345.rtysxs.netwww.faience-porcelaine.comvolnyvideo.czwww.district3.nbed.nb.casharepointvillage.comliugraveitaly.spaces.live.comwww.prjuliana.apeldoorn-onderwijs.nlwww.viajero.co.ukelw1zz4rd.blogspot.comwhyfame.us.intellitxt.compontaillersursaone.stationverte.comwww.keywestseaplanecharters.comwww.childbankaccount.netwww.arrabaldelpuente.comwww.danielmack.comfurthof-antikmoebel.dewww.nikitasam.comwww.sjzw.dewww.s-cat.ne.jpzandarvts.blogspot.comwww.eurosignal.czwww.hand-made.rummmimages.metcentral.comwww.boiron.cawww.environmentalsustainability.infowww.opera-starazagora.bgwww.springspree2011.blogspot.comwww.potless.talktalk.nethansler.co.ukfishfactory.orgwww.thill-optik.detintuc.tintucthitruong.com.vnsardarshahar.orgwww.starlinepacific.comintr.sps-br.czimages-cdn.digu.comwww.ofr.govwww.iproducts.wsjmconline.co.ukwww.tsv-volleyballer.dewww.seolm.orgwww.groovesharing.compartisan63.narod.rulandbank.comwww.t3g.comwww.travaux-et-decoration.comwww.musicoscopio.comokes.co.ukwww.hirvisaari.infowww.acerugcleaners.comsinnet.com.cnwww.tpointsolutions.netwww.bmwland.co.ukavanzacomunicacion.comcounter.mystat-in.netwww.stoffelsweb.dewww.grandsaltsjobaden.setoysforyou.co.ukthepiratebay.co.ukswwe.netwww.zanegreyinc.comwww.codagenomics.comviolet.ruwww.euroabschaffung.dewww.geolineinc.comwww.stmarysschool.co.zawww.initiative-rodachtal.dewww.rivingtonarms.comwww.gemueseschlacht.dewww.skrewdriver.netrecargatederegalos.comlittlemermaidlauren.tumblr.comsamscustoms.co.ukwww.ferialibrovalladolid.eswww.gvrgolf.comwww.welshguardsreunited.co.ukbarbieonline.co.uksovereign1.co.uklarissatora.comamarganim.co.ilfivesimplesteps.orgktengineering.co.ukwww.oldpilot.irdrupalcamp.lvwww.garnerclayton.comwww.history.kemsu.ruteleseminars.co.ukwww.mazeesagapegarden.comwww.polymathconsulting.comwww.runfor.jpdying.about.comleguidedelaville.comiris-fancluboficial.rowww.synaesthesie.chtextiles.org.twwww.broga.chchevalconfiance.free.frwww.pacha-hh.defishing-phuket.comkasu.otaku.atwww.clewetts.co.ukmybbkul.orggooglemini.co.ukjuliecox.free.frtotalybooks.co.ukdaphneanson.blogspot.co.ildundee-florists.co.uknuuvem.com.brwww.roess-nature-group.dewww.gccinn.comwww.swaythegame.comwww.newsmc.netrobertaustin.co.ukarmonster.co.ukeurex-trade.comwww.roemerturm-aufkirch.dewww.19yxw.comkidsgardening.orgwww.germandarioperez-trionuevacolombia.comslimerz.co.ukwww.studentcenter.inall-cube.co.ukwww.lightbox.ptshakeel.co.ukwww.automatenberufe.dewww.representtalent.comflowerhouse.com.twwww.astrazeneca.grcard-express.co.ukwww.theblakesociety.weebly.comsn2000.co.ukwww.imagesdemarc.netwww.fish-design.itwww.keyiforganizasyon.comwww.stribrna.czwww.910theduck.combuffy-uk.co.ukcondense.co.ukeropornonet.comwww.normandylets.comwww.thorupsmed.dkwww.fotoklub-nograd.hutioutwaha.blogspot.frdesign-share.comtwowayzoo.co.ukwww.archivodepunta.com.arpirata-azul.blogspot.comwww.cianbro.comislesford.combagbabes.co.ukhotpics4free.netwinddragon.narod.ruwww.jmhomeowner.comwww.1stopfinance.co.ukwww.riverbendsigns.comadvanceps.co.ukpavemententertainment.bigcartel.comwww.aso-aso.comretirementwatches.infowoodstockstories.comwww.wowter.netwww.mairie-compiegne.frwww.thedawn.co.ukmevo.narod.rusriley.co.ukjewellerycatalogue.co.uktransfusionandalternatives.comwhsmths.co.ukinfographicsonline.comwww.kenthurst-p.schools.nsw.edu.auwww.ecrinet.netreactor.reality-protocol.debip.mrr.gov.plv-italuna.tumblr.comwww.seakeepers-nz.comunifi.myjonnyhall.co.ukwww.eastmanonline.comwww.newspad.rowww.michaelcarman.comwww.uk-dns.comkii.ciao.jpp4-hofed27vwojjm-kj3eflctn4wimfnx-812516-i1-v6exp3-v4.metric.gstatic.comk-ff.comwww.freehotelsearch.commtesh.comwww.sjvianney.orgmultivite.co.ukbierzocomarca.euwww.online-a.comlerma.obspm.friwfestival.co.ukwww.lozere.orgwww.jennaskuchen-derfilm.dewww.dwain-chambers.co.ukwww.gratinglab.combenjamin-meyer.blogspot.comdiscountpal.co.ukwww.arselelektronik.com.trepistemelinks.netsoccercards.co.ukwww.stagehouserestaurant.comsportsuche.infoedenoriginal.co.ukfirmalojik.comsusancicconi.comiczoneradio.comwww.newayseurasia.comseasideparkyc.homestead.comwww.alambre-lashing.comwww.jowga.commaesglas.co.ukbradfordengland.co.ukmifleague.t35.comwww.maxizoo.frfliptopfilms.co.ukww34.jjsrockandpop.cominter01.adultplatinum.comen.kvb.ltwww.serbianunderground.comwww.civ-lauwersoog.nlwww.claroline.beeurobiketrans.co.ukfurukawa.co.jpcs306707.userapi.comwww.affiliatemarketingconference.org.ukwildner-designer.deliveshemaletube.comrioday.ruwww.handcli.comwww.photosynthesisresearch.orgwww.randbholidayhomes.comwww.ripar.comwww.beyond.co.atwww.protestingthedixiechicks.comwww.janegoodall-italia.orgsmspishkhan.irwww.snowblindthemovie.comwww.radiojazzclub.comww2.kolotv.comheartbypass.co.uktheoldparochialhouse.comhutchdano.comfairyblogmother.co.ukwww.jdcasten.infoddl-ltd.co.ukwww.belasica.iag.bgdejavu.vbi.vt.eduwww.jaluzele.shop.go.ronayminthu.blogspot.comwww.prima-messe.defolog.plwww.mitsuoharada.jpwoodwardatelier.comwww.flomag.czwww.davidcairns.comfreshescorts.co.ukwww.feudodegliulivi.comliz-hall.co.ukterryeibeck.comwww.leadprevention.orgspeedshooter.atgreenphotonicsguide.comwww.realpolitik.inbeersofrussia.co.ukgotscrubs.comqesaqead.ru100design.co.ukwww.leespharmacy.comusceline.comwww.gregel.comadam-rouilly.co.ukbrytwire.co.ukplay-lotto-free.co.ukwww.baney.co.ukspovednica.comnavegabem.comwww.agabekov.chwww.sunnygovan.orgkmlz-news.deiodinestudiesoperaalliance.orgcaptaingreen.co.ukstenciletching.co.ukhanazono-church.netandroidmaniac.netwww.kanazawachurch.orgnordseehenswertes.infocarstart.desearch.unive.it077football.co.ukwww.mobilityweek.euwww.kingdomadvisors.orgbestbrushesformakeup.comfunnyterra.ruwww.ottaviano.plwww.villacatalpa.decsaladiudules.huadcomputing.co.uksitysk.mu.nuwww.websoft.com.ngbesarani.co.ukwww.xiugongkong.comdemo.dotcontent.nl999register.co.ukwww.tradicionmorena.comobywatel.org.plwww.andrea-fleischmann.denorthcanton.patch.com31expo.combarbie.ksiezniczka.i.zebraczka.filmweb.plart-vending.ruwww.summitcemeterydistrict.comwww.jensnieth.deguadaloupe.orgwww.basketclubpaterno.itwww.thaipolyester.comrightflue.co.uksportsfish.co.ukhydroflow.co.ukwww.pniiss.ruyupiteru-ity.comwww.h-waki.comendeffect.co.ukwww.poddcorp.comdemo2.wishpond.comwww.oberverwaltungsgericht.bremen.debenjamin.boukpeti.free.frwww.fhs.lyon.k12.nv.uswww.sihorta.combadmanners.wswww2.geog.ucl.ac.ukhttp.www.alfajrnews.netwww.nct.gov.sdwww.diobesity.orgwww.naturns.netwww.superheromovies.netwww.blog.omahaconsort.orgwww.ekischo.dewww.pailton.org.ukwww.texasdhhresources.orgwww.sex2inc.comwww.foston.n-yorks.sch.ukwestpointgrey.orgwww.nlpmp3.comwhitegvardy.nm.ruwww.annenberg.northwestern.edubusinessservicesuk.comwww.freeonlinehitcounters.comlocal083.seiu503.orgwww.espinozapropiedades.com.arwww.read.org.twteach123-school.blogspot.comwww.rony-photographe.comwww.hot1073jamz.comwww.sobieskawola.piotrekb.com.plwww.deutsche-sprachwelt.demeredithhooper.co.ukwww.albergoraffaello.comwww.freebxml.orgttimbers.comyumuseum.orgwww.jspan.orgwww.fishfun.nlwww.rotomer-kayak.comabuse.rizon.netwww.vainzine2.blogspot.comxr1000.co.ukrocketfusion.co.ukwww.pornopedia.comwww.bdsbayern.deseabreezebandb.co.ukkrokodyl.netwww.cnnpolitics.comwww.accreditedcolleges.co.ukwww.aloak.cabcsl.com.cnwww.utahlp.orgleprechaunslinks.comwww.memoriaytolerancia.orgwww.south-africa.co.ukdebrissa.comwww.garyforcehonda.comwww.vittavi.netclickog.comoceanquest.nuwww.tashas-free-xxx-naked-girls.comwww.ciscoathletic.comwww.ganedenmaui.orgwww.anewarrival.comnickyhenderson.co.ukgunco-book.tripod.comwww.heusser-datendesign.dewww.noisetoys.comxvcontraxv.blogspot.comuimpi.netpadepokan-dewandaru.blogspot.comrlthursz.co.ukwww.aspg.comwww.freeling.sa.gov.autalkingmind.co.uklisten-to-music.co.ukemcaz.tvwww.casadeavila.comtemploevora3d.com.sapo.ptwww.quatorze.orgwww.under-horse.comwww.balletliztalfonso.cult.cuwww.peakphysiquefitness.comomertahistoriamafii.fm.interia.plcs323722.userapi.comwww.mlmdialog.ruwww.ve3syb.cawww.caketheater.comgoldencolorado.co.ukbip.lasy.gov.plbrianlynch.co.ukwww.ccnh.cnwww.signworld.com.twm1cs.co.ukwww.watermusicfestival.cominternships.unt.edudmt.mhilfe.dewww.eastsussexcc.gov.ukwww.wildwyrmart.comwww.ilovehongkong.hksamsclub.cnwww.classreunionprogram.comwww.maxiddaa.comwww.b93lazpak.blogspot.comwww.vmrc.nettulsa.backpage.comalloverart-nc.blogspot.co.ilblogopinia24.plpopmyip.comwww.marechalfloriano.es.gov.brring-it.co.ukwww.j-rietec.co.jproguevalleyrunners.blogspot.comunvjobs.comj.navdmp.comwww.wheresnowflakesdanceandswear.comnovtech.co.ukwww.jetsatboyatabancalari.comwww.gunradyotv.com.trsawtbladi.comasociacionbahiensedebasquetbol.netai.netwww.urwis.plwww.vincentsuryanto.combartwest.comguerrerossme.blogspot.mxdampbasement.co.uknev.appsumo.com.s3.amazonaws.comlibnds.devkitpro.orgwww.japanese-online.comwww.yuko-travel.rutwaddles.co.uknew.nostalgie.fmwww.freestylefotbal.czwww.heizoelboerse.dewww.barnsley.hud.ac.ukclick-2-let.co.ukwww.veedeepee.comniceperfume.dkpinot.co.ukwww.milbos-calyde.comwww.becauto.comwww.upse7e.com.brvantage-la.co.ukwww.youtradefx.comlasrminute.co.ukgemsofafrica.co.ukinfo.portea.frmotosmartzone.comwww.rocksite.ief.stf64photography.co.ukwww.kozeletslife.comwww.gekkoproductions.comwww.pausgarden.sewww.casasanchez.comwww.hitfm.comtrip.ustia.orgdiptara.comwww.lumerical.comwww.franklinautomationsystems.comwww.raw-clan.orgfirst-rates.co.ukwww.svorl.orgwww.scholar360.comlaguiadelpecado.netwww.meijishoin.co.jpwww.goinggreenfilmfestival.comfe15.news.sp1.yahoo.comwww.academiapr.org.brwww.willi-und-max.dewww.floridaarchery.orgspam.mypano.eswww.mainfurniturecompany.comwww.ekkemaass.dewww.ouvroirtemporairedephilosophie.comwww-app1.gfz-potsdam.dewww.rum.nlwww.uknowkids.comasy315.comadmision.ulagos.clwww.barra.ba.probrasil.com.brsfb591.ruhr-uni-bochum.dewww.barrioscoello.com.vejameshanby.co.ukwww.sesc.k12.in.uswww.georgemelly.netwww.mircophoto.itorkan.501.plwww.lososina.plwww.manolokabezabolo.esconnect.zazi.commido-sat.comwww.albatros-ul.dkpassport.dict.cnstockholmkarson.sewww.oremdentalcare.comrowery.eko.org.planalytics.lhp.huwww.toshaliholidays.comwww.adasacchi.com.arfilon.com.plwww.nottingham.edu.cnwww.semantic-web-journal.netka-zz.blogspot.jparbiteronline.commonkeydid.co.ukwww.curling.sports.or.krwww.mit.ps.au.dkwww.cilleruelodearriba.com1352.co.ukwww.bizzarriearmoniche.netwww.yumacabana.comcoebasketball.googlepages.comumkd.orgwww.gpaea.k12.ia.usk-x.co.ukwww.cap-net.orggr8christmas.co.uklidosol.co.ukbuchsblog2.blogspot.comnoblepics.co.ukgrayobrien.myfastforum.orgwww.malbud.siedlce.plalbertsao.multiply.comwww.witneyblanketstory.org.ukwww.elperiodicodeecija.eswww.solopan.com.plwww.tetra.netdannysworld.co.ukwww.fjtsc.co.jpwww.doooo.com.cnhooligansusa.comwww.finnmatkat.fiwww.highlandpc.orgwww.healey-classic.dewww.twogirlsonepint.comyazidbahri.jimdo.comskin-infections.co.ukcuttingtools.co.ukwww.consar.suwww.sibprofile.ruwww.optikmeirandres.devb36.co.ukcs302509.userapi.comdhammatalks.orgtown5.netwww.nepaltea.com.npwww.hamptonhill.comcarrollcountyohio.comwww.one-eins.comhappycloudmoments.blogspot.comgcsouth.co.ukwww.vidyaniketans.orgwww.callyourhusband.comkombatsport.luwww.ratemyprofessor.comwww.placeauxjeunes.mb.cahermesbirkin35online.comwww.southcoasttv.comwww.bazensbrewery.co.ukwww.bigfishonguide.comwww.polymathic.compart-two.co.ukforthoodareahabitat.orgairlinebooking.co.ukps3-mods.co.ukwww.slabute.rowww.tplshopping.comflops.nvidia.comwww.cintimha.comwww.davewood.cabeatrizia.lacoctelera.netwww.stevehubbardrealestate.commacef.fmi.itgilchristsoames.co.ukjoydivision-international-ag.dewww.cardingmillvalleynationaltrust.blogspot.comwww.donna-anna.comsuomenniemensrk.netwww.andongkcr.co.krad-networks.co.uktrojandiscounts.co.ukwww.inroadsfestival.comvega-msk.rucedarhr.co.uksodance.ruwww.bobrowen.comwww.suisansc.or.jpwww.wan.bemummydaycare.co.ukwww.weddingmall.co.ukwww.smslane.comjackbowmanofficial.moonfruit.comtarbydavenport.co.uktov.org.ilwww.hillsong.co.zawww.magazynwiatr.plgracecollection.co.ukwww.ihlienworth-info.dewww.jiggies.comthe-feathered-nest.blogspot.co.ilwww.kultur.lu.chwww.newsdownload.itdhf.odu.edu.trwww.schieren.infoaccountnow.tt.omtrdc.neteurobbc.tumblr.comgolfrules.co.ukwww.ursheer.comwww.forumolimpico.orgreadyat.comwww.dankleff-optik.dewww.gedoelektro.nlbigapplejazz.commein3.dasoertliche.deannebraypottery.comwww.cominofoundation.org.ukbadalook.blogspot.co.ilthesundaything.co.uktotallegal.co.ukstan1.upskirts.comwww.brancato.itwww.oshyn.comwww.calvaryindanville.comwww.800app.comastonishme.co.ukwww.tsugaike.netds154.nevinsk.rumarkdeutsch39.blogspot.comc-h-m.co.ukmcc19thbatchwww.islamichomeeducation.co.uksvstv.co.uknewsmall.co.ukwww.montesdelpardo.comwww.e-gyouza.comnovagames.co.ukpro-warez.ruofflineproperty.comfotosense.co.ukwww.knit-a-square.comshoei-helmets.co.ukmoshplant.commonthlyprivatephone.comwww.sportfreunde-goennersdorf.dewww.blueoceandirect.co.uklaotourism.orgwww.uncleboise.comdiplomaonline.co.ukwww.suncountryleisure.comww1.rip-productions.wswoodwardhouse.co.ukjatinsharma.inwww.aqua-specialists.comjustica.co.ukwww.australis.nettheartofmackin.comwww.nalco.com.jowww.mueden-mosel.dewww.heim-polle.comwww3.city.sanda.hyogo.jpwww.phifamily.comwww.takizawa-bokujo.jpaxcamers.comwww.realfootsex.comwww.fanzun.chachart.uswww.praats.bewww.aoglp.comwww.isionline.orgwww.lm-adv.co.ileverydayhosting.co.ukwww.budiutomo.comwww.cherkasy.osp.ua.infowww.festesdelcampello.eswww.perksofbeingawallflowerbook.comwww.timelessjazz.comitwbgk.co.ukremedycentre.co.uksiciliandreams.blogspot.co.ilwww.studiobarzan.itpaved.wordpress.commaghoo.co.ukpoetisasonhadora.blogs.sapo.ptyoulookgreat.co.ukescourt.co.ukwww.sossagradafamilia.orgwww.coalguru.comwww.egenfeldt.eudatindirect.co.ukdaretowear.co.ukprogramyzadarmo.net.pltaxilines.co.ukdptv.co.ukwww.frau-wirtschaft-weserbergland.despacebook.co.ukwww.solvang.dkwww.griffithstownharriers.co.ukwww.belmontdesign.co.ukwww.noescape.rateofinjury.comwww.akwaibomstatemn.orgokwales.co.ukdhanijones.tvwww.anthemon.huwww.national-tours.commga-consulting.co.ukwxguojizhaoshangcheng.soufun.comsedecatastro.gob.eswww.tax.client.jpvotergate.tvphpsource.co.ukwww.bdmanmag.comshopsonline.co.uksavethefraser.caquinsignac-asso.bzh.bzwww.cm-nelas.ptwww.global-net-concept.comitaliaannozero.orgable2know.commidwalesgallery.co.ukre-bornintheusa.co.ukincome-maker.infowww.affiniti-yacht.comsv.globalvoicesonline.orgwww.flwaterfrontdocks.comwww.uks-pingwiny.plqomuast.comto-drunk.blogspot.comwww.twp.cranberry.pa.uswww.snoco.orgbasystems.co.ukwww.bespeka.comwww.bwpr.nowww.solohomes.co.ukwww.zfighters.chgs-heilpraktiker.dewww.vertexsoftware.comja.delta.comwww.chicagobusinesscapital.comwww.junkpickup.capfaffrail.googlepages.comwww.soldout-online.co.ukschoolwearforus.co.ukwww.gorevette.comstylebot.mewww.wildtrout.co.zawww.gadot.org.nzcableracing.co.ukavoid.co.ukbpnavi.jpwww.axis-dp.co.jpwww.lancasterdems.comwww.bestinfo.attreatmentforcoldsore.comteewars.en.softonic.cominfo.walkerart.orgtouchterminal.co.ukstfranciswakefield.comwww.plgna.orgwww.cesmeo.itcidu791.comdtocc.comwww.cdu-bernkastel-wittlich.dewww.cinema-excellence.benancythomas.blogspot.comwww.ungis.orgspam.robertchristgau.comwww.ottawaimprov.comwww.milwaukeepublishers.com92tu.comwww.paardentoerisme.nlwww.casalevillarena.comwww.goreyguardian.iewww.liciopasson.itejobopenings.comnyflw.cnmikemamaril.commyroof.co.ukkkc-eutelsat.euwahyain.comwww.ahlabaht.orgwww.wiiparty.netwww.zfacts.comwww.allsports.nlamazingwelcome.co.ukmoi.gov.omwww.vigga.nowww.maculata.dewww.nichigi.co.jpspanish4u.co.ukwww.martijnrichters.nlcrazy4sport.co.ukyhol.org.ilwomensdev.blogsmith.comwww.hmptech.private.plwww.coping.atwww.halat.co.ilaapkikismat.comwww.health.gov.ckwww.awesomecalgary.orgfleet-dynamix.co.ukwww.fvpreussen-eberswalde.demcfc.comwww.kreativportfolio.decienciaescolar.netbloglatiendadelbebe.blogspot.combigassdating.comfjordline.co.ukwww.campinglido.comwww.lapadania.itwww.hopevalley.sa.scouts.com.auwww.lautlos-durch-deutschland.dewww.argenttosystems.commagnetgroup.co.ukwww.mackmyra.senoldus.comitwalksbyitself.tumblr.comwww.moslim.sekoeche-bayern.dewww.cactusov.netwww.lepsy.comwww.fairplaytennis.supanet.commshindo.netwww.sloka.siwww.pohja.fiparadepharmacy.co.ukwww.muziekverenigingconcordia.nlwww.dalypearson.co.uklifequiz.co.ukstartfarm.co.uknwlimo.co.ukwww.tejascaddo.orgnews.cealo-ngo.orgesprit-sg.nlwww.tolemaritma.comdirectorzone.cyberlink.comwww.feuerwehr-langfoerden.dewww.goitschel.comwww.kaigoai.jpwww.dbrunningcompany.comwelldonegroup.co.ukhowtomobi.comkaren-murphy.co.ukbrookcorporate.co.ukendeavourgroup.co.ukstpetersparishny.orgblog.gwup.netwww.atci.comxxxix.netmail.zsto.ruwww.vimat.chwww.pgaprofessionalsguide.travelwww.scothorn.comolejar.euwww.gamereversal.comcardiff.panthers.com.auseahorse-olympics.tumblr.comwww.okswat.comwetzel.magix.netwww.engelskasetterklubben.sewww.cialne.com.brwww.russadir.netleadnews64.comwww.etl-dom-cup.deleandro.vcwww.kinokami.frukvatrefunds.co.ukblog.lyjy.gov.cnosydenidushi.hit.bgeverquestgames.co.ukwww.pearlflutes.comwww.cellandbioscience.comlakotaonline.comwww.worldprivacyforum.orgnymphenburg.dewww.yamatokougei.co.jpkudoscars.co.ukwww.inspiair.comwww.phonebookofcroatia.comwww.tennis.aedev.junowebdesign.comunofficialsteampacket.webs.comtanzmitmir.netwww.cegos.frwww.durell.co.ukavalonpier.commeri110.blogfa.comwww.aatw.comalphatext.dewww.dennisdawson.comwww.iusc.inbarrelhousemag.comwww.sluttyteenz.comhealthy2wealthy.co.ukwww.docip.orgwww.ssv-didderse.deromashka.org.uawww.cg17.frflykamair.comwww.jongmkbliemersachterhoek.nlwww.michelpoirier.comwww.jjxianglong.comwww.margrethereedtzskolen.skoleintra.dkwww.forosdz.comremedeine.co.ukwww.indianidolfan.comwww.buyskateshoes.comrichwise.com.auwww.foiredesedan.frwww.irfanfilm.comwww.marketing-movil-sms.comwww.fcgardner.co.ukkrypton-der-film.de.vubenvue.co.ukwww.k-i-z.comwww.elektrownie.website.plyoungscience.ruwww.mwbconsulting.co.ukstudz.comhomeofthegroove.blogspot.comdanlambaovn.blogspot.comlistok.comminecraftworldseeds.infosensorynetworks.co.ukpremierpot.co.ukwww.resolutefp.comanzmbos.blogspot.commicrosatzetron.co.ukwww.world-net.netcdn2.spotflux.comnas.co.ukpaulrusca.co.ukwww.chrismclernon.comwww.echtzeitkultur.orgcst-consulting.co.ukwww.futurlab.co.ukisland-images.co.ukcorron.d303.orgmobpartner.comlacasaenlaplaza.co.ukitaniumservers.co.ukslimpickings.co.ukwww.deltapen.itwww.phillipdavies.co.ukbreizh-tifosi.forums-actifs.netanvato.comwww.sgg.ccyomoyama.ajet.netwww.smokingpaper.comrawfooddiettips.comsharesong.orgwaveseat.co.ukwww.probowling.dewww.mairie-cintegabelle.frhotstudpix.comwww.owensports.co.ukwww.sportspalace.nlwww.dr-rittershaus.dewww.barnperspektivet.sef1-autocentre.co.ukreve.sans.fin.free.frvillaventura.co.ukamarsh.co.ukwww.miamilawyer.comblog.manuelbustos.catwww.matag-immobilien.decalegroup.co.ukfour-sided-triangle.tumblr.comcotswoldcakes.co.ukwww.karabuluttekstil.comwww.symiphotos.comfriendesha.comwww.cwspr.comwww.arbeitserzieher.dewww.moringa.com.brwww.hodgesgardens.comwww.scienzeformazione.unifg.itwww.whatgamedoyougame.comwww.duevittorie.itfantasytoyland.comwww.lojatudopesca.comwww.willi-hausmann.dewww.knzb-zro.nlwww.islagrand.comwww.foodleindia.comwww.nirvana-lyrics.netyarnmanufacturing.blogspot.comwww.olistage.itdzien.proby.filmweb.plamrsresearch.co.ukwww.copt.netscottbond.co.ukisheiyisrael.orgwww.topix.itwww.fordsguideservice.comwww.chellozone.comwww.brodart.ruwww.mortifera.infosevilworld.co.ukeagreen.co.ukimg6.macrojuegos.comwww.maturestreamsex.commedicedu.ruwww.blacklightworld.comhimalayatrekkers.comsignitynetworks.posterous.comwww.manorresources.co.ukwww.asterikstudio.comi-pharmacy.co.ukwww.birdinitaly.netwww.croix-blanche-baldersheim.orgwww.2barbowl.comwww.urmakermester.comwebhusband.co.ukwww.altongas.comwww.wrestlerhub.comdrurylane.co.ukwww.universanimal.comthebill.comfilmedu.cnwww.psychotherapiepruefung.online.dewww.parichoy.comlesbianpicture.co.ukwww.kulturzentrum-altstadt.deconwaystewartpens.co.ukwww.rcciws.comthelwordonline.dewww.hakomagazine.netwww.microgenbioproducts.comwww.vandekasteele.comyoulhwadang.co.krwww.noy1877.amwww.psimanpsiman.pwp.blueyonder.co.ukwww.kirklippoldforcongress.comguyellisrocks.comwww.malligihotels.comhostxp.co.ukwww.agorasavaria.huwww.niu.ac.jpshoelovers.co.ukmrsdvd.blogspot.comwww.spd-bgl.demagicinc.co.ukwww.eadshome.comasu.iate.obninsk.rubaike.jinku.comraghugh.photographers.comvkksu.gov.uawww.bigdickbitch.comcasenet.comwww.visitnunavut.comall-articles-directory.comwww.communication-lyon.frwww.advancedphotography.nettab2000.co.uks.nhanhmedia.comlegislatoredmangano.com24gonline.comzygomycetes.orgwww.cqdjournal.cometom.co.ukfunabi.ac.jpuroradiology.co.ukvwl.co.ukspilt.co.ukwww.thepeacefulwarriormovie.comwww.portaldaraquete.com.brwww.kie.nu HTTP/1.0" 403 0 22118 171 "http://transitionnetwork.org/" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; fr-fr) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16" 0.372 "-"

There were 148 403s served to a small number of bots requesting /forum/rss.php, this is a URL from the old forum, eg ​http://2011.archive.transitionnetwork.org/forum/rss.php?forum=20 some examples of these requests:

"46.20.121.175" www.transitionnetwork.org [09/Jul/2013:18:31:25 +0100] "GET /forum/rss.php?forum=20 HTTP/1.0" 403 162 124 328 "" "-" 0.011 "-"
"46.20.114.134" www.transitionnetwork.org [09/Jul/2013:18:49:21 +0100] "GET /forum/rss.php?forum=29 HTTP/1.0" 403 162 124 328 "" "-" 0.013 "-"
"82.31.231.189" www.transitionnetwork.org [09/Jul/2013:21:20:09 +0100] "GET /forum/rss.php?forum=29 HTTP/1.1" 403 134 337 336 "-" "Apple-PubSub/65.28" 0.000 "1.32"
"54.226.44.93" www.transitionnetwork.org [09/Jul/2013:22:35:28 +0100] "GET /forum/rss.php HTTP/1.1" 403 134 210 336 "-" "rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)" 0.000 "1.32"
"54.224.159.218" www.transitionnetwork.org [10/Jul/2013:02:30:40 +0100] "GET /forum/rss.php?topic=236 HTTP/1.0" 403 123 159 292 "-" "linkdex.com/v2.0" 0.080 "1.32"

63 403s were served in response to requests for /robots.txt to two bots, 53 to MJ12bot, ​http://www.majestic12.co.uk/bot.php and 10 to Nutch, I guess there is some setting in BOA to forbid these bots for some reason, a couple of examples of these requests:

"91.250.82.106" www.transitionnetwork.org [09/Jul/2013:11:11:04 +0100] "GET /robots.txt HTTP/1.0" 403 123 257 292 "-" "OpenWebIndex/Nutch-1.6" 0.028 "1.32"
"108.59.8.70" www.transitionnetwork.org [09/Jul/2013:11:43:42 +0100] "GET /robots.txt HTTP/1.0" 403 162 207 328 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)" 0.111 "-"

There were 8 requests for the front page from ​http://crawler.sistrix.net/ which were forbidden, eg:

"5.9.112.180" news.transitionnetwork.org [09/Jul/2013:07:20:27 +0100] "GET / HTTP/1.1" 403 162 192 333 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)" 0.000 "-"

There were 278 requests for a URL with "register" in it which were forbidden, these will be mostly spambots eg:

"218.65.30.239" www.transitionnetwork.org [10/Jul/2013:00:40:47 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 189 356 391 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"218.65.30.239" www.transitionnetwork.org [10/Jul/2013:00:45:21 +0100] "GET /register.php HTTP/1.1" 403 189 261 391 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"175.44.29.253" www.transitionnetwork.org [10/Jul/2013:04:20:26 +0100] "GET /blogs/ucp.php?mode=register&change_lang=en HTTP/1.1" 403 189 297 391 "http://www.transitionnetwork.org/blogs/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"175.44.29.253" www.transitionnetwork.org [10/Jul/2013:04:20:26 +0100] "GET /ingredients/ucp.php?mode=register&change_lang=en HTTP/1.1" 403 189 309 391 "http://www.transitionnetwork.org/ingredients/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"175.44.29.253" www.transitionnetwork.org [10/Jul/2013:04:20:26 +0100] "GET /news/ucp.php?mode=register&change_lang=en HTTP/1.1" 403 189 295 391 "http://www.transitionnetwork.org/news/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"175.44.29.51" www.transitionnetwork.org [10/Jul/2013:06:06:27 +0100] "GET /people/ucp.php?mode=register&change_lang=en HTTP/1.1" 403 189 299 391 "http://www.transitionnetwork.org/people/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"27.153.208.115" www.transitionnetwork.org [10/Jul/2013:05:57:58 +0100] "GET /blog/login.php?part=register HTTP/1.1" 403 189 494 391 "http://www.transitionnetwork.org/blog/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"27.153.208.115" www.transitionnetwork.org [10/Jul/2013:05:54:50 +0100] "GET /blog/tiki-register.php HTTP/1.1" 403 189 417 391 "http://www.transitionnetwork.org/blog/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"

There were 25 requests with "join" in the request, again spambots I expect, eg:

"121.205.249.200" www.transitionnetwork.org [09/Jul/2013:09:19:33 +0100] "GET /stories/join.php HTTP/1.1" 403 134 408 336 "http://www.transitionnetwork.org/stories/join.php" "Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0" 0.000 "1.32"
"27.159.253.82" www.transitionnetwork.org [09/Jul/2013:09:15:43 +0100] "GET /join.php HTTP/1.1" 403 134 392 336 "http://www.transitionnetwork.org/join.php" "Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0" 0.000 "1.32"

There were 34 requests with "signup" in the request, againn spambots, eg:

"121.205.249.200" www.transitionnetwork.org [09/Jul/2013:09:20:37 +0100] "GET /stories/signup.php HTTP/1.1" 403 189 364 391 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"110.90.12.90, 127.0.0.1" www.transitionnetwork.org [09/Jul/2013:08:45:25 +0100] "GET /trac/signup.php HTTP/1.0" 403 178 442 347 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.001 "3.17"

There were 39 requests for a URL containing "login", again spambots, eg:

"121.205.249.200" www.transitionnetwork.org [09/Jul/2013:09:21:33 +0100] "GET /stories/index.php?app=core&module=global&section=login HTTP/1.1" 403 189 471 391 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"27.153.162.75" www.transitionnetwork.org [09/Jul/2013:15:21:48 +0100] "GET /login.php HTTP/1.1" 403 189 214 391 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"
"27.153.162.75" www.transitionnetwork.org [09/Jul/2013:15:21:57 +0100] "GET /member.php?mod=logging&action=login HTTP/1.1" 403 189 284 391 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)" 0.000 "3.17"

There were 8 !WebDAV requests which were rightly forbidden, eg:

"121.134.135.141" www.transitionnetwork.org [09/Jul/2013:11:49:52 +0100] "PROPFIND / HTTP/1.1" 403 162 172 333 "-" "Microsoft-WebDAV-MiniRedir/6.1.7601" 0.000 "-"

The remaining 190 requests are mostly bots requesting URLs related to other CMSs with known vunerabilities, eg:

"5.135.178.45" www.transitionnetwork.org [09/Jul/2013:07:05:48 +0100] "GET /wp-content/themes/network/library/timthumb.php?src=http://picasa.com.mauritanic.com/user.php HTTP/1.1" 403 564 265 730 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320)" 0.011 "-"

And requests for things that are only available to logged in users and are correctly forbidden when people haven't logged in.

I haven't see any things that indicate that 403s need further investigation, but it's possible that I have missed something.

[chris]

This ticket has been added to the list of load related tickets at the end of wiki:PuffinServer#LoadSpikes and a ist of the all the recent 403 totals has been posted for reference on ticket:483#comment:63 -- we are still getting between 3-4k 403's a day but there haven't been complaints from people so most, if not all, of these will be served to robots. I thionk this ticket is now safe to close.