Apache pcre segfaults

[chris]

Yesterday apache / pcre was segfaulting, an extract from /var/log/messages, follows, in all there were 51 segfault log lines, starting at Feb 1 17:44:34 and ending at Feb 1 19:21:08, I haven't been able to work out the cause of this, I'm opening this ticket in case we see more segfaults and do want to follow it up.

Feb  1 17:44:34 quince kernel: [11153359.760960] apache2[8193]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:34 quince drupal: https://workspaces.transitionnetwork.org|1328118274|user|XXX.XXX.XXX.XXX|https://workspaces.transitionnetwork.org/customerror/403?destination=node%2F116|https://workspaces.transitionnetwork.org/ws/national-hubs|0||Login attempt failed for James Samuel.
Feb  1 17:44:35 quince kernel: [11153361.102389] apache2[8182]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:36 quince kernel: [11153362.535735] apache2[8198]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:38 quince kernel: [11153364.036616] apache2[7160]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:43 quince drupal: https://workspaces.transitionnetwork.org|1328118283|user|XXX.XXX.XXX.XXX|https://workspaces.transitionnetwork.org/customerror/403?destination=node%2F116|https://workspaces.transitionnetwork.org/customerror/403?destination=node%2F116|0||Login attempt failed for James Samuel.
Feb  1 17:44:46 quince kernel: [11153372.256252] apache2[7212]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:48 quince kernel: [11153374.450117] apache2[8214]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]
Feb  1 17:44:49 quince kernel: [11153375.778416] apache2[8215]: segfault at 7fffa4382f50 ip 7f950dd7c16e sp 7fffa4382ef0 error 6 in libpcre.so.3.12.1[7f950dd6b000+30000]

An extract from the /var/log/apache/eeeor.log:

[Wed Feb 01 17:44:34 2012] [notice] child pid 8193 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:36 2012] [notice] child pid 8182 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:37 2012] [notice] child pid 8198 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:39 2012] [notice] child pid 7160 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:47 2012] [notice] child pid 7212 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:49 2012] [notice] child pid 8214 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:50 2012] [notice] child pid 8215 exit signal Segmentation fault (11)
[Wed Feb 01 17:44:52 2012] [notice] child pid 8217 exit signal Segmentation fault (11)

The corresponding apache access log for the first min:

127.0.0.1 - - [01/Feb/2012:17:44:00 +0000] "GET /tags/behaviour-change HTTP/1.1" 200 7542 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:01 +0000] "GET /cron.php HTTP/1.0" 302 - "-" "Wget/1.12 (linux-gnu)"
127.0.0.1 - - [01/Feb/2012:17:44:02 +0000] "GET /notifications/anonymous/subscribe/author/author/10548?destination=node%2F17790&timestamp=1327498024&signature=a1db08a827cdbaf608b5ae75df760481 HTTP/1.1" 200 5881 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:04 +0000] "GET /blogs/energy-bulletin?page=73 HTTP/1.1" 200 8935 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:06 +0000] "GET /notifications/anonymous/subscribe/author/author/8220?destination=node%2F13477&timestamp=1328117196&signature=92f5e293767d5afa02d60fae9a72ae0c HTTP/1.1" 200 5878 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:07 +0000] "GET /people/monika-kirsch HTTP/1.1" 200 6504 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:12 +0000] "GET /query/entity_ids-location/json/1/52.511012,-1.469337_0.5/10 HTTP/1.1" 200 17 "-" "Drupal (+http://drupal.org/)"
127.0.0.1 - - [01/Feb/2012:17:44:12 +0000] "GET /initiatives/nuneaton-transition-town HTTP/1.1" 200 9355 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
127.0.0.1 - - [01/Feb/2012:17:44:13 +0000] "GET /favicon.ico HTTP/1.1" 302 272 "-" "Safari/7534.52.7 CFNetwork/520.2.5 Darwin/11.2.0 (x86_64) (MacBookPro8%2C2)"
127.0.0.1 - - [01/Feb/2012:17:44:15 +0000] "GET /node/all/by_subject/1538?order=title&sort=asc HTTP/1.1" 200 6145 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:15 +0000] "GET /news/2011-01-26/send-rob-your-posters HTTP/1.1" 200 7851 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:17 +0000] "GET /node/all/by_subject/428?order=comment_count&sort=desc HTTP/1.1" 200 5992 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:17 +0000] "GET /query/entity_ids-location/json/1/50.633501,3.788273_1/10?page=3 HTTP/1.1" 200 9142 "-" "Drupal (+http://drupal.org/)"
127.0.0.1 - - [01/Feb/2012:17:44:19 +0000] "GET /people/facilitators?page=64 HTTP/1.1" 200 7488 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:19 +0000] "GET /sources/uk/deal-it?page=%2C7%2C9%2C2%2C2%2C7 HTTP/1.1" 200 5364 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:22 +0000] "GET /node/all/by_subject/1798?order=last_comment_timestamp&sort=desc HTTP/1.1" 200 5904 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:23 +0000] "GET /people/mike-miller HTTP/1.1" 200 6540 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:24 +0000] "GET /node/all/by_subject/1350?order=type&sort=desc HTTP/1.1" 200 5689 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:26 +0000] "GET /node/all/by_subject/236?order=last_comment_timestamp&sort=asc HTTP/1.1" 200 5711 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:27 +0000] "GET /cms/bellingen HTTP/1.1" 200 8468 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
127.0.0.1 - - [01/Feb/2012:17:44:28 +0000] "GET /node/all/by_subject/3341?order=last_comment_timestamp&sort=desc HTTP/1.1" 200 5692 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:29 +0000] "GET /people/alex-cicelsky HTTP/1.1" 200 6249 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:34 +0000] "GET /piwik.js HTTP/1.1" 200 7779 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.1; .NET4.0C)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:34 +0000] "GET /piwik.php?action_name=Welcome%20%7C%20Transition%20Network&idsite=1&rec=1&r=680344&h=11&m=44&s=34&url=http%3A%2F%2Fwww.transitionnetwork.org%2F&urlref=http%3A%2F%2Fwww.theglobeandmail.com%2Fnews%2Fopinions%2Fopinion%2Four-peak-oil-premium%2Farticle2321815%2Fcomments%2F&_id=218ab4fa03a9a340&_idts=1328118274&_idvc=1&_idn=1&_refts=1328118274&_viewts=1328118274&_ref=http%3A%2F%2Fwww.theglobeandmail.com%2Fnews%2Fopinions%2Fopinion%2Four-peak-oil-premium%2Farticle2321815%2Fcomments%2F&java=1&res=1069x855&cookie=1 HTTP/1.1" 200 50 "http://www.transitionnetwork.org/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.1; .NET4.0C)"
::1 - - [01/Feb/2012:17:44:35 +0000] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
127.0.0.1 - - [01/Feb/2012:17:44:35 +0000] "GET /notifications/anonymous/subscribe/author/author/4545?destination=node%2F4907&timestamp=1327729023&signature=7de36d2b702fd311d9e29c0b07839c66 HTTP/1.1" 200 5896 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:36 +0000] "GET /people/abduselam-abdai-mohammed HTTP/1.1" 200 7457 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:37 +0000] "GET /node/all/by_subject/2400?order=last_comment_timestamp&sort=asc HTTP/1.1" 200 5707 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:40 +0000] "GET /node/15409/by_subject/3007?order=comment_count&sort=desc HTTP/1.1" 200 5857 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:41 +0000] "GET /people/sophy-banks HTTP/1.1" 200 7352 "http://www.transitionnetwork.org/news" "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"
127.0.0.1 - - [01/Feb/2012:17:44:43 +0000] "GET /user/register?destination=comment%2Freply%2F21723%23comment-form HTTP/1.1" 301 307 "http://www.transitionnetwork.org/news" "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"
127.0.0.1 - - [01/Feb/2012:17:44:42 +0000] "GET /notifications/anonymous/subscribe/author/author/3016?destination=node%2F3378&timestamp=1328116976&signature=135af8f9b16253b1f60ca6161cac7141 HTTP/1.1" 200 5882 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:44 +0000] "GET /blogs/energy-bulletin/2011-12-23?page=22 HTTP/1.1" 200 9119 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:44 +0000] "GET /sharing/widget_news HTTP/1.1" 200 1095 "http://transitionnorwichnews.blogspot.com/2012/01/where-are-all-fruit-and-nut-trees.html" "Mozilla/5.0 (Windows NT 5.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /node/all/by_subject/2409?order=title&sort=asc HTTP/1.1" 200 5818 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /query/entity_ids-location/rss/1/50.431213,-3.685455_1/10 HTTP/1.1" 200 2016 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; Microsoft Outlook 14.0.6109; ms-office; MSOffice 14)"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /notifications/anonymous/subscribe/author/author/1064?destination=node%2F1427&timestamp=1327506623&signature=1b484e663e414fc697ce6363edf7e2b4 HTTP/1.1" 200 5878 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:45 +0000] "GET /notifications/anonymous/subscribe/author/author/3461?destination=node%2F3823&timestamp=1322606638&signature=483b9c34a74cbf27a739746b8ea9588a HTTP/1.1" 200 5874 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:46 +0000] "GET /tags/bus-transportation HTTP/1.1" 200 6911 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:47 +0000] "GET /resources/branding-guidelines?order=field_asset_type_value&sort=asc&page=1&themes_nid[0]=13 HTTP/1.1" 200 10076 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:51 +0000] "GET /piwik.js HTTP/1.1" 200 7779 "http://www.transitionnetwork.org/blogs/ben-brangwyn/2011-03/faith-movie-weve-all-been-waiting" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB7.2; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET4.0C; .NET CLR 3.0.30729; AskTB5.6)"
::1 - - [01/Feb/2012:17:44:52 +0000] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"
XXX.XXX.XXX.XXX - - [01/Feb/2012:17:44:51 +0000] "GET /piwik.php?action_name=The%20faith%20movie%20we've%20all%20been%20waiting%20for!%20%7C%20Transition%20Network&idsite=1&rec=1&r=327050&h=12&m=44&s=53&url=http%3A%2F%2Fwww.transitionnetwork.org%2Fblogs%2Fben-brangwyn%2F2011-03%2Ffaith-movie-weve-all-been-waiting&urlref=http%3A%2F%2Fwww.google.ca%2Fimgres%3Fimgurl%3Dhttp%3A%2F%2Ffr.academic.ru%2Fpictures%2Ffrwiki%2F71%2FGreat_Synagogue_Plzen_CZ.jpg%26imgrefurl%3Dhttp%3A%2F%2Fwww.transitionnetwork.org%2Fblogs%2Fben-brangwyn%2F2011-03%2Ffaith-movie-weve-all-been-waiting%26h%3D1556%26w%3D1387%26sz%3D346%26tbnid%3DDyxrYgfcASljuM%3A%26tbnh%3D90%26tbnw%3D80%26prev%3D%2Fsearch%253Fq%253D(synagogue)%2526tbm%253Disch%2526tbo%253Du%26zoom%3D1%26q%3D(synagogue)%26docid%3D4kDZHm-yl2nChM%26hl%3Dfr%26sa%3DX%26ei%3DvnkpT--mG4rG0AHM1dXUAg%26ved%3D0CFYQ9QEwBw%26dur%3D1076&_id=4ecf13956b5e2ee4&_idts=1328118293&_idvc=1&_idn=1&_refts=1328118293&_viewts=1328118293&_ref=http%3A%2F%2Fwww.google.ca%2Fimgres%3Fimgurl%3Dhttp%3A%2F%2Ffr.academic.ru%2Fpictures%2Ffrwiki%2F71%2FGreat_Synagogue_Plzen_CZ.jpg%26imgrefurl%3Dhttp%3A%2F%2Fwww.transitionnetwork.org%2Fblogs%2Fben-brangwyn%2F2011-03%2Ffaith-movie-weve-all-been-waiting%26h%3D1556%26w%3D1387%26sz%3D346%26tbnid%3DDyxrYgfcASljuM%3A%26tbnh%3D90%26tbnw%3D80%26prev%3D%2Fsearch%253Fq%253D(synagogue)%2526tbm%253Disch%2526tbo%253Du%26zoom%3D1%26q%3D(synagogue)%26docid%3D4kDZHm-yl2nChM%26hl%3Dfr%26sa%3DX%26ei%3DvnkpT--mG4rG0AHM1dXUAg%26ved%3D0CFYQ9QEwBw%26dur%3D1076&java=1&res=1366x768&cookie=1 HTTP/1.1" 200 50 "http://www.transitionnetwork.org/blogs/ben-brangwyn/2011-03/faith-movie-weve-all-been-waiting" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB7.2; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET4.0C; .NET CLR 3.0.30729; AskTB5.6)"
127.0.0.1 - - [01/Feb/2012:17:44:51 +0000] "GET /tags/reedbed HTTP/1.1" 200 6483 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:52 +0000] "GET /news/network/feed HTTP/1.1" 200 6004 "-" "Apple-PubSub/65.28"
127.0.0.1 - - [01/Feb/2012:17:44:52 +0000] "GET /people/mim-meilland HTTP/1.1" 200 6555 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:53 +0000] "GET /tags/may-celebration HTTP/1.1" 200 6412 "https://www.transitionnetwork.org/user/login" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/5.0)"
127.0.0.1 - - [01/Feb/2012:17:44:53 +0000] "GET /people/irene-lim HTTP/1.1" 404 44619 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:55 +0000] "GET /notifications/anonymous/subscribe/author/author/2791?destination=node%2F3153&timestamp=1328109179&signature=bb861cf6a1a3d856bc959f44c72f017e HTTP/1.1" 200 5887 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [01/Feb/2012:17:44:55 +0000] "GET /user/login?destination=comment%2Freply%2F14129%23comment-form HTTP/1.1" 301 307 "http://tones.transitionnetwork.org/tags/may-celebration" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/5.0)"
127.0.0.1 - - [01/Feb/2012:17:44:57 +0000] "GET /sources/uk/deal-it?page=%2C7%2C9%2C2%2C2%2C5 HTTP/1.1" 200 5364 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
127.0.0.1 - - [01/Feb/2012:17:44:59 +0000] "GET /user/12203/contact HTTP/1.1" 301 278 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
127.0.0.1 - - [01/Feb/2012:17:44:59 +0000] "GET /tags/arts-and-culture HTTP/1.1" 200 7813 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"