Transition Technology: Ticket #548: All Admin functions broken on TN.org

hours, totalhours changed

chris — Tue, 14 May 2013 09:11:33 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.0 to 0.25

The only thing that I'm aware of that has changed in the last few days was the BOA upgrade which was done on Sunday night, see ticket:547#comment:4

Browsing the site I keep getting redirects from HTTPS to HTTP, however these are blocked by the https://addons.mozilla.org/en-US/firefox/addon/requestpolicy/ plugin I have in Firefox and this enables me to use the admin interface normally.

Here is an example of the redirect, I clicked a link to https://www.transitionnetwork.org/node/add/blog and my broswer sent the following GET request (the session cookie string has been replaced with a made up one for security):

GET /node/add/blog HTTP/1.1
Host: www.transitionnetwork.org
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20100101 Firefox/20.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Cookie: SESStai8xeimetief7coh8pheev7siu0li2daocailiew7koo0see2xierucheeceTusiW; LOGGED_IN=1; has_js=1
Connection: keep-alive

And the response is a redirect to the home page:

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 14 May 2013 09:05:58 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Backend: C
X-Allow-Redis: YES
X-Cookie-Domain: .transitionnetwork.org
X-Redis-Prefix: www.transitionnetwork.org_
X-Purge-Level: 6
X-Local-Proto: https
Location: http://www.transitionnetwork.org/
Last-Modified: Tue, 14 May 2013 09:05:56 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Etag: "1368522356"
X-Engine: Octopus 1.0 ET
X-Device: normal
X-Speed-Cache: BYPASS
X-Speed-Cache-UID: Olouveikoh7lariCh2eef4iiv2zai2ais;
X-Speed-Cache-Key: /node/add/blog
X-NoCache: Skip
X-This-Proto: https
X-Server-Name: www.transitionnetwork.org
Vary: Accept-Encoding

More investigation is needed...

jim — Tue, 14 May 2013 09:35:08 GMT

I've tried a few things but can't do much from the UI. I'm looking at it
but don't have my SSH key at work so can't do much until tonight.
It feels like a module issue, or a server config thing -- I notice that all
the admin pages are available, provided you use the old, non-'clean url'
urls, e.g.
   - Normal: https://www.transitionnetwork.org/admin/content/node/overview
   - Tweaked: https://www.transitionnetwork.org/*?q=*
   admin/content/node/overview
Note the *?q= *between the domain and the path -- that will work for any
page.
So this means Drupal is fine, but either it, a module or the server is
redirecting /admin and /node/*/edit paths. Odd.
Ed, when did this last work?
On 14 May 2013 10:11, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:
> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.25           |                  Billable?:  1
>         Total Hours:  0              |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.25
>  * totalhours:  0.0 => 0.25
>
>
> Comment:
>
>  The only thing that I'm aware of that has changed in the last few days was
>  the BOA upgrade which was done on Sunday night, see ticket:547#comment:4
>
>  Browsing the site I keep getting redirects from HTTPS to HTTP, however
>  these are blocked by the https://addons.mozilla.org/en-
>  US/firefox/addon/requestpolicy/ plugin I have in Firefox and this enables
>  me to use the admin interface normally.
>
>  Here is an example of the redirect, I clicked a link to
>  https://www.transitionnetwork.org/node/add/blog and my broswer sent the
>  following GET request (the session cookie string has been replaced with a
>  made up one for security):
>
>  {{{
>  GET /node/add/blog HTTP/1.1
>  Host: www.transitionnetwork.org
>  User-Agent: Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20100101
>  Firefox/20.0
>  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>  Accept-Language: en-gb,en;q=0.5
>  Accept-Encoding: gzip, deflate
>  DNT: 1
>  Cookie:
>  SESStai8xeimetief7coh8pheev7siu0li2daocailiew7koo0see2xierucheeceTusiW;
>  LOGGED_IN=1; has_js=1
>  Connection: keep-alive
>  }}}
>
>  And the response is a redirect to the home page:
>
>  {{{
>  HTTP/1.1 301 Moved Permanently
>  Server: nginx
>  Date: Tue, 14 May 2013 09:05:58 GMT
>  Content-Type: text/html; charset=utf-8
>  Transfer-Encoding: chunked
>  Connection: keep-alive
>  X-Backend: C
>  X-Allow-Redis: YES
>  X-Cookie-Domain: .transitionnetwork.org
>  X-Redis-Prefix: www.transitionnetwork.org_
>  X-Purge-Level: 6
>  X-Local-Proto: https
>  Location: http://www.transitionnetwork.org/
>  Last-Modified: Tue, 14 May 2013 09:05:56 +0000
>  Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
>  Etag: "1368522356"
>  X-Engine: Octopus 1.0 ET
>  X-Device: normal
>  X-Speed-Cache: BYPASS
>  X-Speed-Cache-UID: Olouveikoh7lariCh2eef4iiv2zai2ais;
>  X-Speed-Cache-Key: /node/add/blog
>  X-NoCache: Skip
>  X-This-Proto: https
>  X-Server-Name: www.transitionnetwork.org
>  Vary: Accept-Encoding
>  }}}
>
>  More investigation is needed...
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>
--
Jim Kirkpatrick
i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

hours, totalhours changed

chris — Tue, 14 May 2013 09:39:06 GMT

hours changed from 0.0 to 0.2
totalhours changed from 0.25 to 0.45

This ticket might be related, https://drupal.org/node/1989254 the BOA issue queue is here http://bit.ly/boa-iq

ed — Tue, 14 May 2013 09:46:30 GMT

Last time I used it was Thurs 09/05. Social Reporters tried to use it Monday 13/5 with no luck.

Replying to jim:

I've tried a few things but can't do much from the UI. I'm looking at it
but don't have my SSH key at work so can't do much until tonight.
It feels like a module issue, or a server config thing -- I notice that all
the admin pages are available, provided you use the old, non-'clean url'
urls, e.g.
   - Normal: https://www.transitionnetwork.org/admin/content/node/overview
   - Tweaked: https://www.transitionnetwork.org/*?q=*
   admin/content/node/overview
Note the *?q= *between the domain and the path -- that will work for any
page.
So this means Drupal is fine, but either it, a module or the server is
redirecting /admin and /node/*/edit paths. Odd.
Ed, when did this last work?
On 14 May 2013 10:11, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:
> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.25           |                  Billable?:  1
>         Total Hours:  0              |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.25
>  * totalhours:  0.0 => 0.25
>
>
> Comment:
>
>  The only thing that I'm aware of that has changed in the last few days was
>  the BOA upgrade which was done on Sunday night, see ticket:547#comment:4
>
>  Browsing the site I keep getting redirects from HTTPS to HTTP, however
>  these are blocked by the https://addons.mozilla.org/en-
>  US/firefox/addon/requestpolicy/ plugin I have in Firefox and this enables
>  me to use the admin interface normally.
>
>  Here is an example of the redirect, I clicked a link to
>  https://www.transitionnetwork.org/node/add/blog and my broswer sent the
>  following GET request (the session cookie string has been replaced with a
>  made up one for security):
>
>  {{{
>  GET /node/add/blog HTTP/1.1
>  Host: www.transitionnetwork.org
>  User-Agent: Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20100101
>  Firefox/20.0
>  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>  Accept-Language: en-gb,en;q=0.5
>  Accept-Encoding: gzip, deflate
>  DNT: 1
>  Cookie:
>  SESStai8xeimetief7coh8pheev7siu0li2daocailiew7koo0see2xierucheeceTusiW;
>  LOGGED_IN=1; has_js=1
>  Connection: keep-alive
>  }}}
>
>  And the response is a redirect to the home page:
>
>  {{{
>  HTTP/1.1 301 Moved Permanently
>  Server: nginx
>  Date: Tue, 14 May 2013 09:05:58 GMT
>  Content-Type: text/html; charset=utf-8
>  Transfer-Encoding: chunked
>  Connection: keep-alive
>  X-Backend: C
>  X-Allow-Redis: YES
>  X-Cookie-Domain: .transitionnetwork.org
>  X-Redis-Prefix: www.transitionnetwork.org_
>  X-Purge-Level: 6
>  X-Local-Proto: https
>  Location: http://www.transitionnetwork.org/
>  Last-Modified: Tue, 14 May 2013 09:05:56 +0000
>  Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
>  Etag: "1368522356"
>  X-Engine: Octopus 1.0 ET
>  X-Device: normal
>  X-Speed-Cache: BYPASS
>  X-Speed-Cache-UID: Olouveikoh7lariCh2eef4iiv2zai2ais;
>  X-Speed-Cache-Key: /node/add/blog
>  X-NoCache: Skip
>  X-This-Proto: https
>  X-Server-Name: www.transitionnetwork.org
>  Vary: Accept-Encoding
>  }}}
>
>  More investigation is needed...
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>
--
Jim Kirkpatrick
i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

ed — Tue, 14 May 2013 09:48:04 GMT

good tip about URLs - ta - very timely

jim — Tue, 14 May 2013 09:50:03 GMT

Interesting... My initial hunch was a 443 Session module issue, but the
settings on there look good.
However, it needs our local.setting.php in the
~/static/transition-00X/sites/www.transitionnetwork.org/local.setting.phphaving
the following in it:
   if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] != 'off') {
        ini_set('session.cookie_secure', 1);
      }
This was in our /data/conf/override.global.inc file instead, as that's
included in all sites we host and they curently will all need this.
Chris, could you try commenting out the above lines in
/data/conf/override.global.inc and try to use /admin/* URLs? If that
doesn't work we should uncomment them.
Also, have you tried the restarting php53-fpm service? And maybe
redis-server service?

hours, totalhours changed

chris — Tue, 14 May 2013 10:16:16 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.45 to 0.95

I have raised a ticket here https://drupal.org/node/1994346

jim — Tue, 14 May 2013 10:35:03 GMT

Good work on the ticket, I've added extra info.
BUT Chris, you'll need to attach the key files if you want support!!!
From the new Issue page for Octopus (
https://drupal.org/node/add/project-issue/octopus):
> Please note that any bug report or support request failing to follow the
guidelines will be ignored and closed without any answer.
> Before reporting a bug always search for similar bug report before
submitting your own, and include as much information about your context as
possible. Especially please include the contents (anonymized for security
and privacy) of files:
> /data/disk/USER/log/octopus_log.txt
> /var/aegir/config/includes/barracuda_log.txt
> /root/.USER.octopus.cnf
On 14 May 2013 11:16, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:
> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.5            |                  Billable?:  1
>         Total Hours:  0.45           |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.5
>  * totalhours:  0.45 => 0.95
>
>
> Comment:
>
>  I have raised a ticket here https://drupal.org/node/1994346
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:7>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>
--
Jim Kirkpatrick
i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

hours, totalhours changed

chris — Tue, 14 May 2013 11:24:57 GMT

hours changed from 0.0 to 0.35
totalhours changed from 0.95 to 1.3

Replying to jim:

Chris, could you try commenting out the above lines in /data/conf/override.global.inc and try to use /admin/* URLs? If that doesn't work we should uncomment them.

I tried commenting it out, not expecting this to change anything -- all it is doing is setting the secure flag on the admin cookie, and this is something that we want -- and it didn't change the behaviour.

Also, have you tried the restarting php53-fpm service? And maybe redis-server service?

These will have been restarted during the upgrade so I don't think this will help, but I have restarted them in any case.

Replying to jim:

you'll need to attach the key files if you want support!!!

Have done, https://drupal.org/node/1994346#comment-7410060

Next thing to do is to read through the Nginx configuration to see if a Nginx redirect can be found.

jim — Tue, 14 May 2013 12:05:03 GMT

Added a edit to my post to cover that secure session cookie test:
https://drupal.org/node/1994346#comment-7409920
So now we have to wait I suppose... I'll have a look as soon as I get home
around 6pm, and will keep an eye on the various threads.
Best,
Jim
On 14 May 2013 12:24, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:
> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.35           |                  Billable?:  1
>         Total Hours:  0.95           |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.35
>  * totalhours:  0.95 => 1.3
>
>
> Comment:
>
>  Replying to [comment:6 jim]:
>  >
>  > Chris, could you try commenting out the above lines in
>  > /data/conf/override.global.inc and try to use /admin/* URLs? If that
>  > doesn't work we should uncomment them.
>
>  I tried commenting it out, not expecting this to change anything -- all it
>  is doing is setting the secure flag on the admin cookie, and this is
>  something that we want -- and it didn't change the behaviour.
>
>  > Also, have you tried the restarting php53-fpm service? And maybe
>  > redis-server service?
>
>  These will have been restarted during the upgrade so I don't think this
>  will help, but I have restarted them in any case.
>
>  Replying to [comment:8 jim]:
>  >
>  > you'll need to attach the key files if you want support!!!
>
>  Have done, https://drupal.org/node/1994346#comment-7410060
>
>  Next thing to do is to read through the Nginx configuration to see if a
>  Nginx redirect can be found.
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:9>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>
--
Jim Kirkpatrick
i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

hours, totalhours changed

chris — Tue, 14 May 2013 12:08:32 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.3 to 1.55

Following the Nginx config includes:

/etc/nginx/nginx.conf
- /etc/nginx/mime.types
- /etc/nginx/conf.d/aegir.conf
  - /var/aegir/config/server_master/nginx/pre.d/nginx_speed_purge.conf
  - /var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf
  - /var/aegir/config/server_master/nginx/platform.d/tn.conf -> /data/disk/tn/config/tn.nginx.conf
    - /data/disk/tn/config/server_master/nginx/vhost.d/news.transitionnetwork.org
      - /data/disk/tn/config/includes/fastcgi_params.conf
      - /data/disk/tn/config/includes/nginx_octopus_include.conf
    - /data/disk/tn/config/server_master/nginx/vhost.d/stg.transitionnetwork.org
      - /data/disk/tn/config/includes/fastcgi_params.conf
      - /data/disk/tn/config/includes/nginx_octopus_include.conf
    - /data/disk/tn/config/server_master/nginx/vhost.d/tn.puffin.webarch.net
      - /data/disk/tn/config/includes/fastcgi_params.conf
      - /data/disk/tn/config/includes/nginx_modern_include.conf
    - /data/disk/tn/config/server_master/nginx/vhost.d/www.transitionnetwork.org
      - /data/disk/tn/config/includes/fastcgi_params.conf
      - /data/disk/tn/config/includes/nginx_octopus_include.conf
  - /var/aegir/config/server_master/nginx/vhost.d/cgp.master.puffin.webarch.net
    - /var/aegir/config/includes/fastcgi_params.conf
    - /var/aegir/config/includes/nginx_compact_include.conf
  - /var/aegir/config/server_master/nginx/vhost.d/chive.master.puffin.webarch.net
    - /var/aegir/config/includes/fastcgi_params.conf
    - /var/aegir/config/includes/nginx_compact_include.conf
  - /var/aegir/config/server_master/nginx/vhost.d/master.puffin.webarch.net
    - /var/aegir/config/includes/fastcgi_params.conf
    - /var/aegir/config/includes/nginx_modern_include.conf
  - /var/aegir/config/server_master/nginx/post.d/* (yes there is a files called *)
- /etc/nginx/sites-enabled/ (this directory doesn't exist)

This is a sorted unique list of the files to check:

/data/disk/tn/config/includes/fastcgi_params.conf
/data/disk/tn/config/includes/nginx_modern_include.conf
/data/disk/tn/config/includes/nginx_octopus_include.conf
/data/disk/tn/config/server_master/nginx/vhost.d/news.transitionnetwork.org
/data/disk/tn/config/server_master/nginx/vhost.d/stg.transitionnetwork.org
/data/disk/tn/config/server_master/nginx/vhost.d/tn.puffin.webarch.net
/data/disk/tn/config/server_master/nginx/vhost.d/www.transitionnetwork.org
/etc/nginx/conf.d/aegir.conf
/etc/nginx/mime.types
/etc/nginx/nginx.conf
/var/aegir/config/includes/fastcgi_params.conf
/var/aegir/config/includes/nginx_compact_include.conf
/var/aegir/config/includes/nginx_modern_include.conf
/var/aegir/config/server_master/nginx/pre.d/nginx_speed_purge.conf
/var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf
/var/aegir/config/server_master/nginx/vhost.d/cgp.master.puffin.webarch.net
/var/aegir/config/server_master/nginx/vhost.d/chive.master.puffin.webarch.net
/var/aegir/config/server_master/nginx/vhost.d/master.puffin.webarch.net

milestone changed

ed — Tue, 14 May 2013 13:07:13 GMT

milestone changed from Production to Maintenance

hours, totalhours changed

chris — Tue, 14 May 2013 13:49:21 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.55 to 1.8

Reading through the 3000+ lines of Nginx config, the key file for port 443 is /var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf and Nginx is basically set up as a reverse proxy to itself.

But I'm sorry I can't see what is causing the redirects from port 80 to port 443.

hours, totalhours changed

chris — Tue, 14 May 2013 18:17:57 GMT

hours changed from 0.0 to 0.28
totalhours changed from 1.8 to 2.08

The Redirect documented in ticket:548#comment:1 is a 301, so grepping the Nginx for "permanent" (a 301 is a permanent redirect), there are the following redirects in 3 different files, /data/disk/tn/config/includes/nginx_modern_include.conf, /data/disk/tn/config/includes/nginx_octopus_include.conf and /var/aegir/config/includes/nginx_modern_include.conf:

###
### Deny access to Hostmaster web/db server node.
### It is still possible to edit or break web/db server
### node at /node/2/edit, if you know what are you doing.
###
location ^~ /hosting/c/server_master {
  access_log off;
  rewrite ^ $scheme://$host/hosting/sites permanent;
}
###
### Deny access to Hostmaster db server node.
### It is still possible to edit or break db server
### node at /node/4/edit, if you know what are you doing.
###
location ^~ /hosting/c/server_localhost {
  access_log off;
  rewrite ^ $scheme://$host/hosting/sites permanent;
}
###
### Deny cache details display.
###
location ^~ /admin/settings/performance/cache-backend {
  access_log off;
  rewrite ^ $scheme://$host/admin/settings/performance permanent;
}
###
### Deny cache details display.
###
location ^~ /admin/config/development/performance/redis {
  access_log off;
  rewrite ^ $scheme://$host/admin/config/development/performance permanent;
}
###
### Send all known bots to $args free URLs.
###
location @nobots {
  if ($is_bot) {
    rewrite ^ $scheme://$host$uri? permanent;
  }
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

None of the above look to me like they might be the cause of the problem.

ed — Tue, 14 May 2013 19:25:42 GMT

I'm looking for some Jim love here too, yes?

attachment set

jim — Tue, 14 May 2013 19:28:24 GMT

attachment set to 2.0.8-2.0.9.diff

Git Diff of nginx-for-drupal BOA 2.0.8-2.0.9

jim — Tue, 14 May 2013 19:28:48 GMT

I've done a diff on the git tags for BOA 2.0.8 to 2.0.9... That's attached, I'm working my way through to see if anything stands out.

ed — Tue, 14 May 2013 19:31:56 GMT

nice - i'm on mobile if needed

jim — Tue, 14 May 2013 19:51:45 GMT

I've also disabled Session443 on http://stg.transitionnetwork.org/ -- once disabled AND I log in via HTTP, all is well; admin links are fine and I can move between pages no problems.

So we have a SSL/secure cookie configuration issue which is causing the problems. When using Session443 to control HTTPS access, the admin /node/*/edit issue happens. When turned off, provided I login in both HTTP and HTTPS contexts, it's fine again.

I'll keep digging, but could be related to Puffin/Aegir SSL options back in the server move days...

jim — Tue, 14 May 2013 19:58:26 GMT

One section from the .diff I uploaded is new in 2.0.9 and relates to cookies, HTTPS and results in redirects if BOA thinks the request is from a bot... In file aegir/conf/global.inc.aegir.txt:

@@ -374,7 +375,22 @@ if (isset($_SERVER['HTTP_HOST']) && isset($_SERVER['SERVER_NAME'])) {
       }
     }
-    if (empty($known_bot) && !$high_traffic) {
+    if (!isset($_COOKIE[$test_sess_name])) {
+      if (preg_match("/\/(?:node\/[0-9]+\/edit|node\/add)/", $_SERVER['REQUEST_URI'])) {
+        if (!file_exists('sites/'. $_SERVER['SERVER_NAME'] .'/modules/allow_anon_node_add.info')) {
+          $deny_anon = TRUE;
+          header("HTTP/1.1 301 Moved Permanently");
+          header("Location: http://" . $_SERVER['SERVER_NAME'] . "/");
+        }
+      }
+      if (preg_match("/^\/(?:[a-z]{2}\/)?(?:admin|logout|privatemsg|approve)/", $_SERVER['REQUEST_URI'])) {
+        $deny_anon = TRUE;
+        header("HTTP/1.1 301 Moved Permanently");
+        header("Location: http://" . $_SERVER['SERVER_NAME'] . "/");
+      }
+    }
+

Might be a wild goose chase, but I'll review aegir/conf/global.inc.aegir.txt now to see what the whole section of code does.

mark — Tue, 14 May 2013 20:05:03 GMT

Probably a silly suggestion, but if some degree of security is needed while
SSL gets fixed, there's always the Encrypt Submissions module:
http://drupal.org/project/encrypt_submissions
Good luck. Hope the solution is found without too much bother.
M.
On May 14, 2013 8:51 PM, "Transiton Technology Trac" <
trac@tech.transitionnetwork.org> wrote:
> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Maintenance
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0              |                  Billable?:  1
>         Total Hours:  2.08           |
> -------------------------------------+-------------------------------------
>
> Comment (by jim):
>
>  I've also disabled Session443 on http://stg.transitionnetwork.org/ --
> once
>  disabled AND I log in via HTTP, all is well; admin links are fine and I
>  can move between pages no problems.
>
>  So we have a SSL/secure cookie configuration issue which is causing the
>  problems. When using Session443 to control HTTPS access, the admin
>  /node/*/edit issue happens. When turned off, provided I login in both HTTP
>  and HTTPS contexts, it's fine again.
>
>  I'll keep digging, but could be related to [#484 Puffin/Aegir SSL options]
>  back in the server move days...
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:18
> >
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>

jim — Tue, 14 May 2013 20:12:36 GMT

And that's fixed it!

I've reverted commit 03d1a3e: Issue #1962458 by jeremyr - 403 for anonymous users on node/add and it all seems good to me now.

Downgrading, I'll post more on the Drupal.org Octupus issue.

Ed, can you confirm it works for you? Please test some stuff...

owner, priority, status changed

jim — Tue, 14 May 2013 20:12:44 GMT

owner changed from chris to jim
priority changed from blocker to major
status changed from new to assigned

hours, totalhours changed

chris — Tue, 14 May 2013 20:17:08 GMT

hours changed from 0.0 to 0.25
totalhours changed from 2.08 to 2.33

Replying to jim:

And that's fixed it!

Nice one Jim! That's fixed it :-)

hours, totalhours changed

chris — Tue, 14 May 2013 20:26:01 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.33 to 2.43

This is the ticket referenced in the commit which caused the problem: https://drupal.org/node/1962458

I don't understand why all users appear to be classified as bots.

jim — Tue, 14 May 2013 20:31:28 GMT

I don't understand why all users appear to be classified as bots.

Neither do I... Might be an interaction with Session443, or just a bug in the bot detection. Too tired to debug further, but will pick this up again tomorrow.

So in closing for tonight...

I've updated the ticket description and added some comments: https://drupal.org/node/1994346
I'll follow the bug up from here and re-patch our /data/conf/global.inc file when a fix comes out.

ed — Wed, 15 May 2013 06:18:47 GMT

Tested and it works fine as admin and as a social reporter. Leaving open for follow up.

Could this have anything to do with our use of the various anti-spam measures?

Good work.

jim — Fri, 17 May 2013 17:25:43 GMT

This is fixed in this BOA commit: http://drupalcode.org/project/octopus.git/commit/8c79f5e

It requires replacing our /data/conf/global.inc with this version: http://drupalcode.org/project/octopus.git/blob/8c79f5e:/aegir/conf/global.inc

If I don't hear any objections, I'll proceed with the replacement this weekend.

Also I'll backup our current file so we can roll back as needed. I'll then test and either roll back or let everyone know I think it's fine.

hours, totalhours changed

chris — Fri, 17 May 2013 19:33:57 GMT

hours changed from 0.0 to 0.2
totalhours changed from 2.43 to 2.63

Replying to jim:

If I don't hear any objections, I'll proceed with the replacement this weekend.

Go for it, good to see the BOA developers working on this problem, I have posted another comment to the Drupal bug:

https://drupal.org/node/1994346#comment-7424604

If there is a better alternative to Session 443 that would be fine, but I'm not aware of one and it's working well as far as I'm aware -- I don't really understand why you and omega8cc are not happy with it?

hours, totalhours changed

jim — Wed, 22 May 2013 13:15:34 GMT

hours changed from 0.0 to 0.5
totalhours changed from 2.63 to 3.13

FYI lots of good debugging and other comments by me and Omega8cc from comment #11 here: http://drupal.org/node/1994346#comment-7432698

The situation is:

I'll create two control files so we can run with standard global.in, rather than the patched version.
I'll try to establish if we can find a way to either a) replace Session443 (because it mucks about with the session IDs), or b) enhance BOA... There's also an option c): Get BOA to do the job of Session443/Secure Pages under D6 somehow.

I'll do 1 today at some point, and follow up with 2 over the coming weeks.

hours, totalhours changed

chris — Wed, 22 May 2013 18:27:26 GMT

hours changed from 0.0 to 0.25
totalhours changed from 3.13 to 3.38

Another option we could consider is using a STS header to ensure that people who visit the site using HTTPS don't visit the site using HTTP, see the Wikipedia HTTP Strict Transport Security page, the only problem is that there is no IE support and it would depend on people using new browsers, perhaps this means it not yet viable.

If we are able to add to the Nginx configuration for the site then it's just a matter of adding this:

add_header Strict-Transport-Security max-age=31536000;

And if we can't there is this Drupal module: https://drupal.org/project/hsts

hours, totalhours changed

jim — Fri, 24 May 2013 09:16:38 GMT

hours changed from 0.0 to 2.5
totalhours changed from 3.38 to 5.88

Adding my 2 hours that I thought I'd added from the original debugging.

Recent debugging time I'll share 50% voluntary with BOA since I'm helping an OS project that directly benefits me too... There's already .5 hours against me for that, adding .5 more for recent work and the fix (which will go in tonight).

When adding the control files per https://drupal.org/node/1994346#comment-7433174, and judging by the code, this issue will almost certainly be closed. I'll then raise a separate Octopus ticket on D.o to suggest an alternative session ID handling system.

Regarding STS etc... The bottom line is D6 has crappy session/cookie handling for HTTPS. BOA tries to do some nice stuff to take load off/secure the sessions futher, but this doesn't work with Session443 which we need to give us good session handling under D6. The control files mentioned above effectively turn off the redirects in Octopus.

So I aim to close this tonight, and all my time for the ticket is logged against it now.

cc, priority changed

ed — Fri, 24 May 2013 10:59:08 GMT

cc chris added; mark, jim removed
priority changed from major to critical

Since the update today I cannot access any admin functions - please check out lads

jim — Fri, 24 May 2013 11:05:37 GMT

Aaahh piss.

Ok: Chris, please (checking the XXX bit):

touch /data/disk/tn/static/transitionnetwork-XXX/sites/www.transitionnetwork.org/modules/allow_anon_node_add.info
touch /data/disk/tn/static/transitionnetwork-XXX/sites/www.transitionnetwork.org/modules/disable_admin_dos_protection.info

And backup/replace /data/conf/global.inc with /data/conf/global.inc.HEAD.

That's the fix I would have done tonight...

hours, totalhours changed

chris — Fri, 24 May 2013 11:15:58 GMT

hours changed from 0.0 to 0.15
totalhours changed from 5.88 to 6.03

I feared this would happen ticket:218#comment:90 and I'm sorry that I didn't check to see if it had happened, following Jims advice:

touch /data/disk/tn/static/transition-network-d6-s001/sites/www.stg.transitionnetwork.org/modules/allow_anon_node_add.info
touch /data/disk/tn/static/transition-network-d6-s001/sites/www.stg.transitionnetwork.org/modules/disable_admin_dos_protection.info
touch /data/disk/tn/static/transition-network-d6-004/sites/www.transitionnetwork.org/modules/allow_anon_node_add.info
touch /data/disk/tn/static/transition-network-d6-004/sites/www.transitionnetwork.org/modules/disable_admin_dos_protection.info
mv /data/conf/global.inc /data/conf/global.inc.bak
cp /data/conf/global.inc.HEAD /data/conf/global.inc

The fix has done the trick it seems.

jim — Fri, 24 May 2013 11:47:27 GMT

Well done Chris.

Aaaannnd... relax.

jim — Fri, 24 May 2013 11:49:53 GMT

And FYI these tasks are one-off per platform... they won't need doing again for BOA updates, just if we add a D6 platform with Session443-enabled sites.

ed — Fri, 24 May 2013 14:10:11 GMT

working again - good work lads

jim — Mon, 27 May 2013 21:42:03 GMT

This can now be closed... I'm following up on the d.o issue to see if we can get a nicer fix for D6+Session443, but otherwise we're done here.

status changed; resolution set

jim — Mon, 27 May 2013 21:42:12 GMT

status changed from assigned to closed
resolution set to fixed

chris — Tue, 28 May 2013 23:03:57 GMT

Note that the replacing /data/conf/global.inc as done in ticket:548#comment:34 needed the redis password adding back in and this was done on ticket:554#comment:5