All Admin functions broken on TN.org

[ed]

Admins, editors, social reporters cannot create content on TN.org. Choosing to create content leads to homepage. Can't do anything on the admin menu.

Emergency.

[chris]

The only thing that I'm aware of that has changed in the last few days was the BOA upgrade which was done on Sunday night, see ticket:547#comment:4

Browsing the site I keep getting redirects from HTTPS to HTTP, however these are blocked by the ​https://addons.mozilla.org/en-US/firefox/addon/requestpolicy/ plugin I have in Firefox and this enables me to use the admin interface normally.

Here is an example of the redirect, I clicked a link to ​https://www.transitionnetwork.org/node/add/blog and my broswer sent the following GET request (the session cookie string has been replaced with a made up one for security):

GET /node/add/blog HTTP/1.1
Host: www.transitionnetwork.org
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20100101 Firefox/20.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Cookie: SESStai8xeimetief7coh8pheev7siu0li2daocailiew7koo0see2xierucheeceTusiW; LOGGED_IN=1; has_js=1
Connection: keep-alive

And the response is a redirect to the home page:

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 14 May 2013 09:05:58 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Backend: C
X-Allow-Redis: YES
X-Cookie-Domain: .transitionnetwork.org
X-Redis-Prefix: www.transitionnetwork.org_
X-Purge-Level: 6
X-Local-Proto: https
Location: http://www.transitionnetwork.org/
Last-Modified: Tue, 14 May 2013 09:05:56 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Etag: "1368522356"
X-Engine: Octopus 1.0 ET
X-Device: normal
X-Speed-Cache: BYPASS
X-Speed-Cache-UID: Olouveikoh7lariCh2eef4iiv2zai2ais;
X-Speed-Cache-Key: /node/add/blog
X-NoCache: Skip
X-This-Proto: https
X-Server-Name: www.transitionnetwork.org
Vary: Accept-Encoding

More investigation is needed...

[jim]

I've tried a few things but can't do much from the UI. I'm looking at it
but don't have my SSH key at work so can't do much until tonight.

It feels like a module issue, or a server config thing -- I notice that all
the admin pages are available, provided you use the old, non-'clean url'
urls, e.g.

   - Normal: https://www.transitionnetwork.org/admin/content/node/overview
   - Tweaked: https://www.transitionnetwork.org/*?q=*
   admin/content/node/overview

Note the *?q= *between the domain and the path -- that will work for any
page.

So this means Drupal is fine, but either it, a module or the server is
redirecting /admin and /node/*/edit paths. Odd.

Ed, when did this last work?


On 14 May 2013 10:11, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.25           |                  Billable?:  1
>         Total Hours:  0              |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.25
>  * totalhours:  0.0 => 0.25
>
>
> Comment:
>
>  The only thing that I'm aware of that has changed in the last few days was
>  the BOA upgrade which was done on Sunday night, see ticket:547#comment:4
>
>  Browsing the site I keep getting redirects from HTTPS to HTTP, however
>  these are blocked by the https://addons.mozilla.org/en-
>  US/firefox/addon/requestpolicy/ plugin I have in Firefox and this enables
>  me to use the admin interface normally.
>
>  Here is an example of the redirect, I clicked a link to
>  https://www.transitionnetwork.org/node/add/blog and my broswer sent the
>  following GET request (the session cookie string has been replaced with a
>  made up one for security):
>
>  {{{
>  GET /node/add/blog HTTP/1.1
>  Host: www.transitionnetwork.org
>  User-Agent: Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20100101
>  Firefox/20.0
>  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>  Accept-Language: en-gb,en;q=0.5
>  Accept-Encoding: gzip, deflate
>  DNT: 1
>  Cookie:
>  SESStai8xeimetief7coh8pheev7siu0li2daocailiew7koo0see2xierucheeceTusiW;
>  LOGGED_IN=1; has_js=1
>  Connection: keep-alive
>  }}}
>
>  And the response is a redirect to the home page:
>
>  {{{
>  HTTP/1.1 301 Moved Permanently
>  Server: nginx
>  Date: Tue, 14 May 2013 09:05:58 GMT
>  Content-Type: text/html; charset=utf-8
>  Transfer-Encoding: chunked
>  Connection: keep-alive
>  X-Backend: C
>  X-Allow-Redis: YES
>  X-Cookie-Domain: .transitionnetwork.org
>  X-Redis-Prefix: www.transitionnetwork.org_
>  X-Purge-Level: 6
>  X-Local-Proto: https
>  Location: http://www.transitionnetwork.org/
>  Last-Modified: Tue, 14 May 2013 09:05:56 +0000
>  Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
>  Etag: "1368522356"
>  X-Engine: Octopus 1.0 ET
>  X-Device: normal
>  X-Speed-Cache: BYPASS
>  X-Speed-Cache-UID: Olouveikoh7lariCh2eef4iiv2zai2ais;
>  X-Speed-Cache-Key: /node/add/blog
>  X-NoCache: Skip
>  X-This-Proto: https
>  X-Server-Name: www.transitionnetwork.org
>  Vary: Accept-Encoding
>  }}}
>
>  More investigation is needed...
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>



-- 
Jim Kirkpatrick

i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

[chris]

This ticket might be related, ​https://drupal.org/node/1989254 the BOA issue queue is here ​http://bit.ly/boa-iq

[ed]

Last time I used it was Thurs 09/05. Social Reporters tried to use it Monday 13/5 with no luck.

Replying to jim:

I've tried a few things but can't do much from the UI. I'm looking at it
but don't have my SSH key at work so can't do much until tonight.

It feels like a module issue, or a server config thing -- I notice that all
the admin pages are available, provided you use the old, non-'clean url'
urls, e.g.

   - Normal: https://www.transitionnetwork.org/admin/content/node/overview
   - Tweaked: https://www.transitionnetwork.org/*?q=*
   admin/content/node/overview

Note the *?q= *between the domain and the path -- that will work for any
page.

So this means Drupal is fine, but either it, a module or the server is
redirecting /admin and /node/*/edit paths. Odd.

Ed, when did this last work?


On 14 May 2013 10:11, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.25           |                  Billable?:  1
>         Total Hours:  0              |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.25
>  * totalhours:  0.0 => 0.25
>
>
> Comment:
>
>  The only thing that I'm aware of that has changed in the last few days was
>  the BOA upgrade which was done on Sunday night, see ticket:547#comment:4
>
>  Browsing the site I keep getting redirects from HTTPS to HTTP, however
>  these are blocked by the https://addons.mozilla.org/en-
>  US/firefox/addon/requestpolicy/ plugin I have in Firefox and this enables
>  me to use the admin interface normally.
>
>  Here is an example of the redirect, I clicked a link to
>  https://www.transitionnetwork.org/node/add/blog and my broswer sent the
>  following GET request (the session cookie string has been replaced with a
>  made up one for security):
>
>  {{{
>  GET /node/add/blog HTTP/1.1
>  Host: www.transitionnetwork.org
>  User-Agent: Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20100101
>  Firefox/20.0
>  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>  Accept-Language: en-gb,en;q=0.5
>  Accept-Encoding: gzip, deflate
>  DNT: 1
>  Cookie:
>  SESStai8xeimetief7coh8pheev7siu0li2daocailiew7koo0see2xierucheeceTusiW;
>  LOGGED_IN=1; has_js=1
>  Connection: keep-alive
>  }}}
>
>  And the response is a redirect to the home page:
>
>  {{{
>  HTTP/1.1 301 Moved Permanently
>  Server: nginx
>  Date: Tue, 14 May 2013 09:05:58 GMT
>  Content-Type: text/html; charset=utf-8
>  Transfer-Encoding: chunked
>  Connection: keep-alive
>  X-Backend: C
>  X-Allow-Redis: YES
>  X-Cookie-Domain: .transitionnetwork.org
>  X-Redis-Prefix: www.transitionnetwork.org_
>  X-Purge-Level: 6
>  X-Local-Proto: https
>  Location: http://www.transitionnetwork.org/
>  Last-Modified: Tue, 14 May 2013 09:05:56 +0000
>  Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
>  Etag: "1368522356"
>  X-Engine: Octopus 1.0 ET
>  X-Device: normal
>  X-Speed-Cache: BYPASS
>  X-Speed-Cache-UID: Olouveikoh7lariCh2eef4iiv2zai2ais;
>  X-Speed-Cache-Key: /node/add/blog
>  X-NoCache: Skip
>  X-This-Proto: https
>  X-Server-Name: www.transitionnetwork.org
>  Vary: Accept-Encoding
>  }}}
>
>  More investigation is needed...
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:1>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>



-- 
Jim Kirkpatrick

i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

[ed]

good tip about URLs - ta - very timely

[jim]

Interesting... My initial hunch was a 443 Session module issue, but the
settings on there look good.

However, it needs our local.setting.php in the
~/static/transition-00X/sites/www.transitionnetwork.org/local.setting.phphaving
the following in it:
   if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] != 'off') {
        ini_set('session.cookie_secure', 1);
      }

This was in our /data/conf/override.global.inc file instead, as that's
included in all sites we host and they curently will all need this.

Chris, could you try commenting out the above lines in
/data/conf/override.global.inc and try to use /admin/* URLs? If that
doesn't work we should uncomment them.

Also, have you tried the restarting php53-fpm service? And maybe
redis-server service?

[chris]

I have raised a ticket here ​https://drupal.org/node/1994346

[jim]

Good work on the ticket, I've added extra info.

BUT Chris, you'll need to attach the key files if you want support!!!

From the new Issue page for Octopus (
https://drupal.org/node/add/project-issue/octopus):
> Please note that any bug report or support request failing to follow the
guidelines will be ignored and closed without any answer.

> Before reporting a bug always search for similar bug report before
submitting your own, and include as much information about your context as
possible. Especially please include the contents (anonymized for security
and privacy) of files:

> /data/disk/USER/log/octopus_log.txt
> /var/aegir/config/includes/barracuda_log.txt
> /root/.USER.octopus.cnf




On 14 May 2013 11:16, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.5            |                  Billable?:  1
>         Total Hours:  0.45           |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.5
>  * totalhours:  0.45 => 0.95
>
>
> Comment:
>
>  I have raised a ticket here https://drupal.org/node/1994346
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:7>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>



-- 
Jim Kirkpatrick

i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

[chris]

Replying to jim:

Chris, could you try commenting out the above lines in
/data/conf/override.global.inc and try to use /admin/* URLs? If that
doesn't work we should uncomment them.

I tried commenting it out, not expecting this to change anything -- all it is doing is setting the secure flag on the admin cookie, and this is something that we want -- and it didn't change the behaviour.

Also, have you tried the restarting php53-fpm service? And maybe
redis-server service?

These will have been restarted during the upgrade so I don't think this will help, but I have restarted them in any case.

Replying to jim:

you'll need to attach the key files if you want support!!!

Have done, ​https://drupal.org/node/1994346#comment-7410060

Next thing to do is to read through the Nginx configuration to see if a Nginx redirect can be found.

[jim]

Added a edit to my post to cover that secure session cookie test:
https://drupal.org/node/1994346#comment-7409920

So now we have to wait I suppose... I'll have a look as soon as I get home
around 6pm, and will keep an eye on the various threads.

Best,

Jim


On 14 May 2013 12:24, Transiton Technology Trac <
trac@tech.transitionnetwork.org> wrote:

> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Production
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0.35           |                  Billable?:  1
>         Total Hours:  0.95           |
> -------------------------------------+-------------------------------------
> Changes (by chris):
>
>  * hours:  0.0 => 0.35
>  * totalhours:  0.95 => 1.3
>
>
> Comment:
>
>  Replying to [comment:6 jim]:
>  >
>  > Chris, could you try commenting out the above lines in
>  > /data/conf/override.global.inc and try to use /admin/* URLs? If that
>  > doesn't work we should uncomment them.
>
>  I tried commenting it out, not expecting this to change anything -- all it
>  is doing is setting the secure flag on the admin cookie, and this is
>  something that we want -- and it didn't change the behaviour.
>
>  > Also, have you tried the restarting php53-fpm service? And maybe
>  > redis-server service?
>
>  These will have been restarted during the upgrade so I don't think this
>  will help, but I have restarted them in any case.
>
>  Replying to [comment:8 jim]:
>  >
>  > you'll need to attach the key files if you want support!!!
>
>  Have done, https://drupal.org/node/1994346#comment-7410060
>
>  Next thing to do is to read through the Nginx configuration to see if a
>  Nginx redirect can be found.
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:9>
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>



-- 
Jim Kirkpatrick

i-JK Drupal Solutions -- www.i-jk.co.uk
phone: 07728 794171
email: jim@i-jk.co.uk
skype: jim_kirkpatrick

[chris]

Following the Nginx config includes:

• /etc/nginx/nginx.conf
  • /etc/nginx/mime.types
  • /etc/nginx/conf.d/aegir.conf
    • /var/aegir/config/server_master/nginx/pre.d/nginx_speed_purge.conf
    • /var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf
    • /var/aegir/config/server_master/nginx/platform.d/tn.conf -> /data/disk/tn/config/tn.nginx.conf
      • /data/disk/tn/config/server_master/nginx/vhost.d/news.transitionnetwork.org
        • /data/disk/tn/config/includes/fastcgi_params.conf
        • /data/disk/tn/config/includes/nginx_octopus_include.conf
      • /data/disk/tn/config/server_master/nginx/vhost.d/stg.transitionnetwork.org
        • /data/disk/tn/config/includes/fastcgi_params.conf
        • /data/disk/tn/config/includes/nginx_octopus_include.conf
      • /data/disk/tn/config/server_master/nginx/vhost.d/tn.puffin.webarch.net
        • /data/disk/tn/config/includes/fastcgi_params.conf
        • /data/disk/tn/config/includes/nginx_modern_include.conf
      • /data/disk/tn/config/server_master/nginx/vhost.d/www.transitionnetwork.org
        • /data/disk/tn/config/includes/fastcgi_params.conf
        • /data/disk/tn/config/includes/nginx_octopus_include.conf
    • /var/aegir/config/server_master/nginx/vhost.d/cgp.master.puffin.webarch.net
      • /var/aegir/config/includes/fastcgi_params.conf
      • /var/aegir/config/includes/nginx_compact_include.conf
    • /var/aegir/config/server_master/nginx/vhost.d/chive.master.puffin.webarch.net
      • /var/aegir/config/includes/fastcgi_params.conf
      • /var/aegir/config/includes/nginx_compact_include.conf
    • /var/aegir/config/server_master/nginx/vhost.d/master.puffin.webarch.net
      • /var/aegir/config/includes/fastcgi_params.conf
      • /var/aegir/config/includes/nginx_modern_include.conf
    • /var/aegir/config/server_master/nginx/post.d/* (yes there is a files called *)
  • /etc/nginx/sites-enabled/ (this directory doesn't exist)

This is a sorted unique list of the files to check:

/data/disk/tn/config/includes/fastcgi_params.conf
/data/disk/tn/config/includes/nginx_modern_include.conf
/data/disk/tn/config/includes/nginx_octopus_include.conf
/data/disk/tn/config/server_master/nginx/vhost.d/news.transitionnetwork.org
/data/disk/tn/config/server_master/nginx/vhost.d/stg.transitionnetwork.org
/data/disk/tn/config/server_master/nginx/vhost.d/tn.puffin.webarch.net
/data/disk/tn/config/server_master/nginx/vhost.d/www.transitionnetwork.org
/etc/nginx/conf.d/aegir.conf
/etc/nginx/mime.types
/etc/nginx/nginx.conf
/var/aegir/config/includes/fastcgi_params.conf
/var/aegir/config/includes/nginx_compact_include.conf
/var/aegir/config/includes/nginx_modern_include.conf
/var/aegir/config/server_master/nginx/pre.d/nginx_speed_purge.conf
/var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf
/var/aegir/config/server_master/nginx/vhost.d/cgp.master.puffin.webarch.net
/var/aegir/config/server_master/nginx/vhost.d/chive.master.puffin.webarch.net
/var/aegir/config/server_master/nginx/vhost.d/master.puffin.webarch.net

[chris]

Reading through the 3000+ lines of Nginx config, the key file for port 443 is /var/aegir/config/server_master/nginx/pre.d/nginx_wild_ssl.conf and Nginx is basically set up as a reverse proxy to itself.

But I'm sorry I can't see what is causing the redirects from port 80 to port 443.

[chris]

The Redirect documented in ticket:548#comment:1 is a 301, so grepping the Nginx for "permanent" (a 301 is a permanent redirect), there are the following redirects in 3 different files, /data/disk/tn/config/includes/nginx_modern_include.conf, /data/disk/tn/config/includes/nginx_octopus_include.conf and /var/aegir/config/includes/nginx_modern_include.conf:

###
### Deny access to Hostmaster web/db server node.
### It is still possible to edit or break web/db server
### node at /node/2/edit, if you know what are you doing.
###
location ^~ /hosting/c/server_master {
  access_log off;
  rewrite ^ $scheme://$host/hosting/sites permanent;
}

###
### Deny access to Hostmaster db server node.
### It is still possible to edit or break db server
### node at /node/4/edit, if you know what are you doing.
###
location ^~ /hosting/c/server_localhost {
  access_log off;
  rewrite ^ $scheme://$host/hosting/sites permanent;
}

###
### Deny cache details display.
###
location ^~ /admin/settings/performance/cache-backend {
  access_log off;
  rewrite ^ $scheme://$host/admin/settings/performance permanent;
}

###
### Deny cache details display.
###
location ^~ /admin/config/development/performance/redis {
  access_log off;
  rewrite ^ $scheme://$host/admin/config/development/performance permanent;
}

###
### Send all known bots to $args free URLs.
###
location @nobots {
  if ($is_bot) {
    rewrite ^ $scheme://$host$uri? permanent;
  }
  rewrite ^/(.*)$  /index.php?q=$1 last;
}

None of the above look to me like they might be the cause of the problem.

[ed]

I'm looking for some Jim love here too, yes?

[jim]

Git Diff of nginx-for-drupal BOA 2.0.8-2.0.9

[jim]

I've done a diff on the git tags for BOA 2.0.8 to 2.0.9... That's attached, I'm working my way through to see if anything stands out.

[ed]

nice - i'm on mobile if needed

[jim]

I've also disabled Session443 on ​http://stg.transitionnetwork.org/ -- once disabled AND I log in via HTTP, all is well; admin links are fine and I can move between pages no problems.

So we have a SSL/secure cookie configuration issue which is causing the problems. When using Session443 to control HTTPS access, the admin /node/*/edit issue happens. When turned off, provided I login in both HTTP and HTTPS contexts, it's fine again.

I'll keep digging, but could be related to Puffin/Aegir SSL options back in the server move days...

[jim]

One section from the .diff I uploaded is new in 2.0.9 and relates to cookies, HTTPS and results in redirects if BOA thinks the request is from a bot... In file aegir/conf/global.inc.aegir.txt:

@@ -374,7 +375,22 @@ if (isset($_SERVER['HTTP_HOST']) && isset($_SERVER['SERVER_NAME'])) {
       }
     }
 
-    if (empty($known_bot) && !$high_traffic) {
+    if (!isset($_COOKIE[$test_sess_name])) {
+      if (preg_match("/\/(?:node\/[0-9]+\/edit|node\/add)/", $_SERVER['REQUEST_URI'])) {
+        if (!file_exists('sites/'. $_SERVER['SERVER_NAME'] .'/modules/allow_anon_node_add.info')) {
+          $deny_anon = TRUE;
+          header("HTTP/1.1 301 Moved Permanently");
+          header("Location: http://" . $_SERVER['SERVER_NAME'] . "/");
+        }
+      }
+      if (preg_match("/^\/(?:[a-z]{2}\/)?(?:admin|logout|privatemsg|approve)/", $_SERVER['REQUEST_URI'])) {
+        $deny_anon = TRUE;
+        header("HTTP/1.1 301 Moved Permanently");
+        header("Location: http://" . $_SERVER['SERVER_NAME'] . "/");
+      }
+    }
+

Might be a wild goose chase, but I'll review aegir/conf/global.inc.aegir.txt now to see what the whole section of code does.

[mark]

Probably a silly suggestion, but if some degree of security is needed while
SSL gets fixed, there's always the Encrypt Submissions module:
http://drupal.org/project/encrypt_submissions

Good luck. Hope the solution is found without too much bother.

M.
On May 14, 2013 8:51 PM, "Transiton Technology Trac" <
trac@tech.transitionnetwork.org> wrote:

> #548: All Admin functions broken on TN.org
> -------------------------------------+-------------------------------------
>            Reporter:  ed             |                      Owner:  chris
>                Type:  defect         |                     Status:  new
>            Priority:  blocker        |                  Milestone:
>           Component:  Drupal         |  Maintenance
>   modules & settings                 |                 Resolution:
>            Keywords:                 |  Estimated Number of Hours:  0.0
> Add Hours to Ticket:  0              |                  Billable?:  1
>         Total Hours:  2.08           |
> -------------------------------------+-------------------------------------
>
> Comment (by jim):
>
>  I've also disabled Session443 on http://stg.transitionnetwork.org/ --
> once
>  disabled AND I log in via HTTP, all is well; admin links are fine and I
>  can move between pages no problems.
>
>  So we have a SSL/secure cookie configuration issue which is causing the
>  problems. When using Session443 to control HTTPS access, the admin
>  /node/*/edit issue happens. When turned off, provided I login in both HTTP
>  and HTTPS contexts, it's fine again.
>
>  I'll keep digging, but could be related to [#484 Puffin/Aegir SSL options]
>  back in the server move days...
>
> --
> Ticket URL: <https://tech.transitionnetwork.org/trac/ticket/548#comment:18
> >
> Transition Technology <https://tech.transitionnetwork.org/trac>
> Support and issues tracking for the Transition Network Web Project.
>

[jim]

And that's fixed it!

I've reverted ​commit 03d1a3e: Issue #1962458 by jeremyr - 403 for anonymous users on node/add and it all seems good to me now.

Downgrading, I'll post more on the ​Drupal.org Octupus issue.

Ed, can you confirm it works for you? Please test some stuff...

[chris]

Replying to jim:

And that's fixed it!

Nice one Jim! That's fixed it :-)

[chris]

This is the ticket referenced in the commit which caused the problem: ​https://drupal.org/node/1962458

I don't understand why all users appear to be classified as bots.

[jim]

I don't understand why all users appear to be classified as bots.

Neither do I... Might be an interaction with Session443, or just a bug in the bot detection. Too tired to debug further, but will pick this up again tomorrow.

So in closing for tonight...

• I've updated the ticket description and added some comments: ​https://drupal.org/node/1994346
• I'll follow the bug up from here and re-patch our /data/conf/global.inc file when a fix comes out.

[ed]

Tested and it works fine as admin and as a social reporter. Leaving open for follow up.

Could this have anything to do with our use of the various anti-spam measures?

Good work.

[jim]

This is fixed in this BOA commit: ​http://drupalcode.org/project/octopus.git/commit/8c79f5e

It requires replacing our /data/conf/global.inc with this version: ​http://drupalcode.org/project/octopus.git/blob/8c79f5e:/aegir/conf/global.inc

If I don't hear any objections, I'll proceed with the replacement this weekend.

Also I'll backup our current file so we can roll back as needed. I'll then test and either roll back or let everyone know I think it's fine.

[chris]

Replying to jim:

If I don't hear any objections, I'll proceed with the replacement this weekend.

Go for it, good to see the BOA developers working on this problem, I have posted another comment to the Drupal bug:

​https://drupal.org/node/1994346#comment-7424604

If there is a better alternative to Session 443 that would be fine, but I'm not aware of one and it's working well as far as I'm aware -- I don't really understand why you and omega8cc are not happy with it?

[jim]

FYI lots of good debugging and other comments by me and Omega8cc from comment #11 here: ​http://drupal.org/node/1994346#comment-7432698

The situation is:

1. I'll create two control files so we can run with standard global.in, rather than the patched version.
2. I'll try to establish if we can find a way to either a) replace Session443 (because it mucks about with the session IDs), or b) enhance BOA... There's also an option c): Get BOA to do the job of Session443/Secure Pages under D6 somehow.

I'll do 1 today at some point, and follow up with 2 over the coming weeks.

[chris]

Another option we could consider is using a STS header to ensure that people who visit the site using HTTPS don't visit the site using HTTP, see the ​Wikipedia HTTP Strict Transport Security page, the only problem is that there is no IE support and it would depend on people using new browsers, perhaps this means it not yet viable.

If we are able to add to the Nginx configuration for the site then it's just a matter of adding this:

add_header Strict-Transport-Security max-age=31536000;

And if we can't there is this Drupal module: ​https://drupal.org/project/hsts

[jim]

Adding my 2 hours that I thought I'd added from the original debugging.

Recent debugging time I'll share 50% voluntary with BOA since I'm helping an OS project that directly benefits me too... There's already .5 hours against me for that, adding .5 more for recent work and the fix (which will go in tonight).

When adding the control files per ​https://drupal.org/node/1994346#comment-7433174, and judging by the code, this issue will almost certainly be closed. I'll then raise a separate Octopus ticket on D.o to suggest an alternative session ID handling system.

Regarding STS etc... The bottom line is D6 has crappy session/cookie handling for HTTPS. BOA tries to do some nice stuff to take load off/secure the sessions futher, but this doesn't work with Session443 which we need to give us good session handling under D6. The control files mentioned above effectively turn off the redirects in Octopus.

So I aim to close this tonight, and all my time for the ticket is logged against it now.

[ed]

Since the update today I cannot access any admin functions - please check out lads

[jim]

Aaahh piss.

Ok: Chris, please (checking the XXX bit):

• touch /data/disk/tn/static/transitionnetwork-XXX/sites/www.transitionnetwork.org/modules/allow_anon_node_add.info
• touch /data/disk/tn/static/transitionnetwork-XXX/sites/www.transitionnetwork.org/modules/disable_admin_dos_protection.info

And backup/replace /data/conf/global.inc with /data/conf/global.inc.HEAD.

That's the fix I would have done tonight...

[chris]

I feared this would happen ticket:218#comment:90 and I'm sorry that I didn't check to see if it had happened, following Jims advice:

touch /data/disk/tn/static/transition-network-d6-s001/sites/www.stg.transitionnetwork.org/modules/allow_anon_node_add.info
touch /data/disk/tn/static/transition-network-d6-s001/sites/www.stg.transitionnetwork.org/modules/disable_admin_dos_protection.info
touch /data/disk/tn/static/transition-network-d6-004/sites/www.transitionnetwork.org/modules/allow_anon_node_add.info
touch /data/disk/tn/static/transition-network-d6-004/sites/www.transitionnetwork.org/modules/disable_admin_dos_protection.info
mv /data/conf/global.inc /data/conf/global.inc.bak 
cp /data/conf/global.inc.HEAD /data/conf/global.inc

The fix has done the trick it seems.

[jim]

Well done Chris.

Aaaannnd... relax.

[jim]

And FYI these tasks are one-off per platform... they won't need doing again for BOA updates, just if we add a D6 platform with Session443-enabled sites.

[ed]

working again - good work lads

[jim]

This can now be closed... I'm following up on the d.o issue to see if we can get a nicer fix for D6+Session443, but otherwise we're done here.

[chris]

Note that the replacing /data/conf/global.inc as done in ticket:548#comment:34 needed the redis password adding back in and this was done on ticket:554#comment:5