Transition Technology: Ticket #716: Heartbleed

chris — Wed, 09 Apr 2014 17:55:30 GMT

We are still waiting for Ben B to get his password reset at Gandi.net so he can click though a email and we can get a new cert for transitionnetwork.org. Perhaps Ed has a copy of this password somewhere?

Once we have new a new key and cert in place we can look at taking these steps that drupal.org have taken:

Replaced the private strings (drupal_private_key and drupal_hash_salt) which are used for a variety of security related purposes in all Drupal sites
Removed all active sessions
Verified the email addresses in use today match those in use a week ago
Required that all Drupal.org users with administrative or project repository access to reset their passwords

We will also need to change the MediaWiki, PiwikServer, TransitionTrac and TransitionResearchWagn account passwords.

chris — Wed, 09 Apr 2014 18:10:03 GMT

I can't see how to update the drupal_private_key, it appears to be in the database?

https://api.drupal.org/api/drupal/includes!common.inc/function/drupal_get_private_key/6

We can update the drupal_hash_salt in settings.php if we can work out which one to update:

locate \/settings.php | wc -l
23

chris — Wed, 09 Apr 2014 21:00:37 GMT

I have just closed ticket:717 as a dupe. Sam appears to have a problem with email, I did have one to Ed and Ben blocked by the transitionnetwork.org mx server as spam because it contained "heartbleed.com".

hours, totalhours changed

jim — Wed, 09 Apr 2014 21:43:34 GMT

hours changed from 0.0 to 0.05
totalhours changed from 0.0 to 0.05

Be careful with the Drupal variable changes - they may result in all passwords being invalidated... That may be the goal but it's a change that needs management and probably a note on the login page.

Or they will be fine - but double check!

It might be better to set the passwords of all admins and devs to a random hash to force them to request a new one. A DB query is the easiest way to do that.

cc changed

ed — Thu, 10 Apr 2014 09:18:04 GMT

cc paul added

is that a paul or sam job?

ed — Thu, 10 Apr 2014 09:18:14 GMT

adding paul to ticket

chris — Thu, 10 Apr 2014 11:48:11 GMT

There is potential to make a mess with this. The Drupal.org steps make sense, can we document how to do all the steps in ticket:716#comment:1 and thgen work out when we are goingt o do them?

Still not installed the new cert yet, should happen in an hour or two...

sam — Thu, 10 Apr 2014 12:27:10 GMT

Hi Chris

I have been looking at ways to change passwords. I guess changing admin passwords is fairly straightforward and we should do this as soon as the new certificate is up and active sessions dropped?

I'm not clear from the your first post if we need to change all user passwords? It looks like drupal.org have just done admin passwords.

I found this that looks like it would force user passwords if it is indeed necessary: https://drupal.org/project/force_password_change

hours, totalhours changed

paul — Thu, 10 Apr 2014 12:46:56 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.05 to 0.3

@Sam

I think after we have re-issued our SSL certificates (and restarted affected services) it would be enough, at least for now, to just change critical passwords: the administrative user account and other account with privileged roles.

Investigating ..

ed — Thu, 10 Apr 2014 13:20:24 GMT

Thanks Paul, Chris, Sam. Good work. Forcing a password reset on all users would be quite a job (tech and comms), and we'd need to think it through carefully. Once Chris has done the SSL certificates, I'll make a small announcement - and would appreciate some editorial support!

hours, totalhours changed

paul — Thu, 10 Apr 2014 15:11:15 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.3 to 0.8

First draft?

# Security Recommendation

Dear XXX,

Earlier this week, a flaw in software that is widely used to secure Web communications was disclosed. The Heartbleed bug introduced a security vulnerability (http://heartbleed.com) in the widely used OpenSSL cryptographic library. Transition Network resolved this vulnerability on all if its servers within hours of learning of the issue. In addition, we recommend that you take the following precautionary steps as soon as possible.

Reset your Transition Network password as follows:

Log in at https://www.transitionnetwork.org/ From your "My account" page click on the edit link and on the Edit > Account page change your password. To create a secure password use a password generator like http://random.pw/

Please contact the Transition Network team if you have any questions.

Thank you,

TN Team

ed — Thu, 10 Apr 2014 15:15:23 GMT

Also possibly a link to the handy guide: http://www.transitionnetwork.org/blogs/ed-mitchell/2010-03/editing-your-personal-profile#2

ed — Thu, 10 Apr 2014 15:18:31 GMT

Further to phone call:

Sam going to reset all admin, site admins, and editor passwords and contact all relevant people

Chris do you need to restart the service first?

Ed going to do short blog post about this now

Sending out a mailchimp announcement to the subscribers would be good. I'm away Friday but happy for Sam to do this on Friday if you're happy with that. Use the websupport email for contact.

paul — Thu, 10 Apr 2014 15:20:28 GMT

Very useful. I didn't know we had that :)

ed — Thu, 10 Apr 2014 15:34:58 GMT

From Jim about restrting: Restarting will not do this - clearing the Drupal sessions table will.

hours, totalhours changed

paul — Thu, 10 Apr 2014 15:42:49 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.8 to 1.05

@Ed

Sounds good.

I think we need to patch and restart nginx, .. and then remove all active sessions on our websites. Then we can update *our* passwords, and send out an announcement to our users to recommend that they update their passwords.

Perhaps, we should also indicate in the announcement what could happen if they don't update their password, or maybe just add that to the blog post.

Best, Paul

hours deleted

chris — Thu, 10 Apr 2014 16:01:23 GMT

hours 0 deleted

Replying to paul:

I think we need to patch and restart nginx, ..

When the BOA upgrade is done, see ticket:707 that will rebuild Nginx, however as far as I'm aware the upgrading of OpenSSL and restarting Nginx and php-fpm means that we are not currently vunerable.

I could perhaps do the BOA upgrade soon, but due to time constraints would rather wait till the weekend.

remove all active sessions on our websites

That can be done at any time, but I think it needs to be done at the same time as the password reset?

sam — Thu, 10 Apr 2014 16:31:47 GMT

Replying to ed:

Further to phone call:

Sam going to reset all admin, site admins, and editor passwords and contact all relevant people

I'll email people and let them know they are going to get a password request from the site and explain why. I'll request password resets from the site frontend https://www.transitionnetwork.org/user/password by just entering their emails.

I think this is better than manually changing their passwords & then sending the password in plain text email, which is in itself insecure?

Sending out a mailchimp announcement to the subscribers would be good. I'm away Friday but happy for Sam to do this on Friday if you're happy with that. Use the websupport email for contact.

Is it sufficient to send it just to opt-in subscribers? Do we want to send it to all registered users instead? Can we do this? I can't see anything in the terms about registered users consenting to receive mail: https://www.transitionnetwork.org/terms

Maybe it has to just be subscribers. But it seems a bit arbitrary. I'd like to know if my password has potentially been compromised, even if I don't want a regular newsletter.

jim — Thu, 10 Apr 2014 16:40:02 GMT

Thought from Waterloo: Perhaps a new block on the top of the login page world suffice as a reminder? Then no terms to offend and no missing coverage of non-mc subscribers...

ed — Thu, 10 Apr 2014 16:42:42 GMT

Replying to sam:

Replying to ed:

Further to phone call:

Sam going to reset all admin, site admins, and editor passwords and contact all relevant people

I'll email people and let them know they are going to get a password request from the site and explain why. I'll request password resets from the site frontend https://www.transitionnetwork.org/user/password by just entering their emails.

I think this is better than manually changing their passwords & then sending the password in plain text email, which is in itself insecure?

ED: fine by me. To confirm we are only talking admin, site admin, editors.

Sending out a mailchimp announcement to the subscribers would be good. I'm away Friday but happy for Sam to do this on Friday if you're happy with that. Use the websupport email for contact.

Is it sufficient to send it just to opt-in subscribers? Do we want to send it to all registered users instead? Can we do this? I can't see anything in the terms about registered users consenting to receive mail: https://www.transitionnetwork.org/terms

Maybe it has to just be subscribers. But it seems a bit arbitrary. I'd like to know if my password has potentially been compromised, even if I don't want a regular newsletter.

ED: you're correct of course I'm rushed and being sloppy. It would be registered users. Please don't do this if we're not 100% that it's all been cleared and sorted and ready to go. I'm fine to wait until next week - the risk here is not huge, and I would like to participate in the editing of the message...

SO in fact I'm good for you to set this up, but want to see and help edit the message itself. Which is Monday at the earliest.

ed — Thu, 10 Apr 2014 16:43:42 GMT

Replying to jim:

Thought from Waterloo: Perhaps a new block on the top of the login page world suffice as a reminder? Then no terms to offend and no missing coverage of non-mc subscribers...

ED: what would this block say and do?

hours, totalhours changed

chris — Thu, 10 Apr 2014 17:14:32 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.05 to 1.3

I have updated these wiki pages with the new fingerprint:

jim — Fri, 11 Apr 2014 09:01:09 GMT

Chris, you might be interested to know about [http://drupalcode.org/project/barracuda.git/blobdiff/bbe22cb3ad5f94e58db14a8b6497370c5b17b56d..1c54c537485ff525cdd7cb13063f6331f5d1987a:/CHANGELOG.txt

this documentation tweak] making clear building SSL from source is not necessary on Debian Wheezy and Ubuntu Precise for BOA.

sam — Fri, 11 Apr 2014 11:09:15 GMT

remove all active sessions on our websites

That can be done at any time, but I think it needs to be done at the same time as the password reset?

Hi I'd like to do the password reset for admin's, but I don't want to jump the gun.

I can also see whether users are logged out by looking at their user pages: https://www.transitionnetwork.org/users/ben-brangwyn https://www.transitionnetwork.org/users/rob-hopkins

So if I:

1, Check user is logged out 2, Change their password to something they don't know (to ensure they don't continue using the old password) 3, Do a password reset via the site (so they can set their own password and I don't have to send it in plain text email to them) 4, Send them a mail explaining why this is happening.

Does that work?

Thanks

Sam

sam — Fri, 11 Apr 2014 11:41:18 GMT

On password security what does the panel think of this: http://imgs.xkcd.com/comics/password_strength.png ?

As advice to people choosing new passwords?

hours, totalhours changed

paul — Fri, 11 Apr 2014 12:09:09 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.3 to 1.425

@Sam

Maybe, since we can't be sure that they are logged out, and not just in-active for a few hours (maybe taking a break from writing that long blog post) we should send the email first and then do 2,3?

How would you do 3? Not clear to me right now, how you do that :)

I'll of course leave it to Chris to give the green light.

Best, Paul

hours, totalhours changed

paul — Fri, 11 Apr 2014 12:21:25 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.425 to 1.55

Regarding password security: I'm currently using 1Password and using a password generator to generate a unique 16 character password for each website.

https://agilebits.com/onepassword

http://random.pw/

sam — Fri, 11 Apr 2014 14:09:39 GMT

Hi Paul

I did wonder about suggesting a password manager. Do you think average users will get it? I'd be a bit hesitant about recommending a closed source one.

Has anyone used/ have an opinion on http://keepass.info/

It seems to support all platforms..

chris — Fri, 11 Apr 2014 17:24:27 GMT

Replying to sam:

On password security what does the panel think of this: http://imgs.xkcd.com/comics/password_strength.png ?

As advice to people choosing new passwords?

Yes it's good.

chris — Fri, 11 Apr 2014 17:28:59 GMT

Replying to sam:

I did wonder about suggesting a password manager. Do you think average users will get it? I'd be a bit hesitant about recommending a closed source one.

Happy for password managers to be suggested, but best if they are free (freedom) software and encrypt any data before they upload it, if they are uploading data to central servers. I sorry I don't know which one(s) fit in with this.

chris — Fri, 11 Apr 2014 17:35:54 GMT

Replying to jim:

Chris, you might be interested to know about [http://drupalcode.org/project/barracuda.git/blobdiff/bbe22cb3ad5f94e58db14a8b6497370c5b17b56d..1c54c537485ff525cdd7cb13063f6331f5d1987a:/CHANGELOG.txt

this documentation tweak] making clear building SSL from source is not necessary on Debian Wheezy and Ubuntu Precise for BOA.

Thanks for pointing that out, shame however that there is another variable which doesn't do what one would expect...

+  Note that _SSL_FROM_SOURCES=YES will not force the build from sources on
+  Debian Wheezy

chris — Fri, 11 Apr 2014 17:44:49 GMT

Is the email ready to go? Would it be best edited on a wiki page? Have we worked out what all the steps are that we are going to take and who is going to do them and whem? I can probably do the ticket:770 BOA upgrade tonight -- I'm now on the way back from the conference I have been at.

hours, totalhours changed

paul — Fri, 11 Apr 2014 18:00:22 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.55 to 1.675

Replying to sam:

Hi Paul

I did wonder about suggesting a password manager. Do you think average users will get it? I'd be a bit hesitant about recommending a closed source one.

Has anyone used/ have an opinion on http://keepass.info/

It seems to support all platforms..

I would feel uncomfortable recommending a closed source application to store passwords. I did investigate keepass for my mac, but I couldn't figure out quickly how to install the application. It's probably possible to compile from source, but I didn't want to go down that route.

chris — Fri, 11 Apr 2014 20:26:08 GMT

We do need to do a reset on everybodies passwords:

NSA Said to Exploit Heartbleed Bug for Intelligence for Years

The agency found the Heartbleed glitch shortly after its introduction, according to one of the people familiar with the matter, and it became a basic part of the agency’s toolkit for stealing account passwords and other common tasks.

http://www.bloomberg.com/news/2014-04-11/nsa-said-to-have-used-heartbleed-bug-exposing-consumers.html

We have been exposed since the upgrade to Wheezy ticket:535

sam — Sat, 12 Apr 2014 12:02:56 GMT

Hi all

Good idea on wikifying the response. I have done a draft here: https://trac.transitionnetwork.org/trac/wiki/HeartbleedAdminEmail

Let me know when would be a good time to do the resets/ emails

Thanks

Sam

chris — Sat, 12 Apr 2014 13:19:03 GMT

The email looks fine to me, are you clear how to truncate the sessions table and how to force users to reset their passwords?

sam — Sat, 12 Apr 2014 13:52:30 GMT

Hi Chris

I'm not sure on the sessions table. Google suggests; drush sqlq "TRUNCATE sessions" ?

Initially we are just talking about users with admin rights. I was going to:

Change their password to something they don't know (to ensure they don't continue using the old password)
Do a password reset via the site (so they can set their own password and I don't have to send it in plain text email to them)
Send them the mail explaining why this is happening.

If we later need to use something like: https://drupal.org/project/force_password_change to reset user passwords I guess we'd have to drop the sessions again?

Thanks

Sam

hours, totalhours changed

paul — Sat, 12 Apr 2014 16:30:43 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.675 to 1.925

it may be worth doing a little investigation to see how other sites are responding - and a little more reflection.

There may be arguments for only suggesting to users that they reset their passwords - to ensure that their account could not be modified in the future, if their account details have been compromised. If we went down this path we would need to ensure that a user who later gets additional privileges on the site is forced to have a password reset.

Removing all active sessions: drush sqlq "TRUNCATE TABLE sessions;"

An alternative to "Force Password Change" might be to run an SQL query on the user table to write a random password against each account and have a message below the user login block that asks a user to reset their password the next time they login to the site.

paul — Sat, 12 Apr 2014 16:33:03 GMT

Replying to paul:

it may be worth doing a little investigation to see how other sites are responding - and a little more reflection.

There may be arguments for only suggesting to users that they reset their passwords - to ensure that their account could not be modified in the future, if their account details have been compromised. If we went down this path we would need to ensure that a user who later gets additional privileges on the site is forced to have a password reset.

Removing all active sessions: drush sqlq "TRUNCATE TABLE sessions;"

An alternative to "Force Password Change" might be to run an SQL query on the user table to write a random password against each account and have a message below the user login block that requests a user reset their password before they can login to the site.

chris — Sat, 12 Apr 2014 18:25:10 GMT

Replying to paul:

it may be worth doing a little investigation to see how other sites are responding - and a little more reflection

Sounds good to me, I don't think we need to rush this -- rushing it and making a mess would be worse that not rushing and doing it right.

Perhaps the steps should be test on a dev copy of the site first?

I also think that given that the NSA appear to have been exploiting Heartbleed for two years and given that some users might use the same password for the transition site as other sites we really do need to consider doing a password reset for *all* users.

ed — Sun, 13 Apr 2014 06:49:14 GMT

NO action until the latest BOA upgrade problems are resolved

hours, totalhours changed

paul — Sun, 13 Apr 2014 11:03:40 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.925 to 2.175

Replying to chris:

I also think that given that the NSA appear to have been exploiting Heartbleed for two years and given that some users might use the same password for the transition site as other sites we really do need to consider doing a password reset for *all* users.

Agreed.

.. it's best to assume that all your passwords have been compromised, and you should change them, everywhere. http://www.freelock.com/blog/john-locke/2014-04/heartbleed-do-you-need-do-anything

paul — Sun, 13 Apr 2014 11:04:54 GMT

Replying to ed:

NO action until the latest BOA upgrade problems are resolved

I copy that.

ed — Mon, 14 Apr 2014 07:52:58 GMT

Sam: you are doing just the admin passwords ASAP, correct? That is my understanding of our conversation on Thursday.

Anything we do to all the other accounts needs discussion. All of the other service providers I've received emails from have sent a DIY password update only - ie no forced reset.

The password resetting process is dodgy - I've had problems with it from the beginning. Forcing all our users to do it could be utter carnage.

jim — Mon, 14 Apr 2014 08:04:00 GMT

Hi all,

So a few things:

I'm not certain, but pretty sure the session timeouts will have passed for all since the 'outbreak' - so no point truncating Sessions table now as no old ones will exist. But simply truncating Sessions is trivial.
Regarding Ed's points 2 and 3: that's why I proposed a block on the login page. You'd catch people who wanted to change their passwords, but not force them -- Ed's point regarding the HUGE change management piece around resetting all passwords is very important. And there's just no point IMHO -- stick to admins/devs only.
The block could say: "Blah heartbleed blah, we take security seriously blah. If you are in doubt or want to ensure the highest level of security, please reset your password here. Blah blahdy blah-blah, kisses x"

Otherwise the response so far has been excellent.

Best,

Jim

sam — Mon, 14 Apr 2014 11:10:06 GMT

Hi all following Jim's comment on the session tables I have now reset the passwords for all users with the roles: Site Administrator or Developer.

Thanks

Sam

sam — Mon, 14 Apr 2014 14:11:28 GMT

I just reset passwords for users with 'site editor' role too.

Thanks

Sam

ed — Mon, 14 Apr 2014 14:37:43 GMT

Good work Sam. So we're square and safe for now.

I really like Jim's suggestion of adding a block to the login page.

hours, totalhours changed

paul — Mon, 14 Apr 2014 16:02:32 GMT

hours changed from 0.0 to 0.25
totalhours changed from 2.175 to 2.425

I like Jim's suggestion too.

I think we could also rewrite the password against every user account - to something random - and advise users on the login block / page that they will need to reset their password to login again? It would be easy to do this now, and it would avoid awkward conversations later when someone asks if there account is secure . If we don't force a password update on all user accounts now, we will need to ensure that we do this for any user account that later is given more privileges on the site.

Best, Paul

hours, totalhours changed

chris — Thu, 01 May 2014 09:20:30 GMT

hours changed from 0.0 to 0.25
totalhours changed from 2.425 to 2.675

I forgot to update the SSL key and cert on wiki:ParrotServer, it is used for pages like this one:

https://parrot.transitionnetwork.org/myip.shtml

On wiki:ParrotServer, enable root ssh connecting s by editing /etc/ssh/sshd_config and edit:

PermitRootLogin yes

Restart ssh on wiki:ParrotServer, and on wiki:PenguinServer run:

rsync -av /etc/ssl/transitionnetwork.org/transitionnetwork.org.* parrot:/etc/ssl/transitionnetwork.org/

Then on wiki:ParrotServer test and restart apache:

apache2ctl configtest
Syntax OK
apache2ctl restart

And edit /etc/ssh/sshd_config and edit:

PermitRootLogin no

And restart ssh.