Skip to content

Instantly share code, notes, and snippets.

@raggi

raggi/cloudbleed.md

Created Feb 25, 2017
Embed
What would you like to do?
Some advice on appropriate cloudbleed response

Cloudbleed: Users can't fix themselves

TL;DR

Site-owners MUST invalidate all their users sessions and tokens now, and might want to consider doing so again in the not distant future.

Cloudflare domain lists

There has been a great deal of discussion about the Cloudflare domain lists that are being constructed. These lists are not really all that useful for cleaning up this mess. Let me explain.

The exposure of this issue is not limited to Cloudflare frontend domains. It's all the services upon which those domains also rely. Now you might think "ok, so we need a list of domains that depend on stuff served from known Cloudflare domains" - but no, that is also insufficient. Take for example, that a site you depend on, say a customer service SaaS uses Fastly to serve up their assets. What you don't know as a consumer of that service, is that they use an analytics service on their backend administration portal that is served up from Cloudflare. That portals admin capabilities may be compromised, and by implication it's screenshot capabilities may now be in the hands of someone other than the company you have a contractual agreement with. Neither you, or a user can fix this directly, or even identify it, and it has nothing to do with domain lists.

Passwords & Sessions

Take the example scenario above. This could be executed without any knowledge or need for passwords. It's most likely that such a scenario will only involve some cookies, and it's also very likely that those cookies will be long lived, way beyond reason.

There is still good reason to try rotating some passwords. Doing so might invalidate sessions on your other devices and any an attacker might find in a cache. Sadly this only protects you against the scenario where: a) it's your credentials, b) the system actually invalidates your credentials, and c) the system is not presently compromised by this event due to some upstream credential issue.

The fact is that most of the data in caches is response data, not request data, and that most of the time on most sites, response data doesn't contain passwords. It does however contain session credentials.

Let’s talk about the session credentials some more. There's been a ton of malpractice going on in this area for years. We don't like signing in, and transporting passwords regularly is potentially unsafe as well as leading to bad user behavior due to the UX. We still haven't deployed 2fa widely enough, so signing in too regularly actually increases risk. We also don't rotate the system-managed credentials (cookies and tokens) often enough, and you'll observe that most 1p oauth and oauth2 solutions don't even mandate the use of refresh tokens if they even really ever expire any tokens at all. If refresh tokens are pushed to web clients on the regular then those are at higher risk from these events, and mobile apps do this all the damn time.

What's even worse with the credential problem is that I've seen a ton of "authentication libraries", such as middleware for Rack and WSGI applications, among many many other languages and platforms, that don't even handle mass credential invalidation cleanly. What I mean, explicitly, is that if you rotate your master key that encrypts and/or signs your user sessions, there's a decent chance the third party software you rely on will screw things up for your users really badly. Common examples are to cause redirect loops where software doesn't correctly overwrite cookies or handle oauth failures, and in the case of many mobile apps, forcing users back to the first-time UX that clears all user data from the device - sometimes this is a catastrophic data loss for the user.

So should you be rotating your passwords? Probably. When? Probably now and later, because some things might be transitively compromised right now.

As a user though, you have no way to know if your new password was applied safely right now, and you have no way to know if it will have the desired most important effect of clearing all your other sessions. You should be asking your vendors to invalidate all of your sessions. All of your vendors and all of your sessions.

Upstream vendors

If you own a service, you should be asking your upstream vendors if they have rotated their session keys. It doesn't really matter if they use Cloudflare, because you don't know what they use that might depend on it. See the point above.

A truly exceptional vendor should really be clearing sessions now, as a precaution, and once they get the green light from all their dependencies that they have also done so, doing so once more. Something could still be backdoored, but this is so far beyond what most people will do it's about as far as I can bring myself to recommend.

Mobile

On Android you can use the "clear data" feature from application settings, which should wipe out most remote credentials for apps. Unfortunately some apps really don't recover well from this, and I don't really know how to fix that. This also won't do anything to invalidate your upstream sessions, that is, it won't message servers about it. The servers will very likely still trust your old tokens and as a user there's probably absolutely nothing you can do about this. Again, you really should be pressuring vendors into clearing all stored sessions.

"Safe" vendors

Some more major vendors such as Google and Facebook might seem safe, and in some ways they are. They have large world class security teams that ensure that their products have a lot better practices around credential management. That said though, that doesn't mean that something that uses "google sign-in" or "facebook sign-in" is safe now. There are a number of problems around this. If you've read top-to-bottom then you already know that transitive and upstream administrative exposure of credentials puts you at risk, but there are more issues afoot. It's often possible to use these vendors in a configuration where you don't validate the session against a remote server, instead you just validate a cryptographic signature of, for example, a JWT and check a few timestamps. The problem with this approach is that if a user has a known or unknown need to invalidate those credentials, and they've done so already, you're still trusting that cached value, because it's still cryptographically valid and still within its expiry time. Don't do this, you're preventing users from fixing themselves.

Conclusion

Users really can't fix themselves extensively as a result of this event. Cloudflare can't assert that the world is safe again, and neither can anyone else. Vendors need to be taking action, and we should all be expecting to be signed out of all services and apps for a while. Failure of a vendor to do so is either very bold or somewhat irresponsible, unless they have in place a really extensive policy base and very few or no third party dependencies anywhere in their stack, auth and frontend sides. In practical reality, this means that most vendors out there need to be invalidating their session keys. Everyone, everywhere. Get to it, please.

Disclaimer

This is my personal opinion. I've solicited some review from people involved and people who work in security and they agree that this is mostly sane advice. I'm not a full time security person though, and this is my personal opinion. I made this public to provide a friendly suggestion to the industry. You take responsibility for your choices and actions.

@pirate

This comment has been minimized.

Copy link

@pirate pirate commented Feb 25, 2017

This is true, session keys are a much bigger problem than passwords, I retweeted a PSA about it yesterday. We've updated the list at pirate/sites-using-cloudflare to reflect this in the README as well https://github.com/pirate/sites-using-cloudflare#what-should-i-do.

I have an issue with one of the assumptions made here though, and that's that sites are using Cloudflare for internal upstream services. AFAIK Cloudflare is mainly used as the user-facing CDN and DNS provider, very few sites use it for internal API services, as that's not really what it's designed for. The purpose of our domain collection list is to give people an idea of what end-user facing domains may have been affected, we cant possibly hope to scrape all internally used domains, and I don't think it's as important to attempt either. If you know of sites using Cloudflare for internal services, please let me know, as that greatly changes our assumptions about attack surface.

@coderobe

This comment has been minimized.

Copy link

@coderobe coderobe commented Feb 25, 2017

This is a well-written post, although i suggest removing the Mobile part, because locally clearing caches is not going to do anything but cause problems for end-users. This has to be mitigated server-side.

And what @pirate said, CF is mostly used for caching the user-edge services, not APIs.

@raggi

This comment has been minimized.

Copy link
Owner Author

@raggi raggi commented Feb 25, 2017

@pirate: many services have back office Saas products integrated with their web hosted sales, customer service, operations, inventory etc systems. Those back office products are mostly not visible to users, but some might be (e.g. zendesk). When it's a third party UI integration that's just as likely to be CDN hosted as other UI gear for Saas products. This isn't about APIs, it's about back office UI integrations, and those are common across billing, customer service and inventory systems among others. Think the likes of zuora, zendesk, etc.

@coderobe: hoping that a password reset will clear sessions is about as reliable as generating a new mobile session. Neither are really any good here, but if we're recommending password resets it's also valid to recommend trying to let old mobile sessions die by cycling to new ones. As is the core point of all the above though, the only way to get any reasonable confidence of safety is by vendors taking actions, not users.

@pirate

This comment has been minimized.

Copy link

@pirate pirate commented Feb 25, 2017

@raggi true on points. I think many of the "back office" SaaS products are listed in our domain list though, they're equivalent to user-facing products where the user is a business.

Re: password changes, on properly implemented sites, changing your password should rotate session tokens, but very few sites get this right. https://twitter.com/marcan42/status/834990296681193472

@raggi

This comment has been minimized.

Copy link
Owner Author

@raggi raggi commented Feb 26, 2017

Right, as I note above not only to many sites get it wrong, but also many libraries people use actually break when you try to force a mass rotation. Just as common is that folks e2e / web / selenium / manual test cases don't cover this use case and so companies often don't find out until they try to do it in production. That's why I talk about it up there, and the issue of clearance or not is why I'm saying that it is vendors that need to do the work, not users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment