Skip to content

Instantly share code, notes, and snippets.

@rjhansen
Last active October 1, 2022 04:28
Show Gist options
  • Save rjhansen/f716c3ff4a7068b50f2d8896e54e4b7e to your computer and use it in GitHub Desktop.
Save rjhansen/f716c3ff4a7068b50f2d8896e54e4b7e to your computer and use it in GitHub Desktop.
SKS Keyserver Network Attack: Consequences

SKS Keyserver Network Attack: Consequences

This work is released under a Creative Commons Attribution-NoDerivatives 4.0 International License.

Back in late February, the Internet Freedom Festival put together a roundtable of communications security nerds to help dissidents in Venezuela figure out how to organize and communicate in the face of widespread DNS poisoning. I contributed a brief HOWTO explaining what the Maduro regime was doing and some simple, effective mitigations. At the very top of the HOWTO was a paragraph of security considerations. Chief among them was a caution that this document came with an OpenPGP digital signature: before relying on the information in the document they ought ensure nobody had tampered with it, either to install malware into the PDF or to alter the advice I was giving.

I put this HOWTO out in the wild. I've had four people send me thank-you notes for writing it. I figure that means it's been seen by between four and four hundred people. The numbers aren't important, really.

What's important is my instructions told them to check the digital signature. And today, if they do this it is overwhelmingly likely they'll get a poisoned certificate from the keyserver network and their GnuPG installation will break horribly. They're in Venezuela, they're trying to organize for democratic change, they're being surveilled by Maduro, and now this … I don't even know what to call this attacker. There are no polite words. There is only "this sonufabitch."

So this sonufabitch has broken one of the important tools used by Venezuelan activists.

And then I start thinking about all of the other groups I've supported in the last year. Journalists covering Syria and Gaza, for instance. Or …

I gave them excellent advice. Check the signature on this document before you trust it.

And now, thanks to this sonufabitch, if they follow the best practices of communication security they will horribly break their tools.


For much of the keyserver network's existence we had literally no complaints about privacy. Literally none. Around 2000 people began to fear that by putting their email address in the keyserver network they might open themselves to receiving spam. There was some foundation for that fear, but the consensus opinion from the people who actually knew what was going on was that it was foolish to base your anti-spam strategy on keeping the spammers from discovering your email address.

So we ignored it. "Yep, there's a minor risk there, yep, we've done experiments, yep, posting your email address on the keyserver network is associated with a slight increase in spam received. But the increase is so slight we're not going to fix it. Would you like some advice on effective spam filtering?"

People who understood the spam fight agreed with us. But a whole lot of people didn't and made their displeasure — and very often their shocking ignorance — known at high volume.

We ignored them, which was the right thing to do. You don't let people who don't understand problems dictate which problems will be solved.


Then came September 2010. One keyserver operator in the European Union, Peter Pramberger, found himself facing a lawsuit by an OpenPGP user who was angry he could not have his email address deleted from the keyservers. Under EU data privacy regulations he had the right to demand this, but the keyserver network specifically lacked the capability to comply. It was designed that way for a reason.

Peter had no good way out. It would cost him tens of thousands of euros to take this through a court process, and at the end of the court process there was no guarantee he would be excused from an impossible requirement. Peter took down his keyserver.

September 2010 began the Great Assault on the Keyserver Network. All of us knew the handwriting was on the wall. All it took to strike down a keyserver in the EU was a couple of pages of privacy directive and the threat of a lawsuit.


The privacy trolls came out in force after that. They didn't want to hear technical explanations of difficulty, or lack of resources, or lack of competent people, or the fact none of us were paid and none of us could afford to take a year off work to do the overhaul they demanded.


Special criticism goes to the Electronic Frontier Foundation, which paid Micah Lee to publish premade attack tools to exploit these design misfeatures in the keyserver network. Oh, sure, "academic freedom" and "it was about research". I don't know if Micah's trollwot toolkit was used in the most recent attacks. I know that if I was writing an attack tool that's where I'd start from.

Academic freedom should not be construed as permission to publish attack tools against a critical service with known vulnerabilities. Publishing a proof of concept is great and completely within the bounds of acceptable behavior. Publishing attack code is not.

Thanks for nothing, EFF.


January of last year — January 16, 2018 — one user threatened to do what pretty much happened this week. I'm not going to name this jerk, but it was a public message: you can find it in a public mailing list archive somewhere, if your Google-fu is good enough.

How about this; let's make "your" public key the ideal canditate for a global trollwot session, were every GnuPG Linux user can participate and add some funny things to "your" public key. This would be also interesting to see how many signatures a public key can bear.

Maybe people can do also other things with "your" pub key and post the used techniques here, like i did in the past with Erika Mustermann's pub key and the added fake sig from Werner.

This would imho give you and people you talk to in conferences etc. also a better view what i am talking about.


Let me make it clear, I'm not accusing that guy of being the responsible party. I don't have evidence to support that, or any other attribution. That jerk's raw sense of entitlement, and his callous talk about how he's vandalized certificates in the past, is not unusual to see among GnuPG's users. It's a long list.

But I tremble with fear for how people I know in hostile regimes are currently at risk of having their tools broken — and then I look at the preening self-righteousness of those louts who feel entitled to burn things down just to make a point.

They're children, the lot of them, children with matches and gasoline who are prancing and cavorting in our living rooms.

They have no idea the damage they do.

And more to the point, I genuinely don't think they care, either.

@Mikotochan
Copy link

January of last year — January 16, 2018 — one user threatened to do what pretty much happened this week. I'm not going to name this jerk, but it was a public message: you can find it in a public mailing list archive somewhere, if your Google-fu is good enough.

You must mean that one: https://lists.gnupg.org/pipermail/gnupg-users/2018-January/059751.html
No offence but I think that you are the jerk here, if only for the fact that you decided to quote it out of context.

@neilalexander
Copy link

@rjhansen

The privacy trolls came out in force after that. They didn't want to hear technical explanations of difficulty, or lack of resources, or lack of competent people, or the fact none of us were paid and none of us could afford to take a year off work to do the overhaul they demanded.

It is the responsibility as people with significant influence over the SKS network to be competent.

You all seem to have a very slippery attitude and that's what makes SKS dangerous. You don't care about protecting users, you don't care about privacy, you don't care about fixing the flaws in SKS and you don't care about letting anyone else. What exactly do you all care about?

I realise the SKS doesn't make promises to users but keyserver operators also do nothing to advise users of the risks before they take to using the service. Where were the disclaimers over the last couple of decades saying that data submitted is burned into history forever with absolutely no withdrawal or modification capability?

The SKS network has been collecting user data for a very long time in a fashion that can't be reversed and now people are waking up to that. If SKS cannot be fixed to solve that problem then it is right that SKS should die and be superseded.

@DigitalBrains1
Copy link

DigitalBrains1 commented Jun 30, 2019

just to make the problem visible to users (without actually causing much damage)

Really? He mentions the four to four hundred Venezuelans who might refresh rjh's key when verifying the signature on the mentioned PDF document. Similarly, anyone that has done something like --refresh-keys including rjh's or dkg's key, and then a trustdb check, will make their GnuPG stop in its tracks completely (I didn't check, seems logical). Again without checking, I'm pretty sure both keys are part of the strong set, and there are a lot of certification chains including their keys.

People have been working on these keyserver problems, and it would be good if exposure of the issues lead to extra manpower. Unfortunately, there is a distinct shortage of interested qualified developers, and money from industry. This has been the case for a long time in the OpenPGP ecosystem.

But exposing the problem by making the installations of people depending on OpenPGP for their personal safety grind to a halt? Doing this to any system that does a trustdb check with two very prominent public keys in any certification chain? I can completely understand that this type of actions makes rjh blow a fuse and then some. This is making your point over the very backs of very vulnerable third parties, in a truly horrendous way. I'm at a loss for words to express how horrible it is, and I really, really hope there will be no casualties or people getting caught by torturing regimes because of these boys with their toys.

But hey, at least they didn't target a Linux distribution, because that would truly have been bad, eh?

PS: Even though this will dilute the poignancy of my last point, I still can't help but observe that dkg's key is in a public keyring on the machines automatically checking uploads from Debian maintainers. I hope the Debian project doesn't refresh those keyrings with keyserver data, because it would probably prevent any upload of new packages to the distribution.

< edit > Changed "distinct lack of developers" to "distinct shortage of developers". Meant to say "not enough", not "without"; avoid misinterpretation.< /edit >

@horenmar
Copy link

@DigitalBrains1 I am gonna need some proof that GPG actually meaningfully does anything for the activists. As a reminder, this is not some crazy zero day that requires a lot of technical know how to exploit. Nor does it require a lot of resources that only well funded state level actors can bring to bear. This attack can be literally done by a somewhat technically competent individual using the computing power equivalent of a leaf blower.

So here are the possibilities as I see them

  1. The oppressive regime of your choice does not posses a single computer and a technically competent individual to execute this attack -- does not seem likely
  2. The current attacks are indeed done by one of the oppressive regimes you like talking about -- given the targeting, also seems unlikely
  3. GPG encryption is so useless (in the practical terms) that it isn't on the radar of any oppressive regime and your posting about the poor users in danger is just about feeding your ego

@hellerbarde
Copy link

hellerbarde commented Jun 30, 2019

@horenmar

I am gonna need some proof that GPG actually meaningfully does anything for the activists.

Well, at the very least, verify that the info they were given by rjh was correct. I'm not sure why you would be sceptical that a strong encryption would be helpful for political activists under an oppressive regime. Your assessment of the situation in those three points is very short-sighted. I won't address them because they are largely irrelevant to what I want to say next.

I will continue this under the assumption that this was a proof of concept attack to force action.

The silver lining in this attack is that action is now immediately needed.
But what riles rjh (imho understandably) up so much is that potentially, this exploit has now compromised the security (because comms are essentially down now) a number of non-tech-savvy activists or journalists for the very entitled reason of "I think this problem is big enough, so I'll burn it all down, kthxbai". Yes, the system needs to be fixed. Yes, the reason is now clear to everyone. No, this doesn't mean there are magically enough people knowing OCaml stepping up to the plate. No, this was not worth endangering people's lives. Not everyone keeps up with all the tech news and can now just adjust their config and fix this.

There are tools where it just has to work for ordinary people. Where it's vitally important that we don't just metaphorically burn down the town just to make sure the creaking door in John Smith's house is fixed!

PS: I said potentially. That is enough. We can't always have the data. It's enough that someone might be hurt or worse to argue that it's unethical to launch this attack. We are not entitled to break shit just because it's "right there, anyone could have done it too". (because they didn't)

cheers

@horenmar
Copy link

And here I was, thinking that "think of the children" has stopped being an argument years ago.

@706f6c6c7578
Copy link

@rjh what is your point of view (let's forget for the moment that many sigs) about sigs in general now, when someone is looking at your key?

https://keyserver.escomposlinux.org/pks/lookup?search=Robert+J+Hansen&op=vindex

@nealmcb
Copy link

nealmcb commented Jul 8, 2019

I have empathy for all concerned here.
And I think that all concerned would be better served by pointing to more effective ways forward.

I'm glad to see the availability of https://keys.openpgp.org, but I think we should also be identifying more usable and secure options for the freedom fighters of the world, since OpenPGP, while critical and transformative in its day, has more usability problems than it does technical vulnerabilities.

A far, far more usable way to collaborate securely when you don't have some of the more challenging requirements, for not just PGP keys, but chat, document collaboration (encrypted git) and a payment network, is [Keybase[(https://keybase.io). It has some centralization problems, and perhaps others, but has lots of good open-source code also. What would it need to address even more freedom fighting use cases?

What other suitable alternatives are out there now to be recommended or built upon?

I also note that the reference to a "keybase-fs" should be "keyserver-fs" and is actually pointing (appropriately) to https://github.com/yakamok/keyserver-fs".

@lukateras
Copy link

lukateras commented Jul 8, 2019

I also note that the reference to a "keybase-fs" should be "keyserver-fs"

Fixed, thanks!

What other suitable alternatives are out there now to be recommended or built upon?

Outside of OpenPGP ecosystem, depending on your use case: signify for signatures, Signal for forward secure, end-to-end encrypted communications, Borg for backups. Signal is better with regards to metadata leaks than XMPP + OMEMO, because roster, avatar, and message recipient are not known to the server.

For asynchronous messaging specifically, there's nothing well designed available yet but plenty academic research is there that can be leveraged to create a much better end-to-end encrypted MUA via puncturable encryption (paper). Even then, given the best possible implementation, this will still leak some metadata through mandatory plain text recipient header, and due to federation, there's fundamentally no way to address that in full.

@lukateras
Copy link

What would it need to address even more freedom fighting use cases?

The most glaring issue is that Keybase doesn't offer forward secrecy, except for ephemeral (exploding) messages.

@nealmcb
Copy link

nealmcb commented Jul 8, 2019

Yes, there are varying approaches around forward secrecy, with different implications for users, which fit a variety of use cases.

Keybase discusses the options at https://keybase.io/blog/keybase-chat noting:

Apps with FS on all messages must choose one of these:
Choose 1
COMPROMISE #1: all messages, photos, and videos get lost forever on a device upgrade
COMPROMISE #2: violate Forward Secrecy by letting users back up messages or otherwise transfer them around.

I like their approach of letting users choose their option via either device keys (which, being tied to devices, are more secure than OpenPGP keys) or ephemeral exploding messages.

@706f6c6c7578
Copy link

For asynchronous messaging specifically, there's nothing well designed available yet but plenty academic research is there that can be leveraged to create a much better end-to-end encrypted MUA via puncturable encryption (paper). Even then, given the best possible implementation, this will still leak some metadata through mandatory plain text recipient header, and due to federation, there's fundamentally no way to address that in full.

To address the problem of leaking metadata Anonymous Remailers can be used, which are in service since the early '90s, when it comes to smtp protocol usage. Other anonymous services like Bitmessage, not relying on smtp, could be used as well.

https://github.com/merkinmuffley/mixmaster4096
https://github.com/crooks/yamn
https://github.com/Bitmessage/PyBitmessage

@lukateras
Copy link

Keybase discusses the options at https://keybase.io/blog/keybase-chat noting.

Sure, saw that. The problem lies within compromise 2: there is no "forward secrecy violation" in allowing user to backup chats. Not shipping that for all chats is a strict security downgrade from designs like Signal's double ratchet. Non forward secure schemes basically deny user ability to remove a message without destroying the whole identity that it was encrypted for.

@rain-1
Copy link

rain-1 commented Jul 9, 2019

We have a fix in GPG now https://lists.gnupg.org/pipermail/gnupg-announce/2019q3/000439.html

Ignore all key-signatures received from keyservers. This change is required to mitigate a DoS due to keys flooded with faked key-signatures.

@706f6c6c7578
Copy link

You all seem to have a very slippery attitude and that's what makes SKS dangerous. You don't care about protecting users, you don't care about privacy, you don't care about fixing the flaws in SKS and you don't care about letting anyone else. What exactly do you all care about?

This seems to be the case from the early days on. I won't call names but everybody can check this out by themselves.

A co-author from a now obsolete OpenPGP RFC has a nasty signature on his pub key, dated back from 1997. Would you, or
anybody else, then not suggests years later changes to the RFC specs, to protect users (keys) better?

https://tools.ietf.org/html/rfc2440

The same question can be asked the author of GnuPG too after he discovered years ago funny sigs on his pub key.

@nealmcb
Copy link

nealmcb commented Jul 10, 2019

Another best-in-class tool that folks should know about for the signed software release/update use case is The Update Framework (TUF). It address not only signatures on packages, but also critical and complex areas like key rollover, rollback attacks and much more.

@paulyc
Copy link

paulyc commented Jul 13, 2019

It's pretty standard amongst security researchers to publish proof of concept code for security holes that go unfixed after proper notice and waiting period. It's hardly the EFF's fault that OpenPGP is such an unmaintainable POS that not only did this hole go unpatched after YEARS in the wild, but it's unclear that anyone is capable of fixing it at all. Pretty ironic considering all the troubles one must go through with key revocation when a key is compromised. Now we have a key that hasn't been compromised yet is still a security hazard and CAN'T be revoked. Just because it can be a useful tool doesn't exempt it from proper development best practices. I assure you this still just as easily could have and would likely have happened without anyone publishing a POC because it's a known vulnerability, that's why they get published, so the devs can fix the error, or the users can avoid using it and be prepared for attacks.

@oldmud0
Copy link

oldmud0 commented Jul 14, 2019

I don't think the venting seen in this post is effective in the long-run - it exposes one to personal attack. The reality is that gpg is the de-facto standard in the software of today. While it might not be easy to abandon the SKS keyserver overnight, and perhaps I am not following the news around this event very closely, but could it be possible to make a simple modification to SKS so as to push a security (or even deprecation) notice to users when receiving a key? This should accelerate the spreading of this news - and of whatever nasty issues SKS might be hiding.

@paulyc
Copy link

paulyc commented Jul 14, 2019

the only part of "privacy" that really applies to PGP is the content of the messages you're sending. It's not intended to hide the sender and recipient of the messages, or who knows who, in fact the whole keysigning model relies on that being public information for building its trust model: i.e., it's horribly antiquated and broken for all practical purposes and should be abandoned altogether. Want verification, get a domain name and sign it in the DNS, still vulnerable to attack but at least it's been proven scalable, and at least registrars are aware of the fact that people are constantly trying to hijack domains and have some sort of institutional defense against it.

Not saying all PGP, is bad it does its job of encrypting decrypting signing verifying great, but relying on it for any sort of anonymity guarantees you will have none and even trust has little meaning if your keys have been compromised without your knowledge. This attack was just one symptom of a fatally flawed system. Patient DOA. The mere fact that you are sending and receiving PGP messages is more than enough reason for any government to flag you as a potential enemy of the state, and if they ever find messages linked to your public keys in an enemy of the state's data, you're now an enemy of the state as well regardless of the content of the messages.

@yakamok
Copy link

yakamok commented Jul 17, 2019

@706f6c6c7578
Copy link

706f6c6c7578 commented Aug 14, 2019

Regarding GDPR ... SKS key server operators, like Mr. Rude, should better get familiar with this article:

https://www.cmdsonline.com/blog/the-looking-glass/gdpr-us-websites/

because he provides key dumps to the whole world without any consent from EU citizens.

https://keyserver.mattrude.com/dump/

@ILMostro
Copy link

ILMostro commented Oct 27, 2019

This all seems like a masterful scheme of social engineering to implode the oft-used cryptography tools (PGP) through privacy-minded laws (GPDR) and organizations (EFF and others). Divide and conquer, for lack of a better phrase. Ultimately, the end-goal is the same: ensuring that personal freedoms are protected (try not to read that with too much cynicism).

Why not focus on how best to resolve the issues instead of personal attacks?

@paulyc
Copy link

paulyc commented Oct 28, 2019

This is a fantastic article: https://latacora.micro.blog/2019/07/16/the-pgp-problem.html

That pretty much sums it up. I think the only people who actually like it are three/four-letter agency G-man-types. Sorry, G-person-types.

Regarding GDPR ... SKS key server operators, like Mr. Rude, should better get familiar with this article:

https://www.cmdsonline.com/blog/the-looking-glass/gdpr-us-websites/

because he provides key dumps to the whole world without any consent from EU citizens.

https://keyserver.mattrude.com/dump/

Not anymore! I guess the EU bureaucrats finally tracked him down then? He should put up one of those Europol seizure/shutdown notices

@merkinmuffley
Copy link

Around 2000 people began to fear ...

How many people? Ah, around the year 2000 people began ...

@mooleshacat
Copy link

mooleshacat commented Dec 21, 2021

This to me is just like how MSU (I think it was?) tried to poison the Linux Kernel repos. I said about that if they did not ban the entire domain, people would lose trust in the Linux Kernel, and as the Linux Kernel runs most of the internet, it would effectively break the internet. I also said the researchers in this case should find themselves on the other end of a lawsuit from the fortune 500 companies running Linux.

We have a similar situation here, only this time, the researchers were able to exploit the keyservers, and break GPG for everyone. Now installations are left insecure without keys to verify authenticity. I wouldn't be surprised if these researchers also were to find themselves on the other end of a lawsuit from the fortune 500 companies running Linux.

As an end user I can say this keyserver mess has caused me lots of problems just updating my system due to 'NO_PUBKEY' and nowhere to get the keys from. Every keyserver either says 'no name' or 'no data'.

Crossing my fingers...

@Tachi107
Copy link

Tachi107 commented Aug 1, 2022

If you have to rely on people's good will to not break your great tools, maybe your tools aren't that great after all...

To be clear, I'd love if the OpenPGP community found a solid solution to the bad state of OpenPGP and GnuPG, but you can't really blame people for exploiting vulnerabilities, you should acknowledge the facts and fix them.

1PA3PC (Attested Certifications) looks nice btw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment