I want to write software that helps kill people.
Please, before you call the police and get my github account put on lockdown, allow me a moment to explain. What I really want to do is work on projects that advance the human condition and improve people's lives. I've been in a mad dash to learn how to program for the past four or five years exactly because I realized how much good I could do for the world with a computer.
The open source software movement has produced my tools, provided my teachers and mentors, and, recently, has become a major focus of my attention and time. I can whole heartedly say that, at the tender age of 22, open source has already directly had a massive and positive benefit on my life. I'm reasonably certain I would be orders of magnitude less well off and less happy if open source didn't exist as a concept.
Which is why a realization I had a few weeks ago has caused me so much personal strife lately: open source software is used to help kill people. Specifically, Palantir Technologies uses open source to identify who should be killed. In a world where the American military probably has enough power to kill everyone currently alive, the hard questions have shifted from how to effectively kill the most people possible to who should be killed. Palantir is in the business of providing the 33 bits of information needed to identify the people who pose enough threat to America's national security that they need to be eliminated.
Consider the following quote, taken from a profile of Palantir in Business Week:
In Afghanistan, U.S. Special Operations Forces use Palantir to plan assaults. They type a village’s name into the system and a map of the village appears, detailing the locations of all reported shooting skirmishes and IED, or improvised explosive device, incidents. Using the timeline function, the soldiers can see where the most recent attacks originated and plot their takeover of the village accordingly. The Marines have spent years gathering fingerprint and DNA evidence from IEDs and tried to match that against a database of similar information collected from villagers. By the time the analysis results came back, the bombers would be long gone. Now field operatives are uploading the samples from villagers into Palantir and turning up matches from past attacks on the spot, says Samuel Reading, a former Marine who works in Afghanistan for NEK Advanced Securities Group, a U.S. military contractor. “It’s the combination of every analytical tool you could ever dream of,” Reading says. “You will know every single bad guy in your area.”
The last sentence is what got me. Palantir uses open source software to tell soldiers when they should be pulling the trigger. The statement "the Python programming language murdered a 15 year terrorist in Iraq" is rickety logic at best, but it does contain a discomforting modicum of truth.
Here is a list of open source projects that Palantir probably uses or plans to use, as indicated by their jobs posting, blog posts, and various tech talks scattered across the web. While Palantir does much, much more than identify "bad guys", I'm lumping all of the listed projects, tools, and languages into the same conceptual bucket simply for lack of detailed information about what happens behind closed doors. I don't know that these projects are directly involved in informing soldiers, but there is a higher than normal chance that they are directly involved compared to a random project.
Languages:
- Bash
- Coffeescript
- C++
- Groovy
- Java
- Javascript
- Perl
- PHP
- Python
- Ruby
- Scala
Front end[0]:
- Android
- backbone.js
- Java Swing
- less
- rrd4j
Back end:
- Cassandra
- Chef
- Hadoop
- HyperSQL
- JAXB
- jMock
- JMX
- Linux/UNIX
- Log4J
- Lucene
- Nagios
- Postgres
- Puppet
- Rails
- Spring
- Zenoss
The vast majority of the above projects are large and successful. Tens of thousands of developers have been involved with helping build them. In Linux's case alone, there are about 10,000 people who have been directly involved over the past few decades. There's probably an order of magnitude more people who have submitted bug reports (think automated Ubuntu crash reports) and at least another two orders of magnitude who have helped harden Linux via sustained and heavy use.
The questions I've been struggling with is the relationship of the developer and how these libraries are used. The above libraries are general purposes tools that can be used for most anything. They do a tremendous amount of good for humanity as a whole and the world is better off for having them. The question I've been asking myself is "How does the use of a tool reflect back on those who developed the tool?" The purposes of these libraries are not to kill, but they have been used by Palantir to help kill someone[2].
To start with, I've publicly written before about how to use backbone.js. If the front end folks at Palantir saw that[1] and figured out a better way to display lists of "bad guy" names or something trivial, I wouldn't feel so bad about that. That information is out there regardless and the Palantir folks are smart enough that they would've figured it out within a few minutes anyway. Frequenters of stackoverflow needn't worry that their answers have made the difference between life and death for someone.
The same goes for people who have submitted even the most detailed bug reports. These people have pointed out how to harden the library and thus made it more reliable, but that is mostly incremental progress. Just because Postgres happens to not print out a comma when you use a certain SQL statement won't stop Palantir from identifying who should die. Even reports of dire, world ending bugs don't bother me much. I haven't written any of the code that is executed when Palantir executes some query looking for enemy combatants.
Now, let's take the obvious next step up. What if I submit a pull request to Cassandra that fixes a bug? Every time a soldier tells the Palantir suite to go talk to Cassandra, asking for updates on the current status of a battle and find out where the person trying to kill him is hiding, code that I wrote could be executed. In that manner, software that I've written would both hurt (and protect) someone. While I've yet to submit a patch to any of the projects above (at least as far as I can remember), it's not unreasonable to assume that it could happen in the near future.
And now we reach the divide between the trivial and the nontrivial pull requests. Trivial pull requests could be written by anyone. Adding a new command line option that was previously undocumented, fixing a small bug, anything a reasonably competent developer could do in half an afternoon, these tasks are what I call trivial. The trivial pull requests would happen regardless of whether or not I was involved. It would hurt me some to know that a library I had helped improve was used by Palantir in their defense work, but I could get over it in a weekend or so with a bottle of vodka and a good friend. The pull request would have happened anyway by some other developer.
But what if I wrote code that nobody else was likely to write? After a certain point in a project, the number of people who can submit a nontrivial pull request for a useful feature goes down pretty fast. To get a nontrivial pull request into a project like Rails, you have to be pretty damn good and work even harder. And even then, it's not guaranteed at all to go through. At the same moment, you can generalize this idea out to creating the initial seeds of what would become Rails. For the purpose of this post, I don't see much of a difference between making some feature of Postgres 10x faster and making Postgres itself. In a sense, a pull request that makes something 10x better is a big enough change that the request is creating a new project.
At this level, where I'm committing code that nobody else would likely commit, the game changes. I'm now enabling Palantir as they help kill people. And most likely, I would be doing it unintentionally. That's what shocked me. If I wrote a really good open source library that Palantir liked, they would be free to use it to design systems to help kill (or even just straight up kill) people and I would never know. If somebody at Palantir came back to me and said that they've had a great success using a library I had written though, I would probably curl up into a ball and cry for a month. Killing people is the opposite of why I got into software in the first place.
And If I vigorously tried to stop Palantir from using my open source library, I would probably be quickly arrested. Palantir is good at what they do and I think governments around the world will only come to rely on their software even more as time goes on. In some sense, I would be threatening national security by actively trying to prevent Palantir from using software that it needed to use to ensure the continued peace and prosperity of the United States. Although a lack of direct control is the case with open source software in general, in this case it could be illegal and seditious for me to even attempt to prevent Palantir from using my code. I'm not even sure how I could attempt to stop them short of gaining access to their servers and ritually deleting all the code I created (which would be hard, if not impossible, to do).
I've been raised as developer on open source. In my heart of hearts, I've matured as a programmer thinking that the best possible thing I could do to give back is create a powerful and useful open source library for other people to use how they see fit. I still have a decade of learning ahead of me before I think I would be good enough to do so, but I'd like to create a project that could be considered a peer of some of the projects listed above. But Palantir is, and will probably still be, using some of the best software ever written to help kill people. If I were successful and wrote great software, Palantir would probably want to use it. And so, I want to write software that helps kill people.
And that hurts.
Note that this was not written as a slam against Palantir in any way. I respect their work and recognize that what they are doing has to be done. They provide a much needed service and I'm happy that somebody else has to deal with deciding who should die. The core of it is that I'm distressed that work that I want to do to help improve people's lives could be used to end someone's life. I'm not comfortable yet with the idea that I could help end someone's life, even if they wanted to destroy mine. It's been a shock realizing that a fair portion of the top repositories on Github are potentially being used by Palantir to help kill people.
Note to all of those who comment: Thank you for reading this! I'm flattered that you've commented on this, no matter what your reaction was. It's a powerful feeling knowing that what I say affected people all over the globe. Please note though that I don't plan on making public comments on github, hacker news, or, god forbid, reddit. This is a touchy subject and one which requires a fair amount of thought to consider properly. As people have already noted here and in other forums, this strikes at a philosophical problem that depends very much on your system of values and what you think is inherently "good" or "evil". If you really want me to respond, write out your response, find my email, and wait a week or two before sending it. If you aren't trolling, I'll happily respond within a few weeks.
Footnotes:
[0] I'm going to take a wild swing and guess that the Palantir front end folks are currently evaluating or using d3.js on the front end right now. The library is just too damn good not to use for what they are doing. Nobody from the company has mentioned it online as far as I can tell, but I'd be shocked if they aren't using it or haven't built an equivalent library internally.
[1] That's assuming what I have written is worth reading, which is a big assumption, but bear with me here. I try to write things worth reading. I ended up taking down that article because I didn't want to maintain it any longer and I've since stopped using backbone.
[2] I've been unable to find any supporting evidence online that Palantir's tech stack has been used to provide the information needed to identify a specific individual to be killed, i.e. a report that a specific terrorist was identified and neutralized thanks to Palantir's tech. I believe that it is reasonable to assume that the above has happened though. I think that it is also reasonable that a public report detailing the above would be against Palantir's interests and they would work to prevent or suppress it.