zmaril/softwarehelpskill.md

## softwarehelpskill.md

      
    Raw
  

              softwarehelpskill.md
            
          
    I want to write software that helps kill people.
Please, before you call the police and get my
github account put on lockdown, allow me
a moment to explain. What I really want to do is work on projects that
advance the human condition and improve people's lives. I've been in a
mad dash to learn how to program for the past four or five years
exactly because I realized how much good I could do for the world with
a computer.
The open source software movement has produced
my tools, provided
my teachers and mentors,
and, recently, has become
a major focus of my attention and time.
I can whole heartedly say that, at the tender age of 22, open source
has already directly had a massive and positive benefit on my life.
I'm reasonably certain I would be orders of magnitude less well off
and less happy if open source didn't exist as a concept.
Which is why a realization I had a few weeks ago has caused me so much
personal strife lately: open source software is used to help kill
people. Specifically, Palantir Technologies uses open
source to identify who should be killed. In a world where
the American military probably has enough power to kill everyone currently alive,
the hard questions have shifted from
how to effectively kill the most people possible
to who should be killed. Palantir is in the business of providing the
33 bits of information needed to identify
the people who pose enough threat to America's national security that
they need to be eliminated.
Consider the following quote, taken from
a profile of Palantir in Business Week:

In Afghanistan, U.S. Special Operations Forces use Palantir to plan
assaults. They type a village’s name into the system and a map of the
village appears, detailing the locations of all reported shooting
skirmishes and IED, or improvised explosive device, incidents. Using
the timeline function, the soldiers can see where the most recent
attacks originated and plot their takeover of the village accordingly.
The Marines have spent years gathering fingerprint and DNA evidence
from IEDs and tried to match that against a database of similar
information collected from villagers. By the time the analysis results
came back, the bombers would be long gone. Now field operatives are
uploading the samples from villagers into Palantir and turning up
matches from past attacks on the spot, says Samuel Reading, a former
Marine who works in Afghanistan for NEK Advanced Securities Group, a
U.S. military contractor. “It’s the combination of every analytical
tool you could ever dream of,” Reading says. “You will know every
single bad guy in your area.”

The last sentence is what got me. Palantir uses open source software
to tell soldiers when they should be pulling the trigger. The
statement "the Python programming language murdered a 15 year
terrorist in Iraq" is rickety logic at best, but it does contain a
discomforting modicum of truth.
Here is a list of open source projects that Palantir probably uses or
plans to use, as indicated by their
jobs posting,
blog posts, and various
tech talks scattered across the web. While Palantir does much, much
more than identify "bad guys", I'm lumping all of the listed projects,
tools, and languages into the same conceptual bucket simply for lack
of detailed information about what happens behind closed doors. I
don't know that these projects are directly involved in informing
soldiers, but there is a higher than normal chance that they are
directly involved compared to a random project.
Languages:

Bash
Coffeescript
C++
Groovy
Java
Javascript
Perl
PHP
Python
Ruby
Scala

Front end[0]:

Android
backbone.js
Java Swing
less
rrd4j

Back end:

Cassandra
Chef
Hadoop
HyperSQL
JAXB
jMock
JMX
Linux/UNIX
Log4J
Lucene
Nagios
Postgres
Puppet
Rails
Spring
Zenoss

The vast majority of the above projects are large and successful. Tens
of thousands of developers have been involved with helping build them.
In Linux's case alone, there are about 10,000 people who have been
directly involved over the past few decades. There's probably an order
of magnitude more people who have submitted bug reports (think
automated Ubuntu crash reports) and at least another two orders of
magnitude who have helped harden Linux via sustained and heavy use.
The questions I've been struggling with is the relationship of the
developer and how these libraries are used. The above libraries are
general purposes tools that can be used for most anything. They do a
tremendous amount of good for humanity as a whole and the world is
better off for having them. The question I've been asking myself is
"How does the use of a tool reflect back on those who developed the
tool?" The purposes of these libraries are not to kill, but they have
been used by Palantir to help kill someone[2].
To start with, I've publicly written before about how to use
backbone.js. If the front end folks at Palantir saw that[1] and
figured out a better way to display lists of "bad guy" names or
something trivial, I wouldn't feel so bad about that. That information
is out there regardless and the Palantir folks are smart enough that
they would've figured it out within a few minutes anyway. Frequenters
of stackoverflow needn't worry that their answers have made the
difference between life and death for someone.
The same goes for people who have submitted even the most detailed bug
reports. These people have pointed out how to harden the library and
thus made it more reliable, but that is mostly incremental progress.
Just because Postgres happens to not print out a comma when you use a
certain SQL statement won't stop Palantir from identifying who should
die. Even reports of dire, world ending bugs don't bother me much. I
haven't written any of the code that is executed when Palantir
executes some query looking for enemy combatants.
Now, let's take the obvious next step up. What if I submit a pull
request to Cassandra that fixes a bug? Every time a soldier tells the
Palantir suite to go talk to Cassandra, asking for updates on the
current status of a battle and find out where the person trying to
kill him is hiding, code that I wrote could be executed. In that
manner, software that I've written would both hurt (and protect)
someone. While I've yet to submit a patch to any of the projects above
(at least as far as I can remember), it's not unreasonable to assume
that it could happen in the near future.
And now we reach the divide between the trivial and the nontrivial
pull requests. Trivial pull requests could be written by anyone.
Adding a new command line option that was previously undocumented,
fixing a small bug, anything a reasonably competent developer could do
in half an afternoon, these tasks are what I call trivial. The trivial
pull requests would happen regardless of whether or not I was
involved. It would hurt me some to know that a library I had helped
improve was used by Palantir in their defense work, but I could get
over it in a weekend or so with a bottle of vodka and a good friend.
The pull request would have happened anyway by some other developer.
But what if I wrote code that nobody else was likely to write? After a
certain point in a project, the number of people who can submit a
nontrivial pull request for a useful feature goes down pretty fast. To
get a
nontrivial pull request into a project like Rails,
you have to be pretty damn good and work even harder. And even then,
it's not guaranteed at all to go through. At the same moment, you can
generalize this idea out to creating the initial seeds of what would
become Rails. For the purpose of this post, I don't see much of a
difference between making some feature of Postgres 10x faster and
making Postgres itself. In a sense, a pull request that makes
something 10x better is a big enough change that the request is
creating a new project.
At this level, where I'm committing code that nobody else would likely
commit, the game changes. I'm now enabling Palantir as they help kill
people. And most likely, I would be doing it unintentionally. That's
what shocked me. If I wrote a really good open source library that
Palantir liked, they would be free to use it to design systems to help
kill (or even just straight up kill) people and I would never know. If
somebody at Palantir came back to me and said that they've had a great
success using a library I had written though, I would probably curl up
into a ball and cry for a month. Killing people is the opposite of why
I got into software in the first place.
And If I vigorously tried to stop Palantir from using my open source
library, I would probably be quickly arrested. Palantir is good at
what they do and I think governments around the world will only come
to rely on their software even more as time goes on. In some sense, I
would be threatening national security by actively trying to prevent
Palantir from using software that it needed to use to ensure the
continued peace and prosperity of the United States. Although a lack
of direct control is the case with open source software in general, in
this case it could be illegal and seditious for me to even attempt to
prevent Palantir from using my code. I'm not even sure how I could
attempt to stop them short of gaining access to their servers and
ritually deleting all the code I created (which would be hard, if not
impossible, to do).
I've been raised as developer on open source. In my heart of hearts,
I've matured as a programmer thinking that the best possible thing I
could do to give back is create a powerful and useful open source
library for other people to use how they see fit. I still have a
decade of learning ahead of me before I think I would be good enough
to do so, but I'd like to create a project that could be considered a
peer of some of the projects listed above. But Palantir is, and will
probably still be, using some of the best software ever written to
help kill people. If I were successful and wrote great software,
Palantir would probably want to use it. And so, I want to write
software that helps kill people.
And that hurts.

Note that this was not written as a slam against Palantir in any way.
I respect their work and recognize that what they are doing has to be
done. They provide a much needed service and I'm happy that somebody
else has to deal with deciding who should die. The core of it is that
I'm distressed that work that I want to do to help improve people's
lives could be used to end someone's life. I'm not comfortable yet
with the idea that I could help end someone's life, even if they
wanted to destroy mine. It's been a shock realizing that a fair
portion of the
top repositories on Github are
potentially being used by Palantir to help kill people.

Note to all of those who comment: Thank you for reading this! I'm flattered that you've commented on this, no matter what your reaction was. It's a powerful feeling knowing that what I say affected people all over the globe. Please note though that I don't plan on making public comments on github, hacker news, or, god forbid, reddit. This is a touchy subject and one which requires a fair amount of thought to consider properly. As people have already noted here and in other forums, this strikes at a philosophical problem that depends very much on your system of values and what you think is inherently "good" or "evil". If you really want me to respond, write out your response, find my email, and wait a week or two before sending it. If you aren't trolling, I'll happily respond within a few weeks.

Footnotes:
[0] I'm going to take a wild swing and guess that the Palantir front
end folks are currently evaluating or using d3.js on the front end
right now. The library is just too damn good not to use for what they
are doing. Nobody from the company has mentioned it online as far as I
can tell, but I'd be shocked if they aren't using it or haven't built
an equivalent library internally.
[1] That's assuming what I have written is worth reading, which is a
big assumption, but bear with me here. I try to write things worth
reading. I ended up taking down that article because I didn't want to
maintain it any longer and I've since stopped using backbone.
[2] I've been unable to find any supporting evidence online that
Palantir's tech stack has been used to provide the information needed
to identify a specific individual to be killed, i.e. a report that a
specific terrorist was identified and neutralized thanks to Palantir's
tech. I believe that it is reasonable to assume that the above has
happened though. I think that it is also reasonable that a public
report detailing the above would be against Palantir's interests and
they would work to prevent or suppress it.