Mick Stute, BS in CS, PhD work in Bioinformatics
This was mine:
Was hired by a psychologist to fix a program that seemed to have "strange output" written by one of his ex-grad students. It was a program that reads a data files, asks about 50 questions, does some calculations, and comes up with some score based on this PhD's research. It's on a research 3B2 at the university. He demonstrates the program and sure enough there seemed to be strange flashing words on the screen when it moves from question to question and they don't seem nice. I agree to do it, should be pretty straight forward, so he'll pay me by the hour to determine how big the fix is and then we'll agree to a fee.
I sit down at the 3B2 and login to the ex-grad students account that has been given to me. This is where the code resides. I examine the C code. It is written to be hard to read. All the code is squished on one line. It's spread over 15 files with about 3 functions per file -- all on one line. All variable names are just three, seemingly random, letters. I talk to the guy and agree to go with hourly on this (great decision). I untangle all the code and format it nicely so I can see it.
It was done on purpose. It used the curses library to move to a point on the screen print a question and the answers and wait for a response. But it first went to the first line of the question, printed some white supremacy message, waited 1/2 a second, and then over wrote it with the question. This ought to be simple. There's only about 5 places it output anything and all of them had this subliminal flash of a message. Each one was hard coded. No problem. Delete the offending mvpwintw() and all is well. Or should be. I compile thinking I'm done. But when I ran it, there it is again -- the subliminal messages. This time with different text still the same subject, just different messages.
I check my code and believe it or it's back to the initial state I found it. 15 files, mangled, 3-letter variables -- the whole thing right back where I started. I want shoot myself for not making a copy of my code. I unmangle again this time putting it in three files, named differently. I make a copy of the whole directory and I mark the files readable only. I compiler. All looks good. I run the program. There's now a copy of the original 15 files in the directory along with mine and the subliminal messages are back.
Okay so some where on the disk is the source code necessary to keep doing this and he's set the program up to pull in that code when you compile it. I do a full disk search in the include areas (/usr/include) and since this is a research version we have source for just about everything but the kernel itself. That's a lot of header files and this takes some time on the 3B2 so that's day 1.
The disk search showed up nothing. The strings were apparently either encrypted or they are buried in a library somewhere. Because I don't have check sums of all the original executable objects I decide to search all libraries for the text. This is even longer than before, so day two is over.
No results. The strings are encrypted. That means I'm going to have to follow all the header files from each #include and each one they #include to find where this is. And that will, take some time. We do alert the campus computing department that we believe someone has gained root level access to Dr. Phelps research computer which is just a shared lab computer in the science building. They're understandably not convinced.
I start unwinding the #include files. I do that, no where do I find the code. So now I know it's compiled in a library. No problem at all. Why not just recompile all those libraries, we do have the source after all.
The hardest part, convincing the campus nerds they have an issue. But we finally do and Mark, the Unix admin who was hired because he married the Dean's daughter, gets busy learning how to do this. In the end, he agrees to allow me to handle it because he just doesn't really know how to get all that stuff compiled. End of Day 6, all standard libraries are recompiled. Who hoo!
I whip out my modified, cleaned up source and start the compile. All looks good. I run it. O M G. It did it again. 15 messed up source files and the subliminal messages are back. This is suddenly like magic. I investigate very very carefully though I am stumped. This code doesn't exist in source code. I think I might be beaten. Dr. Phelps isn't happy with the hours involved and thinks maybe we ought to just rewrite the program from scratch. "Sure", I say staring at the terminal like a lost puppy too deep in my thoughts to put out of my thinking mode, "I think you're right. That will be quicker." "Good," he says, "we can start tomorrow."
To hell with that. This guy isn't beating me. We are compiling it from his stinking code or not at all! "You don't have to pay me anymore, Dr. Phelps, I just want lab time." This is nerd war.
I get smart, I'm thinking he somehow modified the curses library. I compile the curses code to assembly and though I don't know 3B2 assembly (yet!) I start learning. I read manuals for 6 days, piecing together that assembly code. Waste of time, nothing seems unusual.
I suddenly realize it's in the compiler. It was the compiler. And every time you compile the original code and run it puts in the subliminal message code into the source code. I'd heard of this before.
Ah ah! I've got him!!!! We have the source code for the compiler as well. I search through it looking for a reference. Lo and behold, I find it. Indeed. There is source code in the compiler/linker that does this:
- it examines any call to fopen(), searches the file opened looking for Dr. Phelp's questions; if it finds them then
- it rewrites the 15 files to the current directory when compiling that specific program.
- It then compiles Dr. Phelps program using the 15 files and outputs to the -o name in the link phase.
The compiler was modified to put that code in Dr. Phelps program as written by the man that modified the compiler.
Several days later, an AT&T tech shows up with a disk and loads the proper compile and linker source and we recompile the compiler from the source. That solves it. All the bad source in the compiler is gone and we've got a new clean copy of the compiler.
Except it didn't. Because the compiler was poisoned with other source code that we didn't have. And that source code, that now existed only in the executable compiler, put those changes back into the compiler source before it compiled it. But this time it didn't modify the /usr/src copy, it copied it to a hidden directory, modified the compiler source, compiled itself from there, and deleted the hidden directory. It took an AT&T tech to find this. The ex-grad student had poisoned the compiler to poison itself when it was recompiled. We had to put a new binary version of the compiler on disk from another 3B2 running the same revision before the problem went away.
We also found that if /sbin/login is compiled it puts in a backdoor allowing anyone who uses a specific password to login in as the root user. This computer is accessible by modem and Tymnet. Finally this gets the computing center's attention.
Genius! But put to a horrible cause.