Solving a Java CTF challenge by writing static analysis passes!
As part of team Blue Water (perfect blue + Water Paddler collab), we played Real World CTF 2023 and placed 2nd out of 632 teams.
I, along with teammates Unblvr and Pew, solved a challenge called Dark Portal. The challenge itself is essentially a Java reverse engineering challenge disguised as a web challenge, as we'll quickly see below.
Pew quickly got past the first part of the challenge, which was essentially a local file inclusion using a file:// URL. The second part of the challenge requires reversing an obfuscated Java HTTP handler. This writeup describes how I solved the challenge using the power of static analysis tooling.
In the first part of the challenge, Pew used the LFI vulnerability to download /opt/tomcat/webapps/ROOT.war. Inside ROOT.war, we can find DarkMagic.class, which implements a HTTP request handler. We need to figure out how to exploit this request handler to read the flag from the remote server. However, the DarkMagic class is heavily obfuscated. If we try to decompile it, it gives us total gibberish:
There are several notable forms of obfuscation employed:
- Renaming of fields and methods to gibberish names made of
I,{, and\ncharacters (very annoying); - Addition of dead or useless code, namely the useless
System.out.printlncalls within useless try-catch blocks; - Opaque predicates, which can be seen in the
whileandifstatements; - String obfuscation: strings are stored encrypted at rest and decrypted at runtime;
- Reflection obfuscation; method calls are done through
invokedynamicdispatch orjava.lang.Invoke.
All of these obfuscation methods work in concert to disrupt static analysis.
We can undo each of these through static analysis and binary rewriting. I used the well-known MapleIR framework to do each of these deobfuscation passes.
MapleIR is a Java static analysis framework designed for solving CTF challenges and other real-world deobfuscation challenges. It was originally written as a student project, but has seen real-world applications. MapleIR has a fully-featured bytecode-to-IL lifter and IL-to-bytecode compiler. This makes it extremely suitable for writing deobfuscation passes.
It's impossible to work with the decompilation output without first eliminating these obnoxious member names. Before that, I couldn't even begin to understand the rest of the obfuscation. I took MapleIR's built-in MethodRenamerPass and FieldRenamerPass and slightly modified them for my purposes. The passes simply re-assigned deterministic (f_aaaaa, f_aaaab, ... etc) to all stupidly named fields and methods.
Now, the program looks a lot more like real Java code. But what's this? There's still some II{I{{I{\nII{ nonsense going on. Why is that still there?
It turns out that these wacky name is the callsite passed to a invokedynamic method call. But it turns out there weren't any methods which we renamed that actually match that specified callsite name. Upon further analysis, it turns out these are just decoy names that are totally unused. So next, let's fix the reflection obfuscation.
In the JVM, the invokedynamic opcode provides facilities supporting runtime dynamic call resolution, while avoiding the Java reflection APIs. This is because the use of reflection is slow. On the other hand, invokedynamic calls stay completely within the JVM implementation, which is fast. Thus, invokedynamic is especially beneficial to dynamic languages implemented on the JVM, like JRuby. Because the existing documentation on the internet is limited or difficult to comprehend, I will briefly cover the semantics and calling convention of invokedynamic calls.
An invokedynamic call is essentially a call to a statically-resolved "bootstrap" function, which is a programmer-defined method that returns the actual callsite the call should be resolved to.
The arguments to the bootstrap function are not supplied on the stack, unlike typical method calls in the JVM. Instead, the arguments supplied on the stack are supplied to the call to the final, resolved callsite returned by the bootstrap method. The first arguments supplied to the bootstrap method are passed by the JVM directly (docs). Following those three, the programmer may specify a variable number of additional arguments. These arguments must be static, as they are pulled from the calling class's constant pool, not the stack.
For the purposes of this challenge, we only really care about the bootstrap method and the additional arguments. The other arguments are not used. In this challenge, there are two bootstrap methods present: m_aaaad and m_aaaae. (These names are generated from our renaming pass, and aren't the original names). Let's take a look.
public static Object m_aaaad(final Object o, final String s, final Object o2, final Object o3, final Object o4, final Object o5, final Object o6, final String s2, final Object o7, final Object o8, final Object o9, final Object o10, final Object o11) throws Exception {
final char[] charArray = s2.toCharArray();
final int f_aaaad = DarkMagic.f_aaaad;
final char[] charArray2 = ((String)o4).toCharArray();
final byte[] bytes = new byte[charArray2.length];
for (int i = 0; i < charArray2.length; ++i) {
bytes[i] = (byte)((charArray2[i] ^ charArray[i % charArray.length]) & 0xFF);
}
final Class<?> forName = Class.forName(new String(bytes));
final char[] charArray3 = ((String)o6).toCharArray();
final byte[] bytes2 = new byte[charArray3.length];
for (int j = 0; j < charArray3.length; ++j) {
bytes2[j] = (byte)(charArray3[j] ^ charArray[j % charArray.length]);
}
final MethodType fromMethodDescriptorString = MethodType.fromMethodDescriptorString(new String(bytes2), forName.getClassLoader());
final char[] charArray4 = ((String)o5).toCharArray();
final byte[] array = new byte[charArray4.length];
for (int k = 0; k < charArray4.length; ++k) {
array[k] = (byte)(charArray4[k] ^ charArray[k % charArray.length]);
}
if ((((int)o3 ^ DarkMagic.f_aaaac) & 0xFF) == f_aaaad) {
return new MutableCallSite(((MethodHandles.Lookup)o).findStatic(forName, new String(array), fromMethodDescriptorString).asType((MethodType)o2));
}
return new MutableCallSite(((MethodHandles.Lookup)o).findVirtual(forName, new String(array), fromMethodDescriptorString).asType((MethodType)o2));
}Manually deleting some dead code, we see that this method implements some straightforward xor obfuscation on the final callsite that's passed in.
private static Object m_aaaae(final MethodHandles.Lookup lookup, final String s, final MethodType newType) {
return new MutableCallSite(lookup.findStatic(DarkMagic.class, "I\n{\n{{\nI\n{I{\nII{\n{I{\n{\n{{\n{I\n{", MethodType.fromMethodDescriptorString("(IJ)Ljava/lang/String;", DarkMagic.class.getClassLoader())).asType(newType));
}As for the other bootstrap method, we can see that it just resolves to the same method each time. In other words, we can simply replace all dynamic calls with this bootstrap method with a call to that method. Of course, we also need to adjust the method name, because that method was renamed by our previous pass.
Now that we know what must be done, we can easily implement a deobfuscation pass in MapleIR:
public class UnfuckDarkMagicPass implements IPass {
// ...
private void process(AnalysisContext cxt) {
for(ClassNode cn : cxt.getApplication().iterate()) {
if (cn != darkMagicClass) { continue; }
for(MethodNode m : cn.getMethods()) {
ControlFlowGraph cfg = cxt.getIRCache().getFor(m);
for(BasicBlock b : cfg.vertices()) {
for(Stmt stmt : b) {
for(Expr e : stmt.enumerateOnlyChildren()) {
if (e.getOpcode() == Opcode.INVOKE && ((InvocationExpr) e).isDynamic()) {
visitInvocation((DynamicInvocationExpr) e);
}
}
}
}
}
}
}
// ...
}First, let's clean up the simple m_aaaae case.
// ...
private void visitInvocation(DynamicInvocationExpr die) {
// aaaaeTarget = the method that "I\n{\n{{\nI\n{I{\nII{\n{I{\n{\n{{\n{I\n{" was renamed to
if (die.getName().equals("m_aaaae")) {
Arrays.stream(die.getArgumentExprs()).forEach(argExpr -> argExpr.setParent(null)); // before reparenting IL node, set parent to null
InvocationExpr resolvedCall = new StaticInvocationExpr(die.getArgumentExprs(), aaaaeTarget.getOwner(), aaaaeTarget.getName(), aaaaeTarget.getDesc());
die.getParent().writeAt(resolvedCall, die.getParent().indexOf(die));
System.out.println("Resolved dynamic invoke via m_aaaae to " + aaaaeTarget.getName() + " , args: " + Arrays.toString(resolvedCall.getArgumentExprs()));
}
// ...
}We can also do this for m_aaaad. We do this by emulating the decryption routine. I just copy-pasted it from the decompilation (and renamed some variables).
// ...
else if (die.getName().equals("m_aaaad")) {
// emulate bootstrap function m_aaaad
final int isStaticInt = (int)die.getBootstrapArgs()[0];
final String encryptedClassName = (String)die.getBootstrapArgs()[1];
final String encryptedName = (String)die.getBootstrapArgs()[2];
final String encryptedDesc = (String)die.getBootstrapArgs()[3];
final String xorKey = (String)die.getBootstrapArgs()[4];
final char[] keyArr = xorKey.toCharArray();
final int f_aaaad = 184;
final int f_aaaac = -762998497;
final char[] encryptedClassNameArr = encryptedClassName.toCharArray();
final byte[] decryptedClassNameArr = new byte[encryptedClassNameArr.length];
for (int i = 0; i < encryptedClassNameArr.length; ++i) {
decryptedClassNameArr[i] = (byte)((encryptedClassNameArr[i] ^ keyArr[i % keyArr.length]) & 0xFF);
}
String decryptedClassName = new String(decryptedClassNameArr);
final char[] encryptedDescArr = encryptedDesc.toCharArray();
final byte[] decryptedDescArr = new byte[encryptedDescArr.length];
for (int j = 0; j < encryptedDescArr.length; ++j) {
decryptedDescArr[j] = (byte)(encryptedDescArr[j] ^ keyArr[j % keyArr.length]);
}
String decryptedDesc = new String(decryptedDescArr);
final char[] encryptedNameArr = encryptedName.toCharArray();
final byte[] decryptedNameArr = new byte[encryptedNameArr.length];
for (int k = 0; k < encryptedNameArr.length; ++k) {
decryptedNameArr[k] = (byte)(encryptedNameArr[k] ^ keyArr[k % keyArr.length]);
}
String decryptedName = new String(decryptedNameArr);
boolean isStatic = false;
if (((isStaticInt ^ f_aaaac) & 0xFF) == f_aaaad) {
isStatic = true;
}
// replace invokedynamic with call to resolved callsite
Arrays.stream(die.getArgumentExprs()).forEach(argExpr -> argExpr.setParent(null)); // before reparenting, set parent to null
InvocationExpr resolvedCall;
if (isStatic) {
resolvedCall = new StaticInvocationExpr(die.getArgumentExprs(), decryptedClassName, decryptedName, decryptedDesc);
} else {
resolvedCall = new VirtualInvocationExpr(CallType.VIRTUAL, die.getArgumentExprs(), decryptedClassName, decryptedName, decryptedDesc);
}
die.getParent().writeAt(resolvedCall, die.getParent().indexOf(die));
System.out.println("Resolved dynamic invoke via m_aaaad to " + resolvedCall);
}With this new deobfuscation pass, our decompilation looks waaaaay better:
However, there's still those pesky opaque predicates and dead code!
Dead code elimination (DCE) and constant folding are classic optimization passes implemented in optimizing compilers. In some sense, you can also think of a deobfuscator as an aggressively-targeted optimizing compiler. Dead code elimination does exactly what you think it does. Constant folding essentially eliminates operations between constant expressions by replacing them with their results, at compile time. Luckily, MapleIR has built in DCE and CF passes, which worked out of the box.
I applied first the ConstantExpressionEvaluatorPass, followed by the DeadCodeEliminationPass, after our previous passes. I also quickly wrote a pass to delete calls to System.out.println because they are useless and irrelevant to the challenge. I put that before the DCE pass.
Wow! Our code looks way better! There's still some garbage try-catch blocks. It's possible to write another analysis pass to delete the useless try-catch blocks, but I didn't bother because the challenge is small.
The most important problem left is the string obfuscation. We can see that in the calls to m_aaaaf, which is essentially a string decryption function.
The string encryption is, like many commercial obfuscators, essentially a lookup function. There is also some reflection obfuscation using those encrypted strings. It's straightforward to write a debofuscation pass that emulates the string decryption function, and inlines the results. The same can be done to defeat the reflection obfuscation. For prevalent, commercial obfuscators, this would be the desired approach to take, since the fruits of our labor would be reusable across many applications. However, for a single-use CTF challenge, this is not economical in terms of time. Instead, we opted to reverse the string encryption function by hand and do the rest of the challenge manually.
As a brief aside, it's also possible to write a generic string deobfuscation pass which emulates any given string decryption method. However, this approach quickly runs into the halting problem. How would the obfuscator know whether a call to a supposed decryption method will halt, or run infinitely? An adversarial obfuscator (i.e., nearly all of them) can obstruct generic deobfuscators by inserting dead calls (e.g., guarded behind an opaque predicate) to the decryption method which, upon emulation, would cause the deobfuscator to hang. It's also important to note that opaque predicates are not generally decidable without targeting a specific obfuscator. This follows from Rice's theorem. If you aren't convinced, remember that interprocedural dataflow analysis is in general undecidable, and that it's relatively simple to entangle the resolution of an opaque predicate among some interprocedural mess.
The heart of the challenge, minus all the obfuscation BS, is actually very simple.
String var2 = var0.getHeader("cmd");
String var3 = var0.getHeader("User-Agent");
String var4 = var0.getParameter("curses");
if (!Objects.equals(var3, "The Argent Dawn")) {
var1.getWriter().println(DarkMagic.randomChoice(f_aaaaa));
} else if (var2 == null) {
PrintWriter var23 = var1.getWriter();
String[] var32 = f_aaaaa;
var23.println(DarkMagic.randomChoice(var32));
} else {
String var36 = "Victory or death!";
byte[] var37 = var36.getBytes();
String var42 = "HmacSHA256";
SecretKeySpec var5 = new SecretKeySpec((byte[])var37, var42);
String var14 = "HmacSHA256";
Mac var6 = Mac.getInstance(var14);
var6.init(var5);
byte[] var15 = var6.doFinal((byte[])"Lok-tar ogar".getBytes());
byte[] var16 = Base64.getEncoder().encode((byte[])var15);
boolean var17 = var4.equals(new String((byte[])var16));
// ... (run a user specified command) ...
}To solve this, all we need to do is just make a specially crafted HTTP request.
$ curl "http://198.11.177.96:37837/7he_d4rk_p0rt4l?curses=5IKRjJICv2BPpCEGG1TF5o%2BZ6aCHqifjjvlQVJa7vOI%3D" -H "User-Agent: The Argent Dawn" -H "cmd: /readflag" -v
rwctf{rwctf_N0w_y0u_4RE_prep4red_17a0}
As you can see, the challenge overall feels more like a reversing challenge than a web challenge.
In this writeup, we explored how static analysis can be a powerful tool for dealing with obfuscated binaries in the context of reverse engineering. Using MapleIR, we defeated several annoying obfuscation techniques by using or writing relatively straightforward debofuscation passes. These techniques included member renaming, dead code insertion, opaque predicates, and invokedynamic abuse.