Berkeley's Okpy is an autograder that grades students' code. A student can use it as many times as he/she wants, but there's a catch: the student must wait for one minute before the next try. This was really annoying to deal with, because sometimes it would just be a typo or a missed parentheses. Thus, we as students naturally wanted to get around it somehow.
My initial workaround to this problem was to use the linux command sleep which would sleep for a given number of seconds and then run a command. (The full command was sleep 60s && python3 ok. This worked in that I didn't have to keep running the autograder to see the time remaining, but I still had the 1 minute lock nonetheless. I wanted to find a better way.
My first thought that there could be a better way came when a student had a problem regarding ok (he had nearly infinite remaining time — a weird bugj). An instructor suggested to remove a file on his local machine, which gave me the idea that perhaps the files somehow maintained state of time remaining. The file in question was .ok_storage. It turns out, removing the file did indeed clear the lock.
But was it safe? Could okpy's servers somehow figure out that students were doing this and perhaps invalidate our code?
I then decided to look into Okpy's source code on Github, since it is open sourced. I grepped the source for .ok_storage, and ended up with the file storage.py. It was reasonable that the .ok_storage would be managed in storage.py. The code is essentially as follows:
- There is a security key, which appears to be a constant string throughout each okpy instance.
- A hashing function is generated from this security key, which would later be used to hash values.
- The database is created using python's
shelvemodule, and there are methods to store values into and get values fromshelve.
From this point on, decrypting the .ok_storage file was trivial. It was simply a matter of importing shelve and using shelve.open() on .ok_storage. It turns out, .ok_storage was actually a dictionary with key-value pairs that corresponded to one of two possibilities:
- last time okpy was run on a given question (in unix time), and
- the number of times okpy was run on a given question. There was also a hash that was in the dictionary, and it was the hash generated by running the previous hashing function seen in ## Investigation.
At this point, I realized that exploiting the .ok_storage was trivial. All a person needs to do is to open the .ok_storage file, and rewrite the time into a time at least one minute before current time, and generate the matching hash for that (to pass hash checks). Here is a working proof-of-concept: https://hastebin.com/oxijutunod.py
After seeing this issue, I was curious as to why the hash was even introduced. The source code has a comment "security" after the hmac import, but it provides no such function. Since the key used to generate the hash is constant, it is trivial to break. There is no more functionality from having a hash and not having a hash, other than a larger file size. Furthermore, if ok doesn't find a .ok_storage file, it will simply just generate another one (and also bypassing the time check). Thus, it seems like implementing hmac was a waste of time.
I also began to think of ways to mitigate this problem. Eventually, I came a sad conclusion that there was no simple way. A solely client-side solution would face the same issues, since the client is free to do as he wishes. Making the repo closed source could cause breaking the storage method to be harder, but has much worse drawbacks. The only method I thought to be foolproof is to host all .ok_storages on the okpy server. This has immense drawbacks: namely, that the offline checks (ie. --local) would no longer work. Furthermore, depending on the implementation, it would be a lot more difficult to work on two machines (since one would need to sync .ok_storage).
One method to solve this could be to relock all of the unlocked test cases upon finding that .ok_storage had been deleted, perhaps as some sort of punishment. (However, unlocking test cases works by changing unlocked to True anyways — not that I do that.)
This was quite interesting, and I learned quite a bit as to how okpy and storage of data files work.
Thanks for reading!