MathSH was a very innovative challenge in the category sandbox escape. Three members of our team - ALLES! - worked for several hours and eventually drew first blood on this challenge. This writeup is split into several parts, namely: dumping the binary, analysing the sandbox, gaining a better primitive for code execution and finally escaping the sandbox.
The description Calculator as a Service (CAAS) already hints to CAS, a legacy .NET technology to run code in various level of trusts.
We are given a restricted "shell" to calculate math expressions: