Let $m$ be the message (flag), $n$ be the public modulus and $c$ be the encrypted_flag. We are going to perform a Binary Search to compute the value of $t = \frac{n}{m}$ to sufficient precision to recover $m$.
First, let's define a function is_error(x:int)->bool. is_error(x) will ask the server to decrypt $c x^e \text{ mod } n$, and return True if the server encounters an invalid padding. Recall that in RSA, using $c x^e \text{ mod } n$ as the ciphertext will decrypt into the plaintext $m x \text{ mod } n$. Note that the padding check only checks if the first byte of the plaintext is \x00. I.e., if the plaintext is smaller than some fixed threshold, then it passes the padding check.
The hint to this challenge will be the answers to the following questions:
- What do you expect
is_error(1) to return?
- What do you expect
is_error(x) to return when x is a small integer close to 1?
- As you continue to increase the value of
x, you'd expect is_error(x) to start returning True. Eventual