There was a RSA-like encryption implemented with polynomials over GF(2^2049) instead of integers. With sage it was easy to factor the public key into its two irreducible factors, from which the size of the multiplicative group modulo the public key can be calculated and that can be used to calculate the inverse of the public exponent.
Python (Sage) implementation:
from sage.all import *
from pubkey import P, n, e
R = GF(2**2049, 'a')
c = P(R.fetch_int(int(open('flag.enc', 'rb').read().encode('hex'), 16)))
f = list(factor(n))
p = f[0][0]
q = f[1][0]
assert p*q == n
s = (2**p.degree()-1)*(2**q.degree()-1)
d = inverse_mod(e, s)
print(repr(format(R(pow(c, d, n)).integer_representation(), '0256x').decode('hex')))