I made the assumption that most of the adjacent values of the original signal are somewhat close,
x[n] ≈ x[n−1].
If the “encrypted” y[64n+k] = x[64n+k] xor key[k], then
y[64n+k] xor key[k] ≈ y[64n+k−1] xor key[k−1].
I could not make an assumption about the absolute values of key, but I could define
Δkey[k] = key[k] xor key[k−1], leading to
y[64n+k] xor Δkey[k] xor key[k−1] ≈ y[64n+k−1] xor key[k−1], thus
abs((y[64n+k] xor Δkey[k] xor key[k−1]) − (y[64n+k−1] xor key[k−1])) is small.
If I want to estimate the magnitude of abs((a xor z) − (b xor z)) without knowing z, I can roughly approximate it as the magnitude of the most significant bit of a xor b. I defined
bitdiff(a, b) = msbvalue(a xor b) where msbvalue(n) = n & ~((n>>1) | … | (n>>7)), thus
bitdiff(y[64n+k] xor Δkey[k], y[64n+k−1]) is small.
I was able to brute force Δkey[k] for each k (0…63) by finding the value (0…255) that minimizes the sum of the expression above for all n.
I could almost compute the key, but it is the xor “integral” of Δkey, thus it's floating on top of some yet unknown constant:
key[n] = Δkey[0] xor … xor Δkey[n] xor constant.
I made the assumption that out of all values of constant (0…255) the correct one results in the “smoothest” signal: the sum of the absolute values of the derivative of the signal is the smallest. By brute forcing that I was finally able to compute the key.