We have a program that encrypts files in some unknown way. A good first step is to give it a repeated pattern and see what it outputs:
> python -c 'print "A" * 50' > /tmp/abc
> ./encrypt6 /tmp/abc /tmp/out
> cat /tmp/out
EICTDGYIYZKTHNSIRFXYCPFUEOCKRNEICTDGYIYZKTHNSIRFXY
We get a bunch of capital letters repeating with what seems to be a period of 30:
EICTDGYIYZKTHNSIRFXYCPFUEOCKRN
EICTDGYIYZKTHNSIRFXY...
Now let's see what happens if we give it another character:
> python -c 'print "B" * 50' > /tmp/abc
> ./encrypt6 /tmp/abc /tmp/out
FJDUEHZJZALUIOTJ...
These characters appear to be one character further on the alphabet than the A's were. This is as simple as differentials get: the difference between inputs (shift by 1) directly propagates to the output.
So we subtract the ciphertext from the plaintext to obtain the shift values:
> python
[ord(b) - ord(a) for a,b in zip('BBBBBBBBBBBBBBBB', 'FJDUEHZJZALUIOTJ')]
[4, 8, 2, 19, 3, 6, 24, 8, 24, -1, 10, 19, 7, 13, 18, 8]
Then subtract these from the secret ciphertext:
> cat krypton7
PNUKLYLWRQKGKBE
> python
''.join([chr(ord(b) - a) for a,b in zip([4, 8, 2, 19, 3, 6, 24, 8, 24, -1, 10, 19, 7, 13, 18, 8], 'PNUKLYLWRQKGKBE')])
'LFS8IS4O:RA4D53'
Not quite comprehensible but looks like it could be something... LFSR IS ...?
The characters in the output that look out of place were all shifted by more than 12 originally. The -1 in the shifts for B reveal there is a wraparound happening that isn't being fully seen by using just the first few characters of the alphabet as plaintext. Making the higher shifts signed by subtracting them from 26 is indeed the solution:
''.join([chr(ord(b) - a) for a,b in zip([4, 8, 2, -7, 3, 6, -2, 8, -2, -1, 10, -7, 7, -13, -8], 'PNUKLYLWRQKGKBE')])
LFSRISNOTRANDOM
Not that using an input of MMM... would have avoided this wraparound issue.