Updates on best results so far:
General purpose compression on the original file, using cmix:
\time ./cmix -c /ztmp/mystery_file_difficult.bin /ztmp/mystery_file_difficult.cmix
Detected block types: DEFAULT: 100.0%
2100086 bytes -> 760584 bytes in 5668.77 s.
cross entropy: 2.897
5566.69user 105.07system 1:39:57elapsed 94%CPU (0avgtext+0avgdata 18968788maxresident)k
30749016inputs+5592outputs (3812631major+12008307minor)pagefaults 0swaps
Bulk image data with stripped parity information (and no headers):
./cmix -c /ztmp/pixeldata.bin /ztmp/pixeldata.cmix
Detected block types: DEFAULT: 100.0%
1800000 bytes -> 730424 bytes in 4306.10 s.
cross entropy: 3.246
Source image data:
\time ./cmix -c /ztmp/source.bin /ztmp/source.cmix
Detected block types: DEFAULT: 100.0%
600000 bytes -> 247993 bytes in 1465.30 s.
cross entropy: 3.307
1451.74user 15.60system 24:27.61elapsed 99%CPU (0avgtext+0avgdata 19321988maxresident)k
2264inputs+1664outputs (18major+8034786minor)pagefaults 0swaps
plus isolated noise:
\time ~/shared/dev/cmix/cmix -c noises-nonzero.bin noises-nonzero.cmix
Detected block types: DEFAULT: 100.0%
694950 bytes -> 465886 bytes in 1941.08 s.
cross entropy: 5.363
1927.04user 16.69system 32:24.22elapsed 99%CPU (0avgtext+0avgdata 18628380maxresident)k
6448inputs+2272outputs (23major+7851847minor)pagefaults 0swaps
= 713879 bytes.
Although because the noise is uniform, one can do better just by shoving all the noise into a big integer and writing the binary representation to file, which takes 465255 bytes, exactly as predicted earlier.
This gives a total of 713248 bytes to represent the pixel data.
Another approach - compressing the four 'image planes' given by the bayer filter separately:
png with zopflipng:
$ wc -c ????.zop.png
67171 blue.zop.png
65755 grn1.zop.png
65547 grn2.zop.png
66520 redd.zop.png
264993 total
Fail compared to general purpose compression. Loses against cmix (but wins against zpaq).
lossless webp:
$ wc -c ????.webp
64198 blue.webp
62626 grn1.webp
62588 grn2.webp
63584 redd.webp
252996 total
Again, loses vs cmix.
lossless webp combining three planes into one image:
$ wc -c col-r1b.webp grn2.webp
185246 col-r1b.webp
62588 grn2.webp
247834 total
This just barely wins vs cmix, under 200 bytes difference.
Combining red and two green planes into one webp instead of red, a green and blue plane:
$ wc -c col-r12.webp blue.webp
182986 col-r12.webp
64198 blue.webp
247184 total
Slight improvement.