JPEG Encoder comparison (lena 56KB filesize goal)

comparison

Original file:
Lena.png 512 x 512 Pixel 24 bit Color 386.301 Byte
The standard for image compression tests.
Might be highly optimized in some encoders.

Goal: Get the most quality out of around 56KB.

Look at http://benediktkastl.de/summary_lena.htm for the images.

Metrics explained:

PSNR: Peak signal-to-noise ratio
compare http://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio
Ideal value: infinity, higher is better.

PAE: Peak Absolute Error 
The maximum error from the original value in color
Ideal value: 0, lower is better

MSE: Mean Square Error
compare http://en.wikipedia.org/wiki/Mean_square_error
Ideal value: 0, lower is better

SSIM: structural similarity
compare http://en.wikipedia.org/wiki/Structural_similarity
Ideal value: 1, closer to 1 is better

Compressed with Photoshop CS6:
Settings: Quality 57, Smoothing 0, Progressive
Filesize: 57.025 Byte
PSNR: 76.38 (best)
PAE: 514 (best)
MSE: 0.00151093 (best)
SSIM: 0.9999 (best)

Compressed with Photoshop CS6 and optimized:
Settings: Quality 57, Smoothing 0, Progressive, processed with jpegrescan (ImageOptim)
Filesize: 55.140 Byte 
PSNR: 76.38 (best)
PAE: 514 (best)
MSE: 0.00151093 (best)
SSIM: 0.9999 (best)

Compressed with Fireworks CS6:
Settings: Quality 80, Smoothing 0
Filesize: 56.815 Byte
PSNR: 47.12 (second)
PAE: 2014 (second)
MSE: 1.07039 (second)
SSIM: 0.9975 (second)

Compressed with Fireworks CS6 and optimized:
Settings: Quality 80, Smoothing 0, processed with jpegrescan (ImageOptim)
Filesize: 55.745 Byte
PSNR: 47.12 (second)
PAE: 2014 (second)
MSE: 1.07039 (second)
SSIM: 0.9975 (second)

Compressed with libjpegturbo (XnConvert 1.55):
Settings: Quality 91, Smoothing 0, Progressive, optimized Huffman table, DCT Float, Smoothing 0, subsampling default 
Filesize: 56.524 Byte
PSNR: 38.73 (third)
PAE: 11308 (third)
MSE: 8.7014 (third)
SSIM: 0.9972 (third)

Compressed with libjpegturbo (XnConvert 1.55):
Settings: Quality 91, Smoothing 0, Progressive, optimized Huffman table, DCT Float, Smoothing 0, subsampling default, processed with ImageOptim (no optimization possible)
Filesize: 56.524 Byte
PSNR: 38.73 (third)
PAE: 11308 (third)
MSE: 8.7014 (third)
SSIM: 0.9972 (third)

Tools used:
For encoding:
Photoshop CS6 Mac (13.0.4 x64)
Fireworks CS6 Mac (12.0.0.236)
XnConvert 1.55 Mac (x64)
For optimization:
ImageOptim 1.4.1
For comparison:
ImageMagick 6.8.0-10 2013-03-03 Q16
MSU Video Quality Measurement Tool 3.0

Imagemagick command line options for comparison:
compare -metric psnr original.png compressed.jpg null:
compare -metric pae original.png compressed.jpg null:
compare -metric mse original.png compressed.jpg null:

Feel free to do tests with other images...

Summary:
1. Use optimization tools like ImageOptim, jpegoptim, jpegrescan or jpegtran. They absolutely make no difference in image quality, but in filesize. 
2. Compression settings differ a lot. There are more measurable differences in image quality than you might expect.
3. In this comparison, Photoshop is the clear winner. Fireworks is slightly worse, but still better then libjpeg. The results from libjpeg should be identical in other libjpeg-based tools like GIMP or ImageMagick.

BUT: This is just a start for a testing method. Will need to test more images, more compression settings and use more metrics.