Skip to content

Instantly share code, notes, and snippets.

@kiding
Last active April 2, 2024 08:32
Show Gist options
  • Star 19 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a to your computer and use it in GitHub Desktop.
Save kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a to your computer and use it in GitHub Desktop.
Extracting HDR Gain Map from iOS 14.1+ (iPhone 12+) photos
import UIKit
import MobileCoreServices.UTCoreTypes
if #available(iOS 14.1, *) {
let input = Bundle.main.url(forResource: "IMG_0037", withExtension: "HEIC")!
let output = FileManager().temporaryDirectory.appendingPathComponent("IMG_0037.GAIN_MAP.BMP")
let source = CGImageSourceCreateWithURL(input as CFURL, nil)!
// urn:com:apple:photo:2020:aux:hdrgainmap
let dataInfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary
let data = dataInfo[kCGImageAuxiliaryDataInfoData] as! Data
let description = dataInfo[kCGImageAuxiliaryDataInfoDataDescription]! as! [String: Int]
let size = CGSize(width: description["Width"]!, height: description["Height"]!)
let ciImage = CIImage(bitmapData: data, bytesPerRow: description["BytesPerRow"]!, size: size, format: .L8, colorSpace: nil)
let cgImage = CIContext().createCGImage(ciImage, from: CGRect(origin: CGPoint(x: 0, y: 0), size: size))!
let destRef = CGImageDestinationCreateWithURL(output as CFURL, kUTTypeBMP, 1, nil)!
CGImageDestinationAddImage(destRef, cgImage, [:] as CFDictionary)
CGImageDestinationFinalize(destRef)
print(output)
}
@andsimakov
Copy link

andsimakov commented Sep 12, 2022

@frankschlegel I'm not familiar with the development for Apple devices, I mostly deal with Python and back-end. That's why it's hard for me to decide if something is related or useful regarding this topic. I found those links, probably there's something related:

Explore HDR rendering with EDR
Processing HDR Images with Metal

You can try to reach that developer guy of Radiance+, there's his email on the dedicated page in App Store. Though I'm not sure if you'll have an answer because I wrote him a couple of times, in comments on Reddit where I appeared after searching for something about HLG iPhone photos more HDR (gets brighter) than Lightroom exported photos? The second time was I tried to figure out if I get full size images in his app if I get the pro version. No answer either. :( Actually, I find his app overpriced for that amount of functionality: $2,99 monthly / $29,99 permanent. Too much for me. I already have the working script with all the pipeline, though it is not as convenient as his app.

BTW, I also find useful and cognitive an article mentioned in that Reddit post: Apple’s “EDR” Brings High Dynamic Range to Non-HDR Displays. Even a starting animation is funny. It reminds me of myself in my bed before falling asleep in the darkness when I try to "rethink" all my previous images in Radiance+ app :D.

Regarding "how", I like thoughts from the article:

It’s one thing to see HDR video on an HDR TV, where the entire image appears simply brighter and richer. It’s another thing to see this kind of imagery presented in the long-familiar context of a computer screen full of folder icons and file names. It’s probably the right way to handle HDR in an SDR world, but it’s strange and new, and possibly unique to Apple.

And one more definition I found similar to that I used to describe the effect to other people:

... the highlights in these clips should be visibly brighter than UI “white.”

@amagdach
Copy link

amagdach commented Sep 14, 2022

I think that I found solution how to display image as in Photos thanks to https://pythonawesome.com/convert-hdr-photos-taken-by-iphone-12-or-later-to-regular-hdr-images/

float edr_headroom = [[UIScreen mainScreen] currentEDRHeadroom];
DisplayImage = InputImage(SDR) * pow(edr_headroom, HDR Gain Map)

@frankschlegel
Copy link

@amagdach Good find! It's really hard to find any information about this at all.

Applying the gain map exponentially could be the correct solution. But there are two things I still have questions about:

  1. The gain map stores luminance information (single-channel generic grey with sRGB tone curve). That means adding it to all RGB values can't be quite right. The color of the base image would need to be converted to a color space with separate luma channel like Y'CbCr. Then the gain map can be added to the Y' component with the above formula. Afterward, the color can be converted back to RGB. I think only this way the colors are adjusted correctly since RGB doesn't have an even distribution of luminance.
  2. I wonder how the two {MakerApple} EXIF tags play into the formula. One of them (0x21) seems to add a flat luma boost to all pixels in the image, while the other (0x30) controls how much the luma from the gain map affects the result.

@amagdach
Copy link

amagdach commented Sep 15, 2022

@frankschlegel I tested above formula on real devices and compared with Photos app. It looks nearly identical.

  1. Wikipedia, chapter Luma coefficients https://en.wikipedia.org/wiki/Rec._709
    "When encoding Y’CBCR video, BT.709 creates gamma-encoded luma (Y’) using matrix coefficients 0.2126, 0.7152, and 0.0722 (together they add to 1). BT.709-1 used slightly different 0.2125, 0.7154, 0.0721 (changed to standard ones in BT.709-2). Although worldwide agreement on a single R’G’B’ system was achieved with Rec. 709, adoption of different luma coefficients (as those are derived from primaries and white point for Y’CBCR requires the use of different luma-chroma decoding for standard definition and high definition."
    So, if we multiply input colour by some coefficient (k) then Y’new = k * Y’. But in this case you should know what type of BT### to use.
    I tested weights of BT.709 and got pretty close results in comparison with earlier mentioned formula.

  2. I don't know exactly what they mean. I agree (0x21) seems to add a flat luma boost to all pixels in the image.

@AttilaTheFun
Copy link

Hey folks! I’m interested in this too. What I’d like to do is take the gain map and combine it with the original image to produce a new, actual HDR image in a format that support it (e.g. TIFF or AVIF).

@sobotka
Copy link

sobotka commented Dec 24, 2022

Hey folks! I’m interested in this too. What I’d like to do is take the gain map and combine it with the original image to produce a new, actual HDR image in a format that support it (e.g. TIFF or AVIF).

https://gist.github.com/kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a?permalink_comment_id=4301193#gistcomment-4301193

@m13253
Copy link

m13253 commented Jan 19, 2023

Hey folks! I’m interested in this too. What I’d like to do is take the gain map and combine it with the original image to produce a new, actual HDR image in a format that support it (e.g. TIFF or AVIF).

Hope my code can help you: https://github.com/m13253/heif-hdrgainmap-decode

You can use the y4m variant to generate a 12-bit AVIF:

avifenc --cicp 9/16/9 --min 1 --max 12 IMG_0000.y4m IMG_0000.avif

Alternatively, you can use FFmpeg to convert to 10-bit first, then generate an 10-bit AVIF:

ffmpeg -color_primaries bt2020 -color_trc smpte2084 -colorspace bt2020nc -color_range tv -i IMG_0000.y4m -strict -1 -pix_fmt yuv444p10le IMG_0000_10bit.y4m
avifenc --cicp 9/16/9 --min 1 --max 12 IMG_0000_10bit.y4m IMG_0000.y4m

Alternatively, you can use the aces or scrgb variant:

  1. To generate an OpenEXR, then import into DaVinci Resolve, to do color grading.
  2. To generate 32-bit float TIFF, then import to Affinity Photo, to do color grading.

@bigipalabom
Copy link

bigipalabom commented May 11, 2023

@amagdach Good find! It's really hard to find any information about this at all.

Applying the gain map exponentially could be the correct solution. But there are two things I still have questions about:

  1. The gain map stores luminance information (single-channel generic grey with sRGB tone curve). That means adding it to all RGB values can't be quite right. The color of the base image would need to be converted to a color space with separate luma channel like Y'CbCr. Then the gain map can be added to the Y' component with the above formula. Afterward, the color can be converted back to RGB. I think only this way the colors are adjusted correctly since RGB doesn't have an even distribution of luminance.
  2. I wonder how the two {MakerApple} EXIF tags play into the formula. One of them (0x21) seems to add a flat luma boost to all pixels in the image, while the other (0x30) controls how much the luma from the gain map affects the result.

Hi, thank you for all the information, I have successfully implemented auxiliary hdr gain map into a heif file and get iPhone rendered it in EDR

I tried two different methods to generate hdr gain map:

  1. use Y’ Chanel as map directly
  2. Adjust exposure to -5.6 to the original picture in PQ colorspace, make it grey, Following this reference with detailed explains for how to generate the tone map

https://helpx.adobe.com/camera-raw/using/gain-map.html

Both ways work, EDR activated as expected.

Somehow there is a major bug, in the area where both shadows and highlights exist, quite noticeable amount of extra halos pop out around the shadows, which doesn’t exist in the original picture, and it’s quite annoying.

I figured out above issue was due to aliquant pixels, solved by adjusting image resolution.

@jargonjustin
Copy link

It looks like as of WWDC (June 2023), Apple documented the "Apple HDR Effect" at Applying Apple HDR effect to your photos which specifically covers the gain map format, how it's applied, and the interpretation of the {MakerApple} tags 0x21 (33) and 0x30 (48).

They have a video session Support HDR images in your app and associated sample code that covers new HDR support in image APIs on Apple platforms.

@jiawen
Copy link

jiawen commented Oct 24, 2023

These JPEGs (@frankschlegel's fun one and @andsimakov's from Sept 2022) work great in Apple Photos... but only on my iPhones (14 Pro Max and 15 Pro Max). The HDR effect does not show up properly in Apple Photos on MacOS. I tried both Ventura and Sonoma, on an M1 Macbook Pro's internal display and an Apple XDR monitor.

The iPhone native camera's JPEGs do show up correctly.

Any idea what metadata is missing? Maybe the -CustomRendered=HDR flag they mentioned on this thread?

@m13253
Copy link

m13253 commented Oct 25, 2023

The HDR effect does not show up properly in Apple Photos on MacOS.

From what I observe, it seems that the macOS Photos app gets the render formula wrong.
I am able to reproduce a similar result by deliberately changing * to + somewhere in the formula.

Therefore, I guess even Apple engineers themselves don’t have documentations about the exact design.

@frankschlegel
Copy link

Any idea what metadata is missing?

It might be the HDRGainMapVersion that's missing. From Apple's documentation:

Before you begin, check whether the image’s metadata contains the HDRGainMapVersion key. If this key exists in the metadata, the gain map is valid and you can continue calculating the headroom.

I did not know about this piece of metadata when I was creating the sample image. It seems, on macOS, this is actually required for Photos to recognize the image as HDR.

@andsimakov
Copy link

andsimakov commented Oct 25, 2023

I stopped playing with this script. Adobe Lightroom Classic got the ability to edit HDR images recently, and I got a mirrorless camera Canon EOS R6 Mark II which is capable of shooting 10-bit HEIF. This fact appeared to be useless for me — the effect is barely visible on my iPhone 12 PM but it is present. Old good camera RAW (or multiple shots of a single scene) is still a valid starting point. I don't have nor Apple XDR display, neither Apple Silicon-based MBP with the XDR display, or an iPad Pro 12.9 with the XDR display in order to edit images in the HDR range. However, I have the Dell U3223QE display that is HDR400 certified, and yes, I saw the difference there in HDR mode on the display and enabled HDR in macOS. Though it's not sufficient for confident editing workflow, just a chance to try.

One more thing I'd like to add is that HEIF is now dead for me as a file format. I didn’t manage to wait for its wide spreading. Apple finally added AVIF support in macOS Sonoma, and Adobe Lightroom Classic is able to export HDR images to AVIF. That's enough for my current needs.

@gregbenz
Copy link

Adobe, Android 14, Chrome, Edge, Brave, and Opera all support a JPG gain map standard published by Adobe/Google. That's the ideal solution now (already widely supported, backwards compatible with any viewer, can export with control over the SDR rendition via Lightroom / Adobe Camera RAW, and the spec is part of a draft ISO specification). https://gregbenzphotography.com/hdr-images/jpg-hdr-gain-maps-in-adobe-camera-raw/

Longer term, an AVIF gain map will be even better once the key pieces are in place to support encoding and viewing (this already has support in Chrome under a dev flag).

@andsimakov Shoot RAW and process for HDR. The results are much better than capturing directly as HEIF. I'd only shoot that format for very narrow situations such as sports (ie, the same scenarios where you would use JPG over RAW now for SDR). The HDR AVIF export from Lightroom is great for sharing in the Photos app on Apple devices, but the JPG gain map is better for the web (as it gives you significant control over the SDR rendition, rather than using automatic tone mapping - and it offers much better support for FireFox and Edge which don't support HDR AVIF at this time).

@andsimakov
Copy link

@gregbenz thank you for that link. It seems I missed that article. It looks interesting! Yes, JPEG + Gain Map is now the most versatile solution if it is supported by something other than Apple XDR displays and OSes before them. I didn't know about the FireFox browser. My use case is quite narrow. It's image sharing from Apple iCloud to Apple Photos across Apple devices of my family + Apple TV. My most important goal towards new formats — the thing I wanted to get from HEIF and finally got from AVIF — compressing efficiency. I just want to save regular SDR 8-bit files in AVIF in order to save space in Apple iCloud and to postpone upgrading to the next more expensive tier. 10 bits and EDR/HDR is the next priority for me.

@gregbenz
Copy link

@andsimakov Eventually, I would expect widespread adoption of AVIF and gain maps. For now, JPG gain maps in the browser and Android 14, and HDR AVIF on iOS (or browse in LR iOS).

I'm not aware of a way to show HDR photos on Apple TV, but I haven't yet tested to see if HDR AVIF works there too with the latest TV OS. You cannot use AirPlay 2. HDMI output from at least iPhone 15 should work, but I haven't confirmed (definitely works from MacOS).

AVIF is getting really close to being a reasonable mainstream standard for SDR (MS Edge just needs to drop the dev flag and we should be in great shape for generally widespread usage). AVIF also offers other SDR benefits: transparency and reduced risk of banding (10-bit SDR).

@andsimakov
Copy link

My buddy views my photos from a shared album on Apple TV and Philips OLED TV and enjoys the luma effect on some of them.

@gregbenz
Copy link

@andsimakov Are you seeing HDR with HEIC? I just tried this morning using a shared Photos album with this week's latest iOS / TVOS, and as soon as a valid HDR AVIF is added to a shared album, it loses its magic (looks like clipped SDR). So I've lost the HDR before I could even open the photo on Apple TV. AirPlay doesn't work either for HDR. I've just been bypassing the Apple TV and using an HDMI connection to the TV to show photos from a computer.

@andsimakov
Copy link

@gregbenz oh, I just have to trust his words. I showed him pairs SDR/EDR images in the same album, and he told there was a difference. I cannot confirm it on my own, because I still don't have OLED TV 🤷🏻‍♂️ But I have to add those images were produced with Radiance+ iOS app. Lightroom wasn't capable of editing HDRs at that time yet. So my words are irrelevant for HDR AVIFs from Lightroom.

@gregbenz
Copy link

@andsimakov Gotcha. If he saw both, probably worked. Apple has supported EXR for a while, but I don't recall Radiance working. Will have to try that.

@jiawen
Copy link

jiawen commented Nov 3, 2023

I can't believe I managed to resurrect such an old thread on something so esoteric. Bravo! I love this community!

Any idea what metadata is missing?

It might be the HDRGainMapVersion that's missing. From Apple's documentation:

Before you begin, check whether the image’s metadata contains the HDRGainMapVersion key. If this key exists in the metadata, the gain map is valid and you can continue calculating the headroom.

I did not know about this piece of metadata when I was creating the sample image. It seems, on macOS, this is actually required for Photos to recognize the image as HDR.

@frankschlegel Thanks for the pointer! I poked around and as usual with Apple, the documentation is maddening. There's literally no mention of it in any Apple docs besides the one you linked. I even greped all the headers and there's literally only these two:

$ rg -t h -i hdrgain
Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/CoreImage.framework/Headers/CIImage.h
188:CORE_IMAGE_EXPORT CIImageOption const kCIImageAuxiliaryHDRGainMap NS_AVAILABLE(11_0, 14_1);

Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/ImageIO.framework/Headers/CGImageProperties.h
817:IMAGEIO_EXTERN const CFStringRef kCGImageAuxiliaryDataTypeHDRGainMap   IMAGEIO_AVAILABLE_STARTING(11.0, 14.1);

# (Identical copies in iPhoneSimulator, TVOS, etc).

The only other mentions of HDRGainMapVersion on Google are related to this thread - it's about adding it to the XMP in the auxiliary gain map thread on exiftool. I poked around and found that HDRGainMapVersion is an integer, either 65536 (0x10000) as in that thread, or 131072 (0x20000) in some newer files. The stored format 1278226488 is the integer value of kCVPixelFormatType_OneComponent8, which makes sense since the gain map is a JPEG that decodes to an 8-bit grayscale image encoded with a Rec. 709 transfer function.

Any luck on your side getting Apple Photos to recognize a bespoke image?

@jiawen
Copy link

jiawen commented Nov 3, 2023

Adobe, Android 14, Chrome, Edge, Brave, and Opera all support a JPG gain map standard published by Adobe/Google. That's the ideal solution now (already widely supported, backwards compatible with any viewer, can export with control over the SDR rendition via Lightroom / Adobe Camera RAW, and the spec is part of a draft ISO specification). https://gregbenzphotography.com/hdr-images/jpg-hdr-gain-maps-in-adobe-camera-raw/

Longer term, an AVIF gain map will be even better once the key pieces are in place to support encoding and viewing (this already has support in Chrome under a dev flag).

@andsimakov Shoot RAW and process for HDR. The results are much better than capturing directly as HEIF. I'd only shoot that format for very narrow situations such as sports (ie, the same scenarios where you would use JPG over RAW now for SDR). The HDR AVIF export from Lightroom is great for sharing in the Photos app on Apple devices, but the JPG gain map is better for the web (as it gives you significant control over the SDR rendition, rather than using automatic tone mapping - and it offers much better support for FireFox and Edge which don't support HDR AVIF at this time).

@gregbenz I agree with you completely. I wrote several JPEGs using the Adobe/Google format and it works well in Lightroom/Chrome/Android, as expected. But as the others on this thread have pointed out, Apple doesn't yet support it and we have no idea when they will. And most of my friends and family only use Apple devices so it'd be nice to have a format that works for them on say, iMessage or Safari.

AVIF seems to be only semi-working on both Apple and Google phones, so I wouldn't count on it as a universal solution for a while.

@frankschlegel
Copy link

Any luck on your side getting Apple Photos to recognize a bespoke image?

@jiawen Indeed. It took me a while, but I was able to generate another test image that also works on macOS:
macOS_HDR_Test

The following was needed to make it work:

  • In the XMP metadata, the HDRGainMapVersion key needed to be present. (I set it to 65536)
  • A gain map auxiliary image of type urn:com:apple:photo:2020:aux:hdrgainmap.
  • The {MakerApple} metadata tags 33 and 48 need to have meaningful values according to Apple's documentation. (I set them to 0.8 and 0.0 for maximum effect.)
  • And, new to me, the image must be larger than 360 pixels in each dimension. For smaller images, the HDR effect simply doesn't show. This took me the longest to find out. 😅

The first and last point were needed on macOS, but iOS displayed HDR even without them.

I also found, on both platforms, that only Photos is capable of displaying gain map HDR images. The HDR effect appears nowhere else, neither in Messages, Preview, Files, etc.

either 65536 (0x10000) as in that thread, or 131072 (0x20000) in some newer files

Hu, I haven't seen 131072 yet. Do you have an example of where you've seen it?

@gregbenz
Copy link

gregbenz commented Nov 5, 2023

@jiawen Did you write code to do that, or just use software like Adobe LR / ACR? ImageMagick has an open request for contributors who may be able to help support gain maps. ImageMagick/ImageMagick#6377. ImageMagick support would be key to support for gain maps in the WordPress media library, etc.

Another resource for this group (if the interest isn't just limited to HEIC / Swift) is the HDR JPG gain map library published by Google: https://github.com/google/libultrahdr. This is built to the Google / Adobe gain map spec (which is the basis of a draft ISO specification and is a de facto standard with support in Lightroom, Adobe Camera RAW, Chrome, Edge, Brave, and Opera already).

@jiawen
Copy link

jiawen commented Nov 7, 2023

Hi @frankschlegel - this is amazing! I confirmed that your JPEG renders as HDR in Apple Photos my iPhone 15 Pro Max (running iOS 17.1) and M1 Macbook Pro (running macOS Sonoma). Unsurprisingly, it does not render correctly in Apple Photos on my M1 Mac Studio running macOS Ventura when connected to an Apple XDR display (you see a difference if you toggle HDR on/off in Preferences. But it doesn't show the HDR text. It only shows a flat field of bright white vs less bright white.). Surprisingly, it does render as HDR in Chrome (v118)! Even on Ventura. It looks like Google decided to parse one of the fields, probably the XMP you added.

Amazing!

Hu, I haven't seen 131072 yet. Do you have an example of where you've seen it?

I got the following in the aux image's XMP on a JPEG taken with an iPhone 15 Pro's rear camera. I extracted it using exiftool -MPImage2 -b my_image.jpg > aux.jpg followed by exiftool -xmp -b aux.jpg.

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 6.0.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:HDRGainMap="http://ns.apple.com/HDRGainMap/1.0/"
            xmlns:apdi="http://ns.apple.com/pixeldatainfo/1.0/">
         <HDRGainMap:HDRGainMapVersion>131072</HDRGainMap:HDRGainMapVersion>
         <HDRGainMap:HDRGainMapHeadroom>5.651999</HDRGainMap:HDRGainMapHeadroom>
         <apdi:NativeFormat>1278226488</apdi:NativeFormat>
         <apdi:AuxiliaryImageType>urn:com:apple:photo:2020:aux:hdrgainmap</apdi:AuxiliaryImageType>
         <apdi:StoredFormat>1278226488</apdi:StoredFormat>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

@jiawen
Copy link

jiawen commented Nov 7, 2023

@jiawen Did you write code to do that, or just use software like Adobe LR / ACR? ImageMagick has an open request for contributors who may be able to help support gain maps. ImageMagick/ImageMagick#6377. ImageMagick support would be key to support for gain maps in the WordPress media library, etc.

Another resource for this group (if the interest isn't just limited to HEIC / Swift) is the HDR JPG gain map library published by Google: https://github.com/google/libultrahdr. This is built to the Google / Adobe gain map spec (which is the basis of a draft ISO specification and is a de facto standard with support in Lightroom, Adobe Camera RAW, Chrome, Edge, Brave, and Opera already).

I wrote bespoke code to do this. I would love to open source this code as it is simultaneously useful, esoteric, yet not too difficult when you use the right abstractions. But my manager probably wants me to spend my time elsewhere. I think my employer can be convinced to release this portion of our code and maintain it, but someone else will need to plumb it into ImageMagick as I lack expertise there.

@gregbenz
Copy link

gregbenz commented Nov 7, 2023

@jiawen That's great. If you are able to share it, I think just getting it as it currently stands to ImageMagick via ImageMagick/ImageMagick#6377 may be of great value to help expedite those efforts.

@frankschlegel
Copy link

I got the following in the aux image's XMP on a JPEG taken with an iPhone 15 Pro's rear camera.

@jiawen Very interesting! It seems Apple introduced a new version with the iPhone 15, including the new HDRGainMapHeadroom XMP field. I wonder what this does... It's not documented in their gain map documentation so far.

Surprisingly, it does render as HDR in Chrome (v118)!

Also in Brave. It seems support for Apple's format was added together with the support for Adobe's format to the Chrome engine. Very nice!

I wonder where this is all going. On the developer side, Apple seems to advocate for exporting HDR images in what they call "ISO HDR" format (basically an HDR video still frame). In fact, that's the only official way for exporting HDR images. There are no simple APIs for writing an SDR image + HDR gain map.

@gregbenz You are very deep into this topic. Do you have an idea what will be the dominant standard in the future? This "ISO HDR" format seems to be the obvious candidate, as it's simply the same as HDR video. But it comes with the big disadvantage that there is no clearly defined SDR representation for those images. It's up to the displaying application how to tone-map those to SDR. The gain map approach is much more explicit.

@gregbenz
Copy link

gregbenz commented Nov 8, 2023

I don’t know that I’d quite say ISO HDR is the same as a video still frame, not in a literal sense anyhow. Video tends to define a 100 nits white point, photography is using 203. There are certainly many commonalities (sadly including the lack of a common standard for tone mapping).

The “ISO HDR” refers to encoding of an HDR image. There is another ISO draft for gain maps. These are complimentary, as an ISO gain map can use an ISO HDR as the base image. You can use one without the other, but they work best together (if the base image in a gain map is HDR).

Gain maps are the obvious way forward. They offer a much better experience for viewing on any display which lacks the full HDR headroom encoded in the image. It’s vastly better than tone mapping.

JPG gain map is the obvious choice right now. Browsers representing ~75% of viewers (once fully updated) support it on systems with HDR display. And backwards compatibility is 100% (if your browser / image software doesn’t support HDR or gain maps, you gracefully fall back to a nice SDR).

However, JPG requires large files for high quality. There is a draft AVIF gain map standard (Adobe has shared sample files and Chrome/Brave support it). This is clearly the way things “should” go.

I say “should” because best doesn’t always win. As of now, Microsoft Edge (and the file explorer) doesn’t support any AVIF, and that needs to change for AVIF to be viable. Assuming that happens (they have support under a dev flag), we still need an encoder and more browsers to add support. It’s part of the same draft ISO proposal for gain maps and I hope to see it happen.

Another theoretical candidate would be JXL (JPEG XL). As browser support is zero and Chrome pulled support, it doesn’t seem promising anytime soon.

If it were all up to me: we’d use JPG gain maps for now, transition to AVIF gain maps as soon as MS Edge supports AVIF (encoded with an SDR base image for compatibility), then finally get to the ideal AVIF encoded with an HDR base image when support for AVIF gain maps is widely available.

The other thing I’d like to see is encoder support to let the artist provide the SDR rendition as well. This has many benefits, which I outlined in this feature request for Photoshop (please upvote it if you agree): https://community.adobe.com/t5/photoshop-ecosystem-ideas/allow-full-user-control-of-the-sdr-rendition-in-an-hdr-gain-map-export/idi-p/14205440#M19464

@grapeot
Copy link

grapeot commented Nov 20, 2023

Although I'm late to the party, I must say this thread is the most informative and inspiring resource on the entire internet regarding this specific topic. I've truly learned a lot from all the discussions above and want to sincerely thank every participant and contributor. Inspired by the discussion, it seems I've figured out how to use Apple's public APIs to generate a JPEG with a gain map that renders properly on both iPhone and Mac. It can also be inspected by Adobe's demo app and other third-party software like HoneyView. I've shared the code with detailed documentation, hoping it will be helpful to others: https://github.com/grapeot/AppleJPEGGainMap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment