Skip to content

Instantly share code, notes, and snippets.

@kiding
Last active January 19, 2025 16:03
Show Gist options
  • Save kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a to your computer and use it in GitHub Desktop.
Save kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a to your computer and use it in GitHub Desktop.
Extracting HDR Gain Map from iOS 14.1+ (iPhone 12+) photos
import UIKit
import MobileCoreServices.UTCoreTypes
if #available(iOS 14.1, *) {
let input = Bundle.main.url(forResource: "IMG_0037", withExtension: "HEIC")!
let output = FileManager().temporaryDirectory.appendingPathComponent("IMG_0037.GAIN_MAP.BMP")
let source = CGImageSourceCreateWithURL(input as CFURL, nil)!
// urn:com:apple:photo:2020:aux:hdrgainmap
let dataInfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary
let data = dataInfo[kCGImageAuxiliaryDataInfoData] as! Data
let description = dataInfo[kCGImageAuxiliaryDataInfoDataDescription]! as! [String: Int]
let size = CGSize(width: description["Width"]!, height: description["Height"]!)
let ciImage = CIImage(bitmapData: data, bytesPerRow: description["BytesPerRow"]!, size: size, format: .L8, colorSpace: nil)
let cgImage = CIContext().createCGImage(ciImage, from: CGRect(origin: CGPoint(x: 0, y: 0), size: size))!
let destRef = CGImageDestinationCreateWithURL(output as CFURL, kUTTypeBMP, 1, nil)!
CGImageDestinationAddImage(destRef, cgImage, [:] as CFDictionary)
CGImageDestinationFinalize(destRef)
print(output)
}
@frankschlegel
Copy link

Any luck on your side getting Apple Photos to recognize a bespoke image?

@jiawen Indeed. It took me a while, but I was able to generate another test image that also works on macOS:
macOS_HDR_Test

The following was needed to make it work:

  • In the XMP metadata, the HDRGainMapVersion key needed to be present. (I set it to 65536)
  • A gain map auxiliary image of type urn:com:apple:photo:2020:aux:hdrgainmap.
  • The {MakerApple} metadata tags 33 and 48 need to have meaningful values according to Apple's documentation. (I set them to 0.8 and 0.0 for maximum effect.)
  • And, new to me, the image must be larger than 360 pixels in each dimension. For smaller images, the HDR effect simply doesn't show. This took me the longest to find out. 😅

The first and last point were needed on macOS, but iOS displayed HDR even without them.

I also found, on both platforms, that only Photos is capable of displaying gain map HDR images. The HDR effect appears nowhere else, neither in Messages, Preview, Files, etc.

either 65536 (0x10000) as in that thread, or 131072 (0x20000) in some newer files

Hu, I haven't seen 131072 yet. Do you have an example of where you've seen it?

@gregbenz
Copy link

gregbenz commented Nov 5, 2023

@jiawen Did you write code to do that, or just use software like Adobe LR / ACR? ImageMagick has an open request for contributors who may be able to help support gain maps. ImageMagick/ImageMagick#6377. ImageMagick support would be key to support for gain maps in the WordPress media library, etc.

Another resource for this group (if the interest isn't just limited to HEIC / Swift) is the HDR JPG gain map library published by Google: https://github.com/google/libultrahdr. This is built to the Google / Adobe gain map spec (which is the basis of a draft ISO specification and is a de facto standard with support in Lightroom, Adobe Camera RAW, Chrome, Edge, Brave, and Opera already).

@jiawen
Copy link

jiawen commented Nov 7, 2023

Hi @frankschlegel - this is amazing! I confirmed that your JPEG renders as HDR in Apple Photos my iPhone 15 Pro Max (running iOS 17.1) and M1 Macbook Pro (running macOS Sonoma). Unsurprisingly, it does not render correctly in Apple Photos on my M1 Mac Studio running macOS Ventura when connected to an Apple XDR display (you see a difference if you toggle HDR on/off in Preferences. But it doesn't show the HDR text. It only shows a flat field of bright white vs less bright white.). Surprisingly, it does render as HDR in Chrome (v118)! Even on Ventura. It looks like Google decided to parse one of the fields, probably the XMP you added.

Amazing!

Hu, I haven't seen 131072 yet. Do you have an example of where you've seen it?

I got the following in the aux image's XMP on a JPEG taken with an iPhone 15 Pro's rear camera. I extracted it using exiftool -MPImage2 -b my_image.jpg > aux.jpg followed by exiftool -xmp -b aux.jpg.

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 6.0.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:HDRGainMap="http://ns.apple.com/HDRGainMap/1.0/"
            xmlns:apdi="http://ns.apple.com/pixeldatainfo/1.0/">
         <HDRGainMap:HDRGainMapVersion>131072</HDRGainMap:HDRGainMapVersion>
         <HDRGainMap:HDRGainMapHeadroom>5.651999</HDRGainMap:HDRGainMapHeadroom>
         <apdi:NativeFormat>1278226488</apdi:NativeFormat>
         <apdi:AuxiliaryImageType>urn:com:apple:photo:2020:aux:hdrgainmap</apdi:AuxiliaryImageType>
         <apdi:StoredFormat>1278226488</apdi:StoredFormat>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

@jiawen
Copy link

jiawen commented Nov 7, 2023

@jiawen Did you write code to do that, or just use software like Adobe LR / ACR? ImageMagick has an open request for contributors who may be able to help support gain maps. ImageMagick/ImageMagick#6377. ImageMagick support would be key to support for gain maps in the WordPress media library, etc.

Another resource for this group (if the interest isn't just limited to HEIC / Swift) is the HDR JPG gain map library published by Google: https://github.com/google/libultrahdr. This is built to the Google / Adobe gain map spec (which is the basis of a draft ISO specification and is a de facto standard with support in Lightroom, Adobe Camera RAW, Chrome, Edge, Brave, and Opera already).

I wrote bespoke code to do this. I would love to open source this code as it is simultaneously useful, esoteric, yet not too difficult when you use the right abstractions. But my manager probably wants me to spend my time elsewhere. I think my employer can be convinced to release this portion of our code and maintain it, but someone else will need to plumb it into ImageMagick as I lack expertise there.

@gregbenz
Copy link

gregbenz commented Nov 7, 2023

@jiawen That's great. If you are able to share it, I think just getting it as it currently stands to ImageMagick via ImageMagick/ImageMagick#6377 may be of great value to help expedite those efforts.

@frankschlegel
Copy link

I got the following in the aux image's XMP on a JPEG taken with an iPhone 15 Pro's rear camera.

@jiawen Very interesting! It seems Apple introduced a new version with the iPhone 15, including the new HDRGainMapHeadroom XMP field. I wonder what this does... It's not documented in their gain map documentation so far.

Surprisingly, it does render as HDR in Chrome (v118)!

Also in Brave. It seems support for Apple's format was added together with the support for Adobe's format to the Chrome engine. Very nice!

I wonder where this is all going. On the developer side, Apple seems to advocate for exporting HDR images in what they call "ISO HDR" format (basically an HDR video still frame). In fact, that's the only official way for exporting HDR images. There are no simple APIs for writing an SDR image + HDR gain map.

@gregbenz You are very deep into this topic. Do you have an idea what will be the dominant standard in the future? This "ISO HDR" format seems to be the obvious candidate, as it's simply the same as HDR video. But it comes with the big disadvantage that there is no clearly defined SDR representation for those images. It's up to the displaying application how to tone-map those to SDR. The gain map approach is much more explicit.

@gregbenz
Copy link

gregbenz commented Nov 8, 2023

I don’t know that I’d quite say ISO HDR is the same as a video still frame, not in a literal sense anyhow. Video tends to define a 100 nits white point, photography is using 203. There are certainly many commonalities (sadly including the lack of a common standard for tone mapping).

The “ISO HDR” refers to encoding of an HDR image. There is another ISO draft for gain maps. These are complimentary, as an ISO gain map can use an ISO HDR as the base image. You can use one without the other, but they work best together (if the base image in a gain map is HDR).

Gain maps are the obvious way forward. They offer a much better experience for viewing on any display which lacks the full HDR headroom encoded in the image. It’s vastly better than tone mapping.

JPG gain map is the obvious choice right now. Browsers representing ~75% of viewers (once fully updated) support it on systems with HDR display. And backwards compatibility is 100% (if your browser / image software doesn’t support HDR or gain maps, you gracefully fall back to a nice SDR).

However, JPG requires large files for high quality. There is a draft AVIF gain map standard (Adobe has shared sample files and Chrome/Brave support it). This is clearly the way things “should” go.

I say “should” because best doesn’t always win. As of now, Microsoft Edge (and the file explorer) doesn’t support any AVIF, and that needs to change for AVIF to be viable. Assuming that happens (they have support under a dev flag), we still need an encoder and more browsers to add support. It’s part of the same draft ISO proposal for gain maps and I hope to see it happen.

Another theoretical candidate would be JXL (JPEG XL). As browser support is zero and Chrome pulled support, it doesn’t seem promising anytime soon.

If it were all up to me: we’d use JPG gain maps for now, transition to AVIF gain maps as soon as MS Edge supports AVIF (encoded with an SDR base image for compatibility), then finally get to the ideal AVIF encoded with an HDR base image when support for AVIF gain maps is widely available.

The other thing I’d like to see is encoder support to let the artist provide the SDR rendition as well. This has many benefits, which I outlined in this feature request for Photoshop (please upvote it if you agree): https://community.adobe.com/t5/photoshop-ecosystem-ideas/allow-full-user-control-of-the-sdr-rendition-in-an-hdr-gain-map-export/idi-p/14205440#M19464

@grapeot
Copy link

grapeot commented Nov 20, 2023

Although I'm late to the party, I must say this thread is the most informative and inspiring resource on the entire internet regarding this specific topic. I've truly learned a lot from all the discussions above and want to sincerely thank every participant and contributor. Inspired by the discussion, it seems I've figured out how to use Apple's public APIs to generate a JPEG with a gain map that renders properly on both iPhone and Mac. It can also be inspected by Adobe's demo app and other third-party software like HoneyView. I've shared the code with detailed documentation, hoping it will be helpful to others: https://github.com/grapeot/AppleJPEGGainMap

@Wrestor
Copy link

Wrestor commented Jun 29, 2024

Some more information from the apple's developer site: https://developer.apple.com/videos/play/wwdc2024/10177/

@alexfoxy
Copy link

alexfoxy commented Jan 16, 2025

Does anyone know how to capture a AVCapturePhoto with the HDRGain map? I've been advised by an Apple Engineer that I can access the gain map as follows:

// `photo` is an AVCapturePhoto
let data = photo.fileDataRepresentation()!
        
let gainMap = CIImage(data: data, options: [.auxiliaryHDRGainMap : true])

But it's always nil. I am on an iPhone 16 Pro and the gain map is captured on the default iOS camera app so the hardware is capable. I've looked through all the Apple docs but there's no mention of this.

@rogino
Copy link

rogino commented Jan 16, 2025

I’m on my phone right now so I haven’t tested this, but can you check what the image’s colour space is and transform function is? The HDR gain map is only necessary if the image is encoded in a SDR.

I’ve noticed that when reading images from the photo library, they are in the SDR P3 colour space and have an HDR gain map.

Thus, my theory at your situation is that the AVCaptureImage is in a HDR format and thus have no gain map. However, for compatibility reasons when it is imported into Photos, it gets transformed into SDR with a gain map. Thus, the .auxiliaryHDRGainMap flag works in Photo Library assets (which I have found does work), but not with AvCaptureImages

@alexfoxy
Copy link

Thanks for the reply. I've had a look and I can't see how you could change the colour space of the capture device. If they are already in an HDR format, then there would be a way to extract the HDR data?

@hungandang
Copy link

@rogino ro
I'm working with HDR videos and trying to add emoji into every frames of video to preview and export by AvassetReader and AvassetWriter. But the image I created from emoji string through UIGraphicsImageRenderer always darker than HDR video (it's look normal when add to SDR video) whether I use CIContext to creat new CGImage with colorspace: CGColorSpace.itur_2100_HLG. Is there a way to convert images from SDR to HDR or create HDR images?

@rogino
Copy link

rogino commented Jan 17, 2025

@alexfoxy have you tried the [\[expandToHDR: true\]](https://developer.apple.com/videos/play/wwdc2023/10181/?time=1080) flag on CIImage?

@hungandang I haven't used those specific APIs so I can't offer you any help with those APIs, but if all else fails, you should be able to do the conversion manually with a Metal shader:

  • Linearize the contents - if it is in sRGB or displayP3 (with no suffix) then it probably has a gamma applied, which you need to invert before doing anything else. Link to formula
  • Transform into the output color space. This link has a transform for sRGB to P3 - if you are using Rec. 2020 you'll need to find the correct equation somewhere else
  • Apply the HLG/PQ EOTF function to scale values from SDR to HDR

Example for sRGB to DisplayP3 HLG:

/// sRGB gamma function
float srgbEOTF(float E) {
    // https://www.color.org/srgb.pdf
    if (E <= 0.04045) {
        return E / 12.92;
    } else {
        return pow((E + 0.055) / 1.055, 2.4);
    }
}

float3 srgbEOTF(float3 E) {
    return float3(srgbEOTF(E.r), srgbEOTF(E.g), srgbEOTF(E.b));
}

// http://endavid.com/index.php?entry=79
/// Matrix which transforms (linear) sRGB to (linear) Display P3
constant float3x3 srgbToDisplayP3TransformMatrix = float3x3(
    0.8225, 0.0332, 0.0171,
    0.1774, 0.9669, 0.0724,
    0     , 0     , 0.9108
);

// Constants for hybrid-log-gamma transform, as defined by Rec. 2100
// See: https://en.wikipedia.org/wiki/Hybrid_log%E2%80%93gamma
#define hlgA (0.17883277)
#define hlgB (1 - 4 * hlgA)
#define hlgC (0.5 - hlgA * log(4 * hlgA))

/// Hybrid-log-gamma transform, as defined by Rec. 2100.
float hlgOETF(float O) {
    // Rec. 2100 HLG curve.
    // Points (0, 0) (black), (0.5, 1/12) (reference white), (1, 1) (12x reference white)
    
    if (O < 1.0 / 12.0) {
        return sqrt(3 * O);
    } else {
        return hlgA * log(12.0 * O - hlgB) + hlgC;
    }
}
float3 hlgOETF(float3 O) {
    return float3(hlgOETF(O.r), hlgOETF(O.g), hlgOETF(O.b));
}

float3 sdrToEdr(float3 sample) {
    sample = srgbEOTF(sample);
    sample = srgbToDisplayP3TransformMatrix * sample;
    sample = hlgOETF(sample / 12.0);  // (1/12, 1/2) is SDR white
    return sample;
}

(disclaimer: while this code seems to work okay in my app, I wouldn't be surprised if there are bugs in it)

@alexfoxy
Copy link

@rogino Yeah, no dice with that option unfortunately. I think that the AVCapturePhoto has no HDRGainMap for some reason, I assume it's some configuration issue with the AVCaptureSession, but with no documentation it's pretty hard to work out. The other option is that capturing HDRGainMaps is not a public API, has anyone managed to capture images with a HDRGainMap outside of the default camera app?

One other query I had, do ProRAW images have an HDRGainMap, or only HEIC?

Thank you for the help!

@rogino
Copy link

rogino commented Jan 17, 2025

Hmmm, I’m still leaning towards the photo having HDR metadata but it not being read for some reason. I’m running out of ideas though.

If you haven’t already, can you try watch this video and use the sample app to see if the photos captured in your app have HDR data?
https://developer.apple.com/videos/play/wwdc2023/10181/
(They don’t have screenshots of the sample app and I’m on my phone so I might have linked to the wrong one)

@alexfoxy
Copy link

Just to report back on this. I have managed to get the HDRGainMap from a AVCapturePhoto with the following:

let data = photo.fileDataRepresentation()!
let gainMap = CIImage(data: data, options: [.auxiliaryHDRGainMap : true])

The key thing was that the capture device was configured correctly. From an Apple DTS Engineer:

Make sure your capture device's activeFormat reports true for isHighestPhotoQualitySupported. (or use the ".photo" sessionPreset on the capture session).

The last hurdle I'm encountering is that this still returns nil on an iPhone 13. I'm assuming the older hardware does not support it, but it's strange because I can access some sort of HDRGainMap via the CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary method with a photo captured by the default camera app on the iPhone 13. It does appear different to the iPhone 16 Pro however, so I assume there is some difference in hardware output. See below:

Screenshot 2025-01-19 at 16 03 03

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment