Extracting HDR Gain Map from iOS 14.1+ (iPhone 12+) photos

HDRGainMap.swift

import UIKit
import MobileCoreServices.UTCoreTypes

if #available(iOS 14.1, *) {
    let input = Bundle.main.url(forResource: "IMG_0037", withExtension: "HEIC")!
    let output = FileManager().temporaryDirectory.appendingPathComponent("IMG_0037.GAIN_MAP.BMP")
    let source = CGImageSourceCreateWithURL(input as CFURL, nil)!

    // urn:com:apple:photo:2020:aux:hdrgainmap
    let dataInfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary
    let data = dataInfo[kCGImageAuxiliaryDataInfoData] as! Data
    let description = dataInfo[kCGImageAuxiliaryDataInfoDataDescription]! as! [String: Int]
    let size = CGSize(width: description["Width"]!, height: description["Height"]!)
    
    let ciImage = CIImage(bitmapData: data, bytesPerRow: description["BytesPerRow"]!, size: size, format: .L8, colorSpace: nil)
    let cgImage = CIContext().createCGImage(ciImage, from: CGRect(origin: CGPoint(x: 0, y: 0), size: size))!
    
    let destRef = CGImageDestinationCreateWithURL(output as CFURL, kUTTypeBMP, 1, nil)!
    CGImageDestinationAddImage(destRef, cgImage, [:] as CFDictionary)
    CGImageDestinationFinalize(destRef)

    print(output)
}

Although I'm late to the party, I must say this thread is the most informative and inspiring resource on the entire internet regarding this specific topic. I've truly learned a lot from all the discussions above and want to sincerely thank every participant and contributor. Inspired by the discussion, it seems I've figured out how to use Apple's public APIs to generate a JPEG with a gain map that renders properly on both iPhone and Mac. It can also be inspected by Adobe's demo app and other third-party software like HoneyView. I've shared the code with detailed documentation, hoping it will be helpful to others: https://github.com/grapeot/AppleJPEGGainMap

Some more information from the apple's developer site: https://developer.apple.com/videos/play/wwdc2024/10177/

Does anyone know how to capture a AVCapturePhoto with the HDRGain map? I've been advised by an Apple Engineer that I can access the gain map as follows:

// `photo` is an AVCapturePhoto
let data = photo.fileDataRepresentation()!
        
let gainMap = CIImage(data: data, options: [.auxiliaryHDRGainMap : true])

But it's always nil. I am on an iPhone 16 Pro and the gain map is captured on the default iOS camera app so the hardware is capable. I've looked through all the Apple docs but there's no mention of this.

I’m on my phone right now so I haven’t tested this, but can you check what the image’s colour space is and transform function is? The HDR gain map is only necessary if the image is encoded in a SDR.

I’ve noticed that when reading images from the photo library, they are in the SDR P3 colour space and have an HDR gain map.

Thus, my theory at your situation is that the AVCaptureImage is in a HDR format and thus have no gain map. However, for compatibility reasons when it is imported into Photos, it gets transformed into SDR with a gain map. Thus, the .auxiliaryHDRGainMap flag works in Photo Library assets (which I have found does work), but not with AvCaptureImages

Thanks for the reply. I've had a look and I can't see how you could change the colour space of the capture device. If they are already in an HDR format, then there would be a way to extract the HDR data?

@rogino ro
I'm working with HDR videos and trying to add emoji into every frames of video to preview and export by AvassetReader and AvassetWriter. But the image I created from emoji string through UIGraphicsImageRenderer always darker than HDR video (it's look normal when add to SDR video) whether I use CIContext to creat new CGImage with colorspace: CGColorSpace.itur_2100_HLG. Is there a way to convert images from SDR to HDR or create HDR images?

@alexfoxy have you tried the [\[expandToHDR: true\]](https://developer.apple.com/videos/play/wwdc2023/10181/?time=1080) flag on CIImage?

@hungandang I haven't used those specific APIs so I can't offer you any help with those APIs, but if all else fails, you should be able to do the conversion manually with a Metal shader:

• Linearize the contents - if it is in sRGB or displayP3 (with no suffix) then it probably has a gamma applied, which you need to invert before doing anything else. Link to formula
• Transform into the output color space. This link has a transform for sRGB to P3 - if you are using Rec. 2020 you'll need to find the correct equation somewhere else
• Apply the HLG/PQ EOTF function to scale values from SDR to HDR

Example for sRGB to DisplayP3 HLG:

/// sRGB gamma function
float srgbEOTF(float E) {
    // https://www.color.org/srgb.pdf
    if (E <= 0.04045) {
        return E / 12.92;
    } else {
        return pow((E + 0.055) / 1.055, 2.4);
    }
}

float3 srgbEOTF(float3 E) {
    return float3(srgbEOTF(E.r), srgbEOTF(E.g), srgbEOTF(E.b));
}

// http://endavid.com/index.php?entry=79
/// Matrix which transforms (linear) sRGB to (linear) Display P3
constant float3x3 srgbToDisplayP3TransformMatrix = float3x3(
    0.8225, 0.0332, 0.0171,
    0.1774, 0.9669, 0.0724,
    0     , 0     , 0.9108
);

// Constants for hybrid-log-gamma transform, as defined by Rec. 2100
// See: https://en.wikipedia.org/wiki/Hybrid_log%E2%80%93gamma
#define hlgA (0.17883277)
#define hlgB (1 - 4 * hlgA)
#define hlgC (0.5 - hlgA * log(4 * hlgA))

/// Hybrid-log-gamma transform, as defined by Rec. 2100.
float hlgOETF(float O) {
    // Rec. 2100 HLG curve.
    // Points (0, 0) (black), (0.5, 1/12) (reference white), (1, 1) (12x reference white)
    
    if (O < 1.0 / 12.0) {
        return sqrt(3 * O);
    } else {
        return hlgA * log(12.0 * O - hlgB) + hlgC;
    }
}
float3 hlgOETF(float3 O) {
    return float3(hlgOETF(O.r), hlgOETF(O.g), hlgOETF(O.b));
}

float3 sdrToEdr(float3 sample) {
    sample = srgbEOTF(sample);
    sample = srgbToDisplayP3TransformMatrix * sample;
    sample = hlgOETF(sample / 12.0);  // (1/12, 1/2) is SDR white
    return sample;
}

(disclaimer: while this code seems to work okay in my app, I wouldn't be surprised if there are bugs in it)

@rogino Yeah, no dice with that option unfortunately. I think that the AVCapturePhoto has no HDRGainMap for some reason, I assume it's some configuration issue with the AVCaptureSession, but with no documentation it's pretty hard to work out. The other option is that capturing HDRGainMaps is not a public API, has anyone managed to capture images with a HDRGainMap outside of the default camera app?

One other query I had, do ProRAW images have an HDRGainMap, or only HEIC?

Thank you for the help!

Hmmm, I’m still leaning towards the photo having HDR metadata but it not being read for some reason. I’m running out of ideas though.

If you haven’t already, can you try watch this video and use the sample app to see if the photos captured in your app have HDR data?
https://developer.apple.com/videos/play/wwdc2023/10181/
(They don’t have screenshots of the sample app and I’m on my phone so I might have linked to the wrong one)

Just to report back on this. I have managed to get the HDRGainMap from a AVCapturePhoto with the following:

let data = photo.fileDataRepresentation()!
let gainMap = CIImage(data: data, options: [.auxiliaryHDRGainMap : true])

The key thing was that the capture device was configured correctly. From an Apple DTS Engineer:

Make sure your capture device's activeFormat reports true for isHighestPhotoQualitySupported. (or use the ".photo" sessionPreset on the capture session).

The last hurdle I'm encountering is that this still returns nil on an iPhone 13. I'm assuming the older hardware does not support it, but it's strange because I can access some sort of HDRGainMap via the CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary method with a photo captured by the default camera app on the iPhone 13. It does appear different to the iPhone 16 Pro however, so I assume there is some difference in hardware output. See below: