Skip to content

Instantly share code, notes, and snippets.

@kiding
Last active January 19, 2025 16:03
Show Gist options
  • Save kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a to your computer and use it in GitHub Desktop.
Save kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a to your computer and use it in GitHub Desktop.
Extracting HDR Gain Map from iOS 14.1+ (iPhone 12+) photos
import UIKit
import MobileCoreServices.UTCoreTypes
if #available(iOS 14.1, *) {
let input = Bundle.main.url(forResource: "IMG_0037", withExtension: "HEIC")!
let output = FileManager().temporaryDirectory.appendingPathComponent("IMG_0037.GAIN_MAP.BMP")
let source = CGImageSourceCreateWithURL(input as CFURL, nil)!
// urn:com:apple:photo:2020:aux:hdrgainmap
let dataInfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary
let data = dataInfo[kCGImageAuxiliaryDataInfoData] as! Data
let description = dataInfo[kCGImageAuxiliaryDataInfoDataDescription]! as! [String: Int]
let size = CGSize(width: description["Width"]!, height: description["Height"]!)
let ciImage = CIImage(bitmapData: data, bytesPerRow: description["BytesPerRow"]!, size: size, format: .L8, colorSpace: nil)
let cgImage = CIContext().createCGImage(ciImage, from: CGRect(origin: CGPoint(x: 0, y: 0), size: size))!
let destRef = CGImageDestinationCreateWithURL(output as CFURL, kUTTypeBMP, 1, nil)!
CGImageDestinationAddImage(destRef, cgImage, [:] as CFDictionary)
CGImageDestinationFinalize(destRef)
print(output)
}
@rogino
Copy link

rogino commented Jan 17, 2025

@alexfoxy have you tried the [\[expandToHDR: true\]](https://developer.apple.com/videos/play/wwdc2023/10181/?time=1080) flag on CIImage?

@hungandang I haven't used those specific APIs so I can't offer you any help with those APIs, but if all else fails, you should be able to do the conversion manually with a Metal shader:

  • Linearize the contents - if it is in sRGB or displayP3 (with no suffix) then it probably has a gamma applied, which you need to invert before doing anything else. Link to formula
  • Transform into the output color space. This link has a transform for sRGB to P3 - if you are using Rec. 2020 you'll need to find the correct equation somewhere else
  • Apply the HLG/PQ EOTF function to scale values from SDR to HDR

Example for sRGB to DisplayP3 HLG:

/// sRGB gamma function
float srgbEOTF(float E) {
    // https://www.color.org/srgb.pdf
    if (E <= 0.04045) {
        return E / 12.92;
    } else {
        return pow((E + 0.055) / 1.055, 2.4);
    }
}

float3 srgbEOTF(float3 E) {
    return float3(srgbEOTF(E.r), srgbEOTF(E.g), srgbEOTF(E.b));
}

// http://endavid.com/index.php?entry=79
/// Matrix which transforms (linear) sRGB to (linear) Display P3
constant float3x3 srgbToDisplayP3TransformMatrix = float3x3(
    0.8225, 0.0332, 0.0171,
    0.1774, 0.9669, 0.0724,
    0     , 0     , 0.9108
);

// Constants for hybrid-log-gamma transform, as defined by Rec. 2100
// See: https://en.wikipedia.org/wiki/Hybrid_log%E2%80%93gamma
#define hlgA (0.17883277)
#define hlgB (1 - 4 * hlgA)
#define hlgC (0.5 - hlgA * log(4 * hlgA))

/// Hybrid-log-gamma transform, as defined by Rec. 2100.
float hlgOETF(float O) {
    // Rec. 2100 HLG curve.
    // Points (0, 0) (black), (0.5, 1/12) (reference white), (1, 1) (12x reference white)
    
    if (O < 1.0 / 12.0) {
        return sqrt(3 * O);
    } else {
        return hlgA * log(12.0 * O - hlgB) + hlgC;
    }
}
float3 hlgOETF(float3 O) {
    return float3(hlgOETF(O.r), hlgOETF(O.g), hlgOETF(O.b));
}

float3 sdrToEdr(float3 sample) {
    sample = srgbEOTF(sample);
    sample = srgbToDisplayP3TransformMatrix * sample;
    sample = hlgOETF(sample / 12.0);  // (1/12, 1/2) is SDR white
    return sample;
}

(disclaimer: while this code seems to work okay in my app, I wouldn't be surprised if there are bugs in it)

@alexfoxy
Copy link

@rogino Yeah, no dice with that option unfortunately. I think that the AVCapturePhoto has no HDRGainMap for some reason, I assume it's some configuration issue with the AVCaptureSession, but with no documentation it's pretty hard to work out. The other option is that capturing HDRGainMaps is not a public API, has anyone managed to capture images with a HDRGainMap outside of the default camera app?

One other query I had, do ProRAW images have an HDRGainMap, or only HEIC?

Thank you for the help!

@rogino
Copy link

rogino commented Jan 17, 2025

Hmmm, I’m still leaning towards the photo having HDR metadata but it not being read for some reason. I’m running out of ideas though.

If you haven’t already, can you try watch this video and use the sample app to see if the photos captured in your app have HDR data?
https://developer.apple.com/videos/play/wwdc2023/10181/
(They don’t have screenshots of the sample app and I’m on my phone so I might have linked to the wrong one)

@alexfoxy
Copy link

Just to report back on this. I have managed to get the HDRGainMap from a AVCapturePhoto with the following:

let data = photo.fileDataRepresentation()!
let gainMap = CIImage(data: data, options: [.auxiliaryHDRGainMap : true])

The key thing was that the capture device was configured correctly. From an Apple DTS Engineer:

Make sure your capture device's activeFormat reports true for isHighestPhotoQualitySupported. (or use the ".photo" sessionPreset on the capture session).

The last hurdle I'm encountering is that this still returns nil on an iPhone 13. I'm assuming the older hardware does not support it, but it's strange because I can access some sort of HDRGainMap via the CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeHDRGainMap)! as Dictionary method with a photo captured by the default camera app on the iPhone 13. It does appear different to the iPhone 16 Pro however, so I assume there is some difference in hardware output. See below:

Screenshot 2025-01-19 at 16 03 03

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment