Skip to content

Instantly share code, notes, and snippets.

@steven-michaud
Last active May 2, 2024 05:54
Show Gist options
  • Star 21 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save steven-michaud/fda019a4ae2df3a9295409053a53a65c to your computer and use it in GitHub Desktop.
Save steven-michaud/fda019a4ae2df3a9295409053a53a65c to your computer and use it in GitHub Desktop.
Running Third Party Kernel Extensions on Virtualization Framework macOS Guest VMs

Running Third Party Kernel Extensions on Virtualization Framework macOS Guest VMs

As of macOS 12 (Monterey), Apple's Virtualization framework has nice support for macOS guest virtual machines, but with severe limitations: For example you can't install a macOS guest on Intel Macs, install guests with newer versions of macOS than the host, copy and paste between the host and the guest, or install third party kernel extensions in the guest. As usual for Apple, the functionality they do support is nicely implemented, but they've left out so much that the result is only marginally useful -- at least compared to third-party implementations available on Intel Macs, like VMware Fusion and Parallels.

I've been working on the last of these limitations, and have found a workaround. It's rather complex, but should be very useful for those developing kernel extensions for macOS on Apple Silicon, or (more likely) porting existing Intel kexts to Apple Silicon. Developing kernel extensions on bare metal is a pain -- not least because you might end up damaging your nice new Apple Silicon Mac. But doing it in a virtual machine isolates the potential damage to the VM itself -- at worst you'll only need to trash it and create a replacement.

Parts of my workaround are interesting in themselves, even if you don't use them to run third party kernel extensions on macOS guest VMs. For example I've found a way to patch the VM's kernel cache (containing the kernel and built-in kernel extensions), which should be very useful in reverse engineering macOS.

Most of the discoveries here are ones I've made on my own, by trial and error. But I had a very helpful starting point -- NyanSatan's Virtual-iBoot-Fun project.

Prerequisites

  • An Apple Silicon Mac running macOS 12 or higher. Mine is a 2020 Mac Mini (Macmini9,1).

  • A decent virtual machine host that uses Apple's Virtualization framework. I use UTM.

  • One or more decent disassemblers. I use Ghidra and Hopper Disassembler. I also installed Nick Botticelli's ghidra-iboot plugin

  • Tihmstar's img4tool.

  • A decent hex editor. I use Hex Fiend. I change its default edit mode to "Overwrite" and its line number format to "Hexadecimal".

  • A binary diff tool. I use VBinDiff.

  • A calculator with support for radix modes. Apple's Calculator in Programmer mode will do.

Background

My workaround patches three iBoot modules and the VM's kernel cache.

The iBoot modules load early in the macOS (and IOS) boot sequence, before the kernel. There are at least three variants of iBoot, used in three diffent "stages" -- Stage 0 (as I call it), Stage 1 and Stage 2. On bare metal, Stage 0 is implemented in hardware (in Apple's "Secure Enclave"). Stage 1 (aka LLB) and Stage 2 (iBoot properly so called) are implemented in software, though only the Stage 2 iBoot exists (inside iBoot.img4) in the macOS file system.

By design, the Secure Enclave Processor (SEP) is inaccessible to ordinary mortals. And on bare metal the LLB and iBoot modules are encrypted (using a key held in the SEP). But for a VM, the modules for all three stages are unencrypted. Stage 0 is implemented (on the host) in /System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin. Stage 1 is embedded (in img4 format) in an AuxiliaryStorage file (on the host) associated with the DMG file (with an img extension) that stores the VM's image. Stage 2 (inside iBoot.img4) exists in the VM's file system.

None of the iBoot variants call any external functions, and none have any symbols. So they're difficult to decipher. But the fact that the Virtualization framework uses them without encryption is a golden opportunity to reverse engineer them, and understand them better.

It's apparently common knowledge that the iBoot modules all contain a function to check the "digests" (DGST) of the many img4 images that govern the macOS boot process (including LLB, iBoot.img4 and the kernel cache itself). Each "digest" is a hash, used to ensure that none of these images has changed since they were "signed" by Apple. The source code for a similar function (image4_validate_property_callback()) is available here.

The images in question are those to be loaded at the next stage. So (among other things) Stage 0 iBoot checks the digest of the Stage 1 iBoot img4 image, Stage 1 iBoot checks the digest of the Stage 2 iBoot img4 file, and Stage 2 iBoot checks the digest of the kernelcache file. Aside from the file system itself, nothing checks the integrity of the stage 0 iBoot file (AVPBooter.vmapple2.bin). So all you need to stop these integrity checks is to defeat the file system's protection and patch this function (call it image4_validate_property_callback()) at every "stage".

As NyanSatan mentions, the standard approach is to patch out image4_validate_property_callback() completely -- to replace it with code that "returns 0". I find this causes problems, so my patch is more surgical -- I allow image4_validate_property_callback() to perform all its normal checks, and only patch its return value (to '0') just before it returns.

Some time ago I discovered that failed calls to _validate_acm_context() in the VM's AppleVPBootPolicy.kext stop the VM from creating an auxiliary kext cache. Since third-party kexts are stored here, this prevents them from loading in a Virtualization framework VM. The failed calls originate from two other functions in AppleVPBootPolicy.kext -- _command_create_linked_manifest() and _command_update_local_policy_for_kcos(). My workaround is to patch out the calls to _validate_acm_context() from both of these functions. This is a more "surgical" approach than patching out _validate_acm_context() itself, which is called from many additional functions in AppleVPBootPolicy.kext.

But this isn't enough by itself. An auxiliary kext cache does get created (you can see it using kmutil inspect). But macOS still complains that your third-party kext needs to be rebuilt. To fix this you need to go back to the Stage 2 iBoot module (inside iBoot.img4). One of its purposes is to fill the "device tree" with all the appropriate devices. But it (or the variant available in a Virtualization framework VM) refuses to allow any "AuxKC" entries to be added to chosen/memory-map in a VM, even when the auxiliary kext cache is present. Fixing this requires finding the function that checks whether a given boot object (originally in img4 format) should be processed, and patching out the call to it. Its first parameter is a four-character code -- for example illb, ibot, krnl or auxk. validate_boot_object() (my name for this function) indirectly calls image4_validate_property_callback(). So if you patch out the former you don't need to patch the latter. At some point I'll describe how I found validate_boot_object().

To display the contents of chosen/memory-map, run the following command in a Terminal prompt:

ioreg -p IODeviceTree -n "memory-map" -w 0 -r -t

Settings Changes (Host and Guest)

Some settings changes are required on the host by How to Defang macOS System Protections, which allows changes to AVPBooter.vmapple2.bin.

  • In System Settings, under Software Update, disable "Download new updates when available". Otherwise your host can become unbootable.

  • Boot into Recovery Mode, run Terminal and do the following:

csrutil disable
csrutil authenticated-root disable

You also, of course, need to create a macOS guest VM. Make sure it uses the Virtualization framework. In UTM this is accomplished by choosing "Virtualize" and then "macOS 12+".

Terminal will be used heavily below. So, to avoid lots of annoying prompts to give it permission to access files, it's best to give the Terminal app "full disk access" on the guest (in Privacy and Security in System Settings).

Then boot into Recovery Mode on the guest. On macOS 12 you'll need to use my hack to do this.

  • Run the Startup Security Utility. Choose Reduced Security and "Allow user management of kernel extensions from identified developers".

  • Run Terminal and do the following:

  • csrutil disable, then y to "Allow booting unsigned operating systems and any kernel extensions".

Special Considerations Once You've Made the Changes Described Here

Below you'll create (and boot from) an APFS snapshot on your host computer to accomodate changes to the Stage 0 iBoot module (AVMBooter.vmapple2.bin). You need to keep your host computer in this state as long as you're using third party kernel extensions in your macOS guest VM. (Without the patched AVPBooter.vmapple2.bin it will simply refuse to start.) You also need to keep csrutil's "authenticated-root" disabled, and keep "Download new updates when available" disabled under Software Update. But don't use Software Update to update macOS on your host while it's booted from this custom snapshot. I don't know that this will cause trouble, but I expect it might.

Use the following command to revert your snapshot (and AVPBooter.vmapple2.bin), then reboot your computer:

sudo bless --mount / --last-sealed-snapshot

Release version macOS 13 and 14 guest VMs can be upgraded to newer versions of macOS. But before you do so you should revert this document's changes to the Stage 2 iBoot module (iBoot.img4) and the kernel cache (kernelcache): Boot into Recovery Mode, run the Startup Security Utility, and choose Full Security. Afterwards you'll need to go through all the following steps again, from scratch. Among other things you'll need to create new iBoot.img4.org and kernelcache.org files, and possibly also a new LLB.img4.org file.

macOS 12 guests can't be upgraded. The Virtualization process (com.apple.Virtualization.VirtualMachine) crashes and your VM becomes unbootable. This is an Apple bug. I've seen it myself, and have seen it documented here. The same is true for beta-version macOS 14 guests, though the consequences are less severe.

In this case your only option is to create a new guest VM directly from an IPSW file, then work through the following steps.

Finding the Modules to be Patched

iBoot Stage 0 (AVPBooter.vmapple2.bin)

This one's on the host, and is easy to find. Copy it and rename the copy to AVPBooter.vmapple2.bin.org.

/System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin

iBoot Stage 1 (LLB)

This one's also on the host, but is harder to find. By default, VMs created by UTM are stored in the following directory:

~/Library/Containers/com.utmapp.UTM/Data/Documents

Each VM is stored in a "package" with the extension .utm. Inside the package, in its Data subdirectory, is a file named AuxiliaryStorage. LLB is embedded here, in img4 format. It's followed immediately by an img4 format image of the "logo". You'll need to get copies of both.

If you've upgraded your guest VM to a newer version of macOS, there will be two copies of the LLB image in the AuxiliaryStorage file, each followed by a "logo" image. Only the second set will be active. So that's the one you'll need to work on. And if you've already patched a previous version of macOS in your guest VM (following the instructions in this document), the second LLB image may still contain your earlier patch. If so, you won't need to repatch it.

The first LLB image is always at offset 0x24000. The second, if it exists, should be at offset 0x224000. To be sure you've found all the LLB images, and their correct locations, search (in your hex editor) on "illb" (the four character code for the LLB image).

  • Open AuxiliaryStorage in a hex editor. I use Hex Fiend, and will tailor my steps to it specifically.

  • Jump to offset 0x224000. If an LLB image exists at this location, its first two bytes should be 3083. If not, jump to offset 0x24000.

  • The img4 image is in DER format (used by ASN1), so the first five bytes are 3083 followed by a three digit hexadecimal number in big endian format (for example 037FD1). This number is the length of the image, exclusive of the length of its header. So the total length is 0x37FD6.

  • Select the first five bytes of the image, then choose "Extend Selection". In this case, you'd extend it by 0x37FD1 bytes. Scroll down to the end of the selection (without disturbing it) and check that it's in the correct location (just before the "logo" image). If it is, CMD-C to copy the image, CMD-N to open a new window, CMD-V to paste in its contents, and save the file as LLB.img4.org.

  • The header for the "logo" image should be 3082 followed by a two digit hexdecimal number (for example 36DA). Select the first four bytes, extend the selection by 0x36DA bytes and scroll down to its end. The following bytes should be a bunch of NULLs (00). CMD-C to copy the image, CMD-N to open another new window, CMD-V to paste in its contents, and save this second new file as logo.img4.org.

iBoot Stage 2 (iBoot.img4) and the Kernel Cache (kernelcache)

These modules are stored in files on the VM. So run the VM and do the following in it, at a Terminal prompt:

  • Run kmutil inspect, and observe where the boot kernel cache exists in the VM's file system. The following is an example, which will get used in the following steps. The path's exact contents will differ from case to case.
/System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/System/Library/Caches/com.apple.kernelcaches/kernelcache
  • Note the long hexadecimal number just before /System/Library/Caches -- in this case FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83. This is the "Next Stage Image4 Hash (nsih)", observable in the output of sudo bputil -d.

  • Copy this file and rename it to kernelcache.org.

  • cd / and sudo find . -name iBoot.img4 -exec ls -al \{\} \;. There will be at least two hits. Choose the one whose path contains the Next Stage Image4 Hash. Copy it and rename the copy to iBoot.img4.org. You might find the original iBoot.img4 here, for example:

/System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/usr/standalone/firmware/iBoot.img4

Patching These Modules

iBoot Stage 0 (AVPBooter.vmapple2.bin.org)

For this I use Ghidra with the ghidra-iboot plugin. Copy AVPBooter.vmapple2.bin.org to AVPBooter.vmapple2.bin. Then run Ghidra and create a new project. Then:

  • Run Ghidra's CodeBrowser and import AVPBooter.vmapple2.bin.

  • Choose Search : Program Text : All Fields. Type "0x4447" ('DG' of DGST) in "Search for", then choose "Search All". You should find two hits, both in the same function. Click on one to move the cursor to its location.

  • Once again choose Search : Program Text : All Fields, and type "retab" in "Search For". Then choose "Next". Now you should see the code that runs just before the function returns. It will look something like as follows. In this example, you want to change the instruction at address 0x00102a0c (mov x0,x20) to mov x0,#0x0. (AARCH64 machine code uses the X0 register to store a return value.)

001029f4 a8 83 5a f8     ldur       x8,[x29, #local_68]
001029f8 29 f9 37 d0     adrp       x9,0x70028000
001029fc 1f 20 03 d5     nop
00102a00 29 a1 40 f9     ldr        x9,[x9, #0x140]=>DAT_70028140
00102a04 3f 01 08 eb     cmp        x9,x8
00102a08 21 06 00 54     b.ne       LAB_00102acc
00102a0c e0 03 14 aa     mov        x0,x20
00102a10 fd 7b 4c a9     ldp        x29=>local_10,x30,[sp, #0xc0]
00102a14 f4 4f 4b a9     ldp        x20,x19,[sp, #local_20]
00102a18 f6 57 4a a9     ldp        x22,x21,[sp, #local_30]
00102a1c f8 5f 49 a9     ldp        x24,x23,[sp, #local_40]
00102a20 fa 67 48 a9     ldp        x26,x25,[sp, #local_50]
00102a24 fc 6f 47 a9     ldp        x28,x27,[sp, #local_60]
00102a28 ff 43 03 91     add        sp,sp,#0xd0
00102a2c ff 0f 5f d6     retab
  • Right click on the mov x0,0x20 instruction and choose "Patch Instruction". Then change x0,0x20 to x0,#0x0.

  • In Ghidra's CodeBrowser, choose File : Export Program. Then choose Format : Raw Bytes, and overwrite AVPBooter.vmapple2.bin.

  • Use vbindiff AVPBooter.vmapple2.bin.org AVPBooter.vmapple2.bin to check your results. There should be just one change, to a four-byte value -- the length of one AARCH64 instruction.

iBoot Stage 1 (LLB.img4)

All the other modules that need patching are "wrapped" in img4 format. So to get at their actual content you need to use img4tool to unpack them. Then you'll patch them and rewrap them in new img4 format files.

  • img4tool -e -p LLB.im4p.org LLB.img4.org

  • img4tool -e -m LLB.im4m.org LLB.img4.org

  • img4tool -e -o LLB.bin.org LLB.im4p.org

The iBoot Stage 1 module should now be in LLB.bin.org. Copy it to LLB.bin and patch it according to the instructions for the iBoot Stage 0 binary (AVPBooter.vmapple2.bin) above. If you've upgraded macOS in your guest VM from a version that you already patched, LLB.bin.org (and LLB.bin) may still contain your previous patch. In this case you don't need to repatch it, and may skip ahead to iBoot Stage 2.

  • Run img4tool LLB.im4p.org, which will produce output something like what follows. Use the information from it to run the next command. The value for "desc" differs from one version of macOS to another. Note that you do not want to use compression, even though img4tool supports it. img4tool uses Apple's libcompression.dylib to implement its compression (and decompression). But, even though it's the same type ("bvx2"), it's incompatible with the decompression used by the iBoot binaries, which don't have access to any external modules.
img4tool version: 0.199-ed194718f9d6a035a432f2fdfe9fc639b72cba6c-RELEASE
Compiled with plist: YES
IM4P: ---------
type: illb
desc: iBoot-8422.141.2
size: 0x000369df

Compression: bvx2
Uncompressed size: 0x0006dc90
IM4P does not contain KBAG values
  • img4tool -c LLB.im4p -t illb -d "iBoot-8422.141.2" LLB.bin

  • img4tool -c LLB.img4 -p LLB.im4p -m LLB.im4m.org

The patched iBoot Stage 1 module should now be in LLB.img4.

iBoot Stage 2 (iBoot.img4)

Once again you'll need to unwrap the iBoot Stage 2 binary, patch it, and then rewrap it in img4 format. But this time the process is more complicated: iBoot.im4p.org contains a PAYP structure, tacked on to its end, that you'll need to append to iBoot.im4p by hand.

  • img4tool -e -p iBoot.im4p.org iBoot.img4.org

  • img4tool -e -m iBoot.im4m.org iBoot.img4.org

  • img4tool -e -o iBoot.bin.org iBoot.im4p.org

Here's how to copy the PAYP structure from iBoot.im4p.org to a seperate file, which you'll later append to iBoot.im4p.

  • openssl asn1parse -inform der -in iBoot.im4p.org -i -dump

This outputs the entire iBoot.im4p.org file in human-readable ASN1 format. The PAYP structure appears at the end, and looks something like this. Use the structure's offset (259719 in this case) in the next command.

259719:d=1  hl=2 l=  28 cons:  cont [ 0 ]        
259721:d=2  hl=2 l=  26 cons:   SEQUENCE          
259723:d=3  hl=2 l=   4 prim:    IA5STRING         :PAYP
259729:d=3  hl=2 l=  18 cons:    SET               
259731:d=4  hl=7 l=  11 cons:     priv [ 1768907638 ] 
259738:d=5  hl=2 l=   9 cons:      SEQUENCE          
259740:d=6  hl=2 l=   4 prim:       IA5STRING         :iocv
259746:d=6  hl=2 l=   1 prim:       INTEGER           :03
  • xxd -p -s 259719 iBoot.im4p.org iBoot.payp.hex

iBoot.payp.hex is a hex dump. The following command will convert it to a binary (in DER format):

  • xxd -p -r iBoot.payp.hex iBoot.payp.bin

Check the contents of iBoot.payp.bin by running the following command:

openssl asn1parse -inform der -in iBoot.payp.bin -i

Now copy iBoot.bin.org to iBoot.bin. Then patch out the call to validate_boot_object(). As mentioned above, you can do this instead of patching image4_validate_property_callback().

e5 03 04 aa 
04 00 80 52 
  • In the Ghidra CodeBrowser, choose Search : For Instruction Patterns, then Edit Bytes and Input Mode Hex. Copy the above two lines of hexadecimal code and paste it into the Edit Bytes box, then choose Apply. Choose Search All and you should find one hit, with code that looks like the following:
                     FUN_700ac1fc                                    XREF[1]:     FUN_70063a78:70063fbc(c)  
700ac1fc e5 03 04 aa     mov        x5,x4
700ac200 04 00 80 52     mov        w4,#0x0
700ac204 01 00 00 14     b          LAB_700ac208

                     LAB_700ac208                                    XREF[1]:     700ac204(j)  
700ac208 7f 23 03 d5     pacibsp
700ac20c ff 03 03 d1     sub        sp,sp,#0xc0
  • Double-click on the cross reference (FUN_70063a78:70063fbc(c) in this case). That should take you to code that looks like this:
70063fb0 e1 e3 0c 91     add        param_2,sp,#0x338
70063fb4 e4 43 03 91     add        param_5,sp,#0xd0
70063fb8 e3 03 16 aa     mov        param_4,x22
70063fbc 90 20 01 94     bl         FUN_700ac1fc                                     undefined FUN_700ac1fc()
70063fc0 a0 03 00 34     cbz        param_1,LAB_70064034
70063fc4 14 7b 00 51     sub        w20,w24,#0x1e
70063fc8 07 00 00 14     b          LAB_70063fe4
  • In this example, right-click on the bl FUN_700ac1fc instruction and change it to mov x0,#0x0.

  • On recent versions of macOS, there may be more than one cross reference. In this case you should double-click on each one, in turn, and change its target to mov x0,#0x0.

  • Choose File : Export Program. Then choose Format : Raw Bytes, and overwrite iBoot.bin.

  • Run img4tool iBoot.im4p.org, which will produce output something like this. Use the information from it to run the next command. The value of "desc" differs from one version of macOS to another.

img4tool version: 0.199-ed194718f9d6a035a432f2fdfe9fc639b72cba6c-RELEASE
Compiled with plist: YES
IM4P: ---------
type: ibot
desc: iBoot-8422.141.2
size: 0x0003f655

Compression: bvx2
Uncompressed size: 0x0007dde8
PAYP:
iocv: iocv: 3

IM4P does not contain KBAG values
  • img4tool -c iBoot.im4p -t ibot -d "iBoot-8422.141.2" iBoot.bin

  • dd if=iBoot.payp.bin >> iBoot.im4p

  • Use Hex Fiend to open iBoot.im4p and correct its length value. Make sure File : Mode is set to Override.

  • Observe iBoot.im4p's five-byte header and length value -- for example 308307DE0B. Convert the length (0x7DE0B) to decimal (515595) and add 30 (for the length of iBoot.payp.bin in this case). So the new length in this case is 515625 (== 0x7DE29).

  • Correct the length value. The header in this case will now be 308307DE29. Save iBoot.im4p.

  • img4tool -c iBoot.img4 -p iBoot.im4p -m iBoot.im4m.org

The patched Stage 2 module should now be in iBoot.img4.

The Kernel Cache (kernelcache)

As with the iBoot Stage 2 module, you'll need to unwrap the kernel cache, patch it, and rewrap it in img4 format.

  • img4tool -e -p kernelcache.im4p.org kernelcache.org

  • img4tool -e -m kernelcache.im4m.org kernelcache.org

  • img4tool -e -o kernelcache.bin.org kernelcache.im4p.org

kernelcache.im4p.org, like iBoot.im4p.org, has a PAYP structure at its end. Copy the PAYP structure to a separate file, which you'll later append to kernelcache.im4p.

  • openssl asn1parse -inform der -in kernelcache.im4p.org -i -dump

This outputs the entire kernelcache.im4p.org file in human-readable ASN1 format. The PAYP structure appears at the end, and looks something like this. Use the structure's offset (18030138 in this case) in the next command.

18030138:d=1  hl=3 l= 186 cons:  cont [ 0 ]        
18030141:d=2  hl=3 l= 183 cons:   SEQUENCE          
18030144:d=3  hl=2 l=   4 prim:    IA5STRING         :PAYP
18030150:d=3  hl=3 l= 174 cons:    SET               
18030153:d=4  hl=7 l=  19 cons:     priv [ 1801676144 ] 
18030160:d=5  hl=2 l=  17 cons:      SEQUENCE          
18030162:d=6  hl=2 l=   4 prim:       IA5STRING         :kcep
18030168:d=6  hl=2 l=   9 prim:       INTEGER           :FFFFFE0007B7D488
18030179:d=4  hl=7 l=  14 cons:     priv [ 1801677926 ] 
18030186:d=5  hl=2 l=  12 cons:      SEQUENCE          
18030188:d=6  hl=2 l=   4 prim:       IA5STRING         :kclf
18030194:d=6  hl=2 l=   4 prim:       INTEGER           :030E4000
18030200:d=4  hl=7 l=  19 cons:     priv [ 1801677935 ] 
18030207:d=5  hl=2 l=  17 cons:      SEQUENCE          
18030209:d=6  hl=2 l=   4 prim:       IA5STRING         :kclo
18030215:d=6  hl=2 l=   9 prim:       INTEGER           :FFFFFE0007004000
18030226:d=4  hl=7 l=  14 cons:     priv [ 1801677946 ] 
18030233:d=5  hl=2 l=  12 cons:      SEQUENCE          
18030235:d=6  hl=2 l=   4 prim:       IA5STRING         :kclz
18030241:d=6  hl=2 l=   4 prim:       INTEGER           :AF0000
18030247:d=4  hl=7 l=  11 cons:     priv [ 1801679462 ] 
18030254:d=5  hl=2 l=   9 cons:      SEQUENCE          
18030256:d=6  hl=2 l=   4 prim:       IA5STRING         :kcrf
18030262:d=6  hl=2 l=   1 prim:       INTEGER           :00
18030265:d=4  hl=7 l=  14 cons:     priv [ 1801679482 ] 
18030272:d=5  hl=2 l=  12 cons:      SEQUENCE          
18030274:d=6  hl=2 l=   4 prim:       IA5STRING         :kcrz
18030280:d=6  hl=2 l=   4 prim:       INTEGER           :02C64000
18030286:d=4  hl=7 l=  14 cons:     priv [ 1801680742 ] 
18030293:d=5  hl=2 l=  12 cons:      SEQUENCE          
18030295:d=6  hl=2 l=   4 prim:       IA5STRING         :kcwf
18030301:d=6  hl=2 l=   4 prim:       INTEGER           :02C64000
18030307:d=4  hl=7 l=  13 cons:     priv [ 1801680762 ] 
18030314:d=5  hl=2 l=  11 cons:      SEQUENCE          
18030316:d=6  hl=2 l=   4 prim:       IA5STRING         :kcwz
18030322:d=6  hl=2 l=   3 prim:       INTEGER           :480000
  • xxd -p -s 18030138 kernelcache.im4p.org kernelcache.payp.hex

kernelcache.payp.hex is a hex dump. The following command will convert it to a binary (in DER format):

  • xxd -p -r kernelcache.payp.hex kernelcache.payp.bin

Check the contents of kernelcache.payp.bin by running the following command:

openssl asn1parse -inform der -in kernelcache.payp.bin -i

Now copy kernelcache.bin.org to kernelcache.bin. Ghidra doesn't work well with kernel cache files, so I use Hopper Disassembler to patch kernelcache.bin.

  • In Hopper choose Read Executable to Disassemble (CMD-SHIFT-O) and select kernelcache.bin. Then click on Loader, scroll down to com.apple.security.AppleVPBootPolicy and click OK.

  • Select Labels, type "_validate_acm_context" and click on it in the list below. This should take you to the function, whose top few lines should look like this:

                     __validate_acm_context:
fffffe0008be57ec         pacibsp
fffffe0008be57f0         sub        sp, sp, #0x40
fffffe0008be57f4         stp        x20, x19, [sp, #0x20]
fffffe0008be57f8         stp        fp, lr, [sp, #0x30]
fffffe0008be57fc         add        fp, sp, #0x30
fffffe0008be5800         sturb      wzr, [fp, var_11]
fffffe0008be5804         cbz        w0, loc_fffffe0008be5868
  • You want to find the calls to _validate_acm_context() from _command_create_linked_manifest() and _command_update_local_policy_for_kcos(), and patch each one out by replacing it with a nop instruction.

  • Click on _validate_acm_context()'s first line (pacibsp), then press the "x" key. This should open a dialog listing all the calls to _validate_acm_context() from elsewhere in AppleVPBootPolicy. Find _command_create_linked_manifest() and _command_update_local_policy_for_kcos() in this list, and click on each of them in turn. The calls to _validate_acm_context() should look something like this:

fffffe0008bd4e74         add        x22, x21, #0x66
fffffe0008bd4e78         mov        w0, #0x1
fffffe0008bd4e7c         mov        x1, x22
fffffe0008bd4e80         bl         __validate_acm_context
fffffe0008bd4e84         tbnz       w0, 0x0, loc_fffffe0008bd4ea8
  • Select the call to _validate_acm_context() (bl __validate_acm_context) and change to Hexadecimal mode (from ASM mode). For each of the following four hex values, double-click on it and replace it with values from the following list (binary code for a nop instruction). (Hit Enter to update each value after you've typed in its replacement.)
1f 20 03 d5
  • Repeat the previous two steps until you've patched the calls to _validate_acm_context() from both _command_create_linked_manifest() and _command_update_local_policy_for_kcos().

  • Choose File : New Executable and overwrite kernelcache.bin

  • Use vbindiff kernelcache.bin.org kernelcache.bin to check your results. There should be just two changes, each to a four-byte value -- the length of a single AARCH64 instruction.

  • Run img4tool kernelcache.im4p.org, which will produce output something like this. Use the information from it to run the next command. The value of "desc" differs from one version of macOS to another. Note that the current version of img4tool can't yet deal with kernelcache.im4p.org's PAYP structure.

img4tool version: 0.199-ed194718f9d6a035a432f2fdfe9fc639b72cba6c-RELEASE
Compiled with plist: YES
IM4P: ---------
type: krnl
desc: KernelManagement_host-354.140.3
size: 0x01131df6

Compression: bvx2
Uncompressed size: 0x03bd4000
PAYP:
kcep: kcep: [Error] img4tool: failed with exception:
[exception]:
what=assure failed
code=15597586
line=238
file=ASN1DERElement.cpp
commit count=199
commit sha  =ed194718f9d6a035a432f2fdfe9fc639b72cba6c
  • img4tool -c kernelcache.im4p -t krnl -d "KernelManagement_host-354.140.3" kernelcache.bin

  • dd if=kernelcache.payp.bin >> kernelcache.im4p

  • Use Hex Fiend to open kernelcache.im4p and correct its length value. Make sure File : Mode is set to Override.

  • Observe kernelcache.im4p's six-byte header and length value -- for example 308403BD4033. Convert the length (0x03BD4033) to decimal (62734387) and add 189 (for the length of kernelcache.payp.bin in this case). So the new length in this case is 62734576 (== 0x3BD40F0).

  • Correct the length value. The header in this case will now be 308403BD40F0. Save kernelcache.im4p.

  • img4tool -c kernelcache -p kernelcache.im4p -m kernelcache.im4m.org

The patched kernel cache should now be in kernelcache.

Copying the Patched Modules to their Final Destinations

iBoot Stage 0 (AVPBooter.vmapple2.bin)

AVPBooter.vmapple2.bin is a system file. By default it's protected by the macOS file system, and can't be changed. To get around this I borrow from How to Defang macOS System Protections. Before you follow these steps, you must make the settings changes I described above under Settings Changes.

  • mkdir /tmp/mount

  • Run mount at a Terminal prompt and observe its results, for example as follows. Use the contents of the first line in the next command.

/dev/disk5s1s1 on / (apfs, sealed, local, read-only, journaled)
devfs on /dev (devfs, local, nobrowse)
/dev/disk5s6 on /System/Volumes/VM (apfs, local, noexec, journaled, noatime, nobrowse)
/dev/disk5s2 on /System/Volumes/Preboot (apfs, local, journaled, nobrowse)
/dev/disk5s4 on /System/Volumes/Update (apfs, local, journaled, nobrowse)
/dev/disk1s2 on /System/Volumes/xarts (apfs, local, noexec, journaled, noatime, nobrowse)
/dev/disk1s1 on /System/Volumes/iSCPreboot (apfs, local, journaled, nobrowse)
/dev/disk1s3 on /System/Volumes/Hardware (apfs, local, journaled, nobrowse)
/dev/disk5s5 on /System/Volumes/Data (apfs, local, journaled, nobrowse, protect)
/dev/disk2s3 on /Volumes/Boot3 (apfs, sealed, local, read-only, journaled)
/dev/disk7s3 on /Volumes/Boot2 (apfs, sealed, local, read-only, journaled)
/dev/disk4s1 on /Volumes/Boot4 - Data (apfs, local, journaled, nobrowse, protect)
/dev/disk2s1 on /Volumes/Boot3 - Data (apfs, local, journaled, nobrowse, protect)
/dev/disk4s3 on /Volumes/Boot4 (apfs, sealed, local, read-only, journaled)
/dev/disk6s1 on /Volumes/Boot1 - Data (apfs, local, journaled, nobrowse, protect)
/dev/disk6s3 on /Volumes/Boot1 (apfs, sealed, local, read-only, journaled)
/dev/disk7s1 on /Volumes/Boot2 - Data (apfs, local, journaled, nobrowse, protect)
map auto_home on /System/Volumes/Data/home (autofs, automounted, nobrowse)
  • sudo mount -o nobrowse -t apfs /dev/disk5s1 /tmp/mount

  • cd /tmp/mount/System/Library/Frameworks/Virtualization.framework/Resources

  • sudo cp /path/to/patched/AVPBooter.vmapple2.bin .

  • sudo bless --mount /tmp/mount --bootefi --create-snapshot

  • Reboot your host computer.

On reboot, check the contents of /System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin to make sure it's what you expect.

You need to keep your host computer in this state as long as you're using third party kernel extensions in your macOS guest VM. (Without the patched AVPBooter.vmapple2.bin it will simply refuse to start.)

If need be, use the following command to revert your snapshot (and AVPBooter.vmapple2.bin), then reboot your host computer:

sudo bless --mount / --last-sealed-snapshot

iBoot Stage 1 (LLB.img4)

You'll need to copy both LLB.img4 and logo.img4.org over their original contents in AuxiliaryStorage. This is because LLB.img4 might be a different length than LLB.img4.org, and Apple's virtualization infrastructure expects the logo image to immediately follow the LLB image. You can skip this section if you discovered above that your existing LLB image already contains a patch (applied to an earlier version of macOS in your guest VM).

  • Use Hex Fiend to open LLB.img4, logo.img4.org and AuxiliaryStorage. Make sure Edit : Mode for AuxilaryStorage is Overwrite.

  • Jump to offset 0x224000 in AuxiliaryStorage. If an LLB image exists at this location, its first two bytes should be 3083. If not, jump to offset 0x24000.

  • Copy the contents of LLB.img4 (CMD-A, CMD-C) and paste them into AuxilaryStorage.

  • Copy the contents of logo.img4.org and paste them into AuxiliaryStorage, then save the file.

iBoot Stage 2 (iBoot.img4) and the Kernel Cache (kernelcache)

Copy these files to your macOS guest VM and do the following there:

  • Run kmutil inspect, and observe where the boot kernel cache exists in the VM's file system. The following is an example, which will get used in the following steps. The path's exact contents will differ from case to case.
/System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/System/Library/Caches/com.apple.kernelcaches/kernelcache
  • Note the "Next Stage Image4 Hash" just before /System/Library/Caches -- in this case FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83.

  • sudo cp kernelcache /System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/System/Library/Caches/com.apple.kernelcaches/kernelcache

  • Run cd / and sudo find . -name iBoot.img4 -exec ls -al \{\} \;. Observe the hit whose path contains the Next Stage Image4 Hash. Use the results in the following command. For example:

  • sudo cp iBoot.img4 /System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/usr/standalone/firmware/iBoot.img4

Shut down your your macOS guest VM and reboot it into Recovery Mode. Then perform the following steps. You'll be looking for the same Next Stage Image4 Hash as in the previous steps.

Run Terminal in Recovery Mode, then do the following:

  • Run cd / and find . -name iBoot.img4 -exec ls -al \{\} \;. Expect two hits this time, which should look like the following:
/System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/usr/standalone/firmware/iBoot.img4
/System/Volumes/Data/private/tmp/Recovery/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/usr/standalone/firmware/iBoot.img4
  • Copy iBoot.img4 over both of them.

You may also see another hit that looks like the following. Ignore it. You can tell by its date and file size that you already copied the patched iBoot.img4 over it above.

/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/usr/standalone/firmware/iBoot.img4
  • Run cd / and find . -name kernelcache -exec ls -al \{\} \;. Once again expect two hits, which should look like the following (maybe plus one superfluous hit, as above).
/System/Volumes/Preboot/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/System/Library/Caches/com.apple.kernelcaches/kernelcache
/System/Volumes/Data/private/tmp/Recovery/8467D650-E8D9-4F4C-9403-F51E736C25B0/boot/FB72884642D3490C1D6A0C25D6901AD49BFF7A168B2A851361DD40B393D5FE8E730EF7417B83303610CA04EF15C5CD83/System/Library/Caches/com.apple.kernelcaches/kernelcache
  • Copy kernelcache over both of them.

  • Reboot your macOS guest VM and start playing with third-party kernel extensions on it!

@saagarjha
Copy link

Assuming you have a disk image mounted that contains the patched AVPBooter, you can mount it over /System/Library/Frameworks/Virtualization.framework/Resources/ with mount -t [format] -o union /dev/[disk] /System/Library/Frameworks/Virtualization.framework/Resources/ (where [format] is APFS/HFS/whatever, and [disk] is the device). This works as long as the disk is mounted, so it will go away after a reboot.

@steven-michaud
Copy link
Author

So I suppose I could 1) use hdiutil to create a read-only DMG, or an ISO, containing just AVPBooter.bin, 2) mount that in the normal way, and 3) union mount the resulting device over /System/Library/Frameworks/Virtualization.framework/Resources/.

Frankly, I'd be surprised if this works, even with SIP disabled. And even if it does, I'm not sure it's an improvement.

I've been using my own solution intensively over the last few months. I haven't had any trouble with it since I discovered I need to disable "Download new updates when available". But at some point I'll try out your suggestion. And if it works properly, I'll post it in an appendix, as an alternate way to use an altered AVPBooter.bin.

@Naville
Copy link

Naville commented Nov 9, 2023

Thanks for the effort! Quick an intriguing read.

On the other hand, a multi-billion dollar company can't see to support this feature after 2 calendar years...

@steven-michaud
Copy link
Author

You're most welcome.

Apple provides only the most basic support for macOS guest VMs, and then only on Apple Silicon. How different things were when Apple made the switch from PPC to Intel. Then they worked closely with third parties (i.e. VMware and Parallels) to encourage them to provide full-featured support for all kinds of guest OSes. Now, on Apple Silicon, third parties seem to be meeting serious resistance from Apple. And for macOS guests they're frozen out completely -- unless (like Parallels) they're willing to the use the high-level Virtualization framework with its limited, basic support.

https://communities.vmware.com/t5/VMware-Fusion-Discussions/Any-plans-to-support-macOS-guests-on-Apple-Silicon/m-p/2990900

@JakeBrown2000
Copy link

Thank you so much for this article!
Using this method I have been able to get kexts to load within my VMs!

One incredibly minor thing I would add is around the PAYP structure manipulation.

When running openssl asn1parse -inform der -in iBoot.im4p.org -i -dump (against both iBoot and the kernel cache) it gave me the following PAYP structure:

259709:d=1  hl=2 l=   8 cons:  SEQUENCE
259711:d=2  hl=2 l=   1 prim:   INTEGER           :01
259714:d=2  hl=2 l=   3 prim:   INTEGER           :07DDE8
259719:d=1  hl=2 l=  28 cons:  cont [ 0 ] 
259721:d=2  hl=2 l=  26 cons:   SEQUENCE 
259723:d=3  hl=2 l=   4 prim:    IA5STRING         :PAYP
259729:d=3  hl=2 l=  18 cons:    SET  
259731:d=4  hl=7 l=  11 cons:     priv [ 1768907638 ] 
259738:d=5  hl=2 l=   9 cons:      SEQUENCE 
259740:d=6  hl=2 l=   4 prim:       IA5STRING         :iocv
259746:d=6  hl=2 l=   1 prim:       INTEGER           :03

The important information which I would add is that you want to ignore the top 3 lines - I.E.

259709:d=1  hl=2 l=   8 cons:  SEQUENCE          
259711:d=2  hl=2 l=   1 prim:   INTEGER           :01
259714:d=2  hl=2 l=   3 prim:   INTEGER           :07DDE8

and only perform the manipulation on the line starting with cont [0]:

259719:d=1  hl=2 l=  28 cons:  cont [ 0 ] 
259721:d=2  hl=2 l=  26 cons:   SEQUENCE 
259723:d=3  hl=2 l=   4 prim:    IA5STRING         :PAYP
259729:d=3  hl=2 l=  18 cons:    SET  
259731:d=4  hl=7 l=  11 cons:     priv [ 1768907638 ] 
259738:d=5  hl=2 l=   9 cons:      SEQUENCE 
259740:d=6  hl=2 l=   4 prim:       IA5STRING         :iocv
259746:d=6  hl=2 l=   1 prim:       INTEGER           :03

However, this may have just been me being stupid 😄

Thanks again for this! Do you have a buymeacoffee? I'd be happy to support your work as a token of my gratitude

@steven-michaud
Copy link
Author

I'm glad you found my document useful!

I don't know much about DER format, so I don't know whether or not your suggestion is correct. I also don't currently have time to look into this. For now, all I can say is that my document works as it stands. I've tried it many times on guest VMs running macOS 12, 13 and 14.

I'm retired. My work on debugging and reverse engineering is just for fun. Thanks for the offer, though :-)

@4ch12dy
Copy link

4ch12dy commented Jan 21, 2024

Thank you for your hard work, I learned a lot from the article.
Update: Using the methods in this article, I can load kext normally in VM.

@YungRaj
Copy link

YungRaj commented Mar 6, 2024

Has anyone gotten this working on a 13.4.1 host and 14.3.1 guest? iBoot Stage 0 and 1 work with the patches, but the kernelcache and iBoot stage 2 patches cause the machine to go into recovery and its forces me to select the startup disk, in which it reverts all my original changes.

@steven-michaud
Copy link
Author

I haven't (yet) tried it. But out of curiosity, why are you testing on a macOS 13.4.1 host? The current version is 13.6.4.

@YungRaj
Copy link

YungRaj commented Mar 6, 2024

Using 13.4.1 because I haven't tested http://github.com/YungRaj/MacRootKit on recent versions yet, and was hoping to do so without running on bare metal by using the instructions here. All I know is that works on baremetal up until 13.4.1. I don't want to update just yet until I know this tool works.

@YungRaj
Copy link

YungRaj commented Mar 6, 2024

Screenshot 2024-03-06 at 11 18 18 AM

After applying all the patches, installing them to the appropriate locations on both the host and the VM, I get this error. I am able to boot successfully with the iBoot stage 0 AVPBooter.vmapple2.bin patch using the union mount technique that @saagarjha pointed out and iBoot stage 1 patch. However, when applying the patches to the kernelcache and iBoot.img4 in the 3 locations in the VM, it breaks. I did use Hopper to patch _validate_acm_context() in AppleVPBootPolicy to return 0x1 and appropriately patched both validate_boot_object() and image4_validate_property_callback() in iBoot stage 2

How did you debug this btw? Are you using serial output from the guest and I guess logs from com.apple.Virtualization.VirtualMachine on the host ?

@steven-michaud
Copy link
Author

steven-michaud commented Mar 6, 2024

Have you gotten your method to work anywhere (on any versions of macOS, host and guest)? Have you tried not using the union mount technique?

I myself am never going to test a macOS 14.3.1 guest on a macOS 13.4.1 host. But a new set of macOS updates is due out soon. And once they're available, I'll try a macOS 14.4 guest on a macOS 13.6.5 host -- using the technique from the How to Defang macOS System Protections document. I'll report my results here.

I don't really have any more ways to debug my methods than are already documented above. They either work or they don't. And, of course, the steps are complex and easy to get wrong. Did your guest boot after having applied the iBoot stage 0 patch, and again after having made the iBoot stage 1 changes? (Now I see, from your previous comment, that the answer to both questions is yes.)

@YungRaj
Copy link

YungRaj commented Mar 6, 2024

So I tried booting without the union mount method and I got a kernel panic at boot likely because an update was staged to the system to update to 13.5.1, because the kernel panic message was “Userspace version detected mismatch 22F82 != 22G90”. This is probably because the snapshot I created with the patched AVPBooter.vmapple2.bin included some of the updates. So updating to 13.6.5 is probably a good idea to prevent this. But the union mount method works because I put an empty AVPBooter.vmapple2.bin file in the DMG file and the VM failed to trigger a boot at all. So it seems to be that the changes are being reflected.

I’ll probably also try a 13.x guest and/or bite the bullet and update.

From your experience if I got this far into the boot and the VM fails by entering recovery with the kernelcache and iBoot.img4 files patched what usually is the problem?

@steven-michaud
Copy link
Author

From your experience if I got this far into the boot and the VM fails by entering recovery with the kernelcache and iBoot.img4 files patched what usually is the problem?

It's been several months since I last worked my way through my own document. I've been busy with other things, and haven't had the time. Let me try a macOS 14.4 guest on a macOS 13.6.5 host and see what happens. I may need to update my document.

@YungRaj
Copy link

YungRaj commented Mar 7, 2024

I just updated to macOS 13.6.4, reapplied a fresh set of patches to the AVPBooter.vmapple2.bin file and installed it to /System/Library/Frameworks/Virtualization.framework/Resources and created a snapshot for it. I confirmed the changes were present after a reboot. The AVPBooter.vmapple2.bin file was able to boot in the VM (guest = 14.3.1) but was not able to fix the issue of the kernelcache and iBoot.img4 changes causing a failed boot, resulting the recovery screen shown above.

Tonight, I'm gonna try macOS 13.x as a guest

@hexploitable
Copy link

@YungRaj does your kext run in older macOS guests or your only success is on bare metal?

@YungRaj
Copy link

YungRaj commented Mar 13, 2024

Update: I got third party kexts working in a 13.6 guest, including the one I wrote.

Going to try 14.3.1 next by repeating the instructions. Hopefully it still works

@steven-michaud
Copy link
Author

Update: I got third party kexts working in a 13.6 guest

Going to try 14.3.1 next by repeating the instructions. Hopefully it still works

I'm glad to hear it. It'll still be a few days before I can get back to this. When I do, I'll try creating a macOS 14.4 guest on macOS 13.6.5 and 14.4 hosts. With luck I'll be able to figure out what's going wrong, and update my document.

@steven-michaud
Copy link
Author

steven-michaud commented Mar 19, 2024

I've finally been able to work on this, and I've got macOS 14.4 guests working on both a macOS 14.4 host and a macOS 13.6.5 host. The problem is in the "iBoot Stage 2 (iBoot.img4)" section, where you patch the function that calls validate_boot_object() (as I name it). On macOS 14.4 there are two functions that call validate_boot_object(). So you need to patch out both calls (by replacing each one with mov x0,#0x0).

But I'm still not out of the woods. I can't get macOS 13.6.5 working as a guest. I'll revise my document after I've resolved this additional problem ... and any others I find along the way.

Edit: Oops, it's validate_boot_object() that's called from two different locations on macOS 14.4, not image4_validate_property_callback().

@JakeBrown2000
Copy link

I've finally been able to work on this, and I've got macOS 14.4 guests working on both a macOS 14.4 host and a macOS 13.6.5 host. The problem is in the "iBoot Stage 2 (iBoot.img4)" section, where you patch the function that calls image4_validate_property_callback() (as I name it). On macOS 14.4 there are two functions that call image4_validate_property_callback(). So you need to patch out both calls (by replacing each one with mov x0,#0x0).

But I'm still not out of the woods. I can't get macOS 13.6.5 working as a guest. I'll revise my document after I've resolved this additional problem ... and any others I find along the way.

Interesting that there are now 2 functions as of macOS 14.4 which call image4_validate_property_callback, Thanks for the update

@YungRaj
Copy link

YungRaj commented Mar 23, 2024

Unfortunately, I'm still having issues getting macOS 14.4 to work on a 13.6.5 host even after applying the patches to both calls to validate_boot_object() in iBoot stage 2. All the patches work up until I change iBoot stage 2 (iBoot.img4) and the kernelcache on the guest

And I can confirm that 13.6 still works so far

@steven-michaud
Copy link
Author

steven-michaud commented Mar 23, 2024

Do you get an unbootable 14.4 guest if you only copy the altered iBoot.img4 to the destinations my document describes? That is, if you don't copy both the altered iBoot.img4 and kernelcache? This would indicate problems with LLB.img4 (which, as it happens, I'm having with my macOS 13.6.5 guests, but not my macOS 14.4 guests).

If your 14.4 guest becomes unbootable, allow Apple to force you to re-sign it. This reinstalls the default iBoot.img4 and kernelcache, which will be the same as the ones you originally worked from (to create the patched versions). The altered LLB.img4 stays the same (you don't have to recopy it over). But if you delete the unbootable guest, you need to start over again from scratch, with new copies of LLB.img4.org, iBoot.img4.org and kernelcache.org.

I still don't have a working macOS 13.6.5 guest, but I'm getting closer.

Another thing: Are you using UTM? In general, if you're having trouble you should do things exactly as my document prescribes. Except, of course, where I've found it needs to be revised, as for example in the business of the multiple calls to validate_boot_object() in iBoot.img4 (Stage 2 iBoot).

@YungRaj
Copy link

YungRaj commented Mar 23, 2024

Do you get an unbootable 14.4 guest if you only copy the altered iBoot.img4 to the destinations my document describes? That is, if you don't copy both the altered iBoot.img4 and kernelcache? This would indicate problems with LLB.img4 (which, as it happens, I'm having with my macOS 13.6.5 guests, but not my macOS 14.4 guests).

Yes, I get an unbootable 14.4 guest when I only copy the altered iBoot.img4 to the destinations. This might explain my situation being similar to yours, because I installed a 13.0 guest and updated to 14.4. I cannot install 14.4 directly, as the guest fails to install because a software update is requested.

I'm not worried about restoring the VM to its original state because this VM is empty, but I have been forcing Apple to resign it to continue trying

I am not using UTM I am using https://github.com/saagarjha/VirtualApple

@YungRaj
Copy link

YungRaj commented Mar 24, 2024

Got it working. DO NOT update from a lower macOS version to the target macOS version because the LLB.img4 (iBoot Stage 1 bootloader) is not guaranteed to get updated in the AuxiliaryStorage. In my case, a major mismatch of iBoot Stage 1 LLB.img4 (iBoot-8422.141.2) was found against iBoot Stage 2 iBoot.img4 (iBoot-10151.101.3). As explained below by Steven, this is because when macOS updates are staged, 2 different instances of LLB.img4 are managed in AuxiliaryStorage. This means download the ipsw of the Mac version you want to be installed and do not perform any upgrades at any point in the process.

To fix the issue of not being able to install e.g. 14.4 on a 13.6.5 guest, install the most recent Xcode beta that got launched before macOS 14.x released, that way the device support list from the Xcode install will have 14.x added. In my case it was Xcode 15 beta 8. On 13.6.5, the latest release build of Xcode that is supported is Xcode 14.3.1 and this prevented Virtualization.framework from authorizing a macOS 14.4 guest install.

Also, do not use VirtualApple to create the virtual machine and apply all the necessary patches. There is a script here that links your UTM VM to VirtualApple by creating symbolic links to the VM files in the /Users/user/Library/Containers/com.utmapp.UTM folder

gist by akemin-dayo

This is an important step for the likes of those who want to use the GDB stub offered by Virtualization.framework for kernel debugging, which I will do with MacRootKit

@steven-michaud
Copy link
Author

steven-michaud commented Mar 24, 2024

Actually, you can use my document's instructions on a macOS guest whose version has been updated. An update (at least a single one) creates a second LLB.img4 in AuxiliaryStorage. In my case, and probably also yours, that's the active one, and the one you need to patch. Find it, in HexFiend, by searching on "illb" (the four character code for LLB.im4p and LLB.img4). My second copy is at offset 0x224000.

My macOS 14.4 guests were created directly from a 14.4 IPSW file. But there isn't an IPSW file for macOS 13.6.5. So I first created a macOS 13.6 guest from the 13.6 IPSW file and then upgraded it to macOS 13.6.5.

I've still got more work to do before I revise my document. For example, I need to find out what happens when I do multiple macOS upgrades to a macOS guest. That'll probably take a few more days. But, @YungRaj, I think I've now given you a simpler solution than the one you found yourself.

It's odd you can't use VirtualApple. But I still don't have much time to spend on this, so I won't be trying it myself anytime soon. As for the GDB stub, I've written a patch for UTM that enables it (and also adds a serial port object to generate console output when you run my patched UTM from the command line). But using it is a bit clumsy, because you need to set the amfi_get_out_of_my_way=1 boot arg, which disables a lot of important apps -- most/all web browsers and Ghidra, for example. If you run gdb-remotesoon enough in lldb, you can attach to the GDB stub while it's running the iBoot stages. You can't set breakpoints in the first stage (stage 0). But at least on a macOS 14 host, you can set them (with some difficulty) in the second or third stages (stage 1 and stage 2).

I haven't yet published my patch, though I plan to do so in the not too distant future. UTM will never accept the patch, so first I need to figure out how best to make it public. Right now I'm thinking a permanent fork that I keep up to date with the original UTM, with a branch for my patch (and whatever updates it needs to keep up with UTM). I also need to get a better handle on the limitations of the GDB stub, so I can document them. As best I can tell, it's only really useful on a macOS 14 host. It's available back to macOS 12, but there are just too may weirdnesses to deal with on those older hosts.

@steven-michaud
Copy link
Author

steven-michaud commented Mar 24, 2024

I just did a second macOS update on my macOS 13.6.5 guest -- to macOS 14.4. When I loaded its AuxiliaryStorage file into HexFiend, it still had just two copies of LLB.img4, and the second one was still active. But this second one had not been overwritten -- it was still the patched copy I'd inserted into the AuxiliaryStorage file (at offset 0x224000) when that guest was still at macOS 13.6.5. I started from scratch with new iBoot.img4.org and kernelcache.org files. But once I'd patched them (and copied them into position) things worked just fine, without my having made any (new) changes to the LLB.img4 at offset 0x224000.

I'm not sure guest macOS updates will always have this effect. I suspect that the second LLB.img4 will sometimes get overwritten, and the new one will need to be patched. But I also suspect that the maximum number is two, and that the last one will always be the active one.

@steven-michaud
Copy link
Author

I've updated my document to fix the problems that @YungRaj and I found.

@YungRaj
Copy link

YungRaj commented Mar 29, 2024

Thanks Steven, thanks to you I can now confirm I have MacRootKit working on macOS 14.4

@steven-michaud
Copy link
Author

steven-michaud commented Mar 30, 2024

I just discovered something interesting: Apple has fixed the second of the two major problems I discussed above, though only in a macOS 14.4.1 guest running on a macOS 14.4.1 host. With this configuration, validate_boot_object() (in the guest's iBoot module) no longer rejects a valid auxk boot object. _validate_acm_context() (in the AppleVPBootPolicy system kernel extension) still fails incorrectly, though. So you still need my patches to install third-party kernel extensions, even on a macOS 14.4.1 guest running on a macOS 14.4.1 host.

So Apple still hasn't fully resolved the problem of running third-party kexts in a Virtualization framework VM. But there are signs they may be working on it.

The "fix" for validate_boot_object() now calls it three times on an auxk object -- not just once. Each time one of its parameters is changed (out of six), and it's retried if it fails. It succeeds on the third try.

I know macOS 14.0 didn't have this "fix". But I haven't tested with any other versions earlier than 14.4.1. So it's possible the "fix" was introduced in a somewhat earlier version.

I have tested a macOS 14.4.1 guest on a macOS 13.6.6 host -- validate_boot_object() still works incorrectly there. Which is also interesting, because it suggests there are significant differences between the macOS 13 macOS 14 AVPBooter.vmapple2.bin modules.

@steven-michaud
Copy link
Author

steven-michaud commented Apr 10, 2024

I've just revised my document to use a more "surgical" approach to patching the kernelcache. Rather than patching out the _validate_acm_context() call itself, my instructions now have you patch out calls to it from _command_create_linked_manifest() and _command_update_local_policy_for_kcos().

I found these calls by patching _validate_acm_context() to panic when it would normally do an error return. This gave me two different call stacks, which I painstakingly symbolized by hand. Fortunately I only saw panics in the context of the Apple bug(s) this document works around -- which shows that _validate_acm_context() failures are vanishingly rare.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment