Skip to content

Instantly share code, notes, and snippets.

@mohrezaei
Created March 27, 2022 18:02
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mohrezaei/69dae8c7d43c543b38ee5d33f67400b5 to your computer and use it in GitHub Desktop.
Save mohrezaei/69dae8c7d43c543b38ee5d33f67400b5 to your computer and use it in GitHub Desktop.
CalyxOS 3.x (Android 12) with Magisk

CalyxOS 3.x (Android 12) with Magisk with working signed images and AVB Verity

This idea was inspired by this post topjohnwu/Magisk#509 (comment)

I got this working with CalyxOS 3.3.1 (Android 12) with full AVB Verity enabled and was able to lock the bootloader after flashing and still have su. The OTA update zip generated was also tested against the Calyx 2.11.0 version (with Magisk) and it works well.

Create a working build

First, make sure you can build and sign a proper CalyxOS for your device. This is probably the hardest part.

Prepare Magisk files for rooting

Second, prepare a magisk directory outside your build directory as follows:

mkdir magisk24304
cd magisk24304/
wget https://cdn.jsdelivr.net/gh/topjohnwu/magisk-files@a17271415ec0b3b34fbb5715d92893a1f8c529d0/app-debug.apk
unzip app-debug.apk

Replace the apk URL with whatever version is latest or works best for you. For Android 12, v24+ is a must. The URL for the latest version can be found in the Magisk files repo. https://github.com/topjohnwu/magisk-files

We then need a few helper scripts in the same directory. cat > root-img.sh

#!/bin/bash

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

export PATH=$PATH:$SCRIPT_DIR

export BOOTMODE=true
export KEEPVERITY=true

cp $SCRIPT_DIR/lib/x86/libmagiskboot.so $SCRIPT_DIR/assets/magiskboot
cp $SCRIPT_DIR/lib/arm64-v8a/libmagisk64.so $SCRIPT_DIR/assets/magisk64
cp $SCRIPT_DIR/lib/armeabi-v7a/libmagisk32.so $SCRIPT_DIR/assets/magisk32
cp $SCRIPT_DIR/lib/arm64-v8a/libmagiskinit.so $SCRIPT_DIR/assets/magiskinit

. $SCRIPT_DIR/assets/boot_patch.sh $*

chmod 755 root-img.sh

Make sure magiskinit is correct for your target in root-img.sh.

cat > dos2unix

#!/bin/bash
cat $*

chmod 755 dos2unix

cat > getprop

#!/bin/bash
echo $*

chmod 755 getprop

That's all for preparing magisk.

Prepare signing step

Now we need to intercept avbtool to root the boot.img file just before it's hashed/signed.

In the last step of building the OS, the target files are zipped up and moved into a signing directory, along with the signing keys and binaries. In the bin directory, you should find avbtool which will be used during signing. We're going to replace it with a script that detects boot images, roots them and then continues with the real avbtool.

cd bin
mv avbtool avbtool.real

cat > avbtool

#!/bin/bash

# change this to whereever you created the magisk directory:
MAGISK_DIR=/media/work/magisk24304

echo "%%%%%%%%%%" `date` Running avbtool with "$*" >> $MAGISK_DIR/avbtool-invokes.txt

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
IMG_NAME=`realpath $3`

if [[ $1 == add_hash_footer ]] && [[ $7 == boot ]] ;
then
        echo starting to root $3 >> $MAGISK_DIR/rooting.txt
        $MAGISK_DIR/root-img.sh $IMG_NAME >> $MAGISK_DIR/rooting.txt 2>&1
        cp $MAGISK_DIR/assets/new-boot.img $IMG_NAME
fi

$SCRIPT_DIR/avbtool.real $*

chmod 755 avbtool

We'll do something similar for toybox to avoid an error in the build.

mv toybox toybox.real

cat > toybox

#!/bin/bash

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
echo "%%%%%%%%%%" `date` Running toybox with "$*" >> $SCRIPT_DIR/toybox-invokes.txt

if [[ $1 == cpio ]] && [[ $2 == -F ]] ;
then
        echo ignoring toybox error >> $SCRIPT_DIR/toybox-invokes.txt
        $SCRIPT_DIR/toybox.real $* >> $SCRIPT_DIR/toybox-invokes.txt 2>&1
        exit 0
fi

$SCRIPT_DIR/toybox.real $*

chmod 755 toybox

Now, sign the target files again. If all goes well, that should create a rooted boot.img with the correct signatures. You can check the avbtool-invokes.txt and rooting.txt files to see if everything went well. You can apply the factory image (which will wipe the phone), or the OTA update (no wipe) if you have a previous OS with the same keys.

@sunrez
Copy link

sunrez commented Sep 22, 2023

Hi @mohrezaei ,

Thanks for your hard work. Do you have any updated instructions for using your scripts with CalyxOS 4.1x / Android 13?

I am able to successfully build CalyxOS 4.12 / Android 13 but I noticed there are several 'avbtool' and 'toybox' executables in the source tree and I'm not sure which one(s) to replace. I assume the one in prebuilts & possibly external?

Also, if your scripts are used, I assume that doing a fastboot flashall will use the Magisk-patched boot.img because of your scripts?

~/development/CalyxOS_Android13/calyxos/android13$ find . -name avbtool

./prebuilts/kernel-build-tools/linux-x86/bin/avbtool
./external/avb/avbtool
./out/host/linux-x86/bin/avbtool
./out/host/linux-x86/obj/EXECUTABLES/avbtool_intermediates/avbtool
./out/.path/avbtool
./out/soong/.intermediates/external/avb/avbtool
./out/soong/.intermediates/external/avb/avbtool/linux_glibc_x86_64_PY3/avbtool

~/development/CalyxOS_Android13/calyxos/android13$ find . -name toybox

./prebuilts/build-tools/linux_musl-x86/bin/toybox
./prebuilts/build-tools/linux_musl-x86/asan/bin/toybox
./prebuilts/build-tools/linux-x86/bin/toybox
./prebuilts/build-tools/linux-x86/asan/bin/toybox
./prebuilts/build-tools/darwin-x86/bin/toybox
./external/toybox
./out/target/product/coral/symbols/system/bin/toybox
./out/target/product/coral/symbols/recovery/root/system/bin/toybox
./out/target/product/coral/obj/EXECUTABLES/toybox.recovery_intermediates/toybox
./out/target/product/coral/obj/EXECUTABLES/toybox_intermediates/toybox
./out/target/product/coral/system/bin/toybox
./out/target/product/coral/recovery/root/system/bin/toybox
./out/host/linux-x86/bin/toybox
./out/host/linux-x86/obj/EXECUTABLES/toybox_intermediates/toybox
./out/.path/toybox
./out/soong/.intermediates/external/toybox
./out/soong/.intermediates/external/toybox/toybox
./out/soong/.intermediates/external/toybox/toybox/linux_glibc_x86_64/toybox
./out/soong/.intermediates/external/toybox/toybox/linux_glibc_x86_64/obj/external/toybox
./out/soong/.intermediates/external/toybox/toybox/android_recovery_arm64_armv8-a/toybox
./out/soong/.intermediates/external/toybox/toybox/android_recovery_arm64_armv8-a/unstripped/toybox
./out/soong/.intermediates/external/toybox/toybox/android_recovery_arm64_armv8-a/obj/external/toybox
./out/soong/.intermediates/external/toybox/toybox/android_arm64_armv8-a/toybox
./out/soong/.intermediates/external/toybox/toybox/android_arm64_armv8-a/unstripped/toybox
./out/soong/.intermediates/external/toybox/toybox/android_arm64_armv8-a/obj/external/toybox
./out/soong/.intermediates/external/toybox/toybox_vendor/android_vendor.33_arm64_armv8-a/obj/external/toybox

@mohrezaei
Copy link
Author

The avbtool is not in the source directory. Per the above instructions, this happens after the build during the signing step. A set of files are copied from the build directory into another (top level, outside source) directory for signing. That's where you need to use the scripts.

Everything works fine with Calyx 4.x/Android 13, at least with Magisk v25. v26 has some issues and i haven't tried anything after.

@sunrez
Copy link

sunrez commented Sep 22, 2023

Thanks mohrezaei

If you make the changes you suggest, do you have to manually run the scripts to patch the boot image, or will that happen automatically?

I'm a bit unclear if you're talking about the scripts automatically patching an OTA.zip (which I don't find in my 4.1x builds) or the boot.img which I can manually locate in the out folder.

@mohrezaei
Copy link
Author

it happens as part of the signing -- no extra patching required.

@sunrez
Copy link

sunrez commented Sep 22, 2023

Your instrucitons mention re-naming 'avbtool', do you have a suggestion on which one of these two to rename?

./prebuilts/kernel-build-tools/linux-x86/bin/avbtool
./external/avb/avbtool

@mohrezaei
Copy link
Author

looks like you haven't really finished the packaging process. There is only a single avbtool during the signing step in the bin folder. It's NOT in the source or build directories.
If you have not built, signed and deployed a working calyx image (without magisk) to your phone, you're not far along enough to understand this guide. Do that first.

@sunrez
Copy link

sunrez commented Sep 22, 2023

I am following these instructions ( https://calyxos.org/docs/development/build/ ) through the 'Building' step.

If I understood your comments, you're saying go through this step? : https://calyxos.org/docs/development/build/sign/

@mohrezaei
Copy link
Author

Did you not do this bit:
https://calyxos.org/docs/development/build/sign/
where it says:

Copy:

    $OUT/otatools.zip
    $OUT/obj/PACKAGING/target_files_intermediates/*.zip

to a separate folder for signing.

?

That's the signing folder. if you follow those instructions, you'll have a bin directory in that new folder.
I'm confused as how you could've flashed anything you've built without signing.

@sunrez
Copy link

sunrez commented Sep 22, 2023

Thanks, you confirmed what I mentioned that it's the build/sign step needed that I'm missing.

To answer your question, after building, I issued 'fastboot flashall which flashes the boot.img, etc.

It might be helpful for others to update your top-level instructions to call out work through : https://calyxos.org/docs/development/build/sign/ step because that's not very clear.

@sunrez
Copy link

sunrez commented Sep 24, 2023

@mohrezaei

Thanks for your hard work, after getting CalyxOS signing working, I was able to use your scripts and sideload Magisk without issue. I used Magisk 25.2 successfully with Android 13.

As you noted, getting the signing step solved took a few iterations. I had to comment out two lines in the ./vendor/calyx/scripts/release.sh for MicroG and I also removed Tritium (browser) APKs otherwise I ran into odd issues (which might be user error).

@ubergeek77
Copy link

Hey there @mohrezaei , were you able to get Magisk 26 working with this method? I currently have my build stack configured for Magisk 25, everything is fine, but I'm trying to figure out what I need to change for Magisk 26.

Looks like the Android 14 issue you filed was resolved in mid July:

topjohnwu/Magisk#7109

For the breaking changes in Magisk 26 that you reported on their issue tracker, it seems like all that is needed is to set PREINITDEVICE to i.e. raven?

topjohnwu/Magisk#7125

Are there other issues or Linux utilities that you can't get working on your build system? I'd like to help figure this out so I can have Magisk 26 working by the time the Pixel 8 series releases.

If you have notes, error logs, thoughts, etc. about this - anything you have - I'm very interested in helping you solve this :)

@mohrezaei
Copy link
Author

@ubergeek77 thanks for the offer to help. I haven't tried 26 yet. As you mentioned, the next logical thing to try would be 26300+, with PREINITDEVICE set to a random string. If you try that, let us know.

You don't have to wait for pixel 8 release to figure this out. Just build for 6 or 7.

@ubergeek77
Copy link

Gotcha. I won't be able to actually test this on a live device until the Pixel 8 comes out. I really depend on my phone right now, and with less than a month away, it just feels safer to test something like this on a different phone.

However, I can continue working on my build system and try to get this building for i.e. raven. I won't be able to test if it boots or not, but getting a successful build should be a good sign.

I saw in your previous error logs there were exec format errors. If I run into that as well, I'll work on involving qemu somehow.

@ubergeek77
Copy link

Unfortunately I've not yet been able to get Magisk 26.3 working on Android 14.

I did need to change my avbtool wrapper script to account for multiple calls to avbtool with an --image argument, and additionally fail gracefully if the provided image was not a boot image, thus Magisk returns non-0 when trying to patch.

Additionally, I did set the PREINITDEVICE variable to metadata, as that is the correct value for the Pixel 6 Pro and the Pixel 8 Pro.

I made a "successful" build of Android 14 with a patched boot image, it boots, it works, etc, and the hash doesn't match the stock boot image. However the Magisk app doesn't treat it as truly patched, so something is going wrong here.

I'm not sure how to proceed. Any advice from people coming across this gist would be great.

It seems like other projects are not having these issues, as they support Magisk 26+

@ubergeek77
Copy link

@mohrezaei Alright, I was able to get Magisk 26.3 working with my Pixel 8 Pro on Android 14.

However, unrelated to including Magisk into a ROM like this, Magisk 26 is fairly broken right now. TL;DR is do not enable Zygisk or you will have a bad time. More details about that below.

There were two main problems that I originally had issues with when trying to include Magisk 26:

  • The avbtool wrapper script in the OP needs to be changed. The boot partition is no longer guaranteed to be the 7th argument, and the boot image path is no longer guaranteed to be the 3rd argument. Both need to be searched for programmatically.
  • On certain devices, the ramdisk is no longer loaded from the boot partition. Newer devices use init_boot. In my case, I was patching boot.img which is why Magisk wasn't working, since my Pixel 8 Pro uses init_boot.img for the ramdisk.

I haven't tested the below script, as I use a custom build system which isn't quite compatible with OP's script, however I suggest the following changes to the OP's avbtool wrapper script. Hopefully these changes and comments help point people in the right direction. Let me know if I made any mistakes:

#!/bin/bash

# Specify the correct target partition for your device
# Devices launched with Android 13 and up need "init_boot" here
# If you have an older device, use "boot" instead
TARGET_PARTITION=init_boot

# Specify the correct PREINITDEVICE
# This must match the actual preinitdevice your device uses
# Pixel 6 Pro and Pixel 8 Pro use `metadata`
# To find this value, extract the following file from the latest Magisk APK
# lib/<architecture>/libmagisk*.so
# It might be called libmagisk32 or libmagisk64 depending on your device
# adb push this file to /data/local/tmp/, and rename it to `magisk`
# Full path should be `/data/local/tmp/magisk`
# Finally use adb shell to run /data/local/tmp/magisk --preinit-device
export PREINITDEVICE=metadata

# change this to whereever you created the magisk directory:
MAGISK_DIR=/media/work/magisk

echo "%%%%%%%%%%" `date` Running avbtool with "$*" >> $MAGISK_DIR/avbtool-invokes.txt

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

# Determine the partition name
prev=""
for arg in "$@"; do
	if [[ $prev == "--partition_name" ]]; then
		PARTITION_NAME="$arg"
		break
	fi
	prev="$arg"
done

if [[ $1 == add_hash_footer ]] && [[ $PARTITION_NAME == ${TARGET_PARTITION:?} ]] ;
then
     	# Find the boot image path
	prev=""
	for arg in "$@"; do
		if [[ $prev == "--image" ]]; then
			BOOT_IMG_PATH="$arg"
			break
		fi
		prev="$arg"
	done
        echo starting to root $BOOT_IMG_PATH >> $MAGISK_DIR/rooting.txt
        $MAGISK_DIR/root-img.sh $BOOT_IMG_PATH>> $MAGISK_DIR/rooting.txt 2>&1
        cp $MAGISK_DIR/assets/new-boot.img $BOOT_IMG_PATH
fi

$SCRIPT_DIR/avbtool.real $*

As of this writing, Magisk 26.3 is the current "stable" version, but if you enable Zygisk, at least on my device with my setup, the device will bootloop. It actually does "boot," but for me it was stuck at "Pixel is starting" infinitely.

This Zygisk bootloop is fixed in the latest Magisk Canary build (which is 0352ea2c (26302) as of this writing). However, it brings with it a different problem - with Zygisk enabled, system navigation will not work, meaning you will be unable to navigate "back," open recents, go home, etc. This includes gestures, those will be broken too.

All of these Zygisk issues seem to be related to some FD sanitization that has had all sorts of back and forth on the Magisk tracker lately. Here is the most recent one:

topjohnwu/Magisk#7448

So in short, Magisk 26 is fine if you don't plan on using Zygisk, but unless you are on Android 14 (and therefore need Magisk 26+ for the Android 14 support), I would stick with Magisk 25 until they get their act together.

@mohrezaei
Copy link
Author

@ubergeek77 Thanks for all the info. I'll make a Android 14 specific gist when these issues are all ironed out.

@mohrezaei
Copy link
Author

@ubergeek77 Reading through some Magisk issues, I came across this: https://github.com/Dr-TSNG/ZygiskNext
Maybe that helps?

@ubergeek77
Copy link

ubergeek77 commented Oct 30, 2023

Interesting, that could work. I'd prefer it if Magisk just worked though. I would also be concerned about how I would update if this maintainer decides to stop working on ZygiskNext.

I just found a build for this PR that may fix the issue once and for all:

topjohnwu/Magisk#7464

Artifacts at:

https://github.com/topjohnwu/Magisk/actions/runs/6675654022

I'm compiling a signed image with this included as we speak. I'll report back if it works. If it seems stable, then whenever the next version of Magisk is released, presumably 26.4, then that would be the time to write instructions for including this with Android 14.

@ubergeek77
Copy link

Alright. Sadly with this build I still do not have a navigation bar. I guess we will have to wait for a proper fix.

@ubergeek77
Copy link

@mohrezaei If you want to get a head start on testing this new process, a recently built commit of Magisk solves the Zygisk issues I described:

https://github.com/topjohnwu/Magisk/actions/runs/6725604108

More discussion:

topjohnwu/Magisk#7448 (comment)

I would expect the next stable release of Magisk to include this fix, and probably a Canary build much sooner.

But as far as I can see, I've got a ROM built with Magisk, Zygisk enabled, and everything is working.

@mohrezaei
Copy link
Author

@ubergeek77 thanks. I'll wait a bit longer till this stuff is more stable.

@dvdmle
Copy link

dvdmle commented Dec 15, 2023

Has anybody gotten this to work for Android 14? I got it to work with 13, but no luck with 14 just yet.

@mohrezaei
Copy link
Author

I've been waiting for a new release of canary, which includes the changes ubergeek77 mentioned as required. The release yesterday (26403) might work, so I'm going to try it soon (probably during the holidays). Any earlier release is highly unlikely to work, and you'll need ubergeek's improved script above.

@ubergeek77
Copy link

@dvdmle Yes, I have. Have you tried this with the fixes I mentioned were needed for Android 14?

https://gist.github.com/mohrezaei/69dae8c7d43c543b38ee5d33f67400b5?permalink_comment_id=4743126#gistcomment-4743126

Use the latest Canary build, updated just yesterday. It should work:

https://github.com/topjohnwu/magisk-files/blob/canary/app-release.apk

If you did all this, do you have logs or something that might point out what the issue is?

@dvdmle
Copy link

dvdmle commented Dec 15, 2023

Alright, good to know. I should also note that I constantly get this prompt, but any time I do what it tells me to do it bricks my phone.

Screenshot_20231214-200843

I have a Pixel 5a.

@ubergeek77
Copy link

Did you ensure PREINITDEVICE is correct? The correct value is metadata on at least the Pixel 6 Pro and the Pixel 8 Pro, but I have no idea what it's supposed to be for the 5a.

Beyond that, I'm not sure what to suggest, apologies.

@dvdmle
Copy link

dvdmle commented Dec 15, 2023

I knew something was up with those args! As for PREINITDEVICE, it should just be 'boot,' right? Since boot.img has been what's been being modified all along? Wait, I've got it wrong. I have no idea what it's supposed to be.

@ubergeek77
Copy link

@dvdmle Read the comment I put in the script ;)

@DrMaxNix
Copy link

DrMaxNix commented Feb 4, 2024

Just to provide some feedback: I got it working today on a Google Pixel 8 shiba with CalyxOS 5.3.2 and Magisk 26.4 (26400).
I used the newer script version by ubergeek77 which worked flawlessly. I got this popup on the first boot, but after I executed the requested reboot, it never popped up again. Root is working; I did not try Zygisk though.

@DrMaxNix
Copy link

DrMaxNix commented Feb 21, 2024

For anyone using this method: Don't use the updater in the Magisk App. If you have OEM Unlocking disabled, this will brick your phone pretty badly (failing to boot, unable to reflash via fastboot due to locked bootloader, neither recovery, nor fastbootd working anymore). I had to learn the hard way. My advice is to always enable OEM Unlocking while tinkering with the bootloader.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment