Skip to content

Instantly share code, notes, and snippets.

@mohrezaei
Last active January 13, 2024 23:24
Show Gist options
  • Star 12 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save mohrezaei/fe76ab2091e834b6edb2ab9f11e31c07 to your computer and use it in GitHub Desktop.
Save mohrezaei/fe76ab2091e834b6edb2ab9f11e31c07 to your computer and use it in GitHub Desktop.
CalyxOS with Magisk with working signed images and AVB Verity

CalyxOS with Magisk with working signed images and AVB Verity

This idea was inspired by this post topjohnwu/Magisk#509 (comment)

I got this working with CalyxOS 2.11.0 (Android 11) with full AVB Verity enabled and was able to lock the bootloader after flashing and still have su.

Create a working build

First, make sure you can build and sign a proper CalyxOS for your device. This is probably the hardest part.

Prepare Magisk files for rooting

Second, prepare a magisk directory outside your build directory as follows:

mkdir magisk
cd magisk
wget https://cdn.jsdelivr.net/gh/topjohnwu/magisk-files@55bdc45955e7ba1fe4d296b6fc06f926ebc9ddd1/app-debug.apk
unzip app-debug.apk

Replace the apk URL with whatever version is latest or works best for you. The URL for the latest version can be found in the Magisk files repo. https://github.com/topjohnwu/magisk-files

We then need a few helper scripts in the same directory. cat > root-img.sh

#!/bin/bash

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

export PATH=$PATH:$SCRIPT_DIR

export BOOTMODE=true
export KEEPVERITY=true

cp $SCRIPT_DIR/lib/x86/libmagiskboot.so $SCRIPT_DIR/assets/magiskboot
cp $SCRIPT_DIR/lib/arm64-v8a/libmagisk64.so $SCRIPT_DIR/assets/magisk64
cp $SCRIPT_DIR/lib/armeabi-v7a/libmagisk32.so $SCRIPT_DIR/assets/magisk32
cp $SCRIPT_DIR/lib/arm64-v8a/libmagiskinit.so $SCRIPT_DIR/assets/magiskinit

. $SCRIPT_DIR/assets/boot_patch.sh $*

chmod 755 root-img.sh

Make sure magiskinit is correct for your target in root-img.sh.

cat > dos2unix

#!/bin/bash
cat $*

chmod 755 dos2unix

cat > getprop

#!/bin/bash
echo $*

chmod 755 getprop

That's all for preparing magisk.

Prepare signing step

Now we need to intercept avbtool to root the boot.img file just before it's hashed/signed.

In the last step of building the OS, the target files are zipped up and moved into a signing directory, along with the signing keys and binaries. In the bin directory, you should find avbtool which will be used during signing. We're going to replace it with a script that detects boot images, roots them and then continues with the real avbtool.

cd bin
mv avbtool avbtool.real

cat > avbtool

#!/bin/bash

# change this to whereever you created the magisk directory:
MAGISK_DIR=/media/work/magisk

echo "%%%%%%%%%%" `date` Running avbtool with "$*" >> $MAGISK_DIR/avbtool-invokes.txt

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
IMG_NAME=`realpath $3`

if [[ $1 == add_hash_footer ]] && [[ $7 == boot ]] ;
then
        echo starting to root $3 >> $MAGISK_DIR/rooting.txt
        $MAGISK_DIR/root-img.sh $IMG_NAME >> $MAGISK_DIR/rooting.txt
        cp $MAGISK_DIR/assets/new-boot.img $IMG_NAME
fi

$SCRIPT_DIR/avbtool.real $*

chmod 755 avbtool

Now, sign the target files again. If all goes well, that should create a rooted boot.img with the correct signatures. You can check the avbtool-invokes.txt and rooting.txt files to see if everything went well.

@AlexandraKey
Copy link

So basically if phone is 64bit I need to comment out "cp $SCRIPT_DIR/lib/armeabi-v7a/libmagisk32.so $SCRIPT_DIR/assets/magisk32" ?

@arolano
Copy link

arolano commented Jan 3, 2022

Hi! I've mostly succeeded with your method

thank you very much mohrezaei and happy new year :)

HOWEVER
for some reason it does not work with magisk v 23 full stop.
I've just double checked (previously thought factory image worked, but that's just my fat fingers and was using the version of magisk from your gisk)

I wget the most recent magisk (works)
I place all the necessary files in requisite places (works)
The logs (rooting txt and such) imply it works

but it does not. As in, the app does not detect the magisk at all.

@AlexandraKey
Copy link

@arolano I think you are running into same problem I am right now - the "modern" 64 bit file is called libstub.so and I guess something being messed up about it oddly - not working out for me too. Trying to figure it out.

And yeah, I saw your previous edit - OTA don't work for me too, that IS a separate problem (I tried with old magisk version that was in the example) - I suspect the tool that bakes OTA doesn't invoke avbtool at all.

@arolano
Copy link

arolano commented Jan 3, 2022

Yeah, my problem with magisk 23 seems to come from not being able to correctly re name the magiskinit / magisk64 binaries.
"Brain-dead" approach of just renaming libstub into both magisk64 and magiskinit results in bootloop.

Uh, excuse me @mohrezaei any idea what should be renamed what with v23 magisk?

@mohrezaei
Copy link
Author

What is libstub? I just downloaded the latest magisk (23016) from: https://cdn.jsdelivr.net/gh/topjohnwu/magisk-files@b368d20076a30879ccd289b8a4766ba3b88da547/app-debug.apk (which I got from https://github.com/topjohnwu/magisk-files/blob/master/canary.json), and the apk is a zip file that contains a lib folder with the entries as written in the gist above (no libstub).

@AlexandraKey: for a 64 bit phone, use the script exactly as it is above. for a 32 bit phone, change:

cp $SCRIPT_DIR/lib/arm64-v8a/libmagiskinit.so $SCRIPT_DIR/assets/magiskinit

to

cp $SCRIPT_DIR/lib/armeabi-v7a/libmagiskinit.so $SCRIPT_DIR/assets/magiskinit

@mohrezaei
Copy link
Author

To clarify further, the app-debug.apk comes from the magisk section of the json, not the stub part.

@AlexandraKey
Copy link

I think the issue is that me and arolano seem to be using release magisk, not canary, and that one has weird libstub.so where normal arm-64-v8a .so files should be.

In any case, through pure trial and error I have figured out the content of root-img.sh that works for magisk-23 (the latest version from release channel)

#!/bin/bash

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

export PATH=$PATH:$SCRIPT_DIR

export BOOTMODE=true
export KEEPVERITY=true

cp $SCRIPT_DIR/lib/x86/libmagiskboot.so $SCRIPT_DIR/assets/magiskboot
cp $SCRIPT_DIR/lib/armeabi-v7a/libmagisk64.so $SCRIPT_DIR/assets/magisk64
cp $SCRIPT_DIR/lib/armeabi-v7a/libmagisk32.so $SCRIPT_DIR/assets/magisk32
cp $SCRIPT_DIR/lib/armeabi-v7a/libmagiskinit.so $SCRIPT_DIR/assets/magiskinit

. $SCRIPT_DIR/assets/boot_patch.sh $*

Don't know why it is the way it is (maybe it uses same files for v7a and arm64-v8a ? Dunno. Mystery!) but this works.

OTA still doesn't work (applying OTA zip wipes magisk out)

This will get hairy when another magisk update drops, but for now I am entirely satisfied.

@mohrezaei
Copy link
Author

@AlexandraKey thanks for the script. It looks like your hardware is v7a, and that's why it works. Magisk 23 is pretty old and there have been major changes, which is why I'm using canary.

I'll try the OTA update with the next Calyx release, which ought to be Android 12.

@arolano
Copy link

arolano commented Jan 4, 2022

FWIW Alexandra's script works for me and my phone is very certainly arm64-v8a (Pixel 3 A XL aka bonito, Qualcomm SDM670)

The OTA behavior is really weird because I took apart the OTA (using payload dumper + Boot IMG unpacker) and ramdisk content has telltale signs of having been magisk-ed.

I recovery-sideload that very same OTA I just dissected and magisk is no longer seen by the magisk manager app.

I'll try rebuilding everything from scratch on a clean Ubuntu install and see how that goes

@ubergeek77
Copy link

ubergeek77 commented Feb 21, 2022

Does anyone have any notes on this method with Android 12, as well as getting OTA's to work with Magisk included?

I've been trying my own method out before I came across this. I can't seem to get the build to work without erroring via my own approach, so I was considering using the method described in this Gist, which will avoid modifying the build and allow it to proceed without errors. However, the comments here make it look like OTAs do not work at all, which I need.

I'll go ahead and try to look into what's missing from OTAs, but if anyone has tried this on Android 12 already I'd love to hear your experience.

@mohrezaei
Copy link
Author

I did a quick 12 build and the images seemed ok, but I didn't test the OTA. Do not use old Magisk (v23, etc). Make sure you're on the latest. There have been many fixes post v23, many of which are for 12.

@ubergeek77
Copy link

Good to know. I've got a build running now, but it will have to run for about 3 hours. I'll let you know if the OTA wipes out Magisk or not.

@ubergeek77
Copy link

ubergeek77 commented Feb 21, 2022

Alright, I have a signed Android 12 build with Magisk v24.1. Here are my findings:

During signing, there are errors dealing with the Magisk .backup folder:

Create OTA update zip
2022-02-21 21:25:03 - common.py - WARNING : Failed to read SYSTEM/etc/build.prop
2022-02-21 21:25:03 - common.py - WARNING : Failed to read VENDOR/etc/build.prop
2022-02-21 21:25:03 - common.py - WARNING : Failed to read ODM/etc/build.prop
2022-02-21 21:25:03 - common.py - WARNING : Failed to read ODM/build.prop
2022-02-21 21:25:03 - common.py - WARNING : Failed to read ODM_DLKM/etc/build.prop
2022-02-21 21:25:03 - common.py - WARNING : Failed to read ODM_DLKM/build.prop
2022-02-21 21:25:03 - common.py - WARNING : Unable to get boot image build props: Failed to run command '['toybox', 'cpio', '-F', '/tmp/boot_acnaqg.img/uncompressed_ramdisk', '-i']' (exit code 1):
cpio: create .backup/.magisk: Permission denied
cpio: create .backup/.rmlist: Permission denied
cpio: create .backup/init: Permission denied


Creating factory images
2022-02-21 21:28:39 - common.py - WARNING : Failed to read SYSTEM/etc/build.prop
2022-02-21 21:28:39 - common.py - WARNING : Failed to read VENDOR/etc/build.prop
2022-02-21 21:28:39 - common.py - WARNING : Failed to read ODM/etc/build.prop
2022-02-21 21:28:39 - common.py - WARNING : Failed to read ODM/build.prop
2022-02-21 21:28:39 - common.py - WARNING : Failed to read ODM_DLKM/etc/build.prop
2022-02-21 21:28:39 - common.py - WARNING : Failed to read ODM_DLKM/build.prop
2022-02-21 21:28:40 - common.py - WARNING : Unable to get boot image build props: Failed to run command '['toybox', 'cpio', '-F', '/tmp/boot_ixrBRi.img/uncompressed_ramdisk', '-i']' (exit code 1):
cpio: create .backup/.magisk: Permission denied
cpio: create .backup/.rmlist: Permission denied
cpio: create .backup/init: Permission denied

This happens because Magisk currently makes a .backup folder inside the cpio archive that has 000 permissions - not even the owner can read or modify this folder. However, when it's extracted from the cpio, .backup is created with these permissions, so its subfolders and files cannot be created.

This results in common.py being unable to get the build properties. Granted - these errors do not seem to cause any problems. However, it would be ideal to make including Magisk as close to the official process as possible. Thankfully, patching the script causing this error to be aware of the .backup folder's permissions, thus allowing it to do its job, does not appear to be too complicated:

https://android.googlesource.com/platform/build/+/refs/tags/android-12.0.0_r32/tools/releasetools/common.py#3830


Anyway, I tested my own CalyxOS build for a Pixel 6 Pro using this procedure, and I am happy to report:

  • As of CalyxOS 3.2.0 (Android 12), and Magisk v24.1, this procedure successfully includes Magisk into the factory image.
  • The bootloader was successfully re-locked on my Pixel 6 Pro, with Magisk intact.
  • Despite the boot image being patched with Magisk, the device can be safely factory reset without risk of bootlooping or being unable to enter recovery mode (this was an issue a year or so ago, it seems this has now been fixed).
  • Sideloading the OTA via recovery mode does not result in losing Magisk - Magisk is intact on OTA images!
    • Please note: I simply sideloaded the same OTA archive that was built along with this test factory image, I did not yet test a "real" update.
    • This test was done via sideloading, not the built-in OTA updater in CalyxOS that allows you to host your own signed OTA updates. In all likelihood, that updater also works fine with this method of including Magisk, but I have not yet tested it.
    • Likewise, I have not tested an incremental/delta OTA update, so I do not know if there is anything in the delta process that would cause Magisk to be lost. I will need to get this set up for my own daily use at some point in the future, so when I have done so, I will report back with my results.
  • It seems that installing the OTA properly changes the boot slot, and in addition, manually changing to the previous boot slot keeps Magisk installed. Both slots keep Magisk intact after updates.
  • All of my testing was performed with a locked bootloader and an unlocked bootloader. Magisk works perfectly in both cases.

@mohrezaei
Copy link
Author

Ah yes, I remember seeing the errors from common.py. common.py calls toybox cpio .... My plan is to replace the toybox binary with a script that simply ignores the error, as it does not really interfere with anything.

Thanks for the update and confirming that things work. I'm waiting for 12 to stabilize a bit before I update my phone. I'll test the delta OTA and put a new guide at that time.

Copy link

ghost commented Feb 26, 2022

Any help would be very appreciated please.
Trying to enable diag mode in CalyxOS, but can’t enter it without root, I need the adb shell su command, normally I know how it works but in CalyxOS I’m stuck because can’t install magisk.
I tried to understand the comments but it goes way beyond me. Sorry I’m just not that good in codes that’s why I’m asking you the professionals, thx already if anyone can help me out.

@ubergeek77
Copy link

@Florlorez87 Sorry, but this post is specifically about compiling your own CalyxOS ROM from source. This post will not help you if you are just using CalyxOS's official release you downloaded from their website.

Copy link

ghost commented Feb 26, 2022

Ah ok thank you for the reply.

@Nextross
Copy link

Nextross commented Mar 13, 2022

Total noob here. I'm getting Pixel 6 next week and want to flash CalyxOS with root privileges, but also with locked bootloader. I found your guide, but I'm a little lost.

I'm in the process of creating a build environment. Should I firstly try to build and sign without Magisk? To see if everything works correctly?

@mohrezaei
Copy link
Author

@Nextross yes, it's critical to be able to build with your own keys and be able to install and lock the bootloader. The build instructions for 12 might be a missing a couple of prerequisite steps. Last time I built 12, I had to do this:

 git clone https://github.com/anestisb/android-simg2img
 cd android-simg2img/
 make
 sudo make install
 cd ..
 git clone https://github.com/chirayudesai/qc_image_unpacker
 cd qc_image_unpacker/
 ./make.sh
 cp bin/qc_image_unpacker ~/bin

before running device.sh

@Nextross
Copy link

Nextross commented Mar 14, 2022

@mohrezaei when running make, it shows this error:

g++ -std=gnu++17 -O2 -W -Wall -Werror -Wextra -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -Iinclude -Iandroid-base/include -c backed_block.cpp -o backed_block.o g++ -std=gnu++17 -O2 -W -Wall -Werror -Wextra -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -Iinclude -Iandroid-base/include -c output_file.cpp -o output_file.o output_file.cpp:30:10: fatal error: zlib.h: No such file or directory 30 | #include <zlib.h> | ^~~~~~~~ compilation terminated. make: *** [Makefile:97: output_file.o] Error 1

What should I do?

@mohrezaei
Copy link
Author

@Nextross Sorry, this is not the right forum for dealing with build issues. You should join the Calyx dev discord https://app.element.io/#/room/#calyxos-dev:matrix.org and ask there.

@ubergeek77
Copy link

ubergeek77 commented Mar 18, 2022

EDIT: False alarm. My issue had nothing to do with Magisk. I also add Lawnchair in my builds, and as it turns out, Lawnchair is not compatible with Android 12L, it leads to a bootloop. More info:

LawnchairLauncher/lawnchair#2517

Is anyone having issues with this as of CalyxOS 3.3.0, the 12L update?

This method was working perfectly fine for a while. Bootloader locked, OTA updates working perfectly, Magisk intact, etc.

But I just built 3.3.0 with Magisk, and the OTA update won't boot. It tries to boot, but crashes on the Google bootloader splash logo (and when it turns off, the screen has lines on it, as though it experienced an abrupt power cut). It tries again 2 more times, then gives up and reverts back to the last good boot slot. I've tried an incremental update and then a full OTA update, the result is the same.

I'm not sure if Magisk is causing this, but my phone doesn't even get to SystemUI before it crashes. I don't think Android even gets to start, so I can't think of anything else it could possibly be. And my updates were working fine before this, my signing keys are all good, etc.

I'll try and see if building without Magisk fixes this, but for now, is anyone here running CalyxOS 3.3.0 with Magisk working?

@mohrezaei
Copy link
Author

I'm going to do a 12L build in the coming week. I'll report back.

@ubergeek77
Copy link

Thank you!

But, as it turns out, Magisk wasn't the cause of my issue. In addition to adding Magisk, I also replaced my launcher with Lawnchair, which is not compatible with Android 12L. See my edit above.

So, confirmation from me: Magisk 24.3 is working just fine with this method on Android 12L on my test device, a Pixel 3XL. I don't expect there to be any further issues once I make a build for my Pixel 6 Pro.

@mohrezaei
Copy link
Author

Just added a new gist for Android 12, with the addition of the toybox script. My build went well on my test device and the update was pretty smooth.

@mohrezaei
Copy link
Author

@YetAnotherRandomGuy Android 12 instructions are now in a separate gist . I have a locked bootloader, but admittedly, that was done with Android 11 (Calyx 2.x) and subsequently upgraded.

Are you sure you replaced the avbtool that's used during signing? Do you separate your signing dir from your build dir?

@YetAnotherRandomGuy
Copy link

@mohrezaei actually... realized I screwed up completely during the signing - so embarassingly I will delete the post so no one else wonders "WTF did he do...". Thank you for the response, though. I'll walk it through with your updated gist once I fix my glitch.

@akash07k
Copy link

Can anyone please let me know that how can I use this method with any other rom?
I'm building crDroid and can't these instructions to work with it.
Can't find avbtool in my build directory.

@mohrezaei
Copy link
Author

crDroid is based on LineageOS, which is not targeted at locking bootloaders post install. There is no point in following these instructions if you're not going to lock your bootloader. Just install crDroid, then follow Magisk install directions.
If you've gone through the arduous process of enabling locked bootloaders in LineageOS, you have to look at the build process and insert the magisk installations script right after the boot.img is created.

@akash07k
Copy link

akash07k commented Nov 27, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment