Skip to content

Instantly share code, notes, and snippets.

@jimklimov

jimklimov/gist:e6775212f6c9781173e94f5085f32fdb

Last active May 29, 2023 21:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jimklimov/e6775212f6c9781173e94f5085f32fdb to your computer and use it in GitHub Desktop.
Save jimklimov/e6775212f6c9781173e94f5085f32fdb to your computer and use it in GitHub Desktop.
Download ZIP
Cross-building exfat tools for Android on a PC
Raw
gistfile1.md

CONTEXT

TL;DR: See about installing MUSL tool-chain, and scroll down to exfatprogs build - that's what did work best for me.

Needed fsck.exfat 1.3.0 or newer to actively fix SD card issues on an Android phone. All builds I could find were older, so can only detect problems but not fix them even though these versions are supposed to be able to fix "some corruptions" (assuming exfat-fuse 1.3.0 here), e.g.:

:; which fsck.exfat
/system/bin/fsck.exfat

:; time fsck.exfat -y /dev/block/mmcblk0p1
exfatfsck 1.3.0
Checking file system on /dev/block/mmcblk0p1.
File system version           1.0
Sector size                 512 bytes
Cluster size                256 KB
Volume size                 466 GB
Used space                  436 GB
Available space              30 GB
ERROR: cluster 0x1ac67d of file '2_5312016608254762256.tgs' is not allocated.
ERROR: cluster 0x1ac67e of file '-5442761804112578030_120.jpg' is not allocated.
ERROR: cluster 0x1ac680 of file '2_5350751634102166060.tgs' is not allocated.
ERROR: cluster 0x1ac681 of file '2_5350751634102166060.tgs' is not allocated.
Totally 5803 directories and 59256 files.
File system checking finished. ERRORS FOUND: 4, FIXED: 0.
    0m48.15s real     0m05.73s user     0m22.60s system

Since I want this to run on a system vastly different from the one I build on, getting a static build (so all the bytes needed are in the produced binary alone, no shared libraries needed at run-time) was a goal in itself.

Ended up using WSL2/Ubuntu (amd64) as the build environment, since that was handy, and building a lot of stuff until settling on exfatprogs.

Hint about copying with netcat

The phone has SimpleSSHD package (with DropBear SSH/rsync server) as well as a Terminal Emulator package for local console, and Magisk for root access.

While I have MTPuTTY (and its Pageant) for remote-console access from PC to the phone, I did not go through the hassle to set up Pageant (SSH key management) vs. OpenSSH integration on this particular computer to pass files around, so rsync or scp were out of the question.

Instead, I used tar and netcat available on both systems.

  • On phone, wait for a connection:
:; cd /some/target/dir
:; nc -l -p 12345 | tar xzvf -
  • On PC, send the data:
:; cd /some/source/dir
:; tar czvf - files and dirs/ | netcat 10.1.2.3 12345
  • It tends to linger after sending the data, so wait a bit to make sure it is not unpacking anymore (iostat -z 1 on phone side can help) and press Ctrl+C, if needed.
  • Alternately if your sending netcat build supports an option like -N or -q 1 to close connection after end of stdin, use that.

For example:

  • On phone (as root):
:; mkdir -p /data/exfat
:; cd /data/exfat && (nc -l -p 12345 | tar xzvf -)
  • On PC:
:; cd ~/exfat/.inst/usr/local/sbin && tar czvf - ./ | netcat -N 10.1.2.3 12345

The steps

  • First tried "usual" cross-builds (but that got stuck with dynamic linking to Ubuntu *.so files); keeping here in case something of this mattered (e.g. qemu-static run of ARM binaries seems important for FUSE build routine later), all as root:

    • Added QEMU:
    :; apt install qemu binfmt-support qemu-user-static
    • Added DPKG arch support:
    :; dpkg --add-architecture arm64
    • Added APT sources (arm64 is in "ports"; "focal" is what runs on my Linux VM per /etc/os-release):
    # cat /etc/apt/sources.list.d/focal-arm64.list
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal main restricted universe multiverse
deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal main restricted universe multiverse

deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal-updates main restricted universe multiverse
deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal-updates main restricted universe multiverse

deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal-backports main restricted universe multiverse
deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal-backports main restricted universe multiverse

deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal-security main restricted universe multiverse
deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports focal-security main restricted universe multiverse

deb [arch=arm64] http://archive.canonical.com/ubuntu focal partner
deb-src [arch=arm64] http://archive.canonical.com/ubuntu focal partner
    • Added tools and deps I thought I needed (including some that are indeed needed even with MUSL builds per below):
    # Know current packaging index:
:; apt-get update

# General stuff you probably have:
:; apt-get install gcc make meson ninja

# GNU cross-build packages:
:; apt-get install gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu

# Dependencies for exfat:
:; apt-get install libfuse3-3:arm64 libfuse3-dev:arm64

  • Fiddled with https://github.com/relan/exfat - however it failed to link properly and/or find the symbols used by packaged FUSE shared object from glibc in the MUSL toolkit. So following up with below as if it were the original path I took :)

    • Note that I sort of cheated by having system libfuse*-dev packages for headers, etc. but a different binary to link against.
    • Homework for readers to cleanly use the files from libfuse checkout/build area made below.

Install MUSL toolchain

Build static libfuse.a

On the PC (Linux VM prepared above):

:; cd ~
:; git clone https://github.com/libfuse/libfuse
:; cd libfuse
:; ( rm -rf build || true
    mkdir -p build && cd build && \
    CC="$HOME"/aarch64-linux-musl-cross/bin/aarch64-linux-musl-gcc \
    CXX="$HOME"/aarch64-linux-musl-cross/bin/aarch64-linux-musl-g++ \
    LDFLAGS="-static" \
        meson setup --default-library=static .. && \
    ninja && \
    cp -pf lib/libfuse3.a "$HOME"/aarch64-linux-musl-cross/aarch64-linux-musl/lib/
    )
  • The last bit is a cheat to make the library file easily available to exfat build later. Normally it would be installed into some proto area along with the headers (the routine here would use ones provided by OS package), and the configuration of exfat would be told to use that location.

Build static exfat-fuse

On the PC (Linux VM prepared above):

:; cd ~
:; git clone https://github.com/relan/exfat
:; cd exfat
:; autoreconf --install # or `-i` if you are typing
:; ( make distclean || true
    ./configure --build x86_64-pc-linux-gnu --host aarch64-linux-gnu \
        LDFLAGS="-static -Wl,--no-as-needed -L$HOME/aarch64-linux-musl-cross/aarch64-linux-musl/lib -ldl -lssp" \
        CC="$HOME"/aarch64-linux-musl-cross/bin/aarch64-linux-musl-gcc \
    && ( make -j 4 -k || ( echo  ====== ; make ) ) \
    && make DESTDIR="`pwd`/.inst" install )

You should end up with the binaries in ~/exfat/.inst/usr/local/sbin subdirectory.

Building Samsung exfatprogs

For some reason, the exfat-fuse 1.4.0 did not deal with the corruption my card proclaimed. So trying another toolkit - see exfatprogs/exfatprogs#136 for overview of differences.

Build is similar to that above, but with no dependency mess to think of:

:; cd ~
:; git clone https://github.com/exfatprogs/exfatprogs
:; cd exfatprogs
:; ./autogen.sh
:; ( make distclean || true
    ./configure --build x86_64-pc-linux-gnu --host aarch64-linux-gnu \
        --enable-static --disable-shared \
        LDFLAGS="-L$HOME/aarch64-linux-musl-cross/aarch64-linux-musl/lib" \
        CC="$HOME"/aarch64-linux-musl-cross/bin/aarch64-linux-musl-gcc \
    && find . -name Makefile -exec sed -e 's,\(\$(LINK) \),\1 -all-static ,' -i '{}' \; \
    && ( make -j 4 -k || ( echo  ====== ; make ) ) \
    && make DESTDIR="`pwd`/.inst" install )
  • Note the find ... sed trick above - that is the only way I found to wedge it into building really statically (compiling MUSL libc.a into the binaries and NOT having them dynamically linked, interpreter /lib/ld-musl-aarch64.so.1 per file and at the same time not a dynamic executable per ldd), adding an option for libtool wrapper which barges into the build routine and breaks stuff compared to configured request :) YMMV

And now we're talkin'!

On the phone:

15|:/data/exfatprogs-samsung-1.2.1 # ./fsck.exfat -h
Usage: ./fsck.exfat
        -r | --repair        Repair interactively
        -y | --repair-yes    Repair without ask
        -n | --repair-no     No repair
        -p | --repair-auto   Repair automatically
        -a                   Repair automatically
        -b | --ignore-bad-fs Try to recover even if exfat is not found
        -s | --rescue        Assign orphaned clusters to files
        -V | --version       Show version
        -v | --verbose       Print debug
        -h | --help          Show help

16|:/data/exfatprogs-samsung-1.2.1 # time ./fsck.exfat -ys /dev/block/mmcblk0p1
exfatprogs version : 1.2.1
ERROR: /LOST+FOUND: size 0, but the first cluster 0 at 0x2082b40. Fix (y/N)? y
ERROR: /Android/data/org.telegram.messenger/cache/2_5312016608254762256.tgs: cluster 0x1ac67d is marked as free at 0x6b4385860. Truncate (y/N)? y
ERROR: /Android/data/org.telegram.messenger/cache/-5442761804112578030_120.jpg: cluster 0x1ac67e is marked as free at 0x6b4385c00. Truncate (y/N)? y
ERROR: /Android/data/org.telegram.messenger/cache/2_5350751634102166060.tgs: cluster 0x1ac680 is marked as free at 0x6b4385fa0. Truncate (y/N)? y
ERROR: /Android/data/org.telegram.messenger/cache/2_5350751634102166060.tgs: cluster 0x1ac681 is marked as free at 0x6b4385fa0. Truncate (y/N)? y
/dev/block/mmcblk0p1: clean. directories 5805, files 59256
/dev/block/mmcblk0p1: files corrupted 0, files fixed 1
    0m08.37s real     0m00.21s user     0m00.95s system

The earlier fsck.exfat present in the OS took variably 48-57s to check the filesystem, so the new one taking 5-8s is a notable bonus (the improvement being actually able to fix the SD card without picking it out of the phone into a Windows machine).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment