Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Linux Desktop on Apple Silicon/M1 in Practice

Linux Desktop on Apple Silicon/M1 in Practice

I bought M1 MacBook Air. It is the fastest computer I have, and I have been a GNOME/GNU/Linux user for long time. It is obvious conclusion that I need practical Linux desktop environment on Apple Silicon/M1.

Fortunately, Linux already works on Apple Silicon/M1. But how practical is it?

As I needed Linux desktop right now, I decided to hack QEMU. The most difficult challenge is obviously accelerated graphics, but there is Virgil 3D; a birdge to expose host OpenGL to the guest. https://virgil3d.github.io

It unfortunately didn't work on macOS host. So I just made it work. That's it. Here is a video demonstrating OpenGL on Linux on Apple Silicon/M1:

https://www.youtube.com/watch?v=k0bVlVQU2JQ&list=PLesZxBYUPr3wdU3sONUv4Q7UDOg1Dn_Ue&index=4

Modifications

QEMU

hvf

  • The hvf patches are merged into current master. (@agraf)

ui/cocoa

  • Added OpenGL support.
  • Enforced pixel by pixel display.
  • Tells physical/pixel window size to the guest. Now Fedora can properly deal with Retina and respond to window size change.
  • Added cursor composition.
  • Releases mouse grabs properly when the key window changes. (@kov, 2021-05-12)
  • Allow to capture all keys including ones handled by macOS (@kov, enable it by adding full-grab=on to cocoa display specification, 2021-05-12)
  • Added clipboard sharing. (2021-06-17)
  • Improved key mappings (e.g. Japanese IME keys, 2021-06-17)
  • A bug which keeps e.g. command key pressed is fixed. (merged into 6.0)

coreaudio

  • Fixed occasional hangs (2021-07-21)
  • Fixed sampling rate settings (which was introduced by me when I added device change support..., 2021-06-17)
  • Added device change support. (i.e. you can plug/unplug an earphone while QEMU is running.) (merged into 6.0)

hw/block

  • Added punchhole operation. (The disk image consumes physical storage only when it actually has data.)
  • Optimized transfer unit.
  • File locking on macOS is fixed. (2021-07-07, Add file.locking=on to drive to prevent drive breakage in case you concurrently launch the same virtual machine by mistake.)

net

Virgil 3D renderer

Improved OpenGL ES support.

Do It Yourself

Setup

1. Disable System Integrity Protection

https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection

You can enable it again after setup. It must be disabled to build ANGLE.

2. Open a terminal.

3. Install GLib, Meson, Pixman, pkg-config and spice-protocol with Homebrew.

brew install glib meson pixman pkg-config spice-protocol

4. Make a empty directory and change the working directory to it.

5.

curl https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5/raw/2455a9becd6381a302320abff506d762bf7239be/run.sh | bash -

6.

bin/qemu-img create var/virtio.raw 64G

It doesn't consume the physical space until it has data, so you can make the image very large. However, you will see odd behavior if you try to write data more than the physical disk allows.

7.

curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/34/Silverblue/aarch64/iso/Fedora-Silverblue-ostree-aarch64-34-1.2.iso

8.

./run -cdrom Fedora-Silverblue-ostree-aarch64-34-1.2.iso

Proceed the installation process, and now you can run Fedora by executing ./run.

Updating

Just download the latest run.sh and execute it in your workspace directory.

Choosing OpenGL profile

Edit run.

  • gl=off will disable Virgil 3D GPU. Most stable but laggy.
  • gl=core will enable OpenGL.framework. Unstable.
  • gl=es will enable ANGLE. Stable and fast.

Running piglit

Test OpenGL with piglit.

On Fedora:

piglit run all -x spec@ext_timer_query@time-elapsed -x 'spec@arb_timer_query@query gl_timestamp' --timeout 9 results/all

The disabled tests will cause qemu crash due to the following bug: https://bugs.chromium.org/p/angleproject/issues/detail?id=5701

vhost-user-gpu would prevent QEMU process from crashing by isolating graphics acceleration process, but it needs modifications to run outside Linux because:

  • historically, vhost-user is a reimplementation of Linux kernel's vhost interface, and it relies on kernel headers for interface definitions and
  • vhost-user uses eventfd which is only available on Linux.

It shouldn't be difficult, but I'm satisfied even without process isolation so I don't.

Upstreaming

Upstreaming is in progress. Hopefully the features I implemented will work just by running brew install qemu in the future.

Future Perspective

As I described here, such a virtualization software is practical and efficient approach to run Linux desktop. The performance overhead is also acceptable for daily use, and it even provides better integration of Linux and macOS. For example, you can switch macOS and Linux with three-finger gesture on trackpad. You can use VirtFS.

However, there are complexities that such a virtualization adds. It basically means sharing one hardware with two systems, so you have to allocate them properly or it ends up with bad user experience. The allocation problem happens everywhere (HID like keyboard, computing resource like CPU, power management, etc.). This approach is efficient but not the best.

In long term, native Linux port is the best option. However, it is not practical if it takes too long before it becomes available. Therefore, we should consider hybrid approaches. marcan42, the founder of Asahi Linux, has an idea to run macOS on KVM on Linux to support complex devices, for example: https://twitter.com/marcan42/status/1361999648819269636

Another approach is to get macOS device support softwares work on Linux. In the past, there was NDISWrapper, a project to bring some Windows drivers to Linux. There was also Cider, a research project to "duct-tape" XNU's IOKit to Linux to bring macOS drivers to Android. https://www.cs.columbia.edu/~nieh/pubs/asplos2014_cider.pdf

By taking those approaches, it is possible to reduce dependencies on macOS gradually, not at once.

UPDATE 2021-03-16

@knazarov wrote Homebrew formulae.

https://github.com/knazarov/homebrew-qemu-virgl

set -eux
mkdir -p depot_tools build/qemu source/angle source/libepoxy source/virglrenderer source/qemu
git -C depot_tools init
git -C depot_tools fetch https://chromium.googlesource.com/chromium/tools/depot_tools.git 1cabb17575917b73ec2e270d4187656c20b1ab0c
git -C depot_tools checkout 1cabb17575917b73ec2e270d4187656c20b1ab0c
git -C source/angle init
git -C source/angle fetch https://chromium.googlesource.com/angle/angle d15be77864e18f407c317be6f6bc06ee2b7d070a
git -C source/angle checkout d15be77864e18f407c317be6f6bc06ee2b7d070a
git -C source/libepoxy init
git -C source/libepoxy fetch https://github.com/akihikodaki/libepoxy.git macos
git -C source/libepoxy checkout FETCH_HEAD
git -C source/virglrenderer init
git -C source/virglrenderer fetch https://github.com/akihikodaki/virglrenderer.git macos
git -C source/virglrenderer checkout FETCH_HEAD
git -C source/qemu init
git -C source/qemu fetch https://github.com/akihikodaki/qemu.git macos
git -C source/qemu checkout FETCH_HEAD
export PATH="$PWD/depot_tools:$PATH"
cd source/angle
DEPOT_TOOLS_UPDATE=0 python2 scripts/bootstrap.py
DEPOT_TOOLS_UPDATE=0 gclient sync
gn gen '--args=target_cpu="arm64"' ../../build/angle
cd ../..
ninja -C build/angle
meson "-Dc_args=-I$PWD/source/angle/include" "-Dc_link_args=-L$PWD/build/angle" "--prefix=$PWD" -Degl=yes -Dx11=false build/libepoxy source/libepoxy
meson install -C build/libepoxy
meson "--pkg-config-path=$PWD/lib/pkgconfig" "--prefix=$PWD" build/virglrenderer source/virglrenderer
meson install -C build/virglrenderer
cd build/qemu
PKG_CONFIG_PATH="$PWD/../../lib/pkgconfig" ../../source/qemu/configure "--extra-cflags=-I$PWD/../../source/angle/include -I$PWD/../../include" "--extra-ldflags=-L$PWD/../angle" "--prefix=$PWD/../.."
meson install
cp -i pc-bios/edk2-aarch64-code.fd pc-bios/edk2-arm-vars.fd ../../var
cd ../..
cat > run <<'EOF'
#!/bin/bash
d="$(dirname "{BASH_SOURCE[0]}")"
exec sudo DYLD_FALLBACK_LIBRARY_PATH="$d/build/angle:$d/lib" "$d/bin/qemu-system-aarch64" -machine virt,accel=hvf,highmem=off -cpu cortex-a72 -smp 8 -m 4096 -device ich9-intel-hda -device hda-output,audiodev=audio -device virtio-gpu-gl-pci -device virtio-keyboard-pci -device virtio-net-device,netdev=net -device virtio-rng-device -display cocoa,gl=es -drive "if=pflash,format=raw,file=$d/var/edk2-aarch64-code.fd,readonly=on" -drive "if=pflash,format=raw,file=$d/var/edk2-arm-vars.fd" -drive "if=virtio,format=raw,file=$d/var/virtio.raw,backend_defaults=on,discard=on" -audiodev coreaudio,id=audio,out.fixed-settings=false -netdev vmnet-macos,id=net,mode=shared -chardev qemu-vdagent,id=spice,name=vdagent,clipboard=on -device virtio-serial-pci -device virtserialport,chardev=spice,name=com.redhat.spice.0 -full-screen -runas "$(id -u):$(id -g)" "$@"
EOF
chmod a+x run
@z0mbix

This comment has been minimized.

Copy link

@z0mbix z0mbix commented Mar 3, 2021

Did you try UTM? https://mac.getutm.app/

@jbrooksuk

This comment has been minimized.

Copy link

@jbrooksuk jbrooksuk commented Mar 3, 2021

@z0mbix, UTM is still a wrapper for QEMU, so doesn't support GPU emulation either.

@bugpowder

This comment has been minimized.

Copy link

@bugpowder bugpowder commented Mar 3, 2021

Did you try UTM? https://mac.getutm.app/

He tried the underlying tech, UTM is just a wrapper. What's more, he improved on it (QEMU) to support OpenGL.

@srikantpatnaik

This comment has been minimized.

Copy link

@srikantpatnaik srikantpatnaik commented Mar 3, 2021

Great work!

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 3, 2021

Did you try UTM? https://mac.getutm.app/

I didn't know it has macOS support and I tried it just now. It indeed has a fancy UI, but it has problems critical for my use so I'll instead stick to bare QEMU.

  • It didn't correctly report display resolution when displaying pixel by pixel. Actually I managed to fix this. utmapp/UTM#2361
  • It doesn't tell physical display size to the guest so HiDPI support on the guest doesn't work. I tried but it requires to update spice-gtk, and I'm not motivated enough to do that.
  • Its graphics is laggy. I think one of the causes is lack of hardware acceleration and my patch will improve it, but another possible cause is the IPC between the window on macOS and QEMU. I can improve the IPC by implementing IOSurface passing via Mach port, but it requires platform-dependent extensions for SPICE protocol. It should be difficult to convince the upstream to accept such a change. Beside that, there is libvirt and virt-manager, which are well-proven, if using SPICE protocol.
@relm

This comment has been minimized.

Copy link

@relm relm commented Mar 3, 2021

I can't seem to build this from the script provided, any ideas? Looks like it fails to download a file when running the "download_from_google_storage" script and then the rest fails.

________ running 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-clang-format -s buildtools/mac/clang-format.sha1' in '/Users/mike/vm/ubuntu/source/angle/.'
0> Failed to fetch file gs://chromium-clang-format/62bde1baa7196ad9df969fc1f06b66360b1a927b for buildtools/mac/clang-format, skipping. [Err: [E2021-03-03T15:31:30.061006-07:00 35237 0 venv.go:975] Command (cwd=/tmp/vpython_bootstrap287609297/packages/virtualenv-15.1.0): [/usr/bin/python2.7 -B -E -s virtualenv.py --no-download /Users/mike/.vpython-root/f029f6]
Process output:
New python executable in /Users/mike/.vpython-root/f029f6/bin/python
ERROR: The executable /Users/mike/.vpython-root/f029f6/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/private/tmp/vpython_bootstrap287609297/packages/virtualenv-15.1.0' (should be u'/Users/mike/.vpython-root/f029f6')
ERROR: virtualenv is not compatible with this system or executable
Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
COLORFGBG=15;0
COLORTERM=truecolor
COMMAND_MODE=unix2003
HOME=/tmp/vpython_bootstrap287609297
HOMEBREW_CELLAR=/opt/homebrew/Cellar
HOMEBREW_PREFIX=/opt/homebrew
HOMEBREW_REPOSITORY=/opt/homebrew
INFOPATH=/opt/homebrew/share/info:
ITERM_PROFILE=Default
ITERM_SESSION_ID=w0t4p0:E57D0A70-66CE-4049-A066-5D15E515DD8E
LANG=en_US.UTF-8
LC_TERMINAL=iTerm2
LC_TERMINAL_VERSION=3.4.4
LOGNAME=mike
LaunchInstanceID=2F7CBF4D-D8AA-459F-A7B3-C94D675C96E2
MANPATH=/Users/mike/.nvm/versions/node/v15.4.0/share/man:/opt/homebrew/share/man::
NVM_BIN=/Users/mike/.nvm/versions/node/v15.4.0/bin
NVM_CD_FLAGS=-q
NVM_DIR=/Users/mike/.nvm
NVM_INC=/Users/mike/.nvm/versions/node/v15.4.0/include/node
PATH=/Users/mike/vm/ubuntu/depot_tools:/Users/mike/.yarn/bin:/Users/mike/.config/yarn/global/node_modules/.bin:/Users/mike/.nvm/versions/node/v15.4.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/mike/vm/ubuntu/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/Users/mike/vm/ubuntu/source/angle
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SECURITYSESSIONID=186b8
SHELL=/bin/zsh
SHLVL=2
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.RWHWSUQBUT/Listeners
TEMP=/tmp
TERM=xterm-256color
TERM_PROGRAM=iTerm.app
TERM_PROGRAM_VERSION=3.4.4
TERM_SESSION_ID=w0t4p0:E57D0A70-66CE-4049-A066-5D15E515DD8E
TMP=/tmp
TMPDIR=/tmp
USER=mike
VERSIONER_PYTHON_VERSION=2.7
VIRTUAL_ENV=/Users/mike/.vpython-root/e36bc6
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.googlecode.iterm2
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0

[E2021-03-03T15:31:30.064674-07:00 35237 0 annotate.go:266] original error: exit status 100

goroutine 1:
#0 go.chromium.org/luci/vpython/venv/venv.go:615 - venv.(*Env).installVirtualEnv()
  reason: failed to create VirtualEnv

#1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
  reason: failed to install VirtualEnv

#2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
#3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
#4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
#5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
  reason: failed to create new VirtualEnv

#6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
#7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
#8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
  reason: failed to create empty probe environment

#9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
#10 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
#11 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
#12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
#13 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
#14 vpython/main.go:110 - main.mainImpl()
#15 vpython/main.go:116 - main.main()
#16 runtime/proc.go:204 - runtime.main()
#17 runtime/asm_amd64.s:1374 - runtime.goexit()
]
Downloading 1 files took 0.447183 second(s)
Error: Command 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-clang-format -s buildtools/mac/clang-format.sha1' returned non-zero exit status 1 in /Users/mike/vm/ubuntu/source/angle/.```
@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 4, 2021

@relm It is incompatible with System Integrity Protection and you need to turn it off. Boot into recovery, open a terminal, and run csrutil disable.
Thank you for noting that. I'll update README.md later.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 4, 2021

It would be really great to have this running on a x86_64 mac. I've tried to build your branch of qemu, and after a few minor tweaks to source code I managed to get it to build. Unfortunately, it doesn't run. When I execute it, I get a "floating point exception".

I guess that's because something is hardcoded in your branch that is arm-specific. But my knowledge of qemu codebase is nil, so I couldn't fix it myself.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 4, 2021

@knazarov I haven't hard-coded anything specific for AArch64. Perhaps I made some silly mistake.

Unfortunately I lost access to Intel Mac (I just quitted my job) so I cannot reproduce the problem. If you want to fix it, please share more details. I think the crash reporter gives you tons of logs.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 4, 2021

I'll try to bisect your patchset to see which patch may contain the problem. The first patch from your tree runs fine on my machine (b7885e4370a2fe426e80d32afe6eb5d01a71640d), so I should be able to pin down a suspect.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 4, 2021

The commit where everything breaks on x86 MacOS is akihikodaki/qemu@5713354. This is the commit where you add "punch hole" operations. I'll try to dig deeper.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

The commit where everything breaks on x86 MacOS is akihikodaki/qemu@5713354. This is the commit where you add "punch hole" operations. I'll try to dig deeper.

That is surprising. I thought it may be related with display resolution handling as it does floating point calculation, but the reality is obviously different from my guess. Thank you for sharing info.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

The commit where everything breaks on x86 MacOS is akihikodaki/qemu@5713354. This is the commit where you add "punch hole" operations. I'll try to dig deeper.

That is surprising. I thought it may be related with display resolution handling as it does floating point calculation, but the reality is obviously different from my guess. Thank you for sharing info.

I've figured out the problem. This happens when raw_probe_blocksizes in raw_format.c calls back to raw_probe_blocksizes in file-posix.c.

The problem is that raw_probe_blocksizes from file-posix.c always returns phys and log equal to zero. And the raw_probe_blocksizes in raw-format.c tries to calculate the alignment like this:

static int raw_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
{
    BDRVRawState *s = bs->opaque;
    int ret;

    ret = bdrv_probe_blocksizes(bs->file->bs, bsz);
    if (ret < 0) {
        return ret;
    }

    if (!QEMU_IS_ALIGNED(s->offset, MAX(bsz->log, bsz->phys))) {
        return -ENOTSUP;
    }

    return 0;
}

On the QEMU_IS_ALIGNED it fails with a floating point exception.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

After reverting the "punchhole" patch and trying to run with -display cocoa,gl=es, I get a segfault when qemu tries to load libEGL.dylib. When running under lldb, I get the following stacktrace:

* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00000002649fbe26 libc++.dylib`std::__1::codecvt<char, char, __mbstate_t>::encoding(this=0x00007fff88c0d4c0) const at __locale:951:16
    frame #2: 0x00000002649ffd05 libc++.dylib`std::__1::__stdinbuf<char>::imbue(this=0x0000000264abe4c8, __loc=0x000070000799d6e0) at __std_stream:84:26
    frame #3: 0x00000002649ffc11 libc++.dylib`std::__1::__stdinbuf<char>::__stdinbuf(this=0x0000000264abe4c8, __fp=0x00007fff88c3ee50, __st=0x0000000264abe530) at __std_stream:76:5
    frame #4: 0x00000002649ff885 libc++.dylib`std::__1::__stdinbuf<char>::__stdinbuf(this=0x0000000264abe4c8, __fp=0x00007fff88c3ee50, __st=0x0000000264abe530) at __std_stream:75:1
    frame #5: 0x00000002649ff5ec libc++.dylib`std::__1::DoIOSInit::DoIOSInit(this=0x0000000264abea18) at iostream.cpp:111:59
    frame #6: 0x00000002649ff9e5 libc++.dylib`std::__1::DoIOSInit::DoIOSInit(this=0x0000000264abea18) at iostream.cpp:107:1
    frame #7: 0x00000002649ffafa libc++.dylib`std::__1::ios_base::Init::Init(this=0x0000000264ac0928) at iostream.cpp:152:22
    frame #8: 0x00000002649ff585 libc++.dylib`std::__1::ios_base::Init::Init(this=0x0000000264ac0928) at iostream.cpp:151:1
    frame #9: 0x0000000264a01be0 libc++.dylib`::__cxx_global_var_init() at iostream.cpp:80:31
    frame #10: 0x0000000264a01c09 libc++.dylib`_GLOBAL__I_000101 at iostream.cpp:0
    frame #11: 0x0000000101249079 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 559
    frame #12: 0x0000000101249478 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
    frame #13: 0x0000000101243d1a dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 492
    frame #14: 0x0000000101243c85 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #15: 0x0000000101243c85 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #16: 0x0000000101241b82 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
    frame #17: 0x0000000101241c22 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
    frame #18: 0x00000001012326a2 dyld`dyld::runInitializers(ImageLoader*) + 82
    frame #19: 0x000000010123d569 dyld`dlopen_internal + 609
    frame #20: 0x00007fff20620fc0 libdyld.dylib`dlopen_internal(char const*, int, void*) + 177
    frame #21: 0x00007fff2060f86e libdyld.dylib`dlopen + 28
    frame #22: 0x000000010141d9c7 libepoxy.0.dylib`get_dlopen_handle(handle=0x0000000101448b90, lib_name="libEGL.dylib", exit_on_fail=<unavailable>, load=<unavailable>) at dispatch_common.c:313:19 [opt]
    frame #23: 0x000000010141da86 libepoxy.0.dylib`epoxy_egl_dlsym [inlined] epoxy_load_egl(exit_if_fails=true, load=true) at dispatch_common.c:636:12 [opt]
    frame #24: 0x000000010141da69 libepoxy.0.dylib`epoxy_egl_dlsym [inlined] epoxy_conservative_egl_dlsym(name="eglGetDisplay", exit_if_fails=true) at dispatch_common.c:646 [opt]
    frame #25: 0x000000010141da69 libepoxy.0.dylib`epoxy_egl_dlsym(name="eglGetDisplay") at dispatch_common.c:655 [opt]
    frame #26: 0x000000010141d1e9 libepoxy.0.dylib`egl_provider_resolver(name=<unavailable>, providers=<unavailable>, entrypoints=<unavailable>) at egl_generated_dispatch.c:0:17 [opt] [artificial]
    frame #27: 0x000000010141d10c libepoxy.0.dylib`egl_single_resolver(provider=<unavailable>, entrypoint_offset=1375) at egl_generated_dispatch.c:3924:12 [opt]
    frame #28: 0x000000010141b2ba libepoxy.0.dylib`epoxy_eglGetDisplay_global_rewrite_ptr [inlined] epoxy_eglGetDisplay_resolver at egl_generated_dispatch.c:4397:12 [opt]
    frame #29: 0x000000010141b2ab libepoxy.0.dylib`epoxy_eglGetDisplay_global_rewrite_ptr(display_id=0) at egl_generated_dispatch.c:5077 [opt]
    frame #30: 0x00000001002008a4 qemu-system-x86_64`qemu_egl_init_dpy_cocoa(mode=DISPLAYGL_MODE_ES) at egl-helpers.c:373:22 [opt]
    frame #31: 0x000000010001d727 qemu-system-x86_64`cocoa_display_init(ds=<unavailable>, opts=0x0000000100d11298) at main.m:707:13 [opt]
    frame #32: 0x00000001002b2215 qemu-system-x86_64`qemu_init [inlined] qemu_init_displays at vl.c:2433:5 [opt]
    frame #33: 0x00000001002b2201 qemu-system-x86_64`qemu_init(argc=<unavailable>, argv=0x000000010261f2b0, envp=<unavailable>) at vl.c:3553 [opt]
    frame #34: 0x0000000100008b99 qemu-system-x86_64`qemu_main(argc=16, argv=0x00007ffeefbff610, envp=<unavailable>) at main.c:49:5 [opt]
    frame #35: 0x000000010001d506 qemu-system-x86_64`call_qemu_main(opaque=0x0000000000000000) at main.m:115:14 [opt]
    frame #36: 0x0000000100545c9e qemu-system-x86_64`qemu_thread_start(args=<unavailable>) at qemu-thread-posix.c:521:9 [opt]
    frame #37: 0x00007fff20604950 libsystem_pthread.dylib`_pthread_start + 224
    frame #38: 0x00007fff2060047b libsystem_pthread.dylib`thread_start + 15

From the stacktrace, it looks like the problem is in the standard initialization code somewhere. It's either a memory corruption or some problem with dynamic linking.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

I think what happens in this case is two different libc++ libraries conflict. One from the system, and another from angle (libEGL.dylib). I've inspected the library dependencies in runtime with lldb, and sure enough, there are 2 different versions of libc++ loaded.

It's possible that we have to link the angle library statically with libc++, but I'm not sure yet how to do it. In your particular case on ARM you likely got lucky and the problem didn't happen.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

@knazarov I fixed the bug in punch hole patch and submitted to the upstream ccing you. Thank you for detailed analysis.

Regarding libc++, I gathered some basic information from my setup:

Opened Files from Activity Monitor
cwd
/Users/person/build/qemu
txt
/Users/person/bin/qemu-system-aarch64
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libgmodule-2.0.0.dylib
txt
/opt/homebrew/Cellar/pixman/0.40.0/lib/libpixman-1.0.40.0.dylib
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libgobject-2.0.0.dylib
txt
/opt/homebrew/Cellar/gettext/0.21/lib/libintl.8.dylib
txt
/opt/homebrew/Cellar/libffi/3.3_2/lib/libffi.7.dylib
txt
/usr/lib/dyld
txt
/Users/person/lib/libepoxy.0.dylib
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libgio-2.0.0.dylib
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libglib-2.0.0.dylib
txt
/Users/person/lib/libvirglrenderer.1.dylib
txt
/Users/person/build/angle/libc++.dylib
txt
/opt/homebrew/Cellar/pcre/8.44/lib/libpcre.1.dylib
txt
/Library/Preferences/Logging/.plist-cache.IkJslF0t
txt
/private/var/db/analyticsd/events.whitelist
txt
/private/var/db/timezone/tz/2021a.1.0/icutz/icutz44l.dat
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/SystemAppearance.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantDark.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/DarkAqua.car
txt
/usr/lib/libobjc-trampolines.dylib
txt
/usr/share/icu/icudt66l.dat
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantDark.car
txt
/System/Library/Fonts/Helvetica.ttc
txt
/System/Library/Caches/com.apple.IntlDataCache.le.kbdx
txt
/Users/person/build/angle/libEGL.dylib
txt
/System/Library/Fonts/SFNS.ttf
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Aqua.car
txt
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/0/com.apple.LaunchServices.dv/com.apple.LaunchServices-2543-v2.csstore
txt
/System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle/Contents/Resources/AppleKeyboardLayouts-L.dat
txt
/Users/person/build/angle/libabsl.dylib
txt
/Users/person/build/angle/libchrome_zlib.dylib
txt
/System/Library/Extensions/AppleMetalOpenGLRenderer.bundle/Contents/MacOS/AppleMetalOpenGLRenderer
txt
/System/Library/Extensions/AGXMetal13_3.bundle/Contents/MacOS/AGXMetal13_3
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantLightGraphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Graphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightGraphiteAX.car
txt
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Extras2.rsrc
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantLight.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLight.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightGraphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/AquaAX.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/GraphiteAX.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightAX.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/DarkGraphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Assets.car
txt
/Library/Application Support/CrashReporter/SubmitDiagInfo.domains
txt
/System/Library/CoreServices/CoreGlyphs.bundle/Contents/Resources/Assets.car
txt
/Users/person/build/angle/libGLESv2.dylib
txt
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/16777235_322/functions.data
txt
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/31001/libraries.data
0
/dev/ttys000
1
/dev/ttys000
2
/dev/ttys000
3
->0x63bf3348027b50b
4
->0x9fb1da11cecc57ed
5
/dev/urandom
6
->0xe66580f2b9a0a149
7
->0x795a4dbf6ec99805
8
->0x23e3bee1b8a0b5bf
9
->0x204fe5858f3bb19b
10
->0x33be3e592fffb539
11
->0x4023151e099f0117
12
/Users/person/ubuntu/drive/AAVMF_CODE.fd
13
/Users/person/ubuntu/drive/virtio.raw
14
/System/Library/Extensions/AppleMetalOpenGLRenderer.bundle/Contents/Resources/default.metallib
15
/Users/person/ubuntu/drive/AAVMF_VARS.fd
16
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/16777235_322/functions.data
17
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/16777235_322/functions.list
18
/Library/Application Support/CrashReporter/SubmitDiagInfo.domains
19
->0x5c5de80413a8a7b5
20
->0x70abb0fd8d1acfb2
21
->0x33b1b4d1d48ccea8
22
->0x59cf4f9476d7d47a
23
->0xe0b80e7690083e67
24
->0x3bec23fce34d43eb
25
->0x35e611764070523c
26
->0x9227c35e8692bf8b
27
->0x4626c7be7f59336a
28
->0xf46bddb09841bf7b
29
->0x749b0129635788db
30
->0xf245372d7e8d5031
31
->0x5950f3184c2d8437
32
->0x7d922d1411ec180b
33
->0xb6cba3c6a656ef37
34
->0xf07ae2ea1dec33a5
35
->0xfb19885eb8fade8f
36
->0xaf9a16b9e3dafbb7
37
->0x702c2fd81b7de4a8
38
->0xde81e46b4705288b
39
->0xd789a28517f45c86
40
->0xe4ea2562453f808
41
->0xf7869e00d8751601
42
->0x7f41e669195f98b5
43
->0x5e29686fe964486e
44
->0x1a99bfbb434e04ca
45
->0x581b777ce9c76274
46
->0x6e74bfc40fa93a35
47
->0x26a8257630828525
48
->0x4c5a8632d5d4c289
49
->0xe016e1aaff288267
50
->0x44816a22c218c958
53
*:53863
57
*:64406
59
*:62887
60
*:53738
67
*:51193
68
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/31001/libraries.data
69
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/31001/libraries.list
73
*:64041
82
*:51092
91
*:57121
otool -L qemu-system-aarch64
/Users/person/bin/qemu-system-aarch64:
	/System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/opt/homebrew/opt/pixman/lib/libpixman-1.0.dylib (compatibility version 41.0.0, current version 41.0.0)
	/Users/person/lib/libepoxy.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libsasl2.2.dylib (compatibility version 3.0.0, current version 3.15.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/opt/homebrew/opt/glib/lib/libgio-2.0.0.dylib (compatibility version 6601.0.0, current version 6601.7.0)
	/opt/homebrew/opt/glib/lib/libgobject-2.0.0.dylib (compatibility version 6601.0.0, current version 6601.7.0)
	/opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 6601.0.0, current version 6601.7.0)
	/Users/person/lib/libvirglrenderer.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/usr/lib/libpam.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
	@rpath/libc++.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2022.20.117)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 54.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1463.2.1)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

For comparison, I compiled a simple C++ program:

a.cc
#include <cstdio>

int main()
{
    getc(stdin);
    return 0;
}
c++ a.cc && otool -L a.out
a.out:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 904.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)

Interestingly, /usr/lib/libc++.1.dylib does not exist on the filesystem, but it works. I'm too ignorant about dyld to make sense of this.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

@akihikodaki I'm glad to help with this, and I'd like to see it merged into upstream someday.

As for the libraries - yes, that's what I did first (otool and friends). And it doesn't show a lot of libraries actually loaded.

The way to do this properly is to load qemu under lldb and then do image list. It will show you a complete set of libraries.

my image list

(lldb) image list
[ 0] 37132C63-E1CF-3E7B-B1E3-73590449A3BD 0x0000000100000000 /Users/knazarov/dev/virgil/bin/qemu-system-x86_64
[ 1] 0D4EA85F-7E30-338B-9215-314A5A5539B6 0x000000010122c000 /usr/lib/dyld
[ 2] 10CBB5E9-D5F9-3CFE-B2E4-7C8ADE945AC9 0x00007fff6f99a000 /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor
[ 3] E0A327D0-5F6A-3F6E-A9E9-3F8C52ACFCFF 0x0000000101344000 /usr/local/opt/pixman/lib/libpixman-1.0.dylib
[ 4] 4C036338-56A4-3AFB-8B11-D46D58BAF152 0x00000001013ce000 /Users/knazarov/dev/virgil/lib/libepoxy.0.dylib
[ 5] 7339869B-1C7D-3745-A3BE-0C29867524BE 0x000000010159a000 /usr/local/opt/lzo/lib/liblzo2.2.dylib
[ 6] 6E2BD7A3-DC55-3183-BBF7-3AC367BC1834 0x00007fff2a73e000 /usr/lib/libz.1.dylib
[ 7] F666699A-02D5-3061-ADF9-B7A7E471C1E1 0x00000001015b7000 /usr/local/opt/libpng/lib/libpng16.16.dylib
[ 8] 5500CEBB-26F2-39DF-9364-8903B1C286CE 0x00000001015e3000 /usr/local/opt/jpeg/lib/libjpeg.9.dylib
[ 9] 670EAB28-5540-356B-81C2-07AB29367D6A 0x0000000101619000 /usr/local/opt/gnutls/lib/libgnutls.30.dylib
[ 10] 7EE1EE13-9D86-3612-A6BC-3E6B5D578267 0x00007fff2d0ce000 /usr/lib/libsasl2.2.dylib
[ 11] 801E2D1E-7EA5-37DA-8F44-B6D7DD3CE5B9 0x00007fff21e10000 /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
[ 12] 33B62FB3-9596-3E1E-A715-F6F7B0EFD6DB 0x00000001017b5000 /usr/local/opt/glib/lib/libgio-2.0.0.dylib
[ 13] 229ACFA3-3F2C-3437-8F40-A959714D3AB0 0x0000000101975000 /usr/local/opt/glib/lib/libgobject-2.0.0.dylib
[ 14] 9DF031A5-CD8C-3418-A9E6-7078B345CEA9 0x00000001019c9000 /usr/local/opt/glib/lib/libglib-2.0.0.dylib
[ 15] 5D418C7F-DE76-3D8A-AB9E-CA0377A54322 0x0000000101afd000 /usr/local/opt/zstd/lib/libzstd.1.dylib
[ 16] AF4DC94C-853D-3C9B-BE8F-E71257939B36 0x0000000101bbd000 /usr/local/opt/libusb/lib/libusb-1.0.0.dylib
[ 17] 22CE6991-5E25-37A0-B959-BDA8F3424374 0x0000000101be1000 /Users/knazarov/dev/virgil/lib/libvirglrenderer.1.dylib
[ 18] 83503CE0-32B1-36DB-A4F0-3CC6B7BCF50A 0x00007fff2a806000 /usr/lib/libSystem.B.dylib
[ 19] 88BD7B0F-DFB3-30E8-9909-26F6A5ECCA6B 0x0000000101c59000 /usr/local/opt/nettle/lib/libnettle.8.dylib
[ 20] FA3308D0-E7B4-3A2C-AE39-0E085F6CDEE0 0x00007fff2aa79000 /usr/lib/libpam.2.dylib
[ 21] 2A0E160E-9EE6-3B23-8832-6979A16EC250 0x00007fff2067b000 /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
[ 22] 49AC0177-35A3-3C96-AD9D-3E59923C4761 0x00007fff22dd5000 /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
[ 23] 77010EA9-4AD1-3F4A-B3EF-2FFE739FB349 0x00007fff33260000 /usr/lib/libcurl.4.dylib
[ 24] E163D5F9-E202-3A53-817B-8BC40B9293C0 0x00007fff2a528000 /usr/lib/libbz2.1.0.dylib
[ 25] A8124803-CB85-3F33-8674-16B2A021D71C 0x00007fff22edc000 /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
[ 26] 42B48E32-1918-3178-8DE0-E8126B9EC088 0x00007fff3348d000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
[ 27] 29845645-F6F2-34E0-AC0D-C659773C78ED 0x00007fff251e2000 /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
[ 28] 8D9081B3-3F6A-31A0-9B20-1AE5CD8DD747 0x00007fff21428000 /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
[ 29] EB6B543F-D42C-3FB2-A2EC-43407C5F80D3 0x00007fff2049b000 /usr/lib/libobjc.A.dylib
[ 30] B377D7C7-EDB6-3737-B492-E9872F4C6469 0x00007fff224b5000 /System/Library/Frameworks/Security.framework/Versions/A/Security
[ 31] 44C477D9-013F-3A6D-A9FE-68A89214E6A5 0x00007fff78386000 /usr/lib/liboah.dylib
[ 32] D939A124-9FD5-37A2-B03B-72E98CBC98FE 0x00007fff2a808000 /usr/lib/libfakelink.dylib
[ 33] 8F8D8A8B-4EE0-3C09-9F45-725A1FBDD38C 0x00007fff22807000 /usr/lib/libicucore.A.dylib
[ 34] EF2AC5FF-B98D-3252-ABA8-2FC721CBA942 0x00007fff2a80c000 /System/Library/PrivateFrameworks/SoftLinking.framework/Versions/A/SoftLinking
[ 35] 3C9FE530-3CD2-3A64-8A36-70816AEBDF0D 0x00007fff205b6000 /usr/lib/libc++abi.dylib
[ 36] B217D905-4F9C-3DE0-8844-88FAA3C2C851 0x00007fff20560000 /usr/lib/libc++.1.dylib
[ 37] 1A98B064-8FED-39CF-BB2E-5BDA1EF5B65A 0x00007fff2a800000 /usr/lib/system/libcache.dylib
[ 38] 822A29CE-BF54-35AD-BB15-8FAECB800C7D 0x00007fff2a7bc000 /usr/lib/system/libcommonCrypto.dylib
[ 39] 62EE1D14-5ED7-3CEC-81C0-9C93833641F1 0x00007fff2a7e6000 /usr/lib/system/libcompiler_rt.dylib
[ 40] 39DBE613-135B-3AFE-A6AF-7898A37F70C2 0x00007fff2a7db000 /usr/lib/system/libcopyfile.dylib
[ 41] 1EB11CFB-ABD7-36DD-97C7-C112A6601416 0x00007fff20389000 /usr/lib/system/libcorecrypto.dylib
[ 42] AD988EEA-1A2F-3404-9A6E-390FC2504223 0x00007fff20456000 /usr/lib/system/libdispatch.dylib
[ 43] 4641E48F-75B5-3CC7-8263-47BF79F15394 0x00007fff2060a000 /usr/lib/system/libdyld.dylib
[ 44] 803F6AED-99D5-3CCF-B0FA-361BCF14C8C0 0x00007fff2a7f7000 /usr/lib/system/libkeymgr.dylib
[ 45] C7C51322-8491-3B78-9CFA-2B4753662BCF 0x00007fff2dc3f000 /usr/lib/system/liblaunch.dylib
[ 46] C2584BC4-497B-3170-ADDF-21B8E10B4DFD 0x00007fff2a79a000 /usr/lib/system/libmacho.dylib
[ 47] 40D35D75-524B-3DA6-8159-E7E0FA66F5BC 0x00007fff29fb4000 /usr/lib/system/libquarantine.dylib
[ 48] 5CC12A16-82CB-32F0-9040-72CFC88A48DF 0x00007fff2a7f4000 /usr/lib/system/libremovefile.dylib
[ 49] 5B48071E-85EB-33B0-AE9B-127AEB398AEC 0x00007fff24e4c000 /usr/lib/system/libsystem_asl.dylib
[ 50] E644CAA0-65B7-36E4-8041-520F3301F3DB 0x00007fff20339000 /usr/lib/system/libsystem_blocks.dylib
[ 51] 4AF71812-4099-3E96-B271-1F259491A2B2 0x00007fff204d7000 /usr/lib/system/libsystem_c.dylib
[ 52] C53D5E0C-0C4F-3B35-A24B-E0D7101A3F95 0x00007fff2a7ee000 /usr/lib/system/libsystem_collections.dylib
[ 53] 4917D824-4DE8-32CC-9ED2-1FBF371FEB9F 0x00007fff292a8000 /usr/lib/system/libsystem_configuration.dylib
[ 54] 6F08275F-B912-3D8E-9D74-4845158AE4F3 0x00007fff2859b000 /usr/lib/system/libsystem_containermanager.dylib
[ 55] 529A0663-A936-309C-9318-1B04B7F70658 0x00007fff2a536000 /usr/lib/system/libsystem_coreservices.dylib
[ 56] E016D8F7-C87F-36F8-B8A0-6A61B8E4BACB 0x00007fff22a69000 /usr/lib/system/libsystem_darwin.dylib
[ 57] E0A0CAB3-6779-3C83-AC67-046CFE69F9C2 0x00007fff2a7f8000 /usr/lib/system/libsystem_dnssd.dylib
[ 58] 9CECB43A-094E-3CA9-B730-24DEA1A6DE05 0x00007fff204d4000 /usr/lib/system/libsystem_featureflags.dylib
[ 59] 0C96CFE8-71F5-3335-8423-581BC3DE5846 0x00007fff2064f000 /usr/lib/system/libsystem_info.dylib
[ 60] DD26CC5C-AFF6-305F-A567-14909DD57163 0x00007fff2a751000 /usr/lib/system/libsystem_m.dylib
[ 61] A498D1EF-E43D-310C-84E8-9C0AADA0C475 0x00007fff20429000 /usr/lib/system/libsystem_malloc.dylib
[ 62] 5213D866-7D0E-3FD9-8E1A-03C0E39CEC44 0x00007fff24de0000 /usr/lib/system/libsystem_networkextension.dylib
[ 63] B2BF20C7-448A-3FBD-A2F5-AB7618D173F6 0x00007fff22e84000 /usr/lib/system/libsystem_notify.dylib
[ 64] 20310EE6-2C3F-361A-9ECA-4223AFC03B65 0x00007fff300f3000 /usr/lib/system/libsystem_product_info_filter.dylib
[ 65] 5F7F3DD1-4B38-310C-AA8F-19FF1B0F5276 0x00007fff292ac000 /usr/lib/system/libsystem_sandbox.dylib
[ 66] E05E35BC-1BAB-365B-8BEE-D774189EFD3B 0x00007fff2a7f1000 /usr/lib/system/libsystem_secinit.dylib
[ 67] AB413518-ECDE-3F04-A61C-278D3CF43076 0x00007fff205cf000 /usr/lib/system/libsystem_kernel.dylib
[ 68] 1C3E1A0A-92A8-3CDE-B622-8940B43A5DF2 0x00007fff20645000 /usr/lib/system/libsystem_platform.dylib
[ 69] B989DF6C-ADFE-3AF9-9C91-07D2521F9E47 0x00007fff205fe000 /usr/lib/system/libsystem_pthread.dylib
[ 70] BC85B46C-02EE-31FF-861D-F0DE01E8F6CF 0x00007fff2654b000 /usr/lib/system/libsystem_symptoms.dylib
[ 71] 87FEF600-48D9-31C9-B8FC-D5249B2AE95D 0x00007fff20371000 /usr/lib/system/libsystem_trace.dylib
[ 72] 1D0A4B4A-4370-3548-8DC1-42A7B4BD45D3 0x00007fff2a7c8000 /usr/lib/system/libunwind.dylib
[ 73] 70F26262-01AA-3CEC-9FAD-2701D24096F0 0x00007fff2033b000 /usr/lib/system/libxpc.dylib
[ 74] C4C7A5DC-C3AD-374D-902A-B6DED14FBB67 0x00007fff24df0000 /usr/lib/libMobileGestalt.dylib
[ 75] 879A04AA-F9A1-3640-A9EF-F5D0813C852A 0x00007fff2a518000 /System/Library/PrivateFrameworks/AppleFSCompression.framework/Versions/A/AppleFSCompression
[ 76] 20EDB261-69C7-3E4A-8C00-878814186A62 0x00007fff266fd000 /usr/lib/libDiagnosticMessagesClient.dylib
[ 77] D71EF121-D4B0-306E-9843-9FAFD558E3A4 0x00007fff29fdb000 /usr/lib/libbsm.0.dylib
[ 78] B33BEF87-3275-356D-9815-D0E94122D2EB 0x00007fff29fc3000 /usr/lib/libcoretls.dylib
[ 79] 80AF345B-1989-3C88-AEF4-8FF18A366A9C 0x00007fff2aea2000 /usr/lib/libcoretls_cfhelpers.dylib
[ 80] ACCC6A4C-3257-38A4-8AFB-BA73D906205B 0x00007fff261aa000 /usr/lib/libsqlite3.dylib
[ 81] FAA7B36E-21EC-3ADF-AA6A-E3742737CA3A 0x00007fff2af9f000 /usr/lib/libxar.1.dylib
[ 82] 7A2E42E6-3AF2-3ECB-8A16-5C4AC41206EE 0x00007fff2743e000 /System/Library/PrivateFrameworks/CoreAutoLayout.framework/Versions/A/CoreAutoLayout
[ 83] 3518EA0E-C32D-32CC-81B9-0F3C83B6430C 0x00007fff2113c000 /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
[ 84] 9C42DB2C-0A6D-38CE-BB8D-899A9B34B695 0x00007fff2aa7e000 /usr/lib/libcompression.dylib
[ 85] 04A917FB-DBFB-3432-BA4C-5B860990A420 0x00007fff24942000 /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
[ 86] ACFCD23E-05DF-3F96-BCFA-E294AAAFD2CE 0x00007fff267e8000 /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
[ 87] 9061C767-43FD-3483-9C48-73973AF82DC1 0x00007fff2a948000 /usr/lib/libarchive.2.dylib
[ 88] E0BF29C7-869B-3DD5-82AE-F36E6398091A 0x00007fff27487000 /usr/lib/libxml2.2.dylib
[ 89] E59859C6-7221-3324-BB58-F910B2199959 0x00007fff301f5000 /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
[ 90] E8044658-AE79-3930-A902-07A38BB22382 0x00007fff292b7000 /usr/lib/liblangid.dylib
[ 91] 970D1F7B-5DBD-355F-80D8-C820AA1626D2 0x00007fff211ba000 /usr/lib/libCRFSuite.dylib
[ 92] A3F1E999-6D61-32D7-B3F7-67832CEA53CD 0x00007fff242fe000 /usr/lib/libnetwork.dylib
[ 93] DE5787F8-D011-3E71-8323-936B35976D9E 0x00007fff2a80d000 /usr/lib/libpcap.A.dylib
[ 94] 97EB5DFD-BB41-3834-BF09-1F597D84F324 0x00007fff26544000 /usr/lib/libdns_services.dylib
[ 95] A2F6DE8D-2B38-3669-A16E-E1BD54108F24 0x00007fff2aa4c000 /usr/lib/libapple_nghttp2.dylib
[ 96] 33A9FBF3-A2D3-3ECE-9370-B0ADDE3896DE 0x00007fff24def000 /usr/lib/libenergytrace.dylib
[ 97] DE05C188-D9FE-30DC-8299-DF3DC9F837A5 0x00007fff2a7a0000 /usr/lib/system/libkxld.dylib
[ 98] 168D84A7-8E54-361E-9161-B661D96C0C9A 0x00007fff2ae89000 /usr/lib/liblzma.5.dylib
[ 99] AD10ECF4-E137-3152-9612-7EC548D919E8 0x00007fff2a845000 /usr/lib/libiconv.2.dylib
[100] D14F9D24-693A-37F0-8F92-D260248EB282 0x00007fff2a799000 /usr/lib/libcharset.1.dylib
[101] 6B70B98B-BCC5-3FBE-A6B5-CD0696AA055A 0x00007fff2740d000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
[102] B857EADF-D517-34E8-8053-34C0E6695250 0x00007fff22a73000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
[103] 49187239-597B-31A1-8895-8AA265C818C7 0x00007fff2674d000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
[104] 8D30829B-426E-361E-B2A9-5850F1E5513D 0x00007fff2a53b000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
[105] 6243D4C0-3572-30A6-889A-B8690A5EAC24 0x00007fff2a9ba000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
[106] 9B6B42DB-8768-343B-B10E-A9A5710280CD 0x00007fff264ce000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
[107] 9ACD5692-3B92-3E6E-8B24-56ADCC911556 0x00007fff20b17000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
[108] 5D0B1BB2-995B-32EB-AE75-F5F27127ACAF 0x00007fff2ae3a000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
[109] C861FAD6-D3A5-302C-88AE-B2813F7201A7 0x00007fff27416000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
[110] 74E17BB2-1A3C-3574-92DD-63ABE39E3FF9 0x00007fff29fb7000 /usr/lib/libCheckFix.dylib
[111] 2F48471B-68F3-3017-8B18-BEDD4ED5853F 0x00007fff24e63000 /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
[112] 6C49B26B-CB39-3504-AD5E-9C3333FFE086 0x00007fff292b9000 /System/Library/PrivateFrameworks/CoreNLP.framework/Versions/A/CoreNLP
[113] FB7BF527-2D95-3E82-AAD8-D42B33DAC0F0 0x00007fff26700000 /System/Library/PrivateFrameworks/MetadataUtilities.framework/Versions/A/MetadataUtilities
[114] 102A0AD8-D119-340B-B652-B75F0AAB1C7E 0x00007fff211f0000 /usr/lib/libmecabra.dylib
[115] 01B968D9-A464-3667-8B8E-C99AEBA1FF79 0x00007fff2a2ca000 /System/Library/Frameworks/MLCompute.framework/Versions/A/MLCompute
[116] 2423F0F6-A3A4-37AB-971E-FCFC7645D282 0x00007fff304c3000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
[117] 1F424683-4213-3594-9BF5-EFCCCDAA83A5 0x00007fff29fec000 /usr/lib/libmecab.dylib
[118] E40EFA11-E95C-36D1-A8BE-8CA5B1DD179D 0x00007fff2a036000 /usr/lib/libgermantok.dylib
[119] 47CDE3EE-60AB-38CE-9494-5CC047CACA7F 0x00007fff2aa22000 /usr/lib/libThaiTokenizer.dylib
[120] 718400BF-9911-3CB6-8CCE-4C2D7C35D80A 0x00007fff2afda000 /usr/lib/libChineseTokenizer.dylib
[121] F37DBCE9-20CD-33D4-806B-93B326E08422 0x00007fff2aa77000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/MetalPerformanceShaders
[122] 829B9C4D-2DB6-38C9-9FE8-E6544FB3BDCC 0x00007fff285c7000 /System/Library/Frameworks/Metal.framework/Versions/A/Metal
[123] 3EBF95E1-30EB-3162-A9D4-B97BE581B69F 0x00007fff267ef000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
[124] 8CD6F55A-8E4F-3653-A367-427D0ED3DD16 0x00007fff301ce000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
[125] 9D11FCFC-6C30-32B6-864B-9CC3ED7D1143 0x00007fff2afe5000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
[126] 4CCEE561-A72B-33BC-B9F9-77A6CF2E7AF0 0x00007fff29a1f000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
[127] 92702864-A5C6-3E22-86F6-DB329532674A 0x00007fff20e1b000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
[128] 2F92E8CD-396D-3665-88EB-41A4E1A9CB31 0x00007fff2aa9d000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
[129] 0D017F3F-6FD6-38DB-AA14-A64523A4D093 0x00007fff2a03c000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
[130] FC45F7FD-6B10-3F8D-A553-F57C757F6717 0x00007fff2aa64000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
[131] 55EE1866-623B-3C10-91FB-C8B06EB83995 0x00007fff2aa97000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
[132] C5CD30D8-114B-3104-9020-E5C05E68CEE8 0x00007fff29361000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
[133] B889DE4E-7356-3CC8-A13E-68D15E2024DF 0x00007fff210cd000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib
[134] 287C79B5-4E13-3DF6-BDC4-439EC621E010 0x00007fff2924c000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSCore.framework/Versions/A/MPSCore
[135] D009508B-DCDF-3421-B1CB-40FDF4A68E93 0x00007fff2a487000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSImage.framework/Versions/A/MPSImage
[136] 8FA469B3-48BE-315B-B3A0-C868CD541608 0x00007fff2a052000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSNeuralNetwork.framework/Versions/A/MPSNeuralNetwork
[137] B988D5F6-4E54-335D-A654-12C74FD668EA 0x00007fff2a412000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSMatrix.framework/Versions/A/MPSMatrix
[138] 613BB6F1-3588-3E25-B523-059022FBE56E 0x00007fff2a27a000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSRayIntersector.framework/Versions/A/MPSRayIntersector
[139] 51ECDE6B-39F8-3BAB-ADC1-84C958F40A3F 0x00007fff2a449000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSNDArray.framework/Versions/A/MPSNDArray
[140] B1FDD261-7707-3E20-A6C2-99D9AF46356B 0x00007fff20d47000 /System/Library/PrivateFrameworks/MetalTools.framework/Versions/A/MetalTools
[141] 029C5E16-2E04-3E98-BE25-CC8BAE5E3D19 0x00007fff285ac000 /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
[142] E6C1A91D-4ADE-33CB-AA47-2F573811BFD9 0x00007fff292b1000 /System/Library/PrivateFrameworks/AggregateDictionary.framework/Versions/A/AggregateDictionary
[143] 4FFF906A-FE79-38F9-B777-AE878A0C99F1 0x00007fff266d8000 /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/CoreAnalytics
[144] 4A637FE0-A740-32F0-AE70-4593F6DF1573 0x00007fff2aa24000 /System/Library/PrivateFrameworks/AppleSauce.framework/Versions/A/AppleSauce
[145] 41EDF102-E1B5-3FBA-A74A-6C583CAA5D24 0x00007fff285be000 /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
[146] 44489BD1-3963-35DF-86F1-DE95778AC0DB 0x00007fff6cb52000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreFSCache.dylib
[147] 4F8A1D06-3EFF-3174-BE9C-C4B70A3EA290 0x00007fff292b3000 /System/Library/PrivateFrameworks/AppleSystemInfo.framework/Versions/A/AppleSystemInfo
[148] B15E9DA1-1C81-3ECF-94C7-FD137AD6C5E2 0x00007fff29c06000 /System/Library/PrivateFrameworks/IOMobileFramebuffer.framework/Versions/A/IOMobileFramebuffer
[149] 1A05BCD7-232F-3950-936D-EC0F95BA3FB0 0x00007fff2178c000 /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
[150] D43F3FFA-6692-3D56-ACBA-DDA40C566DDE 0x00007fff29bf3000 /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
[151] B059FF7F-731A-38F1-884A-588C6C8CA7F2 0x00007fff2935a000 /System/Library/PrivateFrameworks/LinguisticData.framework/Versions/A/LinguisticData
[152] 2F429EBD-1BD2-3057-B760-8A81546DBD6F 0x00007fff2107f000 /System/Library/PrivateFrameworks/Lexicon.framework/Versions/A/Lexicon
[153] A5D1EBB3-70A1-3ECA-A3C7-E97FBAD36ECF 0x00007fff2a936000 /usr/lib/libcmph.dylib
[154] 25F138F8-9E55-3749-8271-0FDB275C3E2C 0x00007fff273ed000 /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
[155] 333A7F2E-0F3E-37F1-9E1B-69996F5084D2 0x00007fff273dd000 /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
[156] E48CCAAC-34B5-377E-8E5D-06CDBD411C2F 0x00007fff2aea4000 /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
[157] 72AC63B1-0B72-394F-97E8-BA9B114DD0A9 0x00007fff29f18000 /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
[158] 0A268749-08E7-3551-8969-442210651CCA 0x00007fff2afad000 /usr/lib/libutil.dylib
[159] 2BBB708C-4D91-364E-ABD0-39BF198688A6 0x00007fff2743b000 /usr/lib/libapp_launch_measurement.dylib
[160] 89992991-8538-380B-B2B8-4EA997CBFDBE 0x00007fff22d9a000 /System/Library/PrivateFrameworks/CoreServicesStore.framework/Versions/A/CoreServicesStore
[161] FD031028-B702-3909-B2AF-3916404DD4A8 0x00007fff29faf000 /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
[162] 1C5219FB-B504-3658-B23B-2C45A3BA28F1 0x00007fff2afb1000 /usr/lib/libxslt.1.dylib
[163] EC15118E-62BD-3C24-A880-3B9B74318B0B 0x00007fff29fa5000 /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
[164] 32F2A1A3-F96F-367E-A799-974BEF5267D2 0x0000000101c9d000 /usr/local/opt/p11-kit/lib/libp11-kit.0.dylib
[165] 7C2B2A62-B6F6-37ED-8160-568BB21B7906 0x0000000101d91000 /usr/local/opt/libidn2/lib/libidn2.0.dylib
[166] 15146CD7-FAEB-3DCA-885F-53BC712F8FD8 0x0000000101dbd000 /usr/local/opt/libunistring/lib/libunistring.2.dylib
[167] 0981BD18-599B-3304-B05C-36E001FC741A 0x0000000101f39000 /usr/local/opt/libtasn1/lib/libtasn1.6.dylib
[168] 545E512F-A3A5-321F-ABB9-C45BFBA775FF 0x0000000101f55000 /usr/local/opt/nettle/lib/libhogweed.6.dylib
[169] 3B2C7DB0-5829-3B4C-9522-1EF23C5E3226 0x0000000101fa1000 /usr/local/opt/gmp/lib/libgmp.10.dylib
[170] FA921CC0-395B-3155-8259-EA61DE25C5D2 0x0000000102019000 /usr/local/opt/gettext/lib/libintl.8.dylib
[171] C148976B-BB19-3551-B57F-AA52BAF80F7B 0x0000000102029000 /usr/local/opt/libffi/lib/libffi.7.dylib
[172] C78E9D22-5110-349B-AF8E-6BA893A20214 0x00007fff2b92d000 /System/Library/PrivateFrameworks/AppServerSupport.framework/Versions/A/AppServerSupport
[173] A9D2E8CD-9D74-304E-BD99-A7704F3CD38B 0x00007fff2d9de000 /System/Library/PrivateFrameworks/perfdata.framework/Versions/A/perfdata
[174] A00B802D-7BB8-3F6B-A79D-E9697F7D0ABB 0x00007fff29f07000 /System/Library/PrivateFrameworks/AssertionServices.framework/Versions/A/AssertionServices
[175] DEE2C5BA-BFD1-349F-A10A-B10717BA8F7C 0x00007fff219b0000 /System/Library/PrivateFrameworks/AudioToolboxCore.framework/Versions/A/AudioToolboxCore
[176] 3FFEB564-C7CC-3F3E-B193-3F4E7B7C6764 0x00007fff286ed000 /System/Library/PrivateFrameworks/caulk.framework/Versions/A/caulk
[177] 29F5E3FE-CF12-3242-9FD3-4950DA38E4D0 0x00007fff3d689000 /System/Library/PrivateFrameworks/SystemPolicy.framework/Versions/A/SystemPolicy
[178] 8DEEA01D-B715-340A-900E-B448AB22720E 0x00007fff2b13d000 /usr/lib/libIOReport.dylib
[179] B207FDFF-8D80-3800-B968-03C718FFBF3F 0x00007fff2d200000 /usr/lib/libSMC.dylib
[180] 3A964BB6-7AB8-3B5D-BAF6-40F56D2BA393 0x00007fff26400000 /System/Library/PrivateFrameworks/BaseBoard.framework/Versions/A/BaseBoard
[181] 4B9FFFB3-310F-350A-BEBB-AB779BD3F405 0x00007fff26482000 /System/Library/PrivateFrameworks/RunningBoardServices.framework/Versions/A/RunningBoardServices
[182] B3FE85C5-8034-38D5-8566-5317B49ECF61 0x00007fff2b48d000 /System/Library/PrivateFrameworks/PersistentConnection.framework/Versions/A/PersistentConnection
[183] A865CC93-C2C2-3EF4-8E97-7BEA78970DAA 0x00007fff28c93000 /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
[184] 58CD91DC-64A9-3A97-AE41-6EA30308D617 0x00007fff26193000 /System/Library/PrivateFrameworks/ProtocolBuffer.framework/Versions/A/ProtocolBuffer
[185] 8C96D016-2B40-39C0-A51C-9CF687AE7218 0x00007fff263e7000 /System/Library/PrivateFrameworks/CommonUtilities.framework/Versions/A/CommonUtilities
[186] F4B7C353-8A65-3B54-9E01-2C93CB3F55A3 0x00007fff2bba8000 /System/Library/PrivateFrameworks/Bom.framework/Versions/A/Bom
[187] 8034EAA7-787C-3A3E-A301-FF90376F247D 0x00007fff24e7b000 /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
[188] 2CD3B3C9-7022-34DD-BF53-D48D79DEF726 0x00007fff287f4000 /System/Library/PrivateFrameworks/FontServices.framework/libFontParser.dylib
[189] C257C950-F430-3762-BC72-15D314D99556 0x00007fff2bd3d000 /System/Library/PrivateFrameworks/WatchdogClient.framework/Versions/A/WatchdogClient
[190] 6DD6A260-800F-3284-842C-8E4CB9EA47FF 0x00007fff21879000 /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
[191] 8E50C806-C6A2-3B96-B3D2-DA1FFC73D2A8 0x00007fff2870a000 /System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia
[192] 50857F8D-C7CC-3609-B0DB-FC3C7382243B 0x00007fff2756a000 /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
[193] 4243F7DC-7EB9-3750-BE36-527B0BADF36C 0x00007fff2bd41000 /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
[194] 04D8759B-1119-3E37-B922-32BDECB7C5D2 0x00007fff26e96000 /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
[195] 425BDD2D-6C26-3D09-AD9F-942EA387B2A4 0x00007fff2bd75000 /System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox
[196] 60363868-111B-36CF-9CDF-2643DF82002D 0x00007fff2bf9f000 /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
[197] 5AD7B6E0-1744-3C8D-8A95-FE425DEA005C 0x00007fff2bf82000 /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
[198] 010F8260-BC46-361E-BE59-540F765EFE74 0x00007fff2bfaa000 /System/Library/PrivateFrameworks/DSExternalDisplay.framework/Versions/A/DSExternalDisplay
[199] 2B7E468B-D901-3EA9-85CF-82FD753785BD 0x00007fff2c00a000 /System/Library/PrivateFrameworks/CMCaptureCore.framework/Versions/A/CMCaptureCore
[200] 3B4701CB-8F66-3E50-A38A-BEEC992AAC31 0x00007fff2b946000 /usr/lib/libspindump.dylib
[201] A67D9B0A-E9FB-325A-873B-032DAF45C112 0x00007fff25876000 /System/Library/Frameworks/ColorSync.framework/Versions/A/ColorSync
[202] 42707DFB-86A4-3F56-A5AF-E6DAA7E49DC7 0x00007fff2b0a4000 /usr/lib/libate.dylib
[203] EE4A9D54-C131-336B-8EC0-ECA2BD8A79F4 0x00007fff2bfa6000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
[204] 828DE4EE-7551-37E0-B37D-EC96D1A0754A 0x00007fff2bfb0000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
[205] CB838530-E4AE-3126-BC35-9FBE588303CB 0x00007fff2befb000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
[206] 173EC7B9-E3CD-3ECC-B67D-6EFDCE60E7FA 0x00007fff2bf22000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
[207] 8941531E-2AED-3366-B709-5FE275994FF0 0x00007fff2c005000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
[208] 7C3C2138-1E8C-35B8-8517-240BBC7F165B 0x00007fff2bc5e000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
[209] 1D978F91-674E-3A44-8A93-773C7E3E513C 0x00007fff2b348000 /usr/lib/libexpat.1.dylib
[210] 7F3819DE-BCE7-32B7-BB33-10C1DB2BA512 0x00007fff2bc13000 /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
[211] F70AF1B3-D17A-3A0E-A8AC-9D45C5B52BA9 0x00007fff6cb5e000 /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
[212] F0DD35B5-3692-3BE5-AD3D-4F6B237EF6AD 0x00007fff6cbae000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
[213] 45A1FFDC-03B7-3D64-AFAF-16D321B53D28 0x00007fff6cb71000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
[214] 68ABAADE-2DB4-3412-9F36-CA1AEAC8E9F6 0x00007fff6cd80000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
[215] 16D10CE1-C2A1-3E24-A617-766CB6E9133C 0x00007fff6cb7a000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
[216] AF77D964-9A4C-3690-AF62-4E05825DC9BF 0x00007fff6cb6e000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
[217] 989F6CBF-CCEF-340D-9CB5-EC4133707040 0x00007fff6cb59000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
[218] 29BE82F7-93BE-336A-AF66-9DD3F48B8565 0x00007fff28ece000 /System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage
[219] 9A8DC9A0-1A80-3A26-ACA0-A245D3FFF921 0x00007fff21c27000 /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
[220] 40B70A3B-D981-3E4A-9BEC-F22EF84118F5 0x00007fff6e1c1000 /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
[221] A7393BC4-4441-31B5-BB15-5A5A70738BEC 0x00007fff2b4b7000 /System/Library/PrivateFrameworks/GraphVisualizer.framework/Versions/A/GraphVisualizer
[222] 23CD0C8A-E300-3E28-BD5B-DDCDA4521C54 0x00007fff2b4c6000 /System/Library/PrivateFrameworks/FaceCore.framework/Versions/A/FaceCore
[223] 827396F3-F3A7-3A4F-BE7F-171077501137 0x00007fff2b8e2000 /System/Library/PrivateFrameworks/OTSVG.framework/Versions/A/OTSVG
[224] 9E9329F6-7384-3C7F-83E7-D52CD7FFBC09 0x00007fff27161000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
[225] F0974976-30C0-35EB-92F1-B4CF1E974F11 0x00007fff2b934000 /System/Library/PrivateFrameworks/FontServices.framework/libhvf.dylib
[226] E3BE877B-1C78-3ABF-8C41-66FE9B6971C7 0x00007fff2b2f4000 /System/Library/PrivateFrameworks/AppleVA.framework/Versions/A/AppleVA
[227] 83DBFA98-F85B-3B63-B58F-C9989BAC9611 0x00007fff2bec8000 /usr/lib/libAudioToolboxUtility.dylib
[228] 6C720DC2-E50E-3A8D-8494-F8C007DF8DF4 0x00007fff3dff3000 /usr/lib/libmis.dylib
[229] 52DCC7E4-69F7-36FA-880D-6434E3DB6B58 0x000000010203d000 /usr/local/Cellar/glib/2.66.7/lib/libgmodule-2.0.0.dylib
[230] 7B043B4A-71CE-3F6E-91F0-CBBED47A9EA9 0x00007fff2d0b5000 /usr/lib/libresolv.9.dylib
[231] 5A905085-B5EB-3F71-A663-200428E7A5F4 0x00007fff23c3f000 /System/Library/PrivateFrameworks/UIFoundation.framework/Versions/A/UIFoundation
[232] 661A448B-2901-33CD-B13C-362D61AFD998 0x00007fff2fe6c000 /System/Library/PrivateFrameworks/RemoteViewServices.framework/Versions/A/RemoteViewServices
[233] D279B67B-7108-3A16-93E5-5A429B898DAA 0x00007fff28c65000 /System/Library/PrivateFrameworks/XCTTargetBootstrap.framework/Versions/A/XCTTargetBootstrap
[234] 2AAE247F-1BA0-3E42-A07B-CBE413747B30 0x00007fff23e94000 /System/Library/Frameworks/UniformTypeIdentifiers.framework/Versions/A/UniformTypeIdentifiers
[235] D88B5A7D-F9F7-36C2-A346-87D7163605A0 0x00007fff28c69000 /System/Library/PrivateFrameworks/CoreSVG.framework/Versions/A/CoreSVG
[236] B6859735-701D-307E-8061-298BA2296E47 0x00007fff2b381000 /System/Library/PrivateFrameworks/IconServices.framework/Versions/A/IconServices
[237] 4B26DB42-BFDD-3B0D-8966-3D0926A3A5B7 0x00007fff28c51000 /System/Library/PrivateFrameworks/DFRFoundation.framework/Versions/A/DFRFoundation
[238] 8E53C25F-9FE2-372E-8374-6A598D72C4C1 0x00007fff2ce8d000 /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
[239] 7BE9684B-4439-3008-A2A7-4093BBCA180A 0x00007fff25d74000 /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
[240] A4EBFF22-F861-321A-AF28-1DB8ECB9A4FF 0x00007fff2c132000 /System/Library/PrivateFrameworks/DataDetectorsCore.framework/Versions/A/DataDetectorsCore
[241] 82B316E1-E47F-334D-8C33-DB1B7E8A4F71 0x00007fff28951000 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
[242] 8C33ECFF-2FBF-30FF-87EA-32A508860543 0x00007fff2fea8000 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
[243] 6DB427BB-44CA-3002-A3FF-77F0FEFC2A51 0x00007fff271a3000 /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
[244] DE997943-6AD5-32DA-B48E-78170C1EAA7B 0x00007fff2c129000 /System/Library/PrivateFrameworks/InternationalSupport.framework/Versions/A/InternationalSupport
[245] 2E5E666D-EE93-3949-8C76-692FA748D129 0x00007fff273d1000 /System/Library/PrivateFrameworks/PerformanceAnalysis.framework/Versions/A/PerformanceAnalysis
[246] 12250395-D660-3CC1-81DB-565340F2651B 0x00007fff2c1ae000 /System/Library/PrivateFrameworks/UserActivity.framework/Versions/A/UserActivity
[247] 50BA07A4-02E8-38EE-B0B9-B12C277B563B 0x00007fff2c05c000 /System/Library/PrivateFrameworks/TextureIO.framework/Versions/A/TextureIO
[248] 3DD60168-A0F3-3388-8DDE-69F81280AE91 0x00007fff2d049000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
[249] F3AEB77A-456F-3F9F-98CC-3E29ABAF5D0C 0x00007fff2d35b000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSyncLegacy.framework/Versions/A/ColorSyncLegacy
[250] 780B2B09-80A6-32C8-B9E5-024988196E00 0x00007fff2596d000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
[251] 14703020-3124-3C62-A9DB-6DD28024CD42 0x00007fff2d340000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
[252] DB939B2B-D47D-3035-8C02-01006D21C5A7 0x00007fff2c014000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
[253] F65D0944-C775-33FB-A345-C046789667B7 0x00007fff2d363000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
[254] 14F3382B-2B8E-31E5-8C73-73F0B8DD05F0 0x00007fff2d039000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
[255] 6C45783D-CF90-3F11-BA86-92D31A9B68CA 0x00007fff2bfd5000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATSUI.framework/Versions/A/ATSUI
[256] CDF6F483-017E-3DED-B107-C50FC3CADD65 0x00007fff2d2e0000 /usr/lib/libcups.2.dylib
[257] D40BF56F-DF5F-3ECB-AAC2-FE6F13E9D120 0x00007fff2d350000 /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
[258] 02FFA3CA-F8AF-3211-9159-36BC29612650 0x00007fff2d9fa000 /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
[259] 5D06022E-C963-390A-9487-90BD1B61BC20 0x00007fff2da0a000 /System/Library/Frameworks/GSS.framework/Versions/A/GSS
[260] 69768234-4F4F-3AB6-B116-4AE249E6417A 0x00007fff2b949000 /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
[261] 85526E13-1E48-377A-A700-EE3F5648A6E3 0x00007fff3378e000 /System/Library/Frameworks/Kerberos.framework/Versions/A/Libraries/libHeimdalProxy.dylib
[262] 75C2713C-F0D8-32BA-A839-17A2AE49A061 0x00007fff26553000 /System/Library/Frameworks/Network.framework/Versions/A/Network
[263] 8F6CD867-FB45-375D-82C1-E0AE79558C8C 0x00007fff2b362000 /usr/lib/libheimdal-asn1.dylib
[264] C4418557-FA52-3B80-804F-2BC7AF760FD6 0x00007fff2da5a000 /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
[265] B149BCAF-854C-3820-809D-15C6077FFB8B 0x00007fff275b7000 /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
[266] F696F4EC-9344-3C85-941E-6E5EF5CEF585 0x00007fff2cfbe000 /System/Library/PrivateFrameworks/AudioSession.framework/Versions/A/AudioSession
[267] 303533DD-D001-3CE8-A5A4-EC29F464FC3A 0x00007fff2d026000 /usr/lib/libAudioStatistics.dylib
[268] 14E84D66-28B8-390E-A499-9F24B7628EC7 0x00007fff2b3ef000 /System/Library/PrivateFrameworks/MediaExperience.framework/Versions/A/MediaExperience
[269] 6AAD02A0-9590-3CED-9568-36E05C59AA2B 0x00007fff2ce5b000 /System/Library/PrivateFrameworks/AudioSession.framework/libSessionUtility.dylib
[270] 5AFDC5C6-EBB6-33E4-9215-1AACE6C6F6EE 0x00007fff2d9eb000 /usr/lib/libperfcheck.dylib
[271] 6A293E4C-1E2D-36DA-B416-FBED7286CD6E 0x00007fff2d36f000 /System/Library/PrivateFrameworks/AudioResourceArbitration.framework/Versions/A/AudioResourceArbitration
[272] 3563C4B4-ABF6-3DE0-8381-D3667DFE2B80 0x00007fff2b36c000 /System/Library/PrivateFrameworks/IconFoundation.framework/Versions/A/IconFoundation
[273] B24969AE-43BA-36C2-80F9-2CBBC3F63CC8 0x00007fff2fe98000 /System/Library/PrivateFrameworks/SpeechRecognitionCore.framework/Versions/A/SpeechRecognitionCore
[274] C6441F4D-3ADF-329E-A264-C90A1B132B37 0x000000010204d000 /usr/local/opt/pcre/lib/libpcre.1.dylib
[275] 7D8D4B1B-625D-3D00-A626-C21AFEE0462E 0x00007fff32fc0000 /usr/lib/libcrypto.44.dylib
[276] E02ED3C9-4956-39C4-AF0A-B541EEDD3F1B 0x00007fff330c4000 /usr/lib/libssl.46.dylib
[277] 246874E6-4FD1-35A5-A039-4141999D8CD6 0x00007fff32f86000 /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
[278] 6FEF60A0-F80F-34E3-8845-FEEC92840920 0x00007fff32fbe000 /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
[279] D73E7719-09C8-30E6-B92D-AFC1BB356D92 0x00007fff30535000 /System/Library/PrivateFrameworks/CoreDaemon.framework/Versions/B/CoreDaemon
[280] 3BF95CAF-2EC5-393A-95B7-429C0C5F8F13 0x00007fff30530000 /System/Library/PrivateFrameworks/AppleSRP.framework/Versions/A/AppleSRP
[281] 0DA95074-B315-3B84-B8A3-BAEDC4471A2C 0x00007fff22d5b000 /System/Library/PrivateFrameworks/CoreServicesInternal.framework/Versions/A/CoreServicesInternal
[282] C5007E6E-897B-308F-B173-4F5C6BF3968D 0x00007fff3137e000 /System/Library/Frameworks/FileProvider.framework/Versions/A/FileProvider
[283] BCB8494A-A340-30A1-928C-9B53E953328D 0x00007fff2636a000 /System/Library/Frameworks/Accounts.framework/Versions/A/Accounts
[284] 974A1465-7AA4-347E-8606-CF35160FFA92 0x00007fff3149f000 /System/Library/PrivateFrameworks/GenerationalStorage.framework/Versions/A/GenerationalStorage
[285] F1A41B4E-671A-335E-8AE0-B99688F255C8 0x00007fff31e62000 /System/Library/PrivateFrameworks/CoreSymbolication.framework/Versions/A/CoreSymbolication
[286] 83B5E330-D786-34D2-AEE6-6F7B34D6FE9C 0x00007fff30d9d000 /System/Library/PrivateFrameworks/SymptomDiagnosticReporter.framework/Versions/A/SymptomDiagnosticReporter
[287] F294CD92-94FA-3D7D-9E59-AA5544504264 0x00007fff2e6a3000 /System/Library/PrivateFrameworks/AppContainer.framework/Versions/A/AppContainer
[288] 2144B85E-A433-3F9E-A25D-E8F9D7B6D175 0x00007fff23ea7000 /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
[289] 591F41F3-1FE9-354F-A1F9-BC0CC73333DA 0x00007fff22e90000 /usr/lib/libsandbox.1.dylib
[290] B6B35079-3860-384F-B7A2-584192BAAE70 0x00007fff275e3000 /System/Library/PrivateFrameworks/UserManagement.framework/Versions/A/UserManagement
[291] 0F83FED4-709A-37A0-9E04-EFFD71CB1272 0x00007fff29ee4000 /System/Library/PrivateFrameworks/MobileKeyBag.framework/Versions/A/MobileKeyBag
[292] CBEBDC8A-CCE4-3A21-915C-D2F3FB20B900 0x00007fff347f8000 /System/Library/PrivateFrameworks/ChunkingLibrary.framework/Versions/A/ChunkingLibrary
[293] 12A5A6E2-6C84-3FCA-9927-8B9E241C607F 0x00007fff31e25000 /System/Library/PrivateFrameworks/DebugSymbols.framework/Versions/A/DebugSymbols
[294] 1BEC89D6-6AC8-36C2-89AE-41662D14DC50 0x00007fff2e6b8000 /System/Library/PrivateFrameworks/SecCodeWrapper.framework/Versions/A/SecCodeWrapper
[295] EA306EDB-2A02-3FB6-8812-F5EAA337AEE6 0x00007fff2b145000 /System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
[296] CADD88A2-87A8-3879-8759-D3C112A6844F 0x00007fff4857c000 /System/Library/PrivateFrameworks/OSAnalytics.framework/Versions/A/OSAnalytics
[297] 78AF1606-37D8-3424-92AE-071A9F43AA0F 0x00007fff331d9000 /System/Library/PrivateFrameworks/RemoteServiceDiscovery.framework/Versions/A/RemoteServiceDiscovery
[298] FA39B702-6E08-3F11-953F-5EC1EFB993BB 0x00007fff52553000 /System/Library/PrivateFrameworks/Symbolication.framework/Versions/A/Symbolication
[299] C4BFF5FC-FC9C-3161-9612-7070EDAE9989 0x00007fff331e9000 /System/Library/PrivateFrameworks/RemoteXPC.framework/Versions/A/RemoteXPC
[300] 201350F5-56C4-3342-AD10-4BE20772C52B 0x00007fff396f4000 /System/Library/PrivateFrameworks/OSAServicesClient.framework/Versions/A/OSAServicesClient
[301] 507894FA-6738-372B-9A10-C38AFC9DA079 0x00007fff3dfd2000 /System/Library/PrivateFrameworks/MallocStackLogging.framework/Versions/A/MallocStackLogging
[302] 4925CAB3-09C0-373C-A5D3-9BB5FF025A4F 0x00007fff2e43a000 /System/Library/PrivateFrameworks/Sharing.framework/Versions/A/Sharing
[303] A62689A7-7C22-33B3-90FA-730E4AAEE5B5 0x00007fff32db6000 /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Apple80211
[304] 4A9B2D94-7AB0-3417-8D4E-A9C6C31C31D4 0x00007fff2a569000 /System/Library/PrivateFrameworks/AuthKit.framework/Versions/A/AuthKit
[305] 184C48A4-7F66-395B-81DB-DAA1641383CF 0x00007fff29ce4000 /System/Library/PrivateFrameworks/CoreUtils.framework/Versions/A/CoreUtils
[306] 35927D28-3324-3CAB-9255-276A7022EADC 0x00007fff29c11000 /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN
[307] 070A0F54-5E52-316E-8584-41C0EEDE01F2 0x00007fff2e582000 /System/Library/Frameworks/IOBluetooth.framework/Versions/A/IOBluetooth
[308] F3D8F158-611A-3A5E-9F1D-E5FC273A4C74 0x00007fff32e1a000 /System/Library/PrivateFrameworks/CoreWiFi.framework/Versions/A/CoreWiFi
[309] AA493452-21D3-3EC6-953C-42DBF75F4EB9 0x00007fff2dae2000 /System/Library/PrivateFrameworks/CorePhoneNumbers.framework/Versions/A/CorePhoneNumbers
[310] 9281E3F8-EAFE-3367-B7D5-23687AA6EB70 0x00007fff30ea5000 /System/Library/PrivateFrameworks/DiskManagement.framework/Versions/A/DiskManagement
[311] 964BA0DB-E86E-3F81-BB82-CC27184B6D20 0x00007fff30dbb000 /System/Library/PrivateFrameworks/AppleIDAuthSupport.framework/Versions/A/AppleIDAuthSupport
[312] 6F4B051F-BC33-3880-81B2-5C62DABFF02D 0x00007fff2dabf000 /System/Library/PrivateFrameworks/KeychainCircle.framework/Versions/A/KeychainCircle
[313] D1056292-E38F-313F-B202-EFD871195729 0x00007fff30e71000 /System/Library/PrivateFrameworks/MediaKit.framework/Versions/A/MediaKit
[314] 2B9AA549-63FD-36BB-AE05-ABD5DAC41279 0x00007fff30dc8000 /System/Library/Frameworks/DiscRecording.framework/Versions/A/DiscRecording
[315] 044E6815-6340-31D5-BEE9-760C354E92A3 0x00007fff2b268000 /usr/lib/libCoreStorage.dylib
[316] 9CDB9580-DD1C-347E-91C5-6D92D96F52F5 0x00007fff33254000 /usr/lib/libcsfde.dylib
[317] 88EF8B12-617E-316C-9036-18357E27BC0A 0x00007fff2e6bc000 /System/Library/PrivateFrameworks/ProtectedCloudStorage.framework/Versions/A/ProtectedCloudStorage
[318] B06CCD1C-A4D1-34D2-89C5-0DB69FE65234 0x00007fff3324c000 /System/Library/PrivateFrameworks/EFILogin.framework/Versions/A/EFILogin
[319] 8CF768B8-836B-3DA8-9F0A-8CF840348202 0x00007fff33841000 /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
[320] B88DEE8B-6268-36BA-BF6C-088F3BD8C652 0x00007fff30d3a000 /System/Library/Frameworks/CoreBluetooth.framework/Versions/A/CoreBluetooth
[321] DE8B7844-BFB2-3E3E-9FD0-1C9AD1DC1736 0x00007fff2b260000 /usr/lib/libMatch.1.dylib
[322] 29187EF9-2BD6-3ABF-8088-DE708721B612 0x0000000260003000 /usr/lib/libobjc-trampolines.dylib
[323] 2BC461CE-B7FA-33F3-A36B-64F54FB371B5 0x00007fff41bff000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/Libraries/libCGInterfaces.dylib
[324] B9BEC3A1-688C-3999-B705-2D50B8756E62 0x00007fff2ba0a000 /System/Library/PrivateFrameworks/login.framework/Versions/A/login
[325] 3E7403EB-156C-36C7-93AD-1356447D5310 0x00007fff4420d000 /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
[326] 62F25002-C547-3C6C-87D1-0824DD058C82 0x00007fff2da6b000 /System/Library/PrivateFrameworks/MobileAsset.framework/Versions/A/MobileAsset
[327] 157C7AC0-9F73-3FF5-A276-54EC3CAE958A 0x00007fff33866000 /System/Library/PrivateFrameworks/StreamingZip.framework/Versions/A/StreamingZip
[328] C8645EA0-643D-38CF-AA88-3253339192B1 0x00007fff2c20d000 /System/Library/Frameworks/MediaToolbox.framework/Versions/A/MediaToolbox
[329] B123866C-3945-37E7-92EF-BE36AEDD620D 0x00007fff31350000 /System/Library/PrivateFrameworks/CoreAVCHD.framework/Versions/A/CoreAVCHD
[330] 264B30DF-646A-30EC-98B6-20429E9718CB 0x00007fff30504000 /System/Library/Frameworks/MediaAccessibility.framework/Versions/A/MediaAccessibility
[331] 61919815-50C9-3358-836F-B81ED66CCD84 0x00007fff3134c000 /System/Library/PrivateFrameworks/Mangrove.framework/Versions/A/Mangrove
[332] 8713D2A4-ECB6-3B96-BE39-F47851EEF900 0x00007fff30510000 /System/Library/PrivateFrameworks/AlgosScoreFramework.framework/Versions/A/AlgosScoreFramework
[333] 4E96F781-3689-3086-9DF3-60F20487FB9E 0x00007fff31d15000 /System/Library/PrivateFrameworks/AppleVPA.framework/Versions/A/AppleVPA
[334] DAE981E0-5FED-39FD-8443-BD91D6957901 0x00007fff30f91000 /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC
[335] 9F31CEDB-5B8A-3BAE-A638-E9F352EF4E87 0x00007fff272e5000 /System/Library/PrivateFrameworks/ViewBridge.framework/Versions/A/ViewBridge
[336] 60C32CD9-3757-361B-B9EB-CB004EFBBEA5 0x0000000263ac7000 /Users/knazarov/dev/virgil/build/angle/libEGL.dylib
[337] 85790259-468E-385F-BC52-ADB02FB43BF8 0x0000000267908000 /Users/knazarov/dev/virgil/build/angle/libabsl.dylib
[338] DD16ED67-125D-35E5-958E-AACCE55249E3 0x0000000267be4000 /Users/knazarov/dev/virgil/build/angle/libc++.dylib

The reason I'm suspecting multiple libc++ versions is because I've found a related ticket from another project that has pretty much the same stacktrace I had: darlinghq/darling#879

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Essentially, libobjc depends on libc++.1.dylib. Why it isn't on the filesystem is beyond my understanding as well. But so far I believe the reasoning is solid. If there's the same stacktrace in another project related to multiple libc++ and we see multiple libc++ versions here, then it's a good start.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Yeah, I've figured out a way to test this. You can run the qemu binary with DYLD_PRINT_BINDINGS=YES env variable and see the following in the console:

...
dyld: weak bind: libc++.dylib:0x10E615B38 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED0Ev, *0x10E615B38 = 0x7FFF2057731A
dyld: weak bind: libc++.dylib:0x10E615FD0 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED0Ev, *0x10E615FD0 = 0x7FFF2057731A
dyld: weak bind: libc++.dylib:0x10E6198D8 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED0Ev, *0x10E6198D8 = 0x7FFF2057731A
dyld: weak bind: libc++.dylib:0x10E615B30 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED1Ev, *0x10E615B30 = 0x7FFF205772C2
dyld: weak bind: libc++.dylib:0x10E615FC8 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED1Ev, *0x10E615FC8 = 0x7FFF205772C2
dyld: weak bind: libc++.dylib:0x10E6198D0 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED1Ev, *0x10E6198D0 = 0x7FFF205772C2
...

The answer in this StackOverflow question is spot on. There are multiple definitions of the same symbols, and they resolve to the system's version of libc++.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

My QEMU with DYLD_PRINT_BINDINGS=YES doesn't have the bind: libc++.dylib:? = libc++.1.dylib:? nonsense. I also believe that is the very cause of the problem.

I ran image list but it's just:

(lldb) image list
[  0] D14327A5-32A0-3ACD-9C14-4EB636E6363C 0x0000000100000000 /Users/person/bin/qemu-system-aarch64 
[  1] B7DB9E8A-A898-3C11-91A0-2B0264F05CB6 0x0000000101040000 /usr/lib/dyld

I found --no-dependents option of target create was enabled by default here, so I explicitly set it false, but it didn't make any difference.

% lldb -v
lldb version 11.1.0

I really feel homesick, or gnusick. I could just info proc mappings if it is gdb.

You may check if a simple program with ANGLE works by running:

cd build/angle && ./simple_texture_2d

Or you may just replace gl=es with gl=core in run. It will use OpenGL directly so it should work without ANGLE although you'll see compatibility issues.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

I believe you see very little output from image list because you didn't run the binary. Try typing the run command in lldb, then let it run for a second and then break and type image list. You should see a complete list then.

The simple_texture_2d works fine for me.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Could you please also post the output of otool -L build/angle/libEGL.dylib?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

Actually I executed image list after run. I'm not sure what's going on.

Here is otool -L build/angle/libEGL.dylib:

objc[919]: Class AMSupportURLConnectionDelegate is implemented in both ?? (0x20e1bb8f0) and ?? (0x1204d02b8). One of the two will be used. Which one is undefined.
objc[919]: Class AMSupportURLSession is implemented in both ?? (0x20e1bb940) and ?? (0x1204d0308). One of the two will be used. Which one is undefined.
build/angle/libEGL.dylib:
	@rpath/libEGL.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libabsl.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libc++.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

I finally managed to get it running. What I did is to build angle with system libc++ like this:

gn gen '--args=use_custom_libcxx=false' ../../build/angle

Now I can run the VM and see some of the graphical output, but it's somewhat corrupted, and the machine can't run into X (hangs).

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Good news is, when using gl=core I get things working and even get decent fps in glxgears

@GlenHenshaw

This comment has been minimized.

Copy link

@GlenHenshaw GlenHenshaw commented Mar 6, 2021

I can get it to run, and even install Ubuntu, but the keyboard doesn't work. The touchpad works for about two seconds and then stops.

I know this is a newby question, but I've tried every key grab combination listed for Qemu, nothing I can figure out works. Any ideas?

EDIT:

This appears to only be an issue with the Ubuntu login screen. If I have auto login set to on, there is no problem.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

@knazarov Nice to hear you managed to get it work, at least with gl=core. The problem you see with ANGLE would need serious debugging and unfortunately I cannot help much for it since I don't own Intel Mac.

@GlenHenshaw Unfortunately I have no idea. Maybe you may change the emulated input device specified in run.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 6, 2021

@akihikodaki I'll get it running eventually.

As for the libc++, I believe you should change your build instructions to use_custom_libcxx=false. I still think the version on ARM Macs works by chance. The reason you see few libraries in lldb is likely because qemu stops on SIGUSR2 a few times in the beginning. I had it too, it looks like this:

* thread #3, stop reason = signal SIGUSR2
    frame #0: 0x00007fff205f2002 libsystem_kernel.dylib`__sigsuspend + 10
libsystem_kernel.dylib`__sigsuspend:
->  0x7fff205f2002 <+10>: jae    0x7fff205f200c            ; <+20>
    0x7fff205f2004 <+12>: movq   %rax, %rdi
    0x7fff205f2007 <+15>: jmp    0x7fff205d129d            ; cerror
    0x7fff205f200c <+20>: retq

To overcome that, you need to type continue a few times until it runs properly. Anyway, if you don't have a wish do dig deeper into this particular issue, then I believe using the system version of libc++ is safe for everyone.

As for other things, I have a few patches on top of your patchset that actually allow hvf acceleration to work on x86. There were a few problems introduced during the refactoring of hvf-accel that I fixed in my own copy, but which likely need some thought and care to fix properly. I'll send a pull request to your fork, if you don't mind.

With time, I believe it's possible to get it working on x86 just fine. There isn't much of a difference in the core functionality of your code that depends on the platform.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

@knazarov I'm still wondering if we should have use_custom_libcxx=false. ANGLE bundles libc++ for reason, of course, and I think it is expected to co-exist with libc++ from the system because:

  • darlinghq/darling#879 suggests libobjc depends on libc++,
  • ANGLE uses Objective-C++ for some macOS specific code, and
  • ANGLE does not have its own libobjc.

Therefore, if we will properly fix it, we have to allow libc++ from ANGLE and one from the system to co-exist. However, we may add use_custom_libcxx=false only for Intel if we couldn't figure out a solution.


Actually I executed:

process handle -s false -n false SIGUSR1
process handle -s false -n false SIGUSR2

And sent ^C at some arbitrary timings, so SIGUSR2 shouldn't be a problem.


If you found problems with hvf, please check if that refactoring is already in the upstream. If it is, submit a patch to the upstream. Otherwise, reply to https://patchew.org/QEMU/20210120224444.71840-1-agraf@csgraf.de/. I often rebase and force-push the results to my macos branch to ease upstreaming, so you may not send a pull request. I would appreciate if you CC Akihiko Odaki <akihiko.odaki@gmail.com> when you send patches to the upstream. Maybe I should maintain two branches, one for Apple Silicon and one for Intel.


Indeed, my changes are not really specific to Apple Silicon so it is worrying for me that you have problems I don't see on Apple Silicon. I appreciate your efforts, and I think everyone who uses QEMU on Intel Mac would benefit from them, considering some changes you are inspecting are already on its way to the upstream (punch hole change and hvf).

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

@knazarov Can you try replacing DYLD_FALLBACK_LIBRARY_PATH with DYLD_LIBRARY_PATH? Apparently libc++ from ANGLE is inspected first for simple_texture_2d so that may do some trick.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 6, 2021

I’ll try the trick with DYLD_LIBRARY_PATH when I get to a computer.

For now I’d like to point out one thing about libc++ and angle. This would be a speculation of course, because I didn’t contact the authors, so I can’t say I know their intention for sure.

The reason angle ships their own libc++ is because they are part of chrome project, and have to work on multiple operating systems. Some of those systems don’t have an onboard libc++ (e.g. Android).

The build flag for using a system-wide libc++ is completely legitimate and standard for angle. The reason it’s not set by default may have nothing to do with the intended usage in third-party apps, because the primary goal here is to make it work with Chrome I suppose.

I looked into the ninja code generated for angle and didn’t find any traces of namespacing the library, and that is the gist of my argument. If it was intended to mix multiple c++ libs this way, then I would have found the namespacing.

Anyway, if it doesn’t convince you, I may just go to their Slack channel and ask about it.

Regarding hvf, I believe the upstream in qemu works fine for x86. It may be that just your branch has something missing. I can show you the diff.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

I also think they use their own libc++ because it is used by Chromium. It is the most tested configuration and it's kind of natural for them to say "we only support our own libc++ but go ahead if you are using your own libc++ and you can deal with it".

The other variety of the platforms shouldn't have affected, as the default value of use_custom_libcxx is explicitly defined for each platforms. (See build/config/c++/c++.gni.)

Actually Firefox specifies use_custom_libcxx for ANGLE, but I suspect it has its own custom libc++, and the developers have thoroughly tested its configuration. We don't have our own libc++ and perform such QA, so it is just "safe" to use ANGLE's libc++. If ANGLE still breaks, we can file a bug for ANGLE and just say we use the default configuration of ANGLE so it should work. It something else breaks because of libc++, we may, well, just fix it.

Some changes of hvf for Apple Silicon are already merged, but the others aren't. Probably the changes not merged yet cause the problem. You may git blame or bisect and see the cause. It should be in https://patchew.org/QEMU/20210120224444.71840-1-agraf@csgraf.de/ but not in the upstream.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

@akihikodaki regarding my changes to the x86 hvf support - they are apparently fixed in latest qemu master. It's just that the patch patchew/20210120224444.71840-1-agraf@csgraf.de that you brought into your fork, didn't contain everything to make x86 work.

I've tried your suggestion with DYLD_LIBRARY_PATH, and it didn't work (basically, the same problem). I think it's about time I ask ANGLE maintainers about what to do here.

EDIT: or maybe the changes are not in the upstream and I should let the author know.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

And I went ahead and built qemu without optimizations, and now I see this:

* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x2c)
    frame #0: 0x0000000100844c2c qemu-system-x86_64`timer_mod(ts=0x0000000000000000, expire_time=58878) at qemu-timer.c:481:36
   478 	
   479 	void timer_mod(QEMUTimer *ts, int64_t expire_time)
   480 	{
-> 481 	    timer_mod_ns(ts, expire_time * ts->scale);
   482 	}
   483 	
   484 	void timer_mod_anticipate(QEMUTimer *ts, int64_t expire_time)
Target 0: (qemu-system-x86_64) stopped.
(lldb) p ts
(QEMUTimer *) $0 = 0x0000000000000000
(lldb) bt
* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x2c)
  * frame #0: 0x0000000100844c2c qemu-system-x86_64`timer_mod(ts=0x0000000000000000, expire_time=58878) at qemu-timer.c:481:36
    frame #1: 0x0000000100307ed3 qemu-system-x86_64`virtio_gpu_fence_poll(opaque=0x0000000118d581a0) at virtio-gpu-3d.c:581:9
    frame #2: 0x0000000100307e58 qemu-system-x86_64`virtio_gpu_virgl_fence_poll(g=0x0000000118d581a0) at virtio-gpu-3d.c:587:5
    frame #3: 0x00000001003054e9 qemu-system-x86_64`virtio_gpu_handle_ctrl(vdev=0x0000000118d581a0, vq=0x0000000118d60000) at virtio-gpu.c:889:9
    frame #4: 0x0000000100305388 qemu-system-x86_64`virtio_gpu_ctrl_bh(opaque=0x0000000118d581a0) at virtio-gpu.c:897:5
    frame #5: 0x000000010082bafe qemu-system-x86_64`aio_bh_call(bh=0x0000000102652bc0) at async.c:136:5
    frame #6: 0x000000010082bc54 qemu-system-x86_64`aio_bh_poll(ctx=0x00000001033088e0) at async.c:164:13
    frame #7: 0x000000010080efd7 qemu-system-x86_64`aio_dispatch(ctx=0x00000001033088e0) at aio-posix.c:381:5
    frame #8: 0x000000010082cca8 qemu-system-x86_64`aio_ctx_dispatch(source=0x00000001033088e0, callback=0x0000000000000000, user_data=0x0000000000000000) at async.c:306:5
    frame #9: 0x0000000101e3ea7c libglib-2.0.0.dylib`g_main_context_dispatch + 364
    frame #10: 0x000000010083f958 qemu-system-x86_64`glib_pollfds_poll at main-loop.c:232:9
    frame #11: 0x000000010083f3a0 qemu-system-x86_64`os_host_main_loop_wait(timeout=1701600) at main-loop.c:255:5
    frame #12: 0x000000010083f26c qemu-system-x86_64`main_loop_wait(nonblocking=0) at main-loop.c:531:11
    frame #13: 0x00000001003fbdb4 qemu-system-x86_64`qemu_main_loop at runstate.c:725:9
    frame #14: 0x000000010000bef8 qemu-system-x86_64`qemu_main(argc=22, argv=0x00007ffeefbff5c0, envp=0x00007ffeefbff678) at main.c:50:5
    frame #15: 0x0000000100029b34 qemu-system-x86_64`call_qemu_main(opaque=0x0000000000000000) at main.m:115:14
    frame #16: 0x00000001008154f9 qemu-system-x86_64`qemu_thread_start(args=0x000000010302ab50) at qemu-thread-posix.c:521:9
    frame #17: 0x00007fff20604950 libsystem_pthread.dylib`_pthread_start + 224
    frame #18: 0x00007fff2060047b libsystem_pthread.dylib`thread_start + 15

Looks like somewhere there's a problem with timer handling.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

So the last problem is because I had -vga virtio parameter passed to qemu, which in conjunction with -display cocoa,gl=es causes a crash.

The reason for this is because there are two subsequent calls to virtio_gpu_virgl_init(), only one of which succeeds. And the second one leaves g->fence_poll uninitialized. Which laters gets dereferenced in virtio_gpu_fence_poll().

Without -vga virtio I managed to get qemu running with gl=es, though it doesn't recognize my display resolution properly, and maxes out at 1280 x something. Whereas with -vga virtio and gl=off I get a full 5k resolution detected by Ubuntu.

@akihikodaki do you have any idea how I can get it to detect the resolution properly?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 7, 2021

@knazarov I think what conflicting with -vga virtio here is -device virtio-gpu-pci. -vga virtio actually means -device virtio-gpu-vga -device virtio-vga. Remove -device virtio-gpu-pci if you have.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

Yeah, you were right. Removing -device virtio-gpu-pci helps. Though, it's probably a good idea to add a few checks and print out a clear error.

I managed to get the native 5k resolution running and it's pretty sweet! The UI feels smooth.

For some reason, though, it still didn't get the correct resolution of 5120 x 2880, and only proposed 5120 x 2160. I managed to add it through xrandr manually, and get it working.

I'd like to thank you for the great work here. It's really an achievement. Let me know how I can further help with this.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 8, 2021

@knazarov I'm glad to hear you got it working with 5K. It is great, considering that Ubuntu was laggy even for much smaller display (M1 MacBook Air) without Virgil.

I'm not sure what prevents from propagating the correct resolution. virtio-gpu variants, including virtio-vga, uses EDID to report display modes to the guest. You may have a look at EDID on the guest, or by injecting printf to hw/display/edid-generate.c.

Thank you for your offer.

If you would do favor, please check the problems you see (the crash which occurs when you specify both of virtio-gpu-pci and virtio-vga, and display resolution problem) are reproducible with the upstream. If they are not reproducible, they are regressions of my changes, so I have to do some more detailed analysis. If they are reproducible, actually they should be irrelevant from my changes, but the upstream developers will appreciate if you fix them. Please CC me if you get them fixed and contribute the changes to the upstream.

The incompatibility of hvf changes with x86 is worrying for me. It may prevent the upstream from having Apple Silicon support, or the buggy patches gets merged and breaks hvf on x86 on the upstream, which is devastating. Please report the problem to the original author of the patch set to add hvf support on Apple Silicon. I'm also interested in the problem so CC me for this, too.

I'd also like to hear about any new discovery regarding ANGLE's libc++ problem.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 8, 2021

I have a fix for the x86 part ready (see https://gist.github.com/knazarov/ea99dd6fe0ce2bc4f1f67c72b1780a79). I have no experience with the qemu mailing list. How do I submit this suggestion to the author of the ARM hvf patch?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 8, 2021

@knazarov I saw your patch and found it was my mistake made when merging, not one of the original patch set.
You do not have to submit it to the upstream. I fixed the merge commit according to your patch. Thanks for the patch.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 8, 2021

Since I can't check what specifying both -vga virtio and -device virtio-gpu-pci will lead to on MacOS before your patch (because of the obvious lack of virgl in master), I used a nested virtualization and did the same on Linux. It crashed, but with a different stacktrace. I suppose it doesn't get as far as it did on the MacOS.

It looks like your patch is not the culprit. It's just likely those options should either be mutually exclusive, or there should be a better error handling. I've found a place where it just swallows the error code and doesn't return it to the caller.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 8, 2021

Thanks for the report. It seems QEMU is somewhat vulnerable to strange or exotic host configuration (my MacBook Air, for example), so it is convincing that specifying both -vga virtio and -device virtio-gpu-pci causes a problem on QEMU. However, indeed, it should be handled somehow, and should not cause crash.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 8, 2021

Turns out, the EDID in qemu is somewhat hardcoded. There is currently no way to specify a custom resolution. And to make things worse, the EDID standard is quite complex, and certain resolutions (5120x2880 in my case) require an extension that's not very well documented on the internet (of course there's the formal documentation, but it's not very readable).

By the way, I've figured out that if you pass -device usb-ehci -device usb-tablet instead of -device virtio-mouse-pci then you'll get host mouse pointer integration. Qemu will no longer have to grab your input, which is quite convenient. And the mouse input will become significantly smoother.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 9, 2021

@knazarov QEMU dynamically generates EDID detailed timing descriptor, which has highest-priority for display mode. Toggle "Zoom To Fit" in "View" menu and resize the window to see the guest dynamically responds to window size changes. If you have already choose a different display mode on the guest, choose the display mode which represents the window size.

You can use -device virtio-tablet-pci if you need a absolute pointer input device. An absolute pointer input device provides more seamless integration to the guest desktop environment, but it also makes the guest think you are using "tablet", as its device name implies, and may impose an undesired behavior. For example, gedit provides text selection UI designed for tablet, which makes it impossible to select text with mouse. A relative pointer input device (e.g. virtio-mouse-pci) is better if you use QEMU in full screen mode.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

The EDID DTD is limited to 12 bits of horizontal and vertical resolution, which is maximum 4095 pixels. My screen is 5120x2880. To add resolution this large, a separate extension should be used, called "DisplayID Extension Block".

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 9, 2021

@knazarov Ah, you're right. It is unfortunate that QEMU does not suport DisplayID.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

Here, I've implemented support for the DisplayID extension: https://gist.github.com/knazarov/40348fe805fce1fdaada11c863380109

Now Ubuntu detects my 5k resolution.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 9, 2021

@knazarov Great!

I'm pretty sure others also want the change. I would like you to submit it to the upstream, but actually there is one problem: the dynamic refresh rate definition is in review and not in the upstream yet. You may branch master, cherry-pick akihikodaki/qemu@bc4df9c and akihikodaki/qemu@92890c8, apply your changes, and add a line Based-on: <20210303152948.59943-2-akihiko.odaki@gmail.com> to your commit message.

If you are not familiar with QEMU patch submission, read: https://wiki.qemu.org/Contribute/SubmitAPatch. Please CC me when submitting a patch, of course.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

I'll send the patch, thanks for your help.

In the meantime, I have one more question. I have Caps Lock remapped as Ctrl on my Mac. My Linux VM doesn't even receive the keypress (I tried to use showkey to monitor the scancodes). If I switch it back to Caps Lock, then everything works. Do you know a place in the qemu source code where I could start investigating it? I know other people had this problem as well, so it's not a new one.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

Nevermind, I've figured it out. Pull this patch: https://gist.github.com/knazarov/3c7f1482c9904c953a39c69699c74252

There was a missing case for Right Ctrl in ui/cocoa/view.m

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 10, 2021

@knazarov Actually it was a regression in my patch (again). I applied your change and pushed to GitHub and sent the patch v2 to the upstream. Thanks again.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

Hi @akihikodaki,

I sometimes see a deadlock in view.m when starting qemu. Maybe it's something you've faced yourself. Here's the backtrace

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff205d20c6 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00007fff206022c5 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 81
    frame #2: 0x00007fff206001bc libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 211
    frame #3: 0x000000010081419d qemu-system-x86_64`qemu_mutex_lock_impl(mutex=0x0000000100fc8ff8, file="../../source/qemu/ui/cocoa/view.m", line=588) at qemu-thread-posix.c:79:11
    frame #4: 0x00000001003e90c2 qemu-system-x86_64`qemu_mutex_lock_iothread_impl(file="../../source/qemu/ui/cocoa/view.m", line=588) at cpus.c:491:5
    frame #5: 0x000000010002dba5 qemu-system-x86_64`-[QemuCocoaView handleEvent:](self=0x00000001027870f0, _cmd="handleEvent:", event=0x00000001026c8ce0) at view.m:588:5
    frame #6: 0x000000010002949d qemu-system-x86_64`-[QemuApplication sendEvent:](self=0x000000010350a570, _cmd="sendEvent:", event=0x00000001026c8ce0) at main.m:77:10
    frame #7: 0x00007fff2337af1e AppKit`-[NSApplication _handleEvent:] + 65
    frame #8: 0x00007fff22f0b6af AppKit`-[NSApplication run] + 623
    frame #9: 0x0000000100029645 qemu-system-x86_64`main(argc=23, argv=0x00007ffeefbff5c8) at main.m:166:5
    frame #10: 0x00007fff2061f621 libdyld.dylib`start + 1
    frame #11: 0x00007fff2061f621 libdyld.dylib`start + 1
@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

The reason why view.m blocks is that call_rcu_thread() tries to block the iothread to fetch events:

  thread #2
    frame #0: 0x00007fff205d20c6 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00007fff206022c5 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 81
    frame #2: 0x00007fff206001bc libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 211
    frame #3: 0x000000010081419d qemu-system-x86_64`qemu_mutex_lock_impl(mutex=0x0000000100fc8ff8, file="../../source/qemu/util/rcu.c", line=266) at qemu-thread-posix.c:79:11
    frame #4: 0x00000001003e90c2 qemu-system-x86_64`qemu_mutex_lock_iothread_impl(file="../../source/qemu/util/rcu.c", line=266) at cpus.c:491:5
    frame #5: 0x0000000100823369 qemu-system-x86_64`call_rcu_thread(opaque=0x0000000000000000) at rcu.c:266:9
    frame #6: 0x00000001008154f9 qemu-system-x86_64`qemu_thread_start(args=0x000000010262cbe0) at qemu-thread-posix.c:521:9
    frame #7: 0x00007fff20604950 libsystem_pthread.dylib`_pthread_start + 224
    frame #8: 0x00007fff2060047b libsystem_pthread.dylib`thread_start + 15

But apparently there are no events, so it holds the mutex forever, blocking view.m from progressing.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

No, I'm mistaken. The reason why this happens is because of the e1000 virtualization. It is hvf that blocks that mutex and doesn't release it. I see one of the vCPU threads blocking on sosendto(so=0x000000010357ae60, m=0x00000001058ed400). I don't get the exact reason why.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 10, 2021

Actually I had deadlocks few times, so I debugged and fixed it. -[QemuCocoaView handleEvent:] must be disabled until cocoa_display_init calls dispatch_sync, but it wasn't. Now the fix is pushed to GitHub.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

Maybe there were other deadlocks, but this one is almost surely in the network stack. I'll leave it be for now.

I have another problem with setting the resolution. I don't know what exactly happened, but after some trial and error I started to get weird behavior when I set a 5k resolution, the screen will shortly flicker to 5k, and then back to 4k. After this, the 5k resolution will disappear from the gnome control panel.

After digging through, I've figured out that EDID is apparently not the problem (the guest still has correct values, which I can pull from /sys just fine, and decode them with edid-decode without any issues).

Anyway, I reverted my qemu tree to your macos branch (without any of my changes, including EDID), and tried a few things. Of course without my patches, I can't set a 5k resolution, so I did something different. I've booted into a Wayland session, and tried to change resolution there. And whenever I change the resolution it reverts back to the previous setting after flickering a few times.

Could you please check if you have the same behavior in Wayland too?

The weirdest thing about it is that kernel modesetting and everything before the gnome session (including gdm) don't have any problems with the resolution at all.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 11, 2021

@knazarov You may not use e1000. Generally speaking, virtio-net performs better.

I don't see such a behavior with Wayland.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 11, 2021

It took me a while to figure out. It looks like the refresh rate of 75 Hz is too much for the resolution. Setting 75Hz results in clock speeds around 1.5 GHz. Xorg has a failsafe that refuses to set clock speed that high.

Why I was able to see that mode sometimes, but not always is because you figure out the refresh_rate dynamically, and sometimes it so happens that it gets below 50 Hz or so. And then the generated EDID becomes fine for Xorg. But then when it tries to set the mode, the refresh_rate again gets above 50 Hz, and it just reverts back.

I'm not sure what to do about it, maybe you have an idea. I can just hardcode the lower refresh rate, but it would raise an eyebrow of people who look at the code and see that in one place there's a dynamically set refresh rate, and in others it's hardcoded. What was your reasoning behind setting it dynamically? Does it influence anything at all for a virtual display?

With Wayland there is definitely a bug in Mutter. I've spent a significant time with a debugger and figured out the reason. It temporarily sets the mode without writing it out to the config, to check that the mode is functional, but then a hotplug event comes from udev that causes the config trigger to update the mode from the config (which unsurprisingly contains the old value). I'm not sure what to do about it and whether I should. It's definitely out of scope.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 12, 2021

@knazarov It figures out the refresh rate dynamically in the case that a window is moved across physical displays with different refresh rate.

Knowing the refresh rate of the physical display allows the guest to generate the least required numbers of frames. I also have my selfish reason: the pixel clock in the EDID detailed timing descriptor overflows if it is 75 Hz. I think X.org refuses a high pixel clock to prevent such overflows. Maybe I should have implemented DisplayID as you did or set an upper limit to prevent the overflow at least, but I just implemented dynamic refresh rate which provides the refresh rate of my display (60 Hz) because it is more efficient anyway.

Lowering the refresh rate is not an option for me because I'm upstreaming all of my changes, and it is a breaking change. Someone may actually want 75 Hz, and we cannot break things in such a situation. So I instead decided to lower the refresh rate only when the QEMU display (ui/cocoa) knows it is good for the physical display currently in use.

A problem here is that you apparently have a feedback loop. It means there are two-way interaction:

  • A QEMU display provides QemuUIInfo to the emulated device, and the emulated device (virtio-gpu) tells it to the guest. It is a way from the host to the guest.
  • Another way is the frame from the guest to the host; the guest provides a frame, and the emulated device converts it to a pixman image, which will be passed to dpy_gfx_switch function of the QEMU display. ui/cocoa resizes the window according to the image, which effectively triggers QemuUIInfo change. It is a way from the guest to the host.

The two-way interaction is designed to be immune to a feedback loop by checking there is a difference when updating QemuUIInfo, which is implemented in dpy_set_ui_info function. With a sane QEMU display and a guest, QemuUIInfo should eventually stablize, but apparently it is not the case.

So, I think the next step in debugging this is to find the code path actually leads from dpy_gfx_switch to -[QemuCocoaView updateUIInfo]. Probably observing the change of QemuUIInfo is also a good idea.

By the way, I'm wondering if it really sets a refresh rate lower than 50 Hz. I think the refresh rate of a modern display is 60 Hz or higher, and CGDisplayModeGetRefreshRate, which ui/cocoa uses to get the refresh rate, just tells the number as is. I would like to know if the value of CGDisplayModeGetRefreshRate is stable at least. You can simply add printf after CGDisplayModeGetRefreshRate to see it.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 12, 2021

With a couple of experiments, I found setting display mode with "Settings" application may indeed cause a revert, but it depends on the version.

  • On Ubuntu focal, it does not revert with X11. it reverts with Wayland.
  • On Ubuntu groovy, it reverts with X11. It does not revert with Wayland.
  • On Ubuntu hirsute, it does not revert either with X11 or with Wayland.

So you may use Wayland with Ubuntu groovy or hirsute.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

In view.m you allocate a structure QemuUIInfo on a stack, and don't memset it to 0.
Then, later you try to assign refresh_rate from the result of CGDisplayModeGetRefreshRate if that's not zero. But in my case the result of CGDisplayModeGetRefreshRate is always zero, which leads to info.refresh_rate containing random stuff from the stack.

It's not clear why on my machine the result from CGDisplayModeGetRefreshRate is always zero...

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

Some people say that CGDisplayModeGetRefreshRate is unreliable, and it's better to use CVDisplayLinkGetActualOutputVideoRefreshPeriod. See this for example: glfw/glfw#137

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

Or alternatively, it should be possible to use iokit directly, which is what DisplayLink is doing under the hood.
Maybe this property will do: https://developer.apple.com/documentation/iokit/iodisplaymodeinformation

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 12, 2021

@kanazarov I pushed a fix. Please test it.

I choose Core Video, not I/O Kit, because CGDisplayIOServicePort is deprecated and apparently CVDisplayLink is designed for this kind of purpose, considering CVDisplayLinkCreateWithOpenGLDisplayMask is present. OpenGL rendering relies on CALayer and CADisplayLink seems the best fit for this case, but it was not available on macOS.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

@akihikodaki yeah, it works for me now.

And I figured out the last bit about the Xorg not accepting the mode. It turns out the current EDID settings hardcoded a maximum dot clock to 1200 MHz. I changed it to the appropriate maximum of 2550 MHz, and it was finally accepted. I'm now ready to submit a patch to qemu.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 14, 2021

@knazarov I reviewed the patch. Please check email. It looks fine for me, but did some style nitpicks.

I pushed new changes to GitHub. Briefly:

  • It was rebased to master of the upstream.
  • The fix of the deadlock of -[QemuCocoaView handleEvent:] and cocoa_display_init was incomplete so fixed it again.
@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 14, 2021

@akihikodaki I've just sent corrections to my previous patch according to your comments.

By the way, do you intend to make the qemu window resizable? I only have the ability to make it full-screen and back. In my case there are no active corners that I can drag.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 14, 2021

@knazarov It is now Reviewed-by: Akihiko Odaki <akihiko.odaki@gmail.com>.

Yes, it should be resizable but only if checking "Zoom To Fit" in "View" menu. Also use the top left or top right corner to drag because clicks on the bottom corners are often reacted by the guest.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 14, 2021

Regarding the hanging sendto, I've found a very interesting scenario.

The MacOS reserves .local domain for Bonjour mDNS (looking up nearby devices). On the other hand, in ubuntu there's Avahi that does the same thing. When starting up, Avahi tries to locate qemu.local, and that's when sometimes the sendto call will hang. I managed to workaround it by either disabling ipv6 in qemu, or by changing the default domain for Avahi from local to lan.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 15, 2021

@knazarov I applied a patch to enable the vmnet framework provided by macOS to my tree. You may try how Avahi behaves with it. It should perform better thanks to network stacks integrated into the kernel. See run.sh for the command line with vmnet.

Note that it requires root when initializing. You can drop root before starting the VM by specifying runas option.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 15, 2021

@akihikodaki I can live without ipv6 or Avahi at the moment. The proper way to solve this problem will be to isolate it as a small piece of userspace C code and open a bug as an Apple "radar".

The bug happens when sending a broadcast ipv6 mDNS request, so it's very likely somewhere inside the kernel. The reason qemu hangs here is that slirp doesn't account for the fact that sendto may block for UDP, because almost always it doesn't. The completely proper way of course will be to teach slirp to operate UDP sockets with O_NONBLOCK. That wouldn't be trivial though, so I will delay it for later, until the most important things are complete.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 15, 2021

I see. slirp is the most convenient backend anyway and getting it proper working with IPv6 is still great.

@manu-romero-411

This comment has been minimized.

Copy link

@manu-romero-411 manu-romero-411 commented Apr 10, 2021

Hello @akihikodaki, I've tried your QEMU patch on an M1 MacBook Pro, but when trying to run some graphics demanding job (like the WebGL aquarium) my Debian guest system lags like hell, and I can't figure why. Is it a normal thing in this QEMU build? Thanks in advance

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 10, 2021

@manu-romero-411 You may use gl=off without running a demanding job as the baseline of performance measurements. On my M1 MacBook Air, Fedora 33 running the WebGL aquarium with gl=es is a bit less laggy than one with gl=off.
There are different factors contributing to worse performance. QEMU build configuration is an obvious suspect. Another possible cause is the graphics stack on the guest including the kernel driver and Mesa userspace library. Debian may have somewhat older software compared with Fedora and Ubuntu, which I tested.

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 22, 2021

Thanks a lot for your work! I think I'm finally getting there for my local setup. Unfortunately Fedora 34 very quickly gets unusable, windows go fully transparent and things like that, so I'll stick to 33 for now.

TL;DR for the below: how do you get qemu to grab the keyboard so alt-tab doesn't switch away from it but switches windows inside gnome instead.

One question: my plan is to do like you @akihikodaki, and run my Gnome desktop as my main interface for my Mini while we don't have Asahi. But Mac OS X seems to be getting some of the important keyboard shortcuts like alt-tab/super-tab, so I'm having to use the mouse a lot more than I normally do. Do you know of a way to make qemu grab everything and only let Mac OS get shortcuts after you release the grab with C-A-g?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 23, 2021

@kov My tree contains a fix for Fedora 34, but apparently it is not sufficient in the case. I mainly use Fedora 33 and it is the most tested and recommended version, but I would also appreciate if you share how you reproduce the problem. (You say it "quickly" gets unusable. Can you detail when it actually goes wrong?) I will probably switch to Fedora 34 once its stable version is released, and need a fix for the problem you face now.

I actually run Fedora 33 for major time I use my MacBook Air. (Actually I'm writing this on Fedora.) On my environment, Alt-Tab is grabbed by QEMU while Super-Tab is grabbed by macOS. The exact procedure of README.md (hopefully) creates the same environment which I have.

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 23, 2021

@akihikodaki interesting, so I guess it must be due to keyboard differences, I'm using a Mini with an external USB keyboard. I ran exactly your run.sh script and used your run command line with just a couple changes which should not affect this (I use qcow2, and I use bridged networking, so I can ssh into and from the VM to my work laptop). I think I may do some hacks to the qemu key mapping code and to its grabbing logic just to fit my needs before I learn a bit more about how this works.

By the way, I think I will need to enable and use spice to get the VM to use multiple screens, right? So maybe I'll need to use something like virt-viewer anyway, and maybe that will work better, we'll see.

As to your question regarding Fedora 34: I install Silverblue 34, then I rpm-ostree upgrade, the next boot the login screen already looks weird, with the corners of the selection on the user name not being drawn. If I login, the system menu is partially transparent, and any apps I start have fully transparent windows.

By the way, also writing this from Epiphany on Fedora 33 running on a VM based on your work, thank you so much again =). Firefox crashes on startup, which I think is fixed in Fedora 34. Mind sharing what you are using as your main browser? I like Epiphany, I used to be a contributor to it and WebKitGTK+, so may as well stay with it.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 23, 2021

@kov Thanks for sharing detailed information about the keyboard issue and the Fedora 34 glitch. I'll try them when I have some time. I currently use a keyboard MacBook Air has so I'll try with Lenovo TrackPoint keyboard KU-1255, which has Alt key, not Option.

Actually I use Firefox. I have no idea about the cause of the difference from your environment.

$ rpm-ostree status
State: busy
Transaction: upgrade (download only)
  Initiator: client(id:gnome-software dbus:1.70 unit:app-gnome-gnome\x2dsoftware\x2dservice-1303.scope uid:1000)
Deployments:
● ostree://fedora:fedora/33/aarch64/silverblue
                   Version: 33.20210421.0 (2021-04-21T21:26:41Z)
                BaseCommit: a95f84567dddaa49da604c72c80cd325fda0ec5a0deff19806da64585453f8af
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: cargo code fedora-workstation-repositories 'gcc-c++' glmark2 libsecret-devel make mozilla-openh264 nodejs protobuf-devel redhat-rpm-config ruby-devel rubygem-rake sqlcipher-devel
                            thunderbird yarnpkg

  ostree://fedora:fedora/33/aarch64/silverblue
                   Version: 33.20210416.0 (2021-04-16T14:20:37Z)
                BaseCommit: fb1e1932c03e3ef4b7f488886a54156e42564b93ab58faeae09e08e9331596a6
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: cargo code fedora-workstation-repositories 'gcc-c++' glmark2 libsecret-devel make mozilla-openh264 nodejs protobuf-devel redhat-rpm-config ruby-devel rubygem-rake sqlcipher-devel
                            thunderbird yarnpkg
$ rpm -q firefox
warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
firefox-88.0-1.fc33.aarch64
@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 23, 2021

FWIW, I have a Keychron 2 v2 with the switch set to Mac, so in theory I also have the standard Control/Option/Command keys. Another issue I've been having is Option and Command seem to be mostly ignored. I need to use the right-side command key to get the overview to open, and if I try to use keybindings like alt-f on my fish or bash shells it just types an 'f'. Poking at xev this is what it sees when I press option and then command:

KeyPress event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141168, (54,40), root:(607,775),
    state 0x0, keycode 108 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141233, (54,40), root:(607,775),
    state 0x80, keycode 108 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141825, (54,40), root:(607,775),
    state 0x0, keycode 134 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141936, (54,40), root:(607,775),
    state 0x80, keycode 134 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

About F33 and Firefox, let's see:

kov@localhost ~> rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/33/aarch64/silverblue
                   Version: 33.20210421.0 (2021-04-21T21:26:41Z)
                BaseCommit: a95f84567dddaa49da604c72c80cd325fda0ec5a0deff19806da64585453f8af
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: code epiphany evolution fira-code-fonts fish 'g++' gcc gem
                            gstreamer1-plugin-openh264 iftop iotop langpacks-en make mosh
                            mozilla-openh264 npm openssl-devel redhat-rpm-config ruby-devel strace
                            telnet

  ostree://fedora:fedora/33/aarch64/silverblue
                   Version: 33.20210421.0 (2021-04-21T21:26:41Z)
                BaseCommit: a95f84567dddaa49da604c72c80cd325fda0ec5a0deff19806da64585453f8af
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: code epiphany fira-code-fonts fish 'g++' gcc gem
                            gstreamer1-plugin-openh264 iftop iotop langpacks-en make mosh
                            mozilla-openh264 npm openssl-devel redhat-rpm-config ruby-devel strace
                            telnet
kov@localhost ~> rpm -q firefox
warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
firefox-88.0-1.fc33.aarch64
kov@localhost ~> firefox
fish: Job 1, 'firefox' terminated by signal SIGILL (Illegal instruction)
kov@localhost ~ [SIGILL]>

That's so weird... I wonder if there are small subtle differences between the Mini and the Air that are harming us here, or if it's something I changed in my Mac OS X configuration.... I may decide to reset my Mac OS installation if I can't figure it out, we'll see. This is the command line I'm running (well, ./run is running - added some newlines for easier reading):

kov@Gustavos-Mini /V/E/src> ps aux | grep qemu
kov              19679 144.8 12.3 414477968 2070064 s001  R+    7:27AM  19:59.64 ./bin/qemu-system-aarch64
-machine virt,accel=hvf,highmem=off -cpu cortex-a72 -smp 8 -m 4096 -device intel-hda -device hda-output
-device virtio-gpu-pci -device virtio-net-device,netdev=net -device qemu-xhci -device usb-kbd -device usb-tablet
-k en-us -display cocoa,gl=es -drive if=pflash,format=raw,file=./edk2-aarch64-code.fd,readonly=on
-drive if=pflash,format=raw,file=./edk2-arm-vars.fd,discard=on -drive if=virtio,format=qcow2,file=./Fedora.qcow2,discard=on
-netdev vmnet-macos,id=net,mode=bridged,ifname=en0 -full-screen -runas 501:20

Right now I'm running with a USB kdb and tablet devices and explicitly added -k en-us to see if they would improve anything as you can see, but no dice.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 23, 2021

@kov Thanks for additional information.

Firefox says SIGILL? I'm really curious what is going on. Can you share a core dump or something with more information?

By the way, I failed to answer to your SPICE idea so here is my thought.

My patches for QEMU does not allow to use egl-headless, which enables OpenGL acceleration for SPICE, on macOS, but it should be straightforward to implement it.

Another problem is that SPICE protocol requires to copy a frame buffer, which is very expensive. Linux allows efficient IPC via DMA-BUF and SPICE protocol. macOS also allows IOSurface passing via Mach port, but supporting it requires some modification for SPICE protocol. I decided not to implement such a change because designing a convincing protocol for macOS-specific feature should be somewhat difficult.

Another approach is to modify the SDL display implementation, which spawns different windows for different displays. Actually I first tried to use SDL instead of Cocoa because it already has OpenGL acceleration with X11. I abandoned the idea when I found its default configuration of desktop OpenGL is incompatible with the code base and configuring it correctly requires some code modification anyway, but a chance is that using ANGLE with SDL is not that complicated.

Modifying Cocoa display implementation and letting it create multiple windows is an obvious solution. Its code is a bit messy, however.

It requires some hacking whatever approach you choose. Improving SDL display implementation should be the most easiest and probably only needs editions of tens of lines of code.

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 23, 2021

@akihikodaki thanks for your reply! For the firefox issue, I found this while researching it a bit, which is why I didn't dig into it very much:

https://bugzilla.mozilla.org/show_bug.cgi?id=1609535

They claim it fixed itself in Debian, so likely a transient compiler issue. Also, since I know it doesn't crash straight away on Fedora 34 I am not sure it is worth spending the time, maybe I'll just try and do a selective upgrade. But I can try and grab a decent core dump for you later today.

Thanks for all your help and insights. Being a Collabora contractor I have some colleagues who definitely know about this stuff - I know we did a lot of work on virgl for instance, but I have never worked on it myself, as I am on a different technical domain inside the company heh, so I am very much a noob in terms of qemu, virgl, spice (though I did do some work with gbm and dmabuf).

That said, I am getting so interested in this that I may try and do a proper fix, probably improving the SDL or Cocoa (I know enough Objective-C from my time as a WebKitGTK+ contributor heh) implementations as you suggest, which are closer to my own technical expertise. I don't think I'll have time right away, but I have a sabbatical coming up early-July to mid-Aug, so maybe I'll slowly learn things meanwhile and try and do it during that time.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 23, 2021

@kov I agree that it is likely to be a temporary compiler issue. (It can even be the same problem.)

I really appreciate works from Collabora. Virgl is one of them. I even wonder it could be useful to use macOS's graphics stack on Asahi Linux with some virtualization. Apparently graphics stack from Asahi Linux will provide Vulkan so Zink may become a requirement.

I also have something to be done like upstreaming, possible USB support, fixing occasional crashes, etc, including investigation of your reports, of course. Hopefully I'll work on them in the future.

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 24, 2021

@akihikodaki hah, I figured out the keyboard issue. I had checked several times that I had rolled back the switching of option and command to each other in Mac OS X. Turns out the settings detects my mouse as a keyboard as well and that is the one it was showing me by default and thus the one I was checking. I fixed the associations of the modifiers and now I have the same behaviour as yours!

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 24, 2021

@akihikodaki I made a couple of screenshots of what it looks like if you rebase to Fedora 34: https://cloud.kov.eti.br/s/casx5DAFKC5y9qb

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 26, 2021

@kov I'm glad to hear that the keyboard issue is not caused by a convoluted event handling of Cocoa.

Thanks for the screenshots. I'll refer to them when I try to replicate the issue.

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 27, 2021

@akihikodaki I just realized today when I was about to submit a couple patches upstream that you are the one doing the big rework on ui/cocoa, nice work there! I submitted my changes to your branch instead as a PR. I don't want to rebase them on master as it will conflict with your work, so either we upstream them along with your changes or I'll wait for 'ui/cocoa: Divide ui/cocoa.m' before submitting them.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 28, 2021

@kov I fixed the graphics glitch of Fedora 34 and pushed relevant changes. (Well, I lied. Actually the fixes are made by Ryan Neph https://gitlab.freedesktop.org/ryanneph. Apparently the problem is universal among OpenGL ES hosts, and people working on Chrome OS are facing it. He is still working on it, but fortunately we only need a tiny part of the effort because ANGLE on macOS supports GL_EXT_texture_format_BGRA8888.)

I wrote my opinion how it should be merged and gave some review on the pull request. Please check them out.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 28, 2021

(By the way, I have just moved my main environment to Fedora 34 Silverblue. A user should expect the distribution is the best supported from now.)

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 28, 2021

I'm facing a problem that the sound output is broken and apparently the subsystem is causing crashes on Fedora 34. I wonder if it has a problem with PipeWire. I recommend to stay Fedora 33, disable audio, or fix it yourself ;)

@lbibass

This comment has been minimized.

Copy link

@lbibass lbibass commented Apr 28, 2021

Hi, this unfortunately seems to be unable to compile on macOS 11.3. When attempting to compile QEMU, it fills with errors.
It does seem like these errors could be the result of problems with the initial configuration script.

depot_tools update failed. Conflict in /Users/lbibass/git/gl-qemu/depot_tools
error: pathspec 'origin/main' did not match any file(s) known to git
WARNING: Your metrics.cfg file was invalid or nonexistent. A new one will be created.
created .gclient
+ gclient sync
depot_tools update failed. Conflict in /Users/lbibass/git/gl-qemu/depot_tools
error: pathspec 'origin/main' did not match any file(s) known to git
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:31:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/utils.h:31:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cmath:308:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/math.h:310:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/limits:121:
../../source/qemu/version:1:1: error: expected unqualified-id
5.2.50
^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:31:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/utils.h:31:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cmath:309:
../../source/qemu/version:1:1: error: expected unqualified-id
5.2.50
^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:668:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/typeinfo:60:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/exception:83:
../../source/qemu/version:1:1: error: expected unqualified-id
5.2.50
^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:671:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/new:93:
../../source/qemu/version:1:1: error: expected unqualified-id
5.2.50
^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:205:
../../source/qemu/version:1:1: error: expected unqualified-id
5.2.50
^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:107:48: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
size_t  mbrlen(const char * __restrict, size_t, mbstate_t * __restrict);
                                                ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:109:6: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
            mbstate_t * __restrict);
            ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:110:19: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
int     mbsinit(const mbstate_t *);
                      ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:112:6: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
            mbstate_t * __restrict);
            ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:123:44: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
size_t  wcrtomb(char * __restrict, wchar_t, mbstate_t * __restrict);
                                            ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:139:6: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
            mbstate_t * __restrict);
            ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:196:21: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
            size_t, mbstate_t * __restrict);
                    ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:204:21: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
            size_t, mbstate_t * __restrict);
                    ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:189:14: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
typedef fpos<mbstate_t>    streampos;
             ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:190:14: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
typedef fpos<mbstate_t>    wstreampos;
             ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:195:14: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
typedef fpos<mbstate_t>    u16streampos;
             ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
In file included from ../../source/qemu/disas/libvixl/vixl/a64/disasm-a64.cc:28:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/disasm-a64.h:33:
In file included from /Users/lbibass/git/gl-qemu/source/qemu/disas/libvixl/vixl/a64/decoder-a64.h:30:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/list:185:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:672:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/utility:206:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__debug:14:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:196:14: error: unknown type name 'mbstate_t'; did you mean '__mbstate_t'?
typedef fpos<mbstate_t>    u32streampos;
             ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:55:3: note: '__mbstate_t' declared here
} __mbstate_t;
  ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
[2234/6280] Compiling C object libcommon.fa.p/migration_block-dirty-bitmap.c.o
ninja: build stopped: subcommand failed.
Could not rebuild .```

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 28, 2021

@akihikodaki that's awesome! I'll probably switch my own system to F34 ASAP then! I don't really have much need for sound on the VM and I wouldn't mind a bit of Pipewire debugging tbh. Unfortunately I'm also hitting the issue @lbibass mentions above after upgrading to 11.3. I tried messing around with Xcode versions and Command Line Tools versions to no avail. Even changing the symlinks directly to point MacOSX.sdk to the previous versions didn't work.

I find this one particularly funny:

In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/limits:121:
../../source/qemu/version:1:1: error: expected unqualified-id
5.2.50
^

It's finding qemu's version file which is definitely not a header instead of /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/version which is right there on the same directory as the file including it!

Anyway, thanks for the reviews, I'll work on upstreaming my patches as soon as I figure this mess out.

@kov

This comment has been minimized.

Copy link

@kov kov commented Apr 29, 2021

@lbibass I managed to fix the problem above by removing Xcode and /Library/Developer/CommandLineTools and then installing Xcode 12.4 downloaded from here (you'll need to create an app dev account if you don't have it yet): https://developer.apple.com/download/more/?=12.4

Finally, I ran sudo xcode-select -s /Applications/Xcode.app for good measure. I think upgrading Xcode may not have been the problem, but the OS upgrade installing these separate ones in /Library/Developer, but I'm not sure. I will leave it as is for now and see if I can safely upgrade Xcode later.

@lbibass

This comment has been minimized.

Copy link

@lbibass lbibass commented Apr 29, 2021

@lbibass I managed to fix the problem above by removing Xcode and /Library/Developer/CommandLineTools and then installing Xcode 12.4 downloaded from here (you'll need to create an app dev account if you don't have it yet): https://developer.apple.com/download/more/?=12.4

Finally, I ran sudo xcode-select -s /Applications/Xcode.app for good measure. I think upgrading Xcode may not have been the problem, but the OS upgrade installing these separate ones in /Library/Developer, but I'm not sure. I will leave it as is for now and see if I can safely upgrade Xcode later.

That's good to know. I'm downloading it now, and I'll let you know if it works. :)

@lbibass

This comment has been minimized.

Copy link

@lbibass lbibass commented Apr 29, 2021

@lbibass I managed to fix the problem above by removing Xcode and /Library/Developer/CommandLineTools and then installing Xcode 12.4 downloaded from here (you'll need to create an app dev account if you don't have it yet): https://developer.apple.com/download/more/?=12.4
Finally, I ran sudo xcode-select -s /Applications/Xcode.app for good measure. I think upgrading Xcode may not have been the problem, but the OS upgrade installing these separate ones in /Library/Developer, but I'm not sure. I will leave it as is for now and see if I can safely upgrade Xcode later.

That's good to know. I'm downloading it now, and I'll let you know if it works. :)

No go. I'm still getting the exact same error as before.

EDIT: Note to self: Don't manually specify an LLVM in my path, and don't re-install the command line tools.

It's compiling now. Thanks so much for the help!

EDIT 2: I was overzealous.

/Users/lbibass/git/gl-qemu/source/qemu/include/qemu/osdep.h:58:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>

I think I need the xcode tools. :(

EDIT 3: it's working!

@zoain-it

This comment has been minimized.

Copy link

@zoain-it zoain-it commented Apr 30, 2021

@lbibass I managed to fix the problem above by removing Xcode and /Library/Developer/CommandLineTools and then installing Xcode 12.4 downloaded from here (you'll need to create an app dev account if you don't have it yet): https://developer.apple.com/download/more/?=12.4

Finally, I ran sudo xcode-select -s /Applications/Xcode.app for good measure. I think upgrading Xcode may not have been the problem, but the OS upgrade installing these separate ones in /Library/Developer, but I'm not sure. I will leave it as is for now and see if I can safely upgrade Xcode later.

Is there a way to replace CommandLineTools that doesn't currently work?

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Apr 30, 2021

I’ve made a workaround for xcode 12.4 in my brew formula. If you want to apply the patch yourself, grab it here. Currently just installing from the formula should do the trick.

If someone has a better solution for xcode 12.4, I’d be glad to hear it.

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 3, 2021

@knazarov
Do I need to disable SIP to install with the brew formula? If so, is there any way to get the angle part precompiled somewhere and drop it in the right place?

As a side note, I have seen many projects that simply needed to install somewhere that isn't protected (like /usr/local). Is that the case here, or is it more complex?

@Lunaticf

This comment has been minimized.

Copy link

@Lunaticf Lunaticf commented May 3, 2021

@lbibass I managed to fix the problem above by removing Xcode and /Library/Developer/CommandLineTools and then installing Xcode 12.4 downloaded from here (you'll need to create an app dev account if you don't have it yet): https://developer.apple.com/download/more/?=12.4
Finally, I ran sudo xcode-select -s /Applications/Xcode.app for good measure. I think upgrading Xcode may not have been the problem, but the OS upgrade installing these separate ones in /Library/Developer, but I'm not sure. I will leave it as is for now and see if I can safely upgrade Xcode later.

That's good to know. I'm downloading it now, and I'll let you know if it works. :)

No go. I'm still getting the exact same error as before.

EDIT: Note to self: Don't manually specify an LLVM in my path, and don't re-install the command line tools.

It's compiling now. Thanks so much for the help!

EDIT 2: I was overzealous.

/Users/lbibass/git/gl-qemu/source/qemu/include/qemu/osdep.h:58:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>

I think I need the xcode tools. :(

EDIT 3: it's working!

same compile error,install xcode not working.is there anything else?

@kov

This comment has been minimized.

Copy link

@kov kov commented May 5, 2021

@lbibass good to know it's working!

@akihikodaki I was wondering if you have played with suspending/hibernating the VM. Using qemu's savevm seems to be out of the question as it complains virgl isn't migratable yet, but I wonder if it's possible to have the guest itself hibernate. My attempts with systemctl hibernate hang after the processes are frozen.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented May 5, 2021

@knazarov
Do I need to disable SIP to install with the brew formula? If so, is there any way to get the angle part precompiled somewhere and drop it in the right place?

As a side note, I have seen many projects that simply needed to install somewhere that isn't protected (like /usr/local). Is that the case here, or is it more complex?

It shouldn’t need to disable SIP. If you have problems with it, I can precompile libangle for Big Sur.

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 5, 2021

@knazarov
Do I need to disable SIP to install with the brew formula? If so, is there any way to get the angle part precompiled somewhere and drop it in the right place?
As a side note, I have seen many projects that simply needed to install somewhere that isn't protected (like /usr/local). Is that the case here, or is it more complex?

It shouldn’t need to disable SIP. If you have problems with it, I can precompile libangle for Big Sur.

Awesome! Now I just need to go through the process of switching back to homebrew.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 7, 2021

@kov No, I haven't tried suspending/hibernating. I have no idea if it works or how to fix it if it doesn't.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented May 7, 2021

@akihikodaki Do you plan to rebase your patchset onto the latest qemu release branch? There are important fixes for x86 that improve the hvf support.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 7, 2021

@knazarov I have just rebased to master (akihikodaki/qemu@d90f154). It also adds a patch for the build failure with the latest Xcode (akihikodaki/qemu@c1db57c).

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 9, 2021

@knazarov
Do I need to disable SIP to install with the brew formula? If so, is there any way to get the angle part precompiled somewhere and drop it in the right place?
As a side note, I have seen many projects that simply needed to install somewhere that isn't protected (like /usr/local). Is that the case here, or is it more complex?

It shouldn’t need to disable SIP. If you have problems with it, I can precompile libangle for Big Sur.

Ya, I'm not going to be able to use homebrew. I am stuck on Macports. Would you mind posting a build of libangle for M1 Big Sur and where I should put it?

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented May 10, 2021

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 11, 2021

@knazarov Thanks for the build.

Does this include the GUI where I can manage machines? I intend to use this with VMs provided by Whonix.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented May 11, 2021

@knazarov Thanks for the build.

Does this include the GUI where I can manage machines? I intend to use this with VMs provided by Whonix.

This build is only for libangle (the library that may require disabling system integrity protection to build). The rest you either have to build yourself, following the guide in this readme, or install using the homebrew formula.

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 11, 2021

getting some errors on MacOS 11.3.1 M1

Undefined symbols for architecture arm64:
  "_sasl_decode", referenced from:
      _vnc_client_read_sasl in ui_vnc-auth-sasl.c.o
  "_sasl_dispose", referenced from:
      _vnc_sasl_client_cleanup in ui_vnc-auth-sasl.c.o
      _start_auth_sasl in ui_vnc-auth-sasl.c.o
      _protocol_client_auth_sasl_start in ui_vnc-auth-sasl.c.o
      _protocol_client_auth_sasl_step in ui_vnc-auth-sasl.c.o
  "_sasl_encode", referenced from:
      _vnc_client_write_sasl in ui_vnc-auth-sasl.c.o
  "_sasl_errdetail", referenced from:
      _start_auth_sasl in ui_vnc-auth-sasl.c.o
      _protocol_client_auth_sasl_start in ui_vnc-auth-sasl.c.o
      _protocol_client_auth_sasl_step in ui_vnc-auth-sasl.c.o
  "_sasl_errstring", referenced from:
      _vnc_display_open in ui_vnc.c.o
      _start_auth_sasl in ui_vnc-auth-sasl.c.o
      _vnc_auth_sasl_check_access in ui_vnc-auth-sasl.c.o
  "_sasl_getprop", referenced from:
      _vnc_auth_sasl_check_ssf in ui_vnc-auth-sasl.c.o
      _vnc_auth_sasl_check_access in ui_vnc-auth-sasl.c.o
  "_sasl_listmech", referenced from:
      _start_auth_sasl in ui_vnc-auth-sasl.c.o
  "_sasl_server_init", referenced from:
      _vnc_display_open in ui_vnc.c.o
  "_sasl_server_new", referenced from:
      _start_auth_sasl in ui_vnc-auth-sasl.c.o
  "_sasl_server_start", referenced from:
      _protocol_client_auth_sasl_start in ui_vnc-auth-sasl.c.o
  "_sasl_server_step", referenced from:
      _protocol_client_auth_sasl_step in ui_vnc-auth-sasl.c.o
  "_sasl_setprop", referenced from:
      _start_auth_sasl in ui_vnc-auth-sasl.c.o
ld: symbol(s) not found for architecture arm64

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 11, 2021

@knazarov Thanks for the build.
Does this include the GUI where I can manage machines? I intend to use this with VMs provided by Whonix.

This build is only for libangle (the library that may require disabling system integrity protection to build). The rest you either have to build yourself, following the guide in this readme, or install using the homebrew formula.

sorry, the second question was about the qemu build.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 12, 2021

There is no GUI.

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 14, 2021

Should I install gnu sasl or cyrus sasl library?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 14, 2021

@phirestalker I don't think so. You may disable SASL (or VNC) and try again.

@Transfusion

This comment has been minimized.

Copy link

@Transfusion Transfusion commented May 16, 2021

Has anyone tried to get USB passthrough with libusb? Even aarch64 linux has better support for devices like 2FA USB token, 4G USB modem, etc. compared to Big Sur (11.3.1). I built QEMU with libusb 1.0.24 from Homebrew (no patches), and read this. I tried an ASIX AX88772C USB to LAN adapter since Big Sur does not have any kexts for it, with -device usb-host,vendorid=0x0b95,productid=0x772b,guest-reset=on.

It shows up in the guest VM (ubuntu-20.04.2-live-server-arm64), but above the USB layer nothing is working - I cannot get a DHCP lease, for example. On the other end, I have a linux box running dhcpd. I ran Wireshark on the interface that dhcpd is listening on, and I did not see any DHCPDISCOVER packets at all.

Just to be sure, I disabled SIP and did sudo nvram 40A0DDD2-77F8-4392-B4A3-1E7304206516:boot-args=amfi_get_out_of_my_way=1 to disable all entitlement checking. It seems that this entitlement is related, so I self-signed the qemu-system-aarch64 binary with this too; it would get Killed by amfid, so I disabled entitlement checking, because the only way to get com.apple.vm.* entitlements is to contact Apple... Anyways, I don't think it's related since libusb is not using IOUSBHost at the moment.

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 16, 2021

FWIW, I have a Keychron 2 v2 with the switch set to Mac, so in theory I also have the standard Control/Option/Command keys. Another issue I've been having is Option and Command seem to be mostly ignored. I need to use the right-side command key to get the overview to open, and if I try to use keybindings like alt-f on my fish or bash shells it just types an 'f'. Poking at xev this is what it sees when I press option and then command:

KeyPress event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141168, (54,40), root:(607,775),
    state 0x0, keycode 108 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141233, (54,40), root:(607,775),
    state 0x80, keycode 108 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141825, (54,40), root:(607,775),
    state 0x0, keycode 134 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 37, synthetic NO, window 0x800001,
    root 0x53d, subw 0x800002, time 141936, (54,40), root:(607,775),
    state 0x80, keycode 134 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 92
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

About F33 and Firefox, let's see:

kov@localhost ~> rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/33/aarch64/silverblue
                   Version: 33.20210421.0 (2021-04-21T21:26:41Z)
                BaseCommit: a95f84567dddaa49da604c72c80cd325fda0ec5a0deff19806da64585453f8af
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: code epiphany evolution fira-code-fonts fish 'g++' gcc gem
                            gstreamer1-plugin-openh264 iftop iotop langpacks-en make mosh
                            mozilla-openh264 npm openssl-devel redhat-rpm-config ruby-devel strace
                            telnet

  ostree://fedora:fedora/33/aarch64/silverblue
                   Version: 33.20210421.0 (2021-04-21T21:26:41Z)
                BaseCommit: a95f84567dddaa49da604c72c80cd325fda0ec5a0deff19806da64585453f8af
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: code epiphany fira-code-fonts fish 'g++' gcc gem
                            gstreamer1-plugin-openh264 iftop iotop langpacks-en make mosh
                            mozilla-openh264 npm openssl-devel redhat-rpm-config ruby-devel strace
                            telnet
kov@localhost ~> rpm -q firefox
warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
firefox-88.0-1.fc33.aarch64
kov@localhost ~> firefox
fish: Job 1, 'firefox' terminated by signal SIGILL (Illegal instruction)
kov@localhost ~ [SIGILL]>

That's so weird... I wonder if there are small subtle differences between the Mini and the Air that are harming us here, or if it's something I changed in my Mac OS X configuration.... I may decide to reset my Mac OS installation if I can't figure it out, we'll see. This is the command line I'm running (well, ./run is running - added some newlines for easier reading):

kov@Gustavos-Mini /V/E/src> ps aux | grep qemu
kov              19679 144.8 12.3 414477968 2070064 s001  R+    7:27AM  19:59.64 ./bin/qemu-system-aarch64
-machine virt,accel=hvf,highmem=off -cpu cortex-a72 -smp 8 -m 4096 -device intel-hda -device hda-output
-device virtio-gpu-pci -device virtio-net-device,netdev=net -device qemu-xhci -device usb-kbd -device usb-tablet
-k en-us -display cocoa,gl=es -drive if=pflash,format=raw,file=./edk2-aarch64-code.fd,readonly=on
-drive if=pflash,format=raw,file=./edk2-arm-vars.fd,discard=on -drive if=virtio,format=qcow2,file=./Fedora.qcow2,discard=on
-netdev vmnet-macos,id=net,mode=bridged,ifname=en0 -full-screen -runas 501:20

Right now I'm running with a USB kdb and tablet devices and explicitly added -k en-us to see if they would improve anything as you can see, but no dice.

EDIT: haha, never mind I figured it out.

Would you mind explaining bridge mode settings a little for me? I can just copy and paste your -device virtio-net-device,netdev=net instead of -device virtio-net-pci,netdev=net and replace my -netdev line with yours right? Also, what does ifname refer to? Is it the name of the host interface name to attach to, or the name given to the guest's interface?

@phirestalker

This comment has been minimized.

Copy link

@phirestalker phirestalker commented May 16, 2021

Is there any way to get a shared clipboard working with this? I know spice has one, but that won't work here.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 17, 2021

@phirestalker You may simply use an external application like Synergy or Barrier. They technically should provide the same thing.

@lbibass

This comment has been minimized.

Copy link

@lbibass lbibass commented May 17, 2021

I'm currently getting this error at the moment.


goroutine 1:
#0 go.chromium.org/luci/vpython/venv/venv.go:763 - venv.(*Env).installWheels()
  reason: failed to install wheels

#1 go.chromium.org/luci/vpython/venv/venv.go:545 - venv.(*Env).createLocked.func2()
  reason: failed to install wheels

#2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
#3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
#4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
#5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
  reason: failed to create new VirtualEnv

#6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
#7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
#8 go.chromium.org/luci/vpython/venv/venv.go:328 - venv.(*Env).withImpl()
#9 go.chromium.org/luci/vpython/venv/venv.go:168 - venv.With()
#10 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
#11 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
#12 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
#13 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
#14 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
#15 vpython/main.go:110 - main.mainImpl()
#16 vpython/main.go:116 - main.main()
#17 runtime/proc.go:225 - runtime.main()
#18 runtime/asm_arm64.s:1130 - runtime.goexit()

any ideas on what's going on?

@manu-romero-411

This comment has been minimized.

Copy link

@manu-romero-411 manu-romero-411 commented May 20, 2021

@lbibass I'm experiencing the same problem as you :(
Anyone else?

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented May 22, 2021

The libangle builds became really flaky. If you experience problems, you can grab the pre-built binaries from here: https://github.com/knazarov/homebrew-qemu-virgl/releases/tag/libangle-20210315.1
It should be enough to build qemu by hand.

I’ve got a few reports from the users of the brew formula as well.

@manu-romero-411

This comment has been minimized.

Copy link

@manu-romero-411 manu-romero-411 commented May 22, 2021

I've been facing a strange issue with this QEMU build these last days.
I managed to solve the problem I had (same as @lbibass) by reinstalling macOS and Homebrew, and Angle works as expected (using both self-compiled and pre-compiled libs), but sound is distorted. It's like audio isn't played at the correct frequency. I've been testing in Youtube on Firefox and Chromium so far

EDIT: Misteriously, It has been solved because of no reason, sorry.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 22, 2021

I updated run.sh to fix the build issue of ANGLE.

It now locks the revision of ANGLE and depot_tools to those used by Chromium 90.0.4430.212, which is its latest stable release. In the past, the stable release latest at that time did not support M1 Macs. (I checked my ANGLE checkout to see it was created on "Jan 31", which even predates the first release of this gist for public. Things changed a lot since then.)

The upstream is aware of the problem in master so there is no need for report. (https://crbug.com/1205263)

@lbibass

This comment has been minimized.

Copy link

@lbibass lbibass commented May 24, 2021

I updated run.sh to fix the build issue of ANGLE.

It now locks the revision of ANGLE and depot_tools to those used by Chromium 90.0.4430.212, which is its latest stable release. In the past, the stable release latest at that time did not support M1 Macs. (I checked my ANGLE checkout to see it was created on "Jan 31", which even predates the first release of this gist for public. Things changed a lot since then.)

The upstream is aware of the problem in master so there is no need for report. (https://crbug.com/1205263)

  • branch HEAD -> FETCH_HEAD
  • git -C source/angle checkout ed5f62c36dc056c96e0ceb4d101c0f167903b900
    fatal: reference is not a tree: ed5f62c36dc056c96e0ceb4d101c0f167903b900

It's still broken, unfortunately.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 25, 2021

Oops. Fixed just now.

@akirakyle

This comment has been minimized.

Copy link

@akirakyle akirakyle commented May 29, 2021

Hey @akihikodaki, thanks so much for all the work you've done with this, I just finally got the chance to install it and it's fantastic!
Soon after Alexander Graf's had released his hvf patches, I had made a few hacks to ui/cocoa.m to make the fullscreen hidpi view work better for me, but you've made all those improvements and so much more with usable opengl support!

One question: what are your thoughts regarding the cursor situation? Using sway, I have to turn on a software rendered cursor since the current ui/cocoa.m doesn't define a .dpy_mouse_set .dpy_cursor_define. I have a patch that adds something similar for ui/cocoa.m so that with the show-cursor=on argument to -display, coca can set the cursor to the provided image. I guess this technique only works with a guest tablet input device, but it can be nice for a bit more seamless integration with macOS as the cursor doesn't stay hidden when QEMU has input focus even though your mouse be over some non-qemu ui. On the other hand I think you mentioned issues with guest apps input handling with virtualized tablet input.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented May 29, 2021

@akirakyle I'm glad to hear the patches are useful for you. (I should contribute patches to the upstream considering its usefulness, but I'm being too lazy these days although I do plan to resume the effort soon[tm].)

I use a virtual mouse with a full-screen window. I occasionally uses with a smaller window, and the virtual mouse locks the cursor in the window. It is somewhat not "seamless", but the virtual mouse is the best for a full-screen window as I said in my previous comment you noticed.
There are certainly rooms to improve the handling of a virtual tablet, but I'm not going to do so as it is not my interest (full-screen, independent Linux environment) and it will make the fork more divergent from the upstream (It already has massive changes anyway). You may of course create a patch to achieve your goal and request merging to the upstream and my tree.

Regarding a "seamless" integration of Linux desktop, there are an on-going effort to provide an integration like WSLg and RDP-RAIL (google them and you'll see what I mean here). It is still in an early development, but maybe it may be a solution for you, depending on your goal. lima-vm/lima#2

@akirakyle

This comment has been minimized.