Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Linux Desktop on Apple Silicon/M1 in Practice

Linux Desktop on Apple Silicon/M1 in Practice

I bought M1 MacBook Air. It is the fastest computer I have, and I have been a GNOME/GNU/Linux user for long time. It is obvious conclusion that I need practical Linux desktop environment on Apple Silicon/M1.

Fortunately, Linux already works on Apple Silicon/M1. But how practical is it?

As I needed Linux desktop right now, I decided to hack QEMU. The most difficult challenge is obviously accelerated graphics, but there is Virgil 3D; a birdge to expose host OpenGL to the guest. https://virgil3d.github.io

It unfortunately didn't work on macOS host. So I just made it work. That's it. Here is a video demonstrating OpenGL on Linux on Apple Silicon/M1:

https://www.youtube.com/watch?v=k0bVlVQU2JQ&list=PLesZxBYUPr3wdU3sONUv4Q7UDOg1Dn_Ue&index=4

Modifications

QEMU

hvf

  • The hvf patches are merged into current master.
  • The installation script is fixed.

ui/cocoa

  • Added OpenGL support.
  • Enforced pixel by pixel display.
  • Tells physical/pixel window size to the guest. Now Fedora can properly deal with Retina and respond to window size change.
  • Added cursor composition.
  • A bug which keeps e.g. command key pressed is fixed.

coreaudio

Added device change support. (i.e. you can plug/unplug an earphone while QEMU is running.)

hw/block

  • Add punchhole operation. (The disk image consumes physical storage only when it actually has data.)
  • Optimized transfer unit.

Virgil 3D renderer

Improved OpenGL ES support.

Do It Yourself

Setup

1. Disable System Integrity Protection

https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection

You can enable it again after setup. It must be disabled to build ANGLE.

2. Open a terminal.

3. Install GLib, Meson, Pixman and pkg-config with Homebrew.

brew install glib meson pixman pkg-config

4. Make a empty directory and change the working directory to it.

5.

curl https://gist.githubusercontent.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5/raw/8a27f179aba5564fb1cba77561c65298c0cbce01/run.sh | bash -

6.

bin/qemu-img create var/virtio.raw 64G

It doesn't consume the physical space until it has data, so you can make the image very large. However, you will see odd behavior if you try to write data more than the physical disk allows.

7.

curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/33/Silverblue/aarch64/iso/Fedora-Silverblue-ostree-aarch64-33-1.2.iso

8.

./run -cdrom Fedora-Silverblue-ostree-aarch64-33-1.2.iso

Proceed the installation process, and now you can run Fedora by executing ./run.

Updating

Just download the latest run.sh and execute it in your workspace directory.

Choosing OpenGL profile

Edit run.

  • gl=off will disable Virgil 3D GPU. Most stable but laggy.
  • gl=core will enable OpenGL.framework. Unstable.
  • gl=es will enable ANGLE. Stable and fast.

Running piglit

Test OpenGL with piglit.

On Fedora:

piglit run all -x spec@ext_timer_query@time-elapsed -x 'spec@arb_timer_query@query gl_timestamp' --timeout 9 results/all

The disabled tests will cause qemu crash due to the following bug: https://bugs.chromium.org/p/angleproject/issues/detail?id=5701

vhost-user-gpu would prevent QEMU process from crashing by isolating graphics acceleration process, but it needs modifications to run outside Linux because:

  • historically, vhost-user is a reimplementation of Linux kernel's vhost interface, and it relies on kernel headers for interface definitions and
  • vhost-user uses eventfd which is only available on Linux.

It shouldn't be difficult, but I'm satisfied even without process isolation so I don't.

Upstreaming

Upstreaming is in progress. Hopefully the features I implemented will work just by running brew install qemu in the future.

Future Perspective

As I described here, such a virtualization software is practical and efficient approach to run Linux desktop. The performance overhead is also acceptable for daily use, and it even provides better integration of Linux and macOS. For example, you can switch macOS and Linux with three-finger gesture on trackpad. You can use VirtFS.

However, there are complexities that such a virtualization adds. It basically means sharing one hardware with two systems, so you have to allocate them properly or it ends up with bad user experience. The allocation problem happens everywhere (HID like keyboard, computing resource like CPU, power management, etc.). This approach is efficient but not the best.

In long term, native Linux port is the best option. However, it is not practical if it takes too long before it becomes available. Therefore, we should consider hybrid approaches. marcan42, the founder of Asahi Linux, has an idea to run macOS on KVM on Linux to support complex devices, for example: https://twitter.com/marcan42/status/1361999648819269636

Another approach is to get macOS device support softwares work on Linux. In the past, there was NDISWrapper, a project to bring some Windows drivers to Linux. There was also Cider, a research project to "duct-tape" XNU's IOKit to Linux to bring macOS drivers to Android. https://www.cs.columbia.edu/~nieh/pubs/asplos2014_cider.pdf

By taking those approaches, it is possible to reduce dependencies on macOS gradually, not at once.

UPDATE 2021-03-16

@knazarov wrote Homebrew formulae.

https://github.com/knazarov/homebrew-qemu-virgl

set -eux
mkdir -p depot_tools build/qemu source/angle source/libepoxy source/virglrenderer source/qemu
git -C depot_tools init
git -C depot_tools fetch https://chromium.googlesource.com/chromium/tools/depot_tools.git
git -C depot_tools checkout FETCH_HEAD
git -C source/angle init
git -C source/angle fetch https://chromium.googlesource.com/angle/angle
git -C source/angle checkout FETCH_HEAD
git -C source/libepoxy init
git -C source/libepoxy fetch https://github.com/akihikodaki/libepoxy.git macos
git -C source/libepoxy checkout FETCH_HEAD
git -C source/virglrenderer init
git -C source/virglrenderer fetch https://github.com/akihikodaki/virglrenderer.git macos
git -C source/virglrenderer checkout FETCH_HEAD
git -C source/qemu init
git -C source/qemu fetch https://github.com/akihikodaki/qemu.git macos
git -C source/qemu checkout FETCH_HEAD
export PATH="$PWD/depot_tools:$PATH"
cd source/angle
python2 scripts/bootstrap.py
gclient sync
gn gen '--args=target_cpu="arm64"' ../../build/angle
cd ../..
ninja -C build/angle
meson "-Dc_args=-I$PWD/source/angle/include" "-Dc_link_args=-L$PWD/build/angle" "--prefix=$PWD" -Degl=yes -Dx11=false build/libepoxy source/libepoxy
meson install -C build/libepoxy
meson "--pkg-config-path=$PWD/lib/pkgconfig" "--prefix=$PWD" build/virglrenderer source/virglrenderer
meson install -C build/virglrenderer
cd build/qemu
PKG_CONFIG_PATH="$PWD/../../lib/pkgconfig" ../../source/qemu/configure "--extra-cflags=-I$PWD/../../source/angle/include -I$PWD/../../include" "--extra-ldflags=-L$PWD/../angle" "--prefix=$PWD/../.."
meson install
cp -i pc-bios/edk2-aarch64-code.fd pc-bios/edk2-arm-vars.fd ../../var
cd ../..
cat > run <<'EOF'
#!/bin/bash
d="$(dirname "{BASH_SOURCE[0]}")"
exec sudo DYLD_FALLBACK_LIBRARY_PATH="$d/build/angle:$d/lib" "$d/bin/qemu-system-aarch64" -machine virt,accel=hvf,highmem=off -cpu cortex-a72 -smp 8 -m 4096 -device intel-hda -device hda-output -device virtio-gpu-pci -device virtio-keyboard-pci -device virtio-net-device,netdev=net -device virtio-mouse-pci -device virtio-rng-device -display cocoa,gl=es -drive "if=pflash,format=raw,file=$d/var/edk2-aarch64-code.fd,readonly=on" -drive "if=pflash,format=raw,file=$d/var/edk2-arm-vars.fd,discard=on" -drive "if=virtio,format=raw,file=$d/var/virtio.raw,discard=on" -netdev vmnet-macos,id=net,mode=shared -full-screen -runas "$(id -u):$(id -g)" "$@"
EOF
chmod a+x run
@z0mbix

This comment has been minimized.

Copy link

@z0mbix z0mbix commented Mar 3, 2021

Did you try UTM? https://mac.getutm.app/

@jbrooksuk

This comment has been minimized.

Copy link

@jbrooksuk jbrooksuk commented Mar 3, 2021

@z0mbix, UTM is still a wrapper for QEMU, so doesn't support GPU emulation either.

@bugpowder

This comment has been minimized.

Copy link

@bugpowder bugpowder commented Mar 3, 2021

Did you try UTM? https://mac.getutm.app/

He tried the underlying tech, UTM is just a wrapper. What's more, he improved on it (QEMU) to support OpenGL.

@srikantpatnaik

This comment has been minimized.

Copy link

@srikantpatnaik srikantpatnaik commented Mar 3, 2021

Great work!

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 3, 2021

Did you try UTM? https://mac.getutm.app/

I didn't know it has macOS support and I tried it just now. It indeed has a fancy UI, but it has problems critical for my use so I'll instead stick to bare QEMU.

  • It didn't correctly report display resolution when displaying pixel by pixel. Actually I managed to fix this. utmapp/UTM#2361
  • It doesn't tell physical display size to the guest so HiDPI support on the guest doesn't work. I tried but it requires to update spice-gtk, and I'm not motivated enough to do that.
  • Its graphics is laggy. I think one of the causes is lack of hardware acceleration and my patch will improve it, but another possible cause is the IPC between the window on macOS and QEMU. I can improve the IPC by implementing IOSurface passing via Mach port, but it requires platform-dependent extensions for SPICE protocol. It should be difficult to convince the upstream to accept such a change. Beside that, there is libvirt and virt-manager, which are well-proven, if using SPICE protocol.
@relm

This comment has been minimized.

Copy link

@relm relm commented Mar 3, 2021

I can't seem to build this from the script provided, any ideas? Looks like it fails to download a file when running the "download_from_google_storage" script and then the rest fails.

________ running 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-clang-format -s buildtools/mac/clang-format.sha1' in '/Users/mike/vm/ubuntu/source/angle/.'
0> Failed to fetch file gs://chromium-clang-format/62bde1baa7196ad9df969fc1f06b66360b1a927b for buildtools/mac/clang-format, skipping. [Err: [E2021-03-03T15:31:30.061006-07:00 35237 0 venv.go:975] Command (cwd=/tmp/vpython_bootstrap287609297/packages/virtualenv-15.1.0): [/usr/bin/python2.7 -B -E -s virtualenv.py --no-download /Users/mike/.vpython-root/f029f6]
Process output:
New python executable in /Users/mike/.vpython-root/f029f6/bin/python
ERROR: The executable /Users/mike/.vpython-root/f029f6/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/private/tmp/vpython_bootstrap287609297/packages/virtualenv-15.1.0' (should be u'/Users/mike/.vpython-root/f029f6')
ERROR: virtualenv is not compatible with this system or executable
Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
COLORFGBG=15;0
COLORTERM=truecolor
COMMAND_MODE=unix2003
HOME=/tmp/vpython_bootstrap287609297
HOMEBREW_CELLAR=/opt/homebrew/Cellar
HOMEBREW_PREFIX=/opt/homebrew
HOMEBREW_REPOSITORY=/opt/homebrew
INFOPATH=/opt/homebrew/share/info:
ITERM_PROFILE=Default
ITERM_SESSION_ID=w0t4p0:E57D0A70-66CE-4049-A066-5D15E515DD8E
LANG=en_US.UTF-8
LC_TERMINAL=iTerm2
LC_TERMINAL_VERSION=3.4.4
LOGNAME=mike
LaunchInstanceID=2F7CBF4D-D8AA-459F-A7B3-C94D675C96E2
MANPATH=/Users/mike/.nvm/versions/node/v15.4.0/share/man:/opt/homebrew/share/man::
NVM_BIN=/Users/mike/.nvm/versions/node/v15.4.0/bin
NVM_CD_FLAGS=-q
NVM_DIR=/Users/mike/.nvm
NVM_INC=/Users/mike/.nvm/versions/node/v15.4.0/include/node
PATH=/Users/mike/vm/ubuntu/depot_tools:/Users/mike/.yarn/bin:/Users/mike/.config/yarn/global/node_modules/.bin:/Users/mike/.nvm/versions/node/v15.4.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/mike/vm/ubuntu/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/Users/mike/vm/ubuntu/source/angle
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SECURITYSESSIONID=186b8
SHELL=/bin/zsh
SHLVL=2
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.RWHWSUQBUT/Listeners
TEMP=/tmp
TERM=xterm-256color
TERM_PROGRAM=iTerm.app
TERM_PROGRAM_VERSION=3.4.4
TERM_SESSION_ID=w0t4p0:E57D0A70-66CE-4049-A066-5D15E515DD8E
TMP=/tmp
TMPDIR=/tmp
USER=mike
VERSIONER_PYTHON_VERSION=2.7
VIRTUAL_ENV=/Users/mike/.vpython-root/e36bc6
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.googlecode.iterm2
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0

[E2021-03-03T15:31:30.064674-07:00 35237 0 annotate.go:266] original error: exit status 100

goroutine 1:
#0 go.chromium.org/luci/vpython/venv/venv.go:615 - venv.(*Env).installVirtualEnv()
  reason: failed to create VirtualEnv

#1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
  reason: failed to install VirtualEnv

#2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
#3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
#4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
#5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
  reason: failed to create new VirtualEnv

#6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
#7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
#8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
  reason: failed to create empty probe environment

#9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
#10 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
#11 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
#12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
#13 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
#14 vpython/main.go:110 - main.mainImpl()
#15 vpython/main.go:116 - main.main()
#16 runtime/proc.go:204 - runtime.main()
#17 runtime/asm_amd64.s:1374 - runtime.goexit()
]
Downloading 1 files took 0.447183 second(s)
Error: Command 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-clang-format -s buildtools/mac/clang-format.sha1' returned non-zero exit status 1 in /Users/mike/vm/ubuntu/source/angle/.```
@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 4, 2021

@relm It is incompatible with System Integrity Protection and you need to turn it off. Boot into recovery, open a terminal, and run csrutil disable.
Thank you for noting that. I'll update README.md later.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 4, 2021

It would be really great to have this running on a x86_64 mac. I've tried to build your branch of qemu, and after a few minor tweaks to source code I managed to get it to build. Unfortunately, it doesn't run. When I execute it, I get a "floating point exception".

I guess that's because something is hardcoded in your branch that is arm-specific. But my knowledge of qemu codebase is nil, so I couldn't fix it myself.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 4, 2021

@knazarov I haven't hard-coded anything specific for AArch64. Perhaps I made some silly mistake.

Unfortunately I lost access to Intel Mac (I just quitted my job) so I cannot reproduce the problem. If you want to fix it, please share more details. I think the crash reporter gives you tons of logs.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 4, 2021

I'll try to bisect your patchset to see which patch may contain the problem. The first patch from your tree runs fine on my machine (b7885e4370a2fe426e80d32afe6eb5d01a71640d), so I should be able to pin down a suspect.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 4, 2021

The commit where everything breaks on x86 MacOS is akihikodaki/qemu@5713354. This is the commit where you add "punch hole" operations. I'll try to dig deeper.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

The commit where everything breaks on x86 MacOS is akihikodaki/qemu@5713354. This is the commit where you add "punch hole" operations. I'll try to dig deeper.

That is surprising. I thought it may be related with display resolution handling as it does floating point calculation, but the reality is obviously different from my guess. Thank you for sharing info.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

The commit where everything breaks on x86 MacOS is akihikodaki/qemu@5713354. This is the commit where you add "punch hole" operations. I'll try to dig deeper.

That is surprising. I thought it may be related with display resolution handling as it does floating point calculation, but the reality is obviously different from my guess. Thank you for sharing info.

I've figured out the problem. This happens when raw_probe_blocksizes in raw_format.c calls back to raw_probe_blocksizes in file-posix.c.

The problem is that raw_probe_blocksizes from file-posix.c always returns phys and log equal to zero. And the raw_probe_blocksizes in raw-format.c tries to calculate the alignment like this:

static int raw_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
{
    BDRVRawState *s = bs->opaque;
    int ret;

    ret = bdrv_probe_blocksizes(bs->file->bs, bsz);
    if (ret < 0) {
        return ret;
    }

    if (!QEMU_IS_ALIGNED(s->offset, MAX(bsz->log, bsz->phys))) {
        return -ENOTSUP;
    }

    return 0;
}

On the QEMU_IS_ALIGNED it fails with a floating point exception.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

After reverting the "punchhole" patch and trying to run with -display cocoa,gl=es, I get a segfault when qemu tries to load libEGL.dylib. When running under lldb, I get the following stacktrace:

* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00000002649fbe26 libc++.dylib`std::__1::codecvt<char, char, __mbstate_t>::encoding(this=0x00007fff88c0d4c0) const at __locale:951:16
    frame #2: 0x00000002649ffd05 libc++.dylib`std::__1::__stdinbuf<char>::imbue(this=0x0000000264abe4c8, __loc=0x000070000799d6e0) at __std_stream:84:26
    frame #3: 0x00000002649ffc11 libc++.dylib`std::__1::__stdinbuf<char>::__stdinbuf(this=0x0000000264abe4c8, __fp=0x00007fff88c3ee50, __st=0x0000000264abe530) at __std_stream:76:5
    frame #4: 0x00000002649ff885 libc++.dylib`std::__1::__stdinbuf<char>::__stdinbuf(this=0x0000000264abe4c8, __fp=0x00007fff88c3ee50, __st=0x0000000264abe530) at __std_stream:75:1
    frame #5: 0x00000002649ff5ec libc++.dylib`std::__1::DoIOSInit::DoIOSInit(this=0x0000000264abea18) at iostream.cpp:111:59
    frame #6: 0x00000002649ff9e5 libc++.dylib`std::__1::DoIOSInit::DoIOSInit(this=0x0000000264abea18) at iostream.cpp:107:1
    frame #7: 0x00000002649ffafa libc++.dylib`std::__1::ios_base::Init::Init(this=0x0000000264ac0928) at iostream.cpp:152:22
    frame #8: 0x00000002649ff585 libc++.dylib`std::__1::ios_base::Init::Init(this=0x0000000264ac0928) at iostream.cpp:151:1
    frame #9: 0x0000000264a01be0 libc++.dylib`::__cxx_global_var_init() at iostream.cpp:80:31
    frame #10: 0x0000000264a01c09 libc++.dylib`_GLOBAL__I_000101 at iostream.cpp:0
    frame #11: 0x0000000101249079 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 559
    frame #12: 0x0000000101249478 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
    frame #13: 0x0000000101243d1a dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 492
    frame #14: 0x0000000101243c85 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #15: 0x0000000101243c85 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #16: 0x0000000101241b82 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
    frame #17: 0x0000000101241c22 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
    frame #18: 0x00000001012326a2 dyld`dyld::runInitializers(ImageLoader*) + 82
    frame #19: 0x000000010123d569 dyld`dlopen_internal + 609
    frame #20: 0x00007fff20620fc0 libdyld.dylib`dlopen_internal(char const*, int, void*) + 177
    frame #21: 0x00007fff2060f86e libdyld.dylib`dlopen + 28
    frame #22: 0x000000010141d9c7 libepoxy.0.dylib`get_dlopen_handle(handle=0x0000000101448b90, lib_name="libEGL.dylib", exit_on_fail=<unavailable>, load=<unavailable>) at dispatch_common.c:313:19 [opt]
    frame #23: 0x000000010141da86 libepoxy.0.dylib`epoxy_egl_dlsym [inlined] epoxy_load_egl(exit_if_fails=true, load=true) at dispatch_common.c:636:12 [opt]
    frame #24: 0x000000010141da69 libepoxy.0.dylib`epoxy_egl_dlsym [inlined] epoxy_conservative_egl_dlsym(name="eglGetDisplay", exit_if_fails=true) at dispatch_common.c:646 [opt]
    frame #25: 0x000000010141da69 libepoxy.0.dylib`epoxy_egl_dlsym(name="eglGetDisplay") at dispatch_common.c:655 [opt]
    frame #26: 0x000000010141d1e9 libepoxy.0.dylib`egl_provider_resolver(name=<unavailable>, providers=<unavailable>, entrypoints=<unavailable>) at egl_generated_dispatch.c:0:17 [opt] [artificial]
    frame #27: 0x000000010141d10c libepoxy.0.dylib`egl_single_resolver(provider=<unavailable>, entrypoint_offset=1375) at egl_generated_dispatch.c:3924:12 [opt]
    frame #28: 0x000000010141b2ba libepoxy.0.dylib`epoxy_eglGetDisplay_global_rewrite_ptr [inlined] epoxy_eglGetDisplay_resolver at egl_generated_dispatch.c:4397:12 [opt]
    frame #29: 0x000000010141b2ab libepoxy.0.dylib`epoxy_eglGetDisplay_global_rewrite_ptr(display_id=0) at egl_generated_dispatch.c:5077 [opt]
    frame #30: 0x00000001002008a4 qemu-system-x86_64`qemu_egl_init_dpy_cocoa(mode=DISPLAYGL_MODE_ES) at egl-helpers.c:373:22 [opt]
    frame #31: 0x000000010001d727 qemu-system-x86_64`cocoa_display_init(ds=<unavailable>, opts=0x0000000100d11298) at main.m:707:13 [opt]
    frame #32: 0x00000001002b2215 qemu-system-x86_64`qemu_init [inlined] qemu_init_displays at vl.c:2433:5 [opt]
    frame #33: 0x00000001002b2201 qemu-system-x86_64`qemu_init(argc=<unavailable>, argv=0x000000010261f2b0, envp=<unavailable>) at vl.c:3553 [opt]
    frame #34: 0x0000000100008b99 qemu-system-x86_64`qemu_main(argc=16, argv=0x00007ffeefbff610, envp=<unavailable>) at main.c:49:5 [opt]
    frame #35: 0x000000010001d506 qemu-system-x86_64`call_qemu_main(opaque=0x0000000000000000) at main.m:115:14 [opt]
    frame #36: 0x0000000100545c9e qemu-system-x86_64`qemu_thread_start(args=<unavailable>) at qemu-thread-posix.c:521:9 [opt]
    frame #37: 0x00007fff20604950 libsystem_pthread.dylib`_pthread_start + 224
    frame #38: 0x00007fff2060047b libsystem_pthread.dylib`thread_start + 15

From the stacktrace, it looks like the problem is in the standard initialization code somewhere. It's either a memory corruption or some problem with dynamic linking.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

I think what happens in this case is two different libc++ libraries conflict. One from the system, and another from angle (libEGL.dylib). I've inspected the library dependencies in runtime with lldb, and sure enough, there are 2 different versions of libc++ loaded.

It's possible that we have to link the angle library statically with libc++, but I'm not sure yet how to do it. In your particular case on ARM you likely got lucky and the problem didn't happen.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

@knazarov I fixed the bug in punch hole patch and submitted to the upstream ccing you. Thank you for detailed analysis.

Regarding libc++, I gathered some basic information from my setup:

Opened Files from Activity Monitor
cwd
/Users/person/build/qemu
txt
/Users/person/bin/qemu-system-aarch64
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libgmodule-2.0.0.dylib
txt
/opt/homebrew/Cellar/pixman/0.40.0/lib/libpixman-1.0.40.0.dylib
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libgobject-2.0.0.dylib
txt
/opt/homebrew/Cellar/gettext/0.21/lib/libintl.8.dylib
txt
/opt/homebrew/Cellar/libffi/3.3_2/lib/libffi.7.dylib
txt
/usr/lib/dyld
txt
/Users/person/lib/libepoxy.0.dylib
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libgio-2.0.0.dylib
txt
/opt/homebrew/Cellar/glib/2.66.7/lib/libglib-2.0.0.dylib
txt
/Users/person/lib/libvirglrenderer.1.dylib
txt
/Users/person/build/angle/libc++.dylib
txt
/opt/homebrew/Cellar/pcre/8.44/lib/libpcre.1.dylib
txt
/Library/Preferences/Logging/.plist-cache.IkJslF0t
txt
/private/var/db/analyticsd/events.whitelist
txt
/private/var/db/timezone/tz/2021a.1.0/icutz/icutz44l.dat
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/SystemAppearance.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantDark.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/DarkAqua.car
txt
/usr/lib/libobjc-trampolines.dylib
txt
/usr/share/icu/icudt66l.dat
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantDark.car
txt
/System/Library/Fonts/Helvetica.ttc
txt
/System/Library/Caches/com.apple.IntlDataCache.le.kbdx
txt
/Users/person/build/angle/libEGL.dylib
txt
/System/Library/Fonts/SFNS.ttf
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Aqua.car
txt
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/0/com.apple.LaunchServices.dv/com.apple.LaunchServices-2543-v2.csstore
txt
/System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle/Contents/Resources/AppleKeyboardLayouts-L.dat
txt
/Users/person/build/angle/libabsl.dylib
txt
/Users/person/build/angle/libchrome_zlib.dylib
txt
/System/Library/Extensions/AppleMetalOpenGLRenderer.bundle/Contents/MacOS/AppleMetalOpenGLRenderer
txt
/System/Library/Extensions/AGXMetal13_3.bundle/Contents/MacOS/AGXMetal13_3
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantLightGraphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Graphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightGraphiteAX.car
txt
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Extras2.rsrc
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/FauxVibrantLight.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLight.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightGraphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/AquaAX.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/GraphiteAX.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightAX.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/DarkGraphite.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/Assets.car
txt
/Library/Application Support/CrashReporter/SubmitDiagInfo.domains
txt
/System/Library/CoreServices/CoreGlyphs.bundle/Contents/Resources/Assets.car
txt
/Users/person/build/angle/libGLESv2.dylib
txt
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/16777235_322/functions.data
txt
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/31001/libraries.data
0
/dev/ttys000
1
/dev/ttys000
2
/dev/ttys000
3
->0x63bf3348027b50b
4
->0x9fb1da11cecc57ed
5
/dev/urandom
6
->0xe66580f2b9a0a149
7
->0x795a4dbf6ec99805
8
->0x23e3bee1b8a0b5bf
9
->0x204fe5858f3bb19b
10
->0x33be3e592fffb539
11
->0x4023151e099f0117
12
/Users/person/ubuntu/drive/AAVMF_CODE.fd
13
/Users/person/ubuntu/drive/virtio.raw
14
/System/Library/Extensions/AppleMetalOpenGLRenderer.bundle/Contents/Resources/default.metallib
15
/Users/person/ubuntu/drive/AAVMF_VARS.fd
16
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/16777235_322/functions.data
17
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/16777235_322/functions.list
18
/Library/Application Support/CrashReporter/SubmitDiagInfo.domains
19
->0x5c5de80413a8a7b5
20
->0x70abb0fd8d1acfb2
21
->0x33b1b4d1d48ccea8
22
->0x59cf4f9476d7d47a
23
->0xe0b80e7690083e67
24
->0x3bec23fce34d43eb
25
->0x35e611764070523c
26
->0x9227c35e8692bf8b
27
->0x4626c7be7f59336a
28
->0xf46bddb09841bf7b
29
->0x749b0129635788db
30
->0xf245372d7e8d5031
31
->0x5950f3184c2d8437
32
->0x7d922d1411ec180b
33
->0xb6cba3c6a656ef37
34
->0xf07ae2ea1dec33a5
35
->0xfb19885eb8fade8f
36
->0xaf9a16b9e3dafbb7
37
->0x702c2fd81b7de4a8
38
->0xde81e46b4705288b
39
->0xd789a28517f45c86
40
->0xe4ea2562453f808
41
->0xf7869e00d8751601
42
->0x7f41e669195f98b5
43
->0x5e29686fe964486e
44
->0x1a99bfbb434e04ca
45
->0x581b777ce9c76274
46
->0x6e74bfc40fa93a35
47
->0x26a8257630828525
48
->0x4c5a8632d5d4c289
49
->0xe016e1aaff288267
50
->0x44816a22c218c958
53
*:53863
57
*:64406
59
*:62887
60
*:53738
67
*:51193
68
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/31001/libraries.data
69
/private/var/folders/hy/k0g7jbc92f72lk_qpmn2gbk40000gn/C/com.apple.metal/31001/libraries.list
73
*:64041
82
*:51092
91
*:57121
otool -L qemu-system-aarch64
/Users/person/bin/qemu-system-aarch64:
	/System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/opt/homebrew/opt/pixman/lib/libpixman-1.0.dylib (compatibility version 41.0.0, current version 41.0.0)
	/Users/person/lib/libepoxy.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libsasl2.2.dylib (compatibility version 3.0.0, current version 3.15.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/opt/homebrew/opt/glib/lib/libgio-2.0.0.dylib (compatibility version 6601.0.0, current version 6601.7.0)
	/opt/homebrew/opt/glib/lib/libgobject-2.0.0.dylib (compatibility version 6601.0.0, current version 6601.7.0)
	/opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 6601.0.0, current version 6601.7.0)
	/Users/person/lib/libvirglrenderer.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/usr/lib/libpam.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
	@rpath/libc++.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2022.20.117)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 54.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1463.2.1)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

For comparison, I compiled a simple C++ program:

a.cc
#include <cstdio>

int main()
{
    getc(stdin);
    return 0;
}
c++ a.cc && otool -L a.out
a.out:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 904.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)

Interestingly, /usr/lib/libc++.1.dylib does not exist on the filesystem, but it works. I'm too ignorant about dyld to make sense of this.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

@akihikodaki I'm glad to help with this, and I'd like to see it merged into upstream someday.

As for the libraries - yes, that's what I did first (otool and friends). And it doesn't show a lot of libraries actually loaded.

The way to do this properly is to load qemu under lldb and then do image list. It will show you a complete set of libraries.

my image list

(lldb) image list
[ 0] 37132C63-E1CF-3E7B-B1E3-73590449A3BD 0x0000000100000000 /Users/knazarov/dev/virgil/bin/qemu-system-x86_64
[ 1] 0D4EA85F-7E30-338B-9215-314A5A5539B6 0x000000010122c000 /usr/lib/dyld
[ 2] 10CBB5E9-D5F9-3CFE-B2E4-7C8ADE945AC9 0x00007fff6f99a000 /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor
[ 3] E0A327D0-5F6A-3F6E-A9E9-3F8C52ACFCFF 0x0000000101344000 /usr/local/opt/pixman/lib/libpixman-1.0.dylib
[ 4] 4C036338-56A4-3AFB-8B11-D46D58BAF152 0x00000001013ce000 /Users/knazarov/dev/virgil/lib/libepoxy.0.dylib
[ 5] 7339869B-1C7D-3745-A3BE-0C29867524BE 0x000000010159a000 /usr/local/opt/lzo/lib/liblzo2.2.dylib
[ 6] 6E2BD7A3-DC55-3183-BBF7-3AC367BC1834 0x00007fff2a73e000 /usr/lib/libz.1.dylib
[ 7] F666699A-02D5-3061-ADF9-B7A7E471C1E1 0x00000001015b7000 /usr/local/opt/libpng/lib/libpng16.16.dylib
[ 8] 5500CEBB-26F2-39DF-9364-8903B1C286CE 0x00000001015e3000 /usr/local/opt/jpeg/lib/libjpeg.9.dylib
[ 9] 670EAB28-5540-356B-81C2-07AB29367D6A 0x0000000101619000 /usr/local/opt/gnutls/lib/libgnutls.30.dylib
[ 10] 7EE1EE13-9D86-3612-A6BC-3E6B5D578267 0x00007fff2d0ce000 /usr/lib/libsasl2.2.dylib
[ 11] 801E2D1E-7EA5-37DA-8F44-B6D7DD3CE5B9 0x00007fff21e10000 /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
[ 12] 33B62FB3-9596-3E1E-A715-F6F7B0EFD6DB 0x00000001017b5000 /usr/local/opt/glib/lib/libgio-2.0.0.dylib
[ 13] 229ACFA3-3F2C-3437-8F40-A959714D3AB0 0x0000000101975000 /usr/local/opt/glib/lib/libgobject-2.0.0.dylib
[ 14] 9DF031A5-CD8C-3418-A9E6-7078B345CEA9 0x00000001019c9000 /usr/local/opt/glib/lib/libglib-2.0.0.dylib
[ 15] 5D418C7F-DE76-3D8A-AB9E-CA0377A54322 0x0000000101afd000 /usr/local/opt/zstd/lib/libzstd.1.dylib
[ 16] AF4DC94C-853D-3C9B-BE8F-E71257939B36 0x0000000101bbd000 /usr/local/opt/libusb/lib/libusb-1.0.0.dylib
[ 17] 22CE6991-5E25-37A0-B959-BDA8F3424374 0x0000000101be1000 /Users/knazarov/dev/virgil/lib/libvirglrenderer.1.dylib
[ 18] 83503CE0-32B1-36DB-A4F0-3CC6B7BCF50A 0x00007fff2a806000 /usr/lib/libSystem.B.dylib
[ 19] 88BD7B0F-DFB3-30E8-9909-26F6A5ECCA6B 0x0000000101c59000 /usr/local/opt/nettle/lib/libnettle.8.dylib
[ 20] FA3308D0-E7B4-3A2C-AE39-0E085F6CDEE0 0x00007fff2aa79000 /usr/lib/libpam.2.dylib
[ 21] 2A0E160E-9EE6-3B23-8832-6979A16EC250 0x00007fff2067b000 /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
[ 22] 49AC0177-35A3-3C96-AD9D-3E59923C4761 0x00007fff22dd5000 /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
[ 23] 77010EA9-4AD1-3F4A-B3EF-2FFE739FB349 0x00007fff33260000 /usr/lib/libcurl.4.dylib
[ 24] E163D5F9-E202-3A53-817B-8BC40B9293C0 0x00007fff2a528000 /usr/lib/libbz2.1.0.dylib
[ 25] A8124803-CB85-3F33-8674-16B2A021D71C 0x00007fff22edc000 /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
[ 26] 42B48E32-1918-3178-8DE0-E8126B9EC088 0x00007fff3348d000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
[ 27] 29845645-F6F2-34E0-AC0D-C659773C78ED 0x00007fff251e2000 /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
[ 28] 8D9081B3-3F6A-31A0-9B20-1AE5CD8DD747 0x00007fff21428000 /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
[ 29] EB6B543F-D42C-3FB2-A2EC-43407C5F80D3 0x00007fff2049b000 /usr/lib/libobjc.A.dylib
[ 30] B377D7C7-EDB6-3737-B492-E9872F4C6469 0x00007fff224b5000 /System/Library/Frameworks/Security.framework/Versions/A/Security
[ 31] 44C477D9-013F-3A6D-A9FE-68A89214E6A5 0x00007fff78386000 /usr/lib/liboah.dylib
[ 32] D939A124-9FD5-37A2-B03B-72E98CBC98FE 0x00007fff2a808000 /usr/lib/libfakelink.dylib
[ 33] 8F8D8A8B-4EE0-3C09-9F45-725A1FBDD38C 0x00007fff22807000 /usr/lib/libicucore.A.dylib
[ 34] EF2AC5FF-B98D-3252-ABA8-2FC721CBA942 0x00007fff2a80c000 /System/Library/PrivateFrameworks/SoftLinking.framework/Versions/A/SoftLinking
[ 35] 3C9FE530-3CD2-3A64-8A36-70816AEBDF0D 0x00007fff205b6000 /usr/lib/libc++abi.dylib
[ 36] B217D905-4F9C-3DE0-8844-88FAA3C2C851 0x00007fff20560000 /usr/lib/libc++.1.dylib
[ 37] 1A98B064-8FED-39CF-BB2E-5BDA1EF5B65A 0x00007fff2a800000 /usr/lib/system/libcache.dylib
[ 38] 822A29CE-BF54-35AD-BB15-8FAECB800C7D 0x00007fff2a7bc000 /usr/lib/system/libcommonCrypto.dylib
[ 39] 62EE1D14-5ED7-3CEC-81C0-9C93833641F1 0x00007fff2a7e6000 /usr/lib/system/libcompiler_rt.dylib
[ 40] 39DBE613-135B-3AFE-A6AF-7898A37F70C2 0x00007fff2a7db000 /usr/lib/system/libcopyfile.dylib
[ 41] 1EB11CFB-ABD7-36DD-97C7-C112A6601416 0x00007fff20389000 /usr/lib/system/libcorecrypto.dylib
[ 42] AD988EEA-1A2F-3404-9A6E-390FC2504223 0x00007fff20456000 /usr/lib/system/libdispatch.dylib
[ 43] 4641E48F-75B5-3CC7-8263-47BF79F15394 0x00007fff2060a000 /usr/lib/system/libdyld.dylib
[ 44] 803F6AED-99D5-3CCF-B0FA-361BCF14C8C0 0x00007fff2a7f7000 /usr/lib/system/libkeymgr.dylib
[ 45] C7C51322-8491-3B78-9CFA-2B4753662BCF 0x00007fff2dc3f000 /usr/lib/system/liblaunch.dylib
[ 46] C2584BC4-497B-3170-ADDF-21B8E10B4DFD 0x00007fff2a79a000 /usr/lib/system/libmacho.dylib
[ 47] 40D35D75-524B-3DA6-8159-E7E0FA66F5BC 0x00007fff29fb4000 /usr/lib/system/libquarantine.dylib
[ 48] 5CC12A16-82CB-32F0-9040-72CFC88A48DF 0x00007fff2a7f4000 /usr/lib/system/libremovefile.dylib
[ 49] 5B48071E-85EB-33B0-AE9B-127AEB398AEC 0x00007fff24e4c000 /usr/lib/system/libsystem_asl.dylib
[ 50] E644CAA0-65B7-36E4-8041-520F3301F3DB 0x00007fff20339000 /usr/lib/system/libsystem_blocks.dylib
[ 51] 4AF71812-4099-3E96-B271-1F259491A2B2 0x00007fff204d7000 /usr/lib/system/libsystem_c.dylib
[ 52] C53D5E0C-0C4F-3B35-A24B-E0D7101A3F95 0x00007fff2a7ee000 /usr/lib/system/libsystem_collections.dylib
[ 53] 4917D824-4DE8-32CC-9ED2-1FBF371FEB9F 0x00007fff292a8000 /usr/lib/system/libsystem_configuration.dylib
[ 54] 6F08275F-B912-3D8E-9D74-4845158AE4F3 0x00007fff2859b000 /usr/lib/system/libsystem_containermanager.dylib
[ 55] 529A0663-A936-309C-9318-1B04B7F70658 0x00007fff2a536000 /usr/lib/system/libsystem_coreservices.dylib
[ 56] E016D8F7-C87F-36F8-B8A0-6A61B8E4BACB 0x00007fff22a69000 /usr/lib/system/libsystem_darwin.dylib
[ 57] E0A0CAB3-6779-3C83-AC67-046CFE69F9C2 0x00007fff2a7f8000 /usr/lib/system/libsystem_dnssd.dylib
[ 58] 9CECB43A-094E-3CA9-B730-24DEA1A6DE05 0x00007fff204d4000 /usr/lib/system/libsystem_featureflags.dylib
[ 59] 0C96CFE8-71F5-3335-8423-581BC3DE5846 0x00007fff2064f000 /usr/lib/system/libsystem_info.dylib
[ 60] DD26CC5C-AFF6-305F-A567-14909DD57163 0x00007fff2a751000 /usr/lib/system/libsystem_m.dylib
[ 61] A498D1EF-E43D-310C-84E8-9C0AADA0C475 0x00007fff20429000 /usr/lib/system/libsystem_malloc.dylib
[ 62] 5213D866-7D0E-3FD9-8E1A-03C0E39CEC44 0x00007fff24de0000 /usr/lib/system/libsystem_networkextension.dylib
[ 63] B2BF20C7-448A-3FBD-A2F5-AB7618D173F6 0x00007fff22e84000 /usr/lib/system/libsystem_notify.dylib
[ 64] 20310EE6-2C3F-361A-9ECA-4223AFC03B65 0x00007fff300f3000 /usr/lib/system/libsystem_product_info_filter.dylib
[ 65] 5F7F3DD1-4B38-310C-AA8F-19FF1B0F5276 0x00007fff292ac000 /usr/lib/system/libsystem_sandbox.dylib
[ 66] E05E35BC-1BAB-365B-8BEE-D774189EFD3B 0x00007fff2a7f1000 /usr/lib/system/libsystem_secinit.dylib
[ 67] AB413518-ECDE-3F04-A61C-278D3CF43076 0x00007fff205cf000 /usr/lib/system/libsystem_kernel.dylib
[ 68] 1C3E1A0A-92A8-3CDE-B622-8940B43A5DF2 0x00007fff20645000 /usr/lib/system/libsystem_platform.dylib
[ 69] B989DF6C-ADFE-3AF9-9C91-07D2521F9E47 0x00007fff205fe000 /usr/lib/system/libsystem_pthread.dylib
[ 70] BC85B46C-02EE-31FF-861D-F0DE01E8F6CF 0x00007fff2654b000 /usr/lib/system/libsystem_symptoms.dylib
[ 71] 87FEF600-48D9-31C9-B8FC-D5249B2AE95D 0x00007fff20371000 /usr/lib/system/libsystem_trace.dylib
[ 72] 1D0A4B4A-4370-3548-8DC1-42A7B4BD45D3 0x00007fff2a7c8000 /usr/lib/system/libunwind.dylib
[ 73] 70F26262-01AA-3CEC-9FAD-2701D24096F0 0x00007fff2033b000 /usr/lib/system/libxpc.dylib
[ 74] C4C7A5DC-C3AD-374D-902A-B6DED14FBB67 0x00007fff24df0000 /usr/lib/libMobileGestalt.dylib
[ 75] 879A04AA-F9A1-3640-A9EF-F5D0813C852A 0x00007fff2a518000 /System/Library/PrivateFrameworks/AppleFSCompression.framework/Versions/A/AppleFSCompression
[ 76] 20EDB261-69C7-3E4A-8C00-878814186A62 0x00007fff266fd000 /usr/lib/libDiagnosticMessagesClient.dylib
[ 77] D71EF121-D4B0-306E-9843-9FAFD558E3A4 0x00007fff29fdb000 /usr/lib/libbsm.0.dylib
[ 78] B33BEF87-3275-356D-9815-D0E94122D2EB 0x00007fff29fc3000 /usr/lib/libcoretls.dylib
[ 79] 80AF345B-1989-3C88-AEF4-8FF18A366A9C 0x00007fff2aea2000 /usr/lib/libcoretls_cfhelpers.dylib
[ 80] ACCC6A4C-3257-38A4-8AFB-BA73D906205B 0x00007fff261aa000 /usr/lib/libsqlite3.dylib
[ 81] FAA7B36E-21EC-3ADF-AA6A-E3742737CA3A 0x00007fff2af9f000 /usr/lib/libxar.1.dylib
[ 82] 7A2E42E6-3AF2-3ECB-8A16-5C4AC41206EE 0x00007fff2743e000 /System/Library/PrivateFrameworks/CoreAutoLayout.framework/Versions/A/CoreAutoLayout
[ 83] 3518EA0E-C32D-32CC-81B9-0F3C83B6430C 0x00007fff2113c000 /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
[ 84] 9C42DB2C-0A6D-38CE-BB8D-899A9B34B695 0x00007fff2aa7e000 /usr/lib/libcompression.dylib
[ 85] 04A917FB-DBFB-3432-BA4C-5B860990A420 0x00007fff24942000 /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
[ 86] ACFCD23E-05DF-3F96-BCFA-E294AAAFD2CE 0x00007fff267e8000 /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
[ 87] 9061C767-43FD-3483-9C48-73973AF82DC1 0x00007fff2a948000 /usr/lib/libarchive.2.dylib
[ 88] E0BF29C7-869B-3DD5-82AE-F36E6398091A 0x00007fff27487000 /usr/lib/libxml2.2.dylib
[ 89] E59859C6-7221-3324-BB58-F910B2199959 0x00007fff301f5000 /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
[ 90] E8044658-AE79-3930-A902-07A38BB22382 0x00007fff292b7000 /usr/lib/liblangid.dylib
[ 91] 970D1F7B-5DBD-355F-80D8-C820AA1626D2 0x00007fff211ba000 /usr/lib/libCRFSuite.dylib
[ 92] A3F1E999-6D61-32D7-B3F7-67832CEA53CD 0x00007fff242fe000 /usr/lib/libnetwork.dylib
[ 93] DE5787F8-D011-3E71-8323-936B35976D9E 0x00007fff2a80d000 /usr/lib/libpcap.A.dylib
[ 94] 97EB5DFD-BB41-3834-BF09-1F597D84F324 0x00007fff26544000 /usr/lib/libdns_services.dylib
[ 95] A2F6DE8D-2B38-3669-A16E-E1BD54108F24 0x00007fff2aa4c000 /usr/lib/libapple_nghttp2.dylib
[ 96] 33A9FBF3-A2D3-3ECE-9370-B0ADDE3896DE 0x00007fff24def000 /usr/lib/libenergytrace.dylib
[ 97] DE05C188-D9FE-30DC-8299-DF3DC9F837A5 0x00007fff2a7a0000 /usr/lib/system/libkxld.dylib
[ 98] 168D84A7-8E54-361E-9161-B661D96C0C9A 0x00007fff2ae89000 /usr/lib/liblzma.5.dylib
[ 99] AD10ECF4-E137-3152-9612-7EC548D919E8 0x00007fff2a845000 /usr/lib/libiconv.2.dylib
[100] D14F9D24-693A-37F0-8F92-D260248EB282 0x00007fff2a799000 /usr/lib/libcharset.1.dylib
[101] 6B70B98B-BCC5-3FBE-A6B5-CD0696AA055A 0x00007fff2740d000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
[102] B857EADF-D517-34E8-8053-34C0E6695250 0x00007fff22a73000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
[103] 49187239-597B-31A1-8895-8AA265C818C7 0x00007fff2674d000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
[104] 8D30829B-426E-361E-B2A9-5850F1E5513D 0x00007fff2a53b000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
[105] 6243D4C0-3572-30A6-889A-B8690A5EAC24 0x00007fff2a9ba000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
[106] 9B6B42DB-8768-343B-B10E-A9A5710280CD 0x00007fff264ce000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
[107] 9ACD5692-3B92-3E6E-8B24-56ADCC911556 0x00007fff20b17000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
[108] 5D0B1BB2-995B-32EB-AE75-F5F27127ACAF 0x00007fff2ae3a000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
[109] C861FAD6-D3A5-302C-88AE-B2813F7201A7 0x00007fff27416000 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
[110] 74E17BB2-1A3C-3574-92DD-63ABE39E3FF9 0x00007fff29fb7000 /usr/lib/libCheckFix.dylib
[111] 2F48471B-68F3-3017-8B18-BEDD4ED5853F 0x00007fff24e63000 /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
[112] 6C49B26B-CB39-3504-AD5E-9C3333FFE086 0x00007fff292b9000 /System/Library/PrivateFrameworks/CoreNLP.framework/Versions/A/CoreNLP
[113] FB7BF527-2D95-3E82-AAD8-D42B33DAC0F0 0x00007fff26700000 /System/Library/PrivateFrameworks/MetadataUtilities.framework/Versions/A/MetadataUtilities
[114] 102A0AD8-D119-340B-B652-B75F0AAB1C7E 0x00007fff211f0000 /usr/lib/libmecabra.dylib
[115] 01B968D9-A464-3667-8B8E-C99AEBA1FF79 0x00007fff2a2ca000 /System/Library/Frameworks/MLCompute.framework/Versions/A/MLCompute
[116] 2423F0F6-A3A4-37AB-971E-FCFC7645D282 0x00007fff304c3000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
[117] 1F424683-4213-3594-9BF5-EFCCCDAA83A5 0x00007fff29fec000 /usr/lib/libmecab.dylib
[118] E40EFA11-E95C-36D1-A8BE-8CA5B1DD179D 0x00007fff2a036000 /usr/lib/libgermantok.dylib
[119] 47CDE3EE-60AB-38CE-9494-5CC047CACA7F 0x00007fff2aa22000 /usr/lib/libThaiTokenizer.dylib
[120] 718400BF-9911-3CB6-8CCE-4C2D7C35D80A 0x00007fff2afda000 /usr/lib/libChineseTokenizer.dylib
[121] F37DBCE9-20CD-33D4-806B-93B326E08422 0x00007fff2aa77000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/MetalPerformanceShaders
[122] 829B9C4D-2DB6-38C9-9FE8-E6544FB3BDCC 0x00007fff285c7000 /System/Library/Frameworks/Metal.framework/Versions/A/Metal
[123] 3EBF95E1-30EB-3162-A9D4-B97BE581B69F 0x00007fff267ef000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
[124] 8CD6F55A-8E4F-3653-A367-427D0ED3DD16 0x00007fff301ce000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
[125] 9D11FCFC-6C30-32B6-864B-9CC3ED7D1143 0x00007fff2afe5000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
[126] 4CCEE561-A72B-33BC-B9F9-77A6CF2E7AF0 0x00007fff29a1f000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
[127] 92702864-A5C6-3E22-86F6-DB329532674A 0x00007fff20e1b000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
[128] 2F92E8CD-396D-3665-88EB-41A4E1A9CB31 0x00007fff2aa9d000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
[129] 0D017F3F-6FD6-38DB-AA14-A64523A4D093 0x00007fff2a03c000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
[130] FC45F7FD-6B10-3F8D-A553-F57C757F6717 0x00007fff2aa64000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
[131] 55EE1866-623B-3C10-91FB-C8B06EB83995 0x00007fff2aa97000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
[132] C5CD30D8-114B-3104-9020-E5C05E68CEE8 0x00007fff29361000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
[133] B889DE4E-7356-3CC8-A13E-68D15E2024DF 0x00007fff210cd000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib
[134] 287C79B5-4E13-3DF6-BDC4-439EC621E010 0x00007fff2924c000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSCore.framework/Versions/A/MPSCore
[135] D009508B-DCDF-3421-B1CB-40FDF4A68E93 0x00007fff2a487000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSImage.framework/Versions/A/MPSImage
[136] 8FA469B3-48BE-315B-B3A0-C868CD541608 0x00007fff2a052000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSNeuralNetwork.framework/Versions/A/MPSNeuralNetwork
[137] B988D5F6-4E54-335D-A654-12C74FD668EA 0x00007fff2a412000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSMatrix.framework/Versions/A/MPSMatrix
[138] 613BB6F1-3588-3E25-B523-059022FBE56E 0x00007fff2a27a000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSRayIntersector.framework/Versions/A/MPSRayIntersector
[139] 51ECDE6B-39F8-3BAB-ADC1-84C958F40A3F 0x00007fff2a449000 /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/Frameworks/MPSNDArray.framework/Versions/A/MPSNDArray
[140] B1FDD261-7707-3E20-A6C2-99D9AF46356B 0x00007fff20d47000 /System/Library/PrivateFrameworks/MetalTools.framework/Versions/A/MetalTools
[141] 029C5E16-2E04-3E98-BE25-CC8BAE5E3D19 0x00007fff285ac000 /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
[142] E6C1A91D-4ADE-33CB-AA47-2F573811BFD9 0x00007fff292b1000 /System/Library/PrivateFrameworks/AggregateDictionary.framework/Versions/A/AggregateDictionary
[143] 4FFF906A-FE79-38F9-B777-AE878A0C99F1 0x00007fff266d8000 /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/CoreAnalytics
[144] 4A637FE0-A740-32F0-AE70-4593F6DF1573 0x00007fff2aa24000 /System/Library/PrivateFrameworks/AppleSauce.framework/Versions/A/AppleSauce
[145] 41EDF102-E1B5-3FBA-A74A-6C583CAA5D24 0x00007fff285be000 /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
[146] 44489BD1-3963-35DF-86F1-DE95778AC0DB 0x00007fff6cb52000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreFSCache.dylib
[147] 4F8A1D06-3EFF-3174-BE9C-C4B70A3EA290 0x00007fff292b3000 /System/Library/PrivateFrameworks/AppleSystemInfo.framework/Versions/A/AppleSystemInfo
[148] B15E9DA1-1C81-3ECF-94C7-FD137AD6C5E2 0x00007fff29c06000 /System/Library/PrivateFrameworks/IOMobileFramebuffer.framework/Versions/A/IOMobileFramebuffer
[149] 1A05BCD7-232F-3950-936D-EC0F95BA3FB0 0x00007fff2178c000 /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
[150] D43F3FFA-6692-3D56-ACBA-DDA40C566DDE 0x00007fff29bf3000 /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
[151] B059FF7F-731A-38F1-884A-588C6C8CA7F2 0x00007fff2935a000 /System/Library/PrivateFrameworks/LinguisticData.framework/Versions/A/LinguisticData
[152] 2F429EBD-1BD2-3057-B760-8A81546DBD6F 0x00007fff2107f000 /System/Library/PrivateFrameworks/Lexicon.framework/Versions/A/Lexicon
[153] A5D1EBB3-70A1-3ECA-A3C7-E97FBAD36ECF 0x00007fff2a936000 /usr/lib/libcmph.dylib
[154] 25F138F8-9E55-3749-8271-0FDB275C3E2C 0x00007fff273ed000 /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
[155] 333A7F2E-0F3E-37F1-9E1B-69996F5084D2 0x00007fff273dd000 /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
[156] E48CCAAC-34B5-377E-8E5D-06CDBD411C2F 0x00007fff2aea4000 /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
[157] 72AC63B1-0B72-394F-97E8-BA9B114DD0A9 0x00007fff29f18000 /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
[158] 0A268749-08E7-3551-8969-442210651CCA 0x00007fff2afad000 /usr/lib/libutil.dylib
[159] 2BBB708C-4D91-364E-ABD0-39BF198688A6 0x00007fff2743b000 /usr/lib/libapp_launch_measurement.dylib
[160] 89992991-8538-380B-B2B8-4EA997CBFDBE 0x00007fff22d9a000 /System/Library/PrivateFrameworks/CoreServicesStore.framework/Versions/A/CoreServicesStore
[161] FD031028-B702-3909-B2AF-3916404DD4A8 0x00007fff29faf000 /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
[162] 1C5219FB-B504-3658-B23B-2C45A3BA28F1 0x00007fff2afb1000 /usr/lib/libxslt.1.dylib
[163] EC15118E-62BD-3C24-A880-3B9B74318B0B 0x00007fff29fa5000 /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
[164] 32F2A1A3-F96F-367E-A799-974BEF5267D2 0x0000000101c9d000 /usr/local/opt/p11-kit/lib/libp11-kit.0.dylib
[165] 7C2B2A62-B6F6-37ED-8160-568BB21B7906 0x0000000101d91000 /usr/local/opt/libidn2/lib/libidn2.0.dylib
[166] 15146CD7-FAEB-3DCA-885F-53BC712F8FD8 0x0000000101dbd000 /usr/local/opt/libunistring/lib/libunistring.2.dylib
[167] 0981BD18-599B-3304-B05C-36E001FC741A 0x0000000101f39000 /usr/local/opt/libtasn1/lib/libtasn1.6.dylib
[168] 545E512F-A3A5-321F-ABB9-C45BFBA775FF 0x0000000101f55000 /usr/local/opt/nettle/lib/libhogweed.6.dylib
[169] 3B2C7DB0-5829-3B4C-9522-1EF23C5E3226 0x0000000101fa1000 /usr/local/opt/gmp/lib/libgmp.10.dylib
[170] FA921CC0-395B-3155-8259-EA61DE25C5D2 0x0000000102019000 /usr/local/opt/gettext/lib/libintl.8.dylib
[171] C148976B-BB19-3551-B57F-AA52BAF80F7B 0x0000000102029000 /usr/local/opt/libffi/lib/libffi.7.dylib
[172] C78E9D22-5110-349B-AF8E-6BA893A20214 0x00007fff2b92d000 /System/Library/PrivateFrameworks/AppServerSupport.framework/Versions/A/AppServerSupport
[173] A9D2E8CD-9D74-304E-BD99-A7704F3CD38B 0x00007fff2d9de000 /System/Library/PrivateFrameworks/perfdata.framework/Versions/A/perfdata
[174] A00B802D-7BB8-3F6B-A79D-E9697F7D0ABB 0x00007fff29f07000 /System/Library/PrivateFrameworks/AssertionServices.framework/Versions/A/AssertionServices
[175] DEE2C5BA-BFD1-349F-A10A-B10717BA8F7C 0x00007fff219b0000 /System/Library/PrivateFrameworks/AudioToolboxCore.framework/Versions/A/AudioToolboxCore
[176] 3FFEB564-C7CC-3F3E-B193-3F4E7B7C6764 0x00007fff286ed000 /System/Library/PrivateFrameworks/caulk.framework/Versions/A/caulk
[177] 29F5E3FE-CF12-3242-9FD3-4950DA38E4D0 0x00007fff3d689000 /System/Library/PrivateFrameworks/SystemPolicy.framework/Versions/A/SystemPolicy
[178] 8DEEA01D-B715-340A-900E-B448AB22720E 0x00007fff2b13d000 /usr/lib/libIOReport.dylib
[179] B207FDFF-8D80-3800-B968-03C718FFBF3F 0x00007fff2d200000 /usr/lib/libSMC.dylib
[180] 3A964BB6-7AB8-3B5D-BAF6-40F56D2BA393 0x00007fff26400000 /System/Library/PrivateFrameworks/BaseBoard.framework/Versions/A/BaseBoard
[181] 4B9FFFB3-310F-350A-BEBB-AB779BD3F405 0x00007fff26482000 /System/Library/PrivateFrameworks/RunningBoardServices.framework/Versions/A/RunningBoardServices
[182] B3FE85C5-8034-38D5-8566-5317B49ECF61 0x00007fff2b48d000 /System/Library/PrivateFrameworks/PersistentConnection.framework/Versions/A/PersistentConnection
[183] A865CC93-C2C2-3EF4-8E97-7BEA78970DAA 0x00007fff28c93000 /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
[184] 58CD91DC-64A9-3A97-AE41-6EA30308D617 0x00007fff26193000 /System/Library/PrivateFrameworks/ProtocolBuffer.framework/Versions/A/ProtocolBuffer
[185] 8C96D016-2B40-39C0-A51C-9CF687AE7218 0x00007fff263e7000 /System/Library/PrivateFrameworks/CommonUtilities.framework/Versions/A/CommonUtilities
[186] F4B7C353-8A65-3B54-9E01-2C93CB3F55A3 0x00007fff2bba8000 /System/Library/PrivateFrameworks/Bom.framework/Versions/A/Bom
[187] 8034EAA7-787C-3A3E-A301-FF90376F247D 0x00007fff24e7b000 /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
[188] 2CD3B3C9-7022-34DD-BF53-D48D79DEF726 0x00007fff287f4000 /System/Library/PrivateFrameworks/FontServices.framework/libFontParser.dylib
[189] C257C950-F430-3762-BC72-15D314D99556 0x00007fff2bd3d000 /System/Library/PrivateFrameworks/WatchdogClient.framework/Versions/A/WatchdogClient
[190] 6DD6A260-800F-3284-842C-8E4CB9EA47FF 0x00007fff21879000 /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
[191] 8E50C806-C6A2-3B96-B3D2-DA1FFC73D2A8 0x00007fff2870a000 /System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia
[192] 50857F8D-C7CC-3609-B0DB-FC3C7382243B 0x00007fff2756a000 /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
[193] 4243F7DC-7EB9-3750-BE36-527B0BADF36C 0x00007fff2bd41000 /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
[194] 04D8759B-1119-3E37-B922-32BDECB7C5D2 0x00007fff26e96000 /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
[195] 425BDD2D-6C26-3D09-AD9F-942EA387B2A4 0x00007fff2bd75000 /System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox
[196] 60363868-111B-36CF-9CDF-2643DF82002D 0x00007fff2bf9f000 /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
[197] 5AD7B6E0-1744-3C8D-8A95-FE425DEA005C 0x00007fff2bf82000 /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
[198] 010F8260-BC46-361E-BE59-540F765EFE74 0x00007fff2bfaa000 /System/Library/PrivateFrameworks/DSExternalDisplay.framework/Versions/A/DSExternalDisplay
[199] 2B7E468B-D901-3EA9-85CF-82FD753785BD 0x00007fff2c00a000 /System/Library/PrivateFrameworks/CMCaptureCore.framework/Versions/A/CMCaptureCore
[200] 3B4701CB-8F66-3E50-A38A-BEEC992AAC31 0x00007fff2b946000 /usr/lib/libspindump.dylib
[201] A67D9B0A-E9FB-325A-873B-032DAF45C112 0x00007fff25876000 /System/Library/Frameworks/ColorSync.framework/Versions/A/ColorSync
[202] 42707DFB-86A4-3F56-A5AF-E6DAA7E49DC7 0x00007fff2b0a4000 /usr/lib/libate.dylib
[203] EE4A9D54-C131-336B-8EC0-ECA2BD8A79F4 0x00007fff2bfa6000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
[204] 828DE4EE-7551-37E0-B37D-EC96D1A0754A 0x00007fff2bfb0000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
[205] CB838530-E4AE-3126-BC35-9FBE588303CB 0x00007fff2befb000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
[206] 173EC7B9-E3CD-3ECC-B67D-6EFDCE60E7FA 0x00007fff2bf22000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
[207] 8941531E-2AED-3366-B709-5FE275994FF0 0x00007fff2c005000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
[208] 7C3C2138-1E8C-35B8-8517-240BBC7F165B 0x00007fff2bc5e000 /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
[209] 1D978F91-674E-3A44-8A93-773C7E3E513C 0x00007fff2b348000 /usr/lib/libexpat.1.dylib
[210] 7F3819DE-BCE7-32B7-BB33-10C1DB2BA512 0x00007fff2bc13000 /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
[211] F70AF1B3-D17A-3A0E-A8AC-9D45C5B52BA9 0x00007fff6cb5e000 /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
[212] F0DD35B5-3692-3BE5-AD3D-4F6B237EF6AD 0x00007fff6cbae000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
[213] 45A1FFDC-03B7-3D64-AFAF-16D321B53D28 0x00007fff6cb71000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
[214] 68ABAADE-2DB4-3412-9F36-CA1AEAC8E9F6 0x00007fff6cd80000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
[215] 16D10CE1-C2A1-3E24-A617-766CB6E9133C 0x00007fff6cb7a000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
[216] AF77D964-9A4C-3690-AF62-4E05825DC9BF 0x00007fff6cb6e000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
[217] 989F6CBF-CCEF-340D-9CB5-EC4133707040 0x00007fff6cb59000 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
[218] 29BE82F7-93BE-336A-AF66-9DD3F48B8565 0x00007fff28ece000 /System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage
[219] 9A8DC9A0-1A80-3A26-ACA0-A245D3FFF921 0x00007fff21c27000 /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
[220] 40B70A3B-D981-3E4A-9BEC-F22EF84118F5 0x00007fff6e1c1000 /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
[221] A7393BC4-4441-31B5-BB15-5A5A70738BEC 0x00007fff2b4b7000 /System/Library/PrivateFrameworks/GraphVisualizer.framework/Versions/A/GraphVisualizer
[222] 23CD0C8A-E300-3E28-BD5B-DDCDA4521C54 0x00007fff2b4c6000 /System/Library/PrivateFrameworks/FaceCore.framework/Versions/A/FaceCore
[223] 827396F3-F3A7-3A4F-BE7F-171077501137 0x00007fff2b8e2000 /System/Library/PrivateFrameworks/OTSVG.framework/Versions/A/OTSVG
[224] 9E9329F6-7384-3C7F-83E7-D52CD7FFBC09 0x00007fff27161000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
[225] F0974976-30C0-35EB-92F1-B4CF1E974F11 0x00007fff2b934000 /System/Library/PrivateFrameworks/FontServices.framework/libhvf.dylib
[226] E3BE877B-1C78-3ABF-8C41-66FE9B6971C7 0x00007fff2b2f4000 /System/Library/PrivateFrameworks/AppleVA.framework/Versions/A/AppleVA
[227] 83DBFA98-F85B-3B63-B58F-C9989BAC9611 0x00007fff2bec8000 /usr/lib/libAudioToolboxUtility.dylib
[228] 6C720DC2-E50E-3A8D-8494-F8C007DF8DF4 0x00007fff3dff3000 /usr/lib/libmis.dylib
[229] 52DCC7E4-69F7-36FA-880D-6434E3DB6B58 0x000000010203d000 /usr/local/Cellar/glib/2.66.7/lib/libgmodule-2.0.0.dylib
[230] 7B043B4A-71CE-3F6E-91F0-CBBED47A9EA9 0x00007fff2d0b5000 /usr/lib/libresolv.9.dylib
[231] 5A905085-B5EB-3F71-A663-200428E7A5F4 0x00007fff23c3f000 /System/Library/PrivateFrameworks/UIFoundation.framework/Versions/A/UIFoundation
[232] 661A448B-2901-33CD-B13C-362D61AFD998 0x00007fff2fe6c000 /System/Library/PrivateFrameworks/RemoteViewServices.framework/Versions/A/RemoteViewServices
[233] D279B67B-7108-3A16-93E5-5A429B898DAA 0x00007fff28c65000 /System/Library/PrivateFrameworks/XCTTargetBootstrap.framework/Versions/A/XCTTargetBootstrap
[234] 2AAE247F-1BA0-3E42-A07B-CBE413747B30 0x00007fff23e94000 /System/Library/Frameworks/UniformTypeIdentifiers.framework/Versions/A/UniformTypeIdentifiers
[235] D88B5A7D-F9F7-36C2-A346-87D7163605A0 0x00007fff28c69000 /System/Library/PrivateFrameworks/CoreSVG.framework/Versions/A/CoreSVG
[236] B6859735-701D-307E-8061-298BA2296E47 0x00007fff2b381000 /System/Library/PrivateFrameworks/IconServices.framework/Versions/A/IconServices
[237] 4B26DB42-BFDD-3B0D-8966-3D0926A3A5B7 0x00007fff28c51000 /System/Library/PrivateFrameworks/DFRFoundation.framework/Versions/A/DFRFoundation
[238] 8E53C25F-9FE2-372E-8374-6A598D72C4C1 0x00007fff2ce8d000 /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
[239] 7BE9684B-4439-3008-A2A7-4093BBCA180A 0x00007fff25d74000 /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
[240] A4EBFF22-F861-321A-AF28-1DB8ECB9A4FF 0x00007fff2c132000 /System/Library/PrivateFrameworks/DataDetectorsCore.framework/Versions/A/DataDetectorsCore
[241] 82B316E1-E47F-334D-8C33-DB1B7E8A4F71 0x00007fff28951000 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
[242] 8C33ECFF-2FBF-30FF-87EA-32A508860543 0x00007fff2fea8000 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
[243] 6DB427BB-44CA-3002-A3FF-77F0FEFC2A51 0x00007fff271a3000 /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
[244] DE997943-6AD5-32DA-B48E-78170C1EAA7B 0x00007fff2c129000 /System/Library/PrivateFrameworks/InternationalSupport.framework/Versions/A/InternationalSupport
[245] 2E5E666D-EE93-3949-8C76-692FA748D129 0x00007fff273d1000 /System/Library/PrivateFrameworks/PerformanceAnalysis.framework/Versions/A/PerformanceAnalysis
[246] 12250395-D660-3CC1-81DB-565340F2651B 0x00007fff2c1ae000 /System/Library/PrivateFrameworks/UserActivity.framework/Versions/A/UserActivity
[247] 50BA07A4-02E8-38EE-B0B9-B12C277B563B 0x00007fff2c05c000 /System/Library/PrivateFrameworks/TextureIO.framework/Versions/A/TextureIO
[248] 3DD60168-A0F3-3388-8DDE-69F81280AE91 0x00007fff2d049000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
[249] F3AEB77A-456F-3F9F-98CC-3E29ABAF5D0C 0x00007fff2d35b000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSyncLegacy.framework/Versions/A/ColorSyncLegacy
[250] 780B2B09-80A6-32C8-B9E5-024988196E00 0x00007fff2596d000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
[251] 14703020-3124-3C62-A9DB-6DD28024CD42 0x00007fff2d340000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
[252] DB939B2B-D47D-3035-8C02-01006D21C5A7 0x00007fff2c014000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
[253] F65D0944-C775-33FB-A345-C046789667B7 0x00007fff2d363000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
[254] 14F3382B-2B8E-31E5-8C73-73F0B8DD05F0 0x00007fff2d039000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
[255] 6C45783D-CF90-3F11-BA86-92D31A9B68CA 0x00007fff2bfd5000 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATSUI.framework/Versions/A/ATSUI
[256] CDF6F483-017E-3DED-B107-C50FC3CADD65 0x00007fff2d2e0000 /usr/lib/libcups.2.dylib
[257] D40BF56F-DF5F-3ECB-AAC2-FE6F13E9D120 0x00007fff2d350000 /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
[258] 02FFA3CA-F8AF-3211-9159-36BC29612650 0x00007fff2d9fa000 /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
[259] 5D06022E-C963-390A-9487-90BD1B61BC20 0x00007fff2da0a000 /System/Library/Frameworks/GSS.framework/Versions/A/GSS
[260] 69768234-4F4F-3AB6-B116-4AE249E6417A 0x00007fff2b949000 /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
[261] 85526E13-1E48-377A-A700-EE3F5648A6E3 0x00007fff3378e000 /System/Library/Frameworks/Kerberos.framework/Versions/A/Libraries/libHeimdalProxy.dylib
[262] 75C2713C-F0D8-32BA-A839-17A2AE49A061 0x00007fff26553000 /System/Library/Frameworks/Network.framework/Versions/A/Network
[263] 8F6CD867-FB45-375D-82C1-E0AE79558C8C 0x00007fff2b362000 /usr/lib/libheimdal-asn1.dylib
[264] C4418557-FA52-3B80-804F-2BC7AF760FD6 0x00007fff2da5a000 /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
[265] B149BCAF-854C-3820-809D-15C6077FFB8B 0x00007fff275b7000 /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
[266] F696F4EC-9344-3C85-941E-6E5EF5CEF585 0x00007fff2cfbe000 /System/Library/PrivateFrameworks/AudioSession.framework/Versions/A/AudioSession
[267] 303533DD-D001-3CE8-A5A4-EC29F464FC3A 0x00007fff2d026000 /usr/lib/libAudioStatistics.dylib
[268] 14E84D66-28B8-390E-A499-9F24B7628EC7 0x00007fff2b3ef000 /System/Library/PrivateFrameworks/MediaExperience.framework/Versions/A/MediaExperience
[269] 6AAD02A0-9590-3CED-9568-36E05C59AA2B 0x00007fff2ce5b000 /System/Library/PrivateFrameworks/AudioSession.framework/libSessionUtility.dylib
[270] 5AFDC5C6-EBB6-33E4-9215-1AACE6C6F6EE 0x00007fff2d9eb000 /usr/lib/libperfcheck.dylib
[271] 6A293E4C-1E2D-36DA-B416-FBED7286CD6E 0x00007fff2d36f000 /System/Library/PrivateFrameworks/AudioResourceArbitration.framework/Versions/A/AudioResourceArbitration
[272] 3563C4B4-ABF6-3DE0-8381-D3667DFE2B80 0x00007fff2b36c000 /System/Library/PrivateFrameworks/IconFoundation.framework/Versions/A/IconFoundation
[273] B24969AE-43BA-36C2-80F9-2CBBC3F63CC8 0x00007fff2fe98000 /System/Library/PrivateFrameworks/SpeechRecognitionCore.framework/Versions/A/SpeechRecognitionCore
[274] C6441F4D-3ADF-329E-A264-C90A1B132B37 0x000000010204d000 /usr/local/opt/pcre/lib/libpcre.1.dylib
[275] 7D8D4B1B-625D-3D00-A626-C21AFEE0462E 0x00007fff32fc0000 /usr/lib/libcrypto.44.dylib
[276] E02ED3C9-4956-39C4-AF0A-B541EEDD3F1B 0x00007fff330c4000 /usr/lib/libssl.46.dylib
[277] 246874E6-4FD1-35A5-A039-4141999D8CD6 0x00007fff32f86000 /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
[278] 6FEF60A0-F80F-34E3-8845-FEEC92840920 0x00007fff32fbe000 /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
[279] D73E7719-09C8-30E6-B92D-AFC1BB356D92 0x00007fff30535000 /System/Library/PrivateFrameworks/CoreDaemon.framework/Versions/B/CoreDaemon
[280] 3BF95CAF-2EC5-393A-95B7-429C0C5F8F13 0x00007fff30530000 /System/Library/PrivateFrameworks/AppleSRP.framework/Versions/A/AppleSRP
[281] 0DA95074-B315-3B84-B8A3-BAEDC4471A2C 0x00007fff22d5b000 /System/Library/PrivateFrameworks/CoreServicesInternal.framework/Versions/A/CoreServicesInternal
[282] C5007E6E-897B-308F-B173-4F5C6BF3968D 0x00007fff3137e000 /System/Library/Frameworks/FileProvider.framework/Versions/A/FileProvider
[283] BCB8494A-A340-30A1-928C-9B53E953328D 0x00007fff2636a000 /System/Library/Frameworks/Accounts.framework/Versions/A/Accounts
[284] 974A1465-7AA4-347E-8606-CF35160FFA92 0x00007fff3149f000 /System/Library/PrivateFrameworks/GenerationalStorage.framework/Versions/A/GenerationalStorage
[285] F1A41B4E-671A-335E-8AE0-B99688F255C8 0x00007fff31e62000 /System/Library/PrivateFrameworks/CoreSymbolication.framework/Versions/A/CoreSymbolication
[286] 83B5E330-D786-34D2-AEE6-6F7B34D6FE9C 0x00007fff30d9d000 /System/Library/PrivateFrameworks/SymptomDiagnosticReporter.framework/Versions/A/SymptomDiagnosticReporter
[287] F294CD92-94FA-3D7D-9E59-AA5544504264 0x00007fff2e6a3000 /System/Library/PrivateFrameworks/AppContainer.framework/Versions/A/AppContainer
[288] 2144B85E-A433-3F9E-A25D-E8F9D7B6D175 0x00007fff23ea7000 /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
[289] 591F41F3-1FE9-354F-A1F9-BC0CC73333DA 0x00007fff22e90000 /usr/lib/libsandbox.1.dylib
[290] B6B35079-3860-384F-B7A2-584192BAAE70 0x00007fff275e3000 /System/Library/PrivateFrameworks/UserManagement.framework/Versions/A/UserManagement
[291] 0F83FED4-709A-37A0-9E04-EFFD71CB1272 0x00007fff29ee4000 /System/Library/PrivateFrameworks/MobileKeyBag.framework/Versions/A/MobileKeyBag
[292] CBEBDC8A-CCE4-3A21-915C-D2F3FB20B900 0x00007fff347f8000 /System/Library/PrivateFrameworks/ChunkingLibrary.framework/Versions/A/ChunkingLibrary
[293] 12A5A6E2-6C84-3FCA-9927-8B9E241C607F 0x00007fff31e25000 /System/Library/PrivateFrameworks/DebugSymbols.framework/Versions/A/DebugSymbols
[294] 1BEC89D6-6AC8-36C2-89AE-41662D14DC50 0x00007fff2e6b8000 /System/Library/PrivateFrameworks/SecCodeWrapper.framework/Versions/A/SecCodeWrapper
[295] EA306EDB-2A02-3FB6-8812-F5EAA337AEE6 0x00007fff2b145000 /System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
[296] CADD88A2-87A8-3879-8759-D3C112A6844F 0x00007fff4857c000 /System/Library/PrivateFrameworks/OSAnalytics.framework/Versions/A/OSAnalytics
[297] 78AF1606-37D8-3424-92AE-071A9F43AA0F 0x00007fff331d9000 /System/Library/PrivateFrameworks/RemoteServiceDiscovery.framework/Versions/A/RemoteServiceDiscovery
[298] FA39B702-6E08-3F11-953F-5EC1EFB993BB 0x00007fff52553000 /System/Library/PrivateFrameworks/Symbolication.framework/Versions/A/Symbolication
[299] C4BFF5FC-FC9C-3161-9612-7070EDAE9989 0x00007fff331e9000 /System/Library/PrivateFrameworks/RemoteXPC.framework/Versions/A/RemoteXPC
[300] 201350F5-56C4-3342-AD10-4BE20772C52B 0x00007fff396f4000 /System/Library/PrivateFrameworks/OSAServicesClient.framework/Versions/A/OSAServicesClient
[301] 507894FA-6738-372B-9A10-C38AFC9DA079 0x00007fff3dfd2000 /System/Library/PrivateFrameworks/MallocStackLogging.framework/Versions/A/MallocStackLogging
[302] 4925CAB3-09C0-373C-A5D3-9BB5FF025A4F 0x00007fff2e43a000 /System/Library/PrivateFrameworks/Sharing.framework/Versions/A/Sharing
[303] A62689A7-7C22-33B3-90FA-730E4AAEE5B5 0x00007fff32db6000 /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Apple80211
[304] 4A9B2D94-7AB0-3417-8D4E-A9C6C31C31D4 0x00007fff2a569000 /System/Library/PrivateFrameworks/AuthKit.framework/Versions/A/AuthKit
[305] 184C48A4-7F66-395B-81DB-DAA1641383CF 0x00007fff29ce4000 /System/Library/PrivateFrameworks/CoreUtils.framework/Versions/A/CoreUtils
[306] 35927D28-3324-3CAB-9255-276A7022EADC 0x00007fff29c11000 /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN
[307] 070A0F54-5E52-316E-8584-41C0EEDE01F2 0x00007fff2e582000 /System/Library/Frameworks/IOBluetooth.framework/Versions/A/IOBluetooth
[308] F3D8F158-611A-3A5E-9F1D-E5FC273A4C74 0x00007fff32e1a000 /System/Library/PrivateFrameworks/CoreWiFi.framework/Versions/A/CoreWiFi
[309] AA493452-21D3-3EC6-953C-42DBF75F4EB9 0x00007fff2dae2000 /System/Library/PrivateFrameworks/CorePhoneNumbers.framework/Versions/A/CorePhoneNumbers
[310] 9281E3F8-EAFE-3367-B7D5-23687AA6EB70 0x00007fff30ea5000 /System/Library/PrivateFrameworks/DiskManagement.framework/Versions/A/DiskManagement
[311] 964BA0DB-E86E-3F81-BB82-CC27184B6D20 0x00007fff30dbb000 /System/Library/PrivateFrameworks/AppleIDAuthSupport.framework/Versions/A/AppleIDAuthSupport
[312] 6F4B051F-BC33-3880-81B2-5C62DABFF02D 0x00007fff2dabf000 /System/Library/PrivateFrameworks/KeychainCircle.framework/Versions/A/KeychainCircle
[313] D1056292-E38F-313F-B202-EFD871195729 0x00007fff30e71000 /System/Library/PrivateFrameworks/MediaKit.framework/Versions/A/MediaKit
[314] 2B9AA549-63FD-36BB-AE05-ABD5DAC41279 0x00007fff30dc8000 /System/Library/Frameworks/DiscRecording.framework/Versions/A/DiscRecording
[315] 044E6815-6340-31D5-BEE9-760C354E92A3 0x00007fff2b268000 /usr/lib/libCoreStorage.dylib
[316] 9CDB9580-DD1C-347E-91C5-6D92D96F52F5 0x00007fff33254000 /usr/lib/libcsfde.dylib
[317] 88EF8B12-617E-316C-9036-18357E27BC0A 0x00007fff2e6bc000 /System/Library/PrivateFrameworks/ProtectedCloudStorage.framework/Versions/A/ProtectedCloudStorage
[318] B06CCD1C-A4D1-34D2-89C5-0DB69FE65234 0x00007fff3324c000 /System/Library/PrivateFrameworks/EFILogin.framework/Versions/A/EFILogin
[319] 8CF768B8-836B-3DA8-9F0A-8CF840348202 0x00007fff33841000 /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
[320] B88DEE8B-6268-36BA-BF6C-088F3BD8C652 0x00007fff30d3a000 /System/Library/Frameworks/CoreBluetooth.framework/Versions/A/CoreBluetooth
[321] DE8B7844-BFB2-3E3E-9FD0-1C9AD1DC1736 0x00007fff2b260000 /usr/lib/libMatch.1.dylib
[322] 29187EF9-2BD6-3ABF-8088-DE708721B612 0x0000000260003000 /usr/lib/libobjc-trampolines.dylib
[323] 2BC461CE-B7FA-33F3-A36B-64F54FB371B5 0x00007fff41bff000 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/Libraries/libCGInterfaces.dylib
[324] B9BEC3A1-688C-3999-B705-2D50B8756E62 0x00007fff2ba0a000 /System/Library/PrivateFrameworks/login.framework/Versions/A/login
[325] 3E7403EB-156C-36C7-93AD-1356447D5310 0x00007fff4420d000 /System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
[326] 62F25002-C547-3C6C-87D1-0824DD058C82 0x00007fff2da6b000 /System/Library/PrivateFrameworks/MobileAsset.framework/Versions/A/MobileAsset
[327] 157C7AC0-9F73-3FF5-A276-54EC3CAE958A 0x00007fff33866000 /System/Library/PrivateFrameworks/StreamingZip.framework/Versions/A/StreamingZip
[328] C8645EA0-643D-38CF-AA88-3253339192B1 0x00007fff2c20d000 /System/Library/Frameworks/MediaToolbox.framework/Versions/A/MediaToolbox
[329] B123866C-3945-37E7-92EF-BE36AEDD620D 0x00007fff31350000 /System/Library/PrivateFrameworks/CoreAVCHD.framework/Versions/A/CoreAVCHD
[330] 264B30DF-646A-30EC-98B6-20429E9718CB 0x00007fff30504000 /System/Library/Frameworks/MediaAccessibility.framework/Versions/A/MediaAccessibility
[331] 61919815-50C9-3358-836F-B81ED66CCD84 0x00007fff3134c000 /System/Library/PrivateFrameworks/Mangrove.framework/Versions/A/Mangrove
[332] 8713D2A4-ECB6-3B96-BE39-F47851EEF900 0x00007fff30510000 /System/Library/PrivateFrameworks/AlgosScoreFramework.framework/Versions/A/AlgosScoreFramework
[333] 4E96F781-3689-3086-9DF3-60F20487FB9E 0x00007fff31d15000 /System/Library/PrivateFrameworks/AppleVPA.framework/Versions/A/AppleVPA
[334] DAE981E0-5FED-39FD-8443-BD91D6957901 0x00007fff30f91000 /System/Library/PrivateFrameworks/CoreAUC.framework/Versions/A/CoreAUC
[335] 9F31CEDB-5B8A-3BAE-A638-E9F352EF4E87 0x00007fff272e5000 /System/Library/PrivateFrameworks/ViewBridge.framework/Versions/A/ViewBridge
[336] 60C32CD9-3757-361B-B9EB-CB004EFBBEA5 0x0000000263ac7000 /Users/knazarov/dev/virgil/build/angle/libEGL.dylib
[337] 85790259-468E-385F-BC52-ADB02FB43BF8 0x0000000267908000 /Users/knazarov/dev/virgil/build/angle/libabsl.dylib
[338] DD16ED67-125D-35E5-958E-AACCE55249E3 0x0000000267be4000 /Users/knazarov/dev/virgil/build/angle/libc++.dylib

The reason I'm suspecting multiple libc++ versions is because I've found a related ticket from another project that has pretty much the same stacktrace I had: darlinghq/darling#879

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Essentially, libobjc depends on libc++.1.dylib. Why it isn't on the filesystem is beyond my understanding as well. But so far I believe the reasoning is solid. If there's the same stacktrace in another project related to multiple libc++ and we see multiple libc++ versions here, then it's a good start.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Yeah, I've figured out a way to test this. You can run the qemu binary with DYLD_PRINT_BINDINGS=YES env variable and see the following in the console:

...
dyld: weak bind: libc++.dylib:0x10E615B38 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED0Ev, *0x10E615B38 = 0x7FFF2057731A
dyld: weak bind: libc++.dylib:0x10E615FD0 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED0Ev, *0x10E615FD0 = 0x7FFF2057731A
dyld: weak bind: libc++.dylib:0x10E6198D8 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED0Ev, *0x10E6198D8 = 0x7FFF2057731A
dyld: weak bind: libc++.dylib:0x10E615B30 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED1Ev, *0x10E615B30 = 0x7FFF205772C2
dyld: weak bind: libc++.dylib:0x10E615FC8 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED1Ev, *0x10E615FC8 = 0x7FFF205772C2
dyld: weak bind: libc++.dylib:0x10E6198D0 = libc++.1.dylib:__ZTv0_n24_NSt3__114basic_iostreamIcNS_11char_traitsIcEEED1Ev, *0x10E6198D0 = 0x7FFF205772C2
...

The answer in this StackOverflow question is spot on. There are multiple definitions of the same symbols, and they resolve to the system's version of libc++.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

My QEMU with DYLD_PRINT_BINDINGS=YES doesn't have the bind: libc++.dylib:? = libc++.1.dylib:? nonsense. I also believe that is the very cause of the problem.

I ran image list but it's just:

(lldb) image list
[  0] D14327A5-32A0-3ACD-9C14-4EB636E6363C 0x0000000100000000 /Users/person/bin/qemu-system-aarch64 
[  1] B7DB9E8A-A898-3C11-91A0-2B0264F05CB6 0x0000000101040000 /usr/lib/dyld

I found --no-dependents option of target create was enabled by default here, so I explicitly set it false, but it didn't make any difference.

% lldb -v
lldb version 11.1.0

I really feel homesick, or gnusick. I could just info proc mappings if it is gdb.

You may check if a simple program with ANGLE works by running:

cd build/angle && ./simple_texture_2d

Or you may just replace gl=es with gl=core in run. It will use OpenGL directly so it should work without ANGLE although you'll see compatibility issues.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

I believe you see very little output from image list because you didn't run the binary. Try typing the run command in lldb, then let it run for a second and then break and type image list. You should see a complete list then.

The simple_texture_2d works fine for me.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Could you please also post the output of otool -L build/angle/libEGL.dylib?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 5, 2021

Actually I executed image list after run. I'm not sure what's going on.

Here is otool -L build/angle/libEGL.dylib:

objc[919]: Class AMSupportURLConnectionDelegate is implemented in both ?? (0x20e1bb8f0) and ?? (0x1204d02b8). One of the two will be used. Which one is undefined.
objc[919]: Class AMSupportURLSession is implemented in both ?? (0x20e1bb940) and ?? (0x1204d0308). One of the two will be used. Which one is undefined.
build/angle/libEGL.dylib:
	@rpath/libEGL.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libabsl.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libc++.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

I finally managed to get it running. What I did is to build angle with system libc++ like this:

gn gen '--args=use_custom_libcxx=false' ../../build/angle

Now I can run the VM and see some of the graphical output, but it's somewhat corrupted, and the machine can't run into X (hangs).

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 5, 2021

Good news is, when using gl=core I get things working and even get decent fps in glxgears

@GlenHenshaw

This comment has been minimized.

Copy link

@GlenHenshaw GlenHenshaw commented Mar 6, 2021

I can get it to run, and even install Ubuntu, but the keyboard doesn't work. The touchpad works for about two seconds and then stops.

I know this is a newby question, but I've tried every key grab combination listed for Qemu, nothing I can figure out works. Any ideas?

EDIT:

This appears to only be an issue with the Ubuntu login screen. If I have auto login set to on, there is no problem.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

@knazarov Nice to hear you managed to get it work, at least with gl=core. The problem you see with ANGLE would need serious debugging and unfortunately I cannot help much for it since I don't own Intel Mac.

@GlenHenshaw Unfortunately I have no idea. Maybe you may change the emulated input device specified in run.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 6, 2021

@akihikodaki I'll get it running eventually.

As for the libc++, I believe you should change your build instructions to use_custom_libcxx=false. I still think the version on ARM Macs works by chance. The reason you see few libraries in lldb is likely because qemu stops on SIGUSR2 a few times in the beginning. I had it too, it looks like this:

* thread #3, stop reason = signal SIGUSR2
    frame #0: 0x00007fff205f2002 libsystem_kernel.dylib`__sigsuspend + 10
libsystem_kernel.dylib`__sigsuspend:
->  0x7fff205f2002 <+10>: jae    0x7fff205f200c            ; <+20>
    0x7fff205f2004 <+12>: movq   %rax, %rdi
    0x7fff205f2007 <+15>: jmp    0x7fff205d129d            ; cerror
    0x7fff205f200c <+20>: retq

To overcome that, you need to type continue a few times until it runs properly. Anyway, if you don't have a wish do dig deeper into this particular issue, then I believe using the system version of libc++ is safe for everyone.

As for other things, I have a few patches on top of your patchset that actually allow hvf acceleration to work on x86. There were a few problems introduced during the refactoring of hvf-accel that I fixed in my own copy, but which likely need some thought and care to fix properly. I'll send a pull request to your fork, if you don't mind.

With time, I believe it's possible to get it working on x86 just fine. There isn't much of a difference in the core functionality of your code that depends on the platform.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

@knazarov I'm still wondering if we should have use_custom_libcxx=false. ANGLE bundles libc++ for reason, of course, and I think it is expected to co-exist with libc++ from the system because:

  • darlinghq/darling#879 suggests libobjc depends on libc++,
  • ANGLE uses Objective-C++ for some macOS specific code, and
  • ANGLE does not have its own libobjc.

Therefore, if we will properly fix it, we have to allow libc++ from ANGLE and one from the system to co-exist. However, we may add use_custom_libcxx=false only for Intel if we couldn't figure out a solution.


Actually I executed:

process handle -s false -n false SIGUSR1
process handle -s false -n false SIGUSR2

And sent ^C at some arbitrary timings, so SIGUSR2 shouldn't be a problem.


If you found problems with hvf, please check if that refactoring is already in the upstream. If it is, submit a patch to the upstream. Otherwise, reply to https://patchew.org/QEMU/20210120224444.71840-1-agraf@csgraf.de/. I often rebase and force-push the results to my macos branch to ease upstreaming, so you may not send a pull request. I would appreciate if you CC Akihiko Odaki <akihiko.odaki@gmail.com> when you send patches to the upstream. Maybe I should maintain two branches, one for Apple Silicon and one for Intel.


Indeed, my changes are not really specific to Apple Silicon so it is worrying for me that you have problems I don't see on Apple Silicon. I appreciate your efforts, and I think everyone who uses QEMU on Intel Mac would benefit from them, considering some changes you are inspecting are already on its way to the upstream (punch hole change and hvf).

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

@knazarov Can you try replacing DYLD_FALLBACK_LIBRARY_PATH with DYLD_LIBRARY_PATH? Apparently libc++ from ANGLE is inspected first for simple_texture_2d so that may do some trick.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 6, 2021

I’ll try the trick with DYLD_LIBRARY_PATH when I get to a computer.

For now I’d like to point out one thing about libc++ and angle. This would be a speculation of course, because I didn’t contact the authors, so I can’t say I know their intention for sure.

The reason angle ships their own libc++ is because they are part of chrome project, and have to work on multiple operating systems. Some of those systems don’t have an onboard libc++ (e.g. Android).

The build flag for using a system-wide libc++ is completely legitimate and standard for angle. The reason it’s not set by default may have nothing to do with the intended usage in third-party apps, because the primary goal here is to make it work with Chrome I suppose.

I looked into the ninja code generated for angle and didn’t find any traces of namespacing the library, and that is the gist of my argument. If it was intended to mix multiple c++ libs this way, then I would have found the namespacing.

Anyway, if it doesn’t convince you, I may just go to their Slack channel and ask about it.

Regarding hvf, I believe the upstream in qemu works fine for x86. It may be that just your branch has something missing. I can show you the diff.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 6, 2021

I also think they use their own libc++ because it is used by Chromium. It is the most tested configuration and it's kind of natural for them to say "we only support our own libc++ but go ahead if you are using your own libc++ and you can deal with it".

The other variety of the platforms shouldn't have affected, as the default value of use_custom_libcxx is explicitly defined for each platforms. (See build/config/c++/c++.gni.)

Actually Firefox specifies use_custom_libcxx for ANGLE, but I suspect it has its own custom libc++, and the developers have thoroughly tested its configuration. We don't have our own libc++ and perform such QA, so it is just "safe" to use ANGLE's libc++. If ANGLE still breaks, we can file a bug for ANGLE and just say we use the default configuration of ANGLE so it should work. It something else breaks because of libc++, we may, well, just fix it.

Some changes of hvf for Apple Silicon are already merged, but the others aren't. Probably the changes not merged yet cause the problem. You may git blame or bisect and see the cause. It should be in https://patchew.org/QEMU/20210120224444.71840-1-agraf@csgraf.de/ but not in the upstream.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

@akihikodaki regarding my changes to the x86 hvf support - they are apparently fixed in latest qemu master. It's just that the patch patchew/20210120224444.71840-1-agraf@csgraf.de that you brought into your fork, didn't contain everything to make x86 work.

I've tried your suggestion with DYLD_LIBRARY_PATH, and it didn't work (basically, the same problem). I think it's about time I ask ANGLE maintainers about what to do here.

EDIT: or maybe the changes are not in the upstream and I should let the author know.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

And I went ahead and built qemu without optimizations, and now I see this:

* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x2c)
    frame #0: 0x0000000100844c2c qemu-system-x86_64`timer_mod(ts=0x0000000000000000, expire_time=58878) at qemu-timer.c:481:36
   478 	
   479 	void timer_mod(QEMUTimer *ts, int64_t expire_time)
   480 	{
-> 481 	    timer_mod_ns(ts, expire_time * ts->scale);
   482 	}
   483 	
   484 	void timer_mod_anticipate(QEMUTimer *ts, int64_t expire_time)
Target 0: (qemu-system-x86_64) stopped.
(lldb) p ts
(QEMUTimer *) $0 = 0x0000000000000000
(lldb) bt
* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x2c)
  * frame #0: 0x0000000100844c2c qemu-system-x86_64`timer_mod(ts=0x0000000000000000, expire_time=58878) at qemu-timer.c:481:36
    frame #1: 0x0000000100307ed3 qemu-system-x86_64`virtio_gpu_fence_poll(opaque=0x0000000118d581a0) at virtio-gpu-3d.c:581:9
    frame #2: 0x0000000100307e58 qemu-system-x86_64`virtio_gpu_virgl_fence_poll(g=0x0000000118d581a0) at virtio-gpu-3d.c:587:5
    frame #3: 0x00000001003054e9 qemu-system-x86_64`virtio_gpu_handle_ctrl(vdev=0x0000000118d581a0, vq=0x0000000118d60000) at virtio-gpu.c:889:9
    frame #4: 0x0000000100305388 qemu-system-x86_64`virtio_gpu_ctrl_bh(opaque=0x0000000118d581a0) at virtio-gpu.c:897:5
    frame #5: 0x000000010082bafe qemu-system-x86_64`aio_bh_call(bh=0x0000000102652bc0) at async.c:136:5
    frame #6: 0x000000010082bc54 qemu-system-x86_64`aio_bh_poll(ctx=0x00000001033088e0) at async.c:164:13
    frame #7: 0x000000010080efd7 qemu-system-x86_64`aio_dispatch(ctx=0x00000001033088e0) at aio-posix.c:381:5
    frame #8: 0x000000010082cca8 qemu-system-x86_64`aio_ctx_dispatch(source=0x00000001033088e0, callback=0x0000000000000000, user_data=0x0000000000000000) at async.c:306:5
    frame #9: 0x0000000101e3ea7c libglib-2.0.0.dylib`g_main_context_dispatch + 364
    frame #10: 0x000000010083f958 qemu-system-x86_64`glib_pollfds_poll at main-loop.c:232:9
    frame #11: 0x000000010083f3a0 qemu-system-x86_64`os_host_main_loop_wait(timeout=1701600) at main-loop.c:255:5
    frame #12: 0x000000010083f26c qemu-system-x86_64`main_loop_wait(nonblocking=0) at main-loop.c:531:11
    frame #13: 0x00000001003fbdb4 qemu-system-x86_64`qemu_main_loop at runstate.c:725:9
    frame #14: 0x000000010000bef8 qemu-system-x86_64`qemu_main(argc=22, argv=0x00007ffeefbff5c0, envp=0x00007ffeefbff678) at main.c:50:5
    frame #15: 0x0000000100029b34 qemu-system-x86_64`call_qemu_main(opaque=0x0000000000000000) at main.m:115:14
    frame #16: 0x00000001008154f9 qemu-system-x86_64`qemu_thread_start(args=0x000000010302ab50) at qemu-thread-posix.c:521:9
    frame #17: 0x00007fff20604950 libsystem_pthread.dylib`_pthread_start + 224
    frame #18: 0x00007fff2060047b libsystem_pthread.dylib`thread_start + 15

Looks like somewhere there's a problem with timer handling.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

So the last problem is because I had -vga virtio parameter passed to qemu, which in conjunction with -display cocoa,gl=es causes a crash.

The reason for this is because there are two subsequent calls to virtio_gpu_virgl_init(), only one of which succeeds. And the second one leaves g->fence_poll uninitialized. Which laters gets dereferenced in virtio_gpu_fence_poll().

Without -vga virtio I managed to get qemu running with gl=es, though it doesn't recognize my display resolution properly, and maxes out at 1280 x something. Whereas with -vga virtio and gl=off I get a full 5k resolution detected by Ubuntu.

@akihikodaki do you have any idea how I can get it to detect the resolution properly?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 7, 2021

@knazarov I think what conflicting with -vga virtio here is -device virtio-gpu-pci. -vga virtio actually means -device virtio-gpu-vga -device virtio-vga. Remove -device virtio-gpu-pci if you have.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 7, 2021

Yeah, you were right. Removing -device virtio-gpu-pci helps. Though, it's probably a good idea to add a few checks and print out a clear error.

I managed to get the native 5k resolution running and it's pretty sweet! The UI feels smooth.

For some reason, though, it still didn't get the correct resolution of 5120 x 2880, and only proposed 5120 x 2160. I managed to add it through xrandr manually, and get it working.

I'd like to thank you for the great work here. It's really an achievement. Let me know how I can further help with this.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 8, 2021

@knazarov I'm glad to hear you got it working with 5K. It is great, considering that Ubuntu was laggy even for much smaller display (M1 MacBook Air) without Virgil.

I'm not sure what prevents from propagating the correct resolution. virtio-gpu variants, including virtio-vga, uses EDID to report display modes to the guest. You may have a look at EDID on the guest, or by injecting printf to hw/display/edid-generate.c.

Thank you for your offer.

If you would do favor, please check the problems you see (the crash which occurs when you specify both of virtio-gpu-pci and virtio-vga, and display resolution problem) are reproducible with the upstream. If they are not reproducible, they are regressions of my changes, so I have to do some more detailed analysis. If they are reproducible, actually they should be irrelevant from my changes, but the upstream developers will appreciate if you fix them. Please CC me if you get them fixed and contribute the changes to the upstream.

The incompatibility of hvf changes with x86 is worrying for me. It may prevent the upstream from having Apple Silicon support, or the buggy patches gets merged and breaks hvf on x86 on the upstream, which is devastating. Please report the problem to the original author of the patch set to add hvf support on Apple Silicon. I'm also interested in the problem so CC me for this, too.

I'd also like to hear about any new discovery regarding ANGLE's libc++ problem.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 8, 2021

I have a fix for the x86 part ready (see https://gist.github.com/knazarov/ea99dd6fe0ce2bc4f1f67c72b1780a79). I have no experience with the qemu mailing list. How do I submit this suggestion to the author of the ARM hvf patch?

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 8, 2021

@knazarov I saw your patch and found it was my mistake made when merging, not one of the original patch set.
You do not have to submit it to the upstream. I fixed the merge commit according to your patch. Thanks for the patch.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 8, 2021

Since I can't check what specifying both -vga virtio and -device virtio-gpu-pci will lead to on MacOS before your patch (because of the obvious lack of virgl in master), I used a nested virtualization and did the same on Linux. It crashed, but with a different stacktrace. I suppose it doesn't get as far as it did on the MacOS.

It looks like your patch is not the culprit. It's just likely those options should either be mutually exclusive, or there should be a better error handling. I've found a place where it just swallows the error code and doesn't return it to the caller.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 8, 2021

Thanks for the report. It seems QEMU is somewhat vulnerable to strange or exotic host configuration (my MacBook Air, for example), so it is convincing that specifying both -vga virtio and -device virtio-gpu-pci causes a problem on QEMU. However, indeed, it should be handled somehow, and should not cause crash.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 8, 2021

Turns out, the EDID in qemu is somewhat hardcoded. There is currently no way to specify a custom resolution. And to make things worse, the EDID standard is quite complex, and certain resolutions (5120x2880 in my case) require an extension that's not very well documented on the internet (of course there's the formal documentation, but it's not very readable).

By the way, I've figured out that if you pass -device usb-ehci -device usb-tablet instead of -device virtio-mouse-pci then you'll get host mouse pointer integration. Qemu will no longer have to grab your input, which is quite convenient. And the mouse input will become significantly smoother.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 9, 2021

@knazarov QEMU dynamically generates EDID detailed timing descriptor, which has highest-priority for display mode. Toggle "Zoom To Fit" in "View" menu and resize the window to see the guest dynamically responds to window size changes. If you have already choose a different display mode on the guest, choose the display mode which represents the window size.

You can use -device virtio-tablet-pci if you need a absolute pointer input device. An absolute pointer input device provides more seamless integration to the guest desktop environment, but it also makes the guest think you are using "tablet", as its device name implies, and may impose an undesired behavior. For example, gedit provides text selection UI designed for tablet, which makes it impossible to select text with mouse. A relative pointer input device (e.g. virtio-mouse-pci) is better if you use QEMU in full screen mode.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

The EDID DTD is limited to 12 bits of horizontal and vertical resolution, which is maximum 4095 pixels. My screen is 5120x2880. To add resolution this large, a separate extension should be used, called "DisplayID Extension Block".

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 9, 2021

@knazarov Ah, you're right. It is unfortunate that QEMU does not suport DisplayID.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

Here, I've implemented support for the DisplayID extension: https://gist.github.com/knazarov/40348fe805fce1fdaada11c863380109

Now Ubuntu detects my 5k resolution.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 9, 2021

@knazarov Great!

I'm pretty sure others also want the change. I would like you to submit it to the upstream, but actually there is one problem: the dynamic refresh rate definition is in review and not in the upstream yet. You may branch master, cherry-pick akihikodaki/qemu@bc4df9c and akihikodaki/qemu@92890c8, apply your changes, and add a line Based-on: <20210303152948.59943-2-akihiko.odaki@gmail.com> to your commit message.

If you are not familiar with QEMU patch submission, read: https://wiki.qemu.org/Contribute/SubmitAPatch. Please CC me when submitting a patch, of course.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

I'll send the patch, thanks for your help.

In the meantime, I have one more question. I have Caps Lock remapped as Ctrl on my Mac. My Linux VM doesn't even receive the keypress (I tried to use showkey to monitor the scancodes). If I switch it back to Caps Lock, then everything works. Do you know a place in the qemu source code where I could start investigating it? I know other people had this problem as well, so it's not a new one.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 9, 2021

Nevermind, I've figured it out. Pull this patch: https://gist.github.com/knazarov/3c7f1482c9904c953a39c69699c74252

There was a missing case for Right Ctrl in ui/cocoa/view.m

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 10, 2021

@knazarov Actually it was a regression in my patch (again). I applied your change and pushed to GitHub and sent the patch v2 to the upstream. Thanks again.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

Hi @akihikodaki,

I sometimes see a deadlock in view.m when starting qemu. Maybe it's something you've faced yourself. Here's the backtrace

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff205d20c6 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00007fff206022c5 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 81
    frame #2: 0x00007fff206001bc libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 211
    frame #3: 0x000000010081419d qemu-system-x86_64`qemu_mutex_lock_impl(mutex=0x0000000100fc8ff8, file="../../source/qemu/ui/cocoa/view.m", line=588) at qemu-thread-posix.c:79:11
    frame #4: 0x00000001003e90c2 qemu-system-x86_64`qemu_mutex_lock_iothread_impl(file="../../source/qemu/ui/cocoa/view.m", line=588) at cpus.c:491:5
    frame #5: 0x000000010002dba5 qemu-system-x86_64`-[QemuCocoaView handleEvent:](self=0x00000001027870f0, _cmd="handleEvent:", event=0x00000001026c8ce0) at view.m:588:5
    frame #6: 0x000000010002949d qemu-system-x86_64`-[QemuApplication sendEvent:](self=0x000000010350a570, _cmd="sendEvent:", event=0x00000001026c8ce0) at main.m:77:10
    frame #7: 0x00007fff2337af1e AppKit`-[NSApplication _handleEvent:] + 65
    frame #8: 0x00007fff22f0b6af AppKit`-[NSApplication run] + 623
    frame #9: 0x0000000100029645 qemu-system-x86_64`main(argc=23, argv=0x00007ffeefbff5c8) at main.m:166:5
    frame #10: 0x00007fff2061f621 libdyld.dylib`start + 1
    frame #11: 0x00007fff2061f621 libdyld.dylib`start + 1
@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

The reason why view.m blocks is that call_rcu_thread() tries to block the iothread to fetch events:

  thread #2
    frame #0: 0x00007fff205d20c6 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00007fff206022c5 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 81
    frame #2: 0x00007fff206001bc libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 211
    frame #3: 0x000000010081419d qemu-system-x86_64`qemu_mutex_lock_impl(mutex=0x0000000100fc8ff8, file="../../source/qemu/util/rcu.c", line=266) at qemu-thread-posix.c:79:11
    frame #4: 0x00000001003e90c2 qemu-system-x86_64`qemu_mutex_lock_iothread_impl(file="../../source/qemu/util/rcu.c", line=266) at cpus.c:491:5
    frame #5: 0x0000000100823369 qemu-system-x86_64`call_rcu_thread(opaque=0x0000000000000000) at rcu.c:266:9
    frame #6: 0x00000001008154f9 qemu-system-x86_64`qemu_thread_start(args=0x000000010262cbe0) at qemu-thread-posix.c:521:9
    frame #7: 0x00007fff20604950 libsystem_pthread.dylib`_pthread_start + 224
    frame #8: 0x00007fff2060047b libsystem_pthread.dylib`thread_start + 15

But apparently there are no events, so it holds the mutex forever, blocking view.m from progressing.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

No, I'm mistaken. The reason why this happens is because of the e1000 virtualization. It is hvf that blocks that mutex and doesn't release it. I see one of the vCPU threads blocking on sosendto(so=0x000000010357ae60, m=0x00000001058ed400). I don't get the exact reason why.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 10, 2021

Actually I had deadlocks few times, so I debugged and fixed it. -[QemuCocoaView handleEvent:] must be disabled until cocoa_display_init calls dispatch_sync, but it wasn't. Now the fix is pushed to GitHub.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 10, 2021

Maybe there were other deadlocks, but this one is almost surely in the network stack. I'll leave it be for now.

I have another problem with setting the resolution. I don't know what exactly happened, but after some trial and error I started to get weird behavior when I set a 5k resolution, the screen will shortly flicker to 5k, and then back to 4k. After this, the 5k resolution will disappear from the gnome control panel.

After digging through, I've figured out that EDID is apparently not the problem (the guest still has correct values, which I can pull from /sys just fine, and decode them with edid-decode without any issues).

Anyway, I reverted my qemu tree to your macos branch (without any of my changes, including EDID), and tried a few things. Of course without my patches, I can't set a 5k resolution, so I did something different. I've booted into a Wayland session, and tried to change resolution there. And whenever I change the resolution it reverts back to the previous setting after flickering a few times.

Could you please check if you have the same behavior in Wayland too?

The weirdest thing about it is that kernel modesetting and everything before the gnome session (including gdm) don't have any problems with the resolution at all.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 11, 2021

@knazarov You may not use e1000. Generally speaking, virtio-net performs better.

I don't see such a behavior with Wayland.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 11, 2021

It took me a while to figure out. It looks like the refresh rate of 75 Hz is too much for the resolution. Setting 75Hz results in clock speeds around 1.5 GHz. Xorg has a failsafe that refuses to set clock speed that high.

Why I was able to see that mode sometimes, but not always is because you figure out the refresh_rate dynamically, and sometimes it so happens that it gets below 50 Hz or so. And then the generated EDID becomes fine for Xorg. But then when it tries to set the mode, the refresh_rate again gets above 50 Hz, and it just reverts back.

I'm not sure what to do about it, maybe you have an idea. I can just hardcode the lower refresh rate, but it would raise an eyebrow of people who look at the code and see that in one place there's a dynamically set refresh rate, and in others it's hardcoded. What was your reasoning behind setting it dynamically? Does it influence anything at all for a virtual display?

With Wayland there is definitely a bug in Mutter. I've spent a significant time with a debugger and figured out the reason. It temporarily sets the mode without writing it out to the config, to check that the mode is functional, but then a hotplug event comes from udev that causes the config trigger to update the mode from the config (which unsurprisingly contains the old value). I'm not sure what to do about it and whether I should. It's definitely out of scope.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 12, 2021

@knazarov It figures out the refresh rate dynamically in the case that a window is moved across physical displays with different refresh rate.

Knowing the refresh rate of the physical display allows the guest to generate the least required numbers of frames. I also have my selfish reason: the pixel clock in the EDID detailed timing descriptor overflows if it is 75 Hz. I think X.org refuses a high pixel clock to prevent such overflows. Maybe I should have implemented DisplayID as you did or set an upper limit to prevent the overflow at least, but I just implemented dynamic refresh rate which provides the refresh rate of my display (60 Hz) because it is more efficient anyway.

Lowering the refresh rate is not an option for me because I'm upstreaming all of my changes, and it is a breaking change. Someone may actually want 75 Hz, and we cannot break things in such a situation. So I instead decided to lower the refresh rate only when the QEMU display (ui/cocoa) knows it is good for the physical display currently in use.

A problem here is that you apparently have a feedback loop. It means there are two-way interaction:

  • A QEMU display provides QemuUIInfo to the emulated device, and the emulated device (virtio-gpu) tells it to the guest. It is a way from the host to the guest.
  • Another way is the frame from the guest to the host; the guest provides a frame, and the emulated device converts it to a pixman image, which will be passed to dpy_gfx_switch function of the QEMU display. ui/cocoa resizes the window according to the image, which effectively triggers QemuUIInfo change. It is a way from the guest to the host.

The two-way interaction is designed to be immune to a feedback loop by checking there is a difference when updating QemuUIInfo, which is implemented in dpy_set_ui_info function. With a sane QEMU display and a guest, QemuUIInfo should eventually stablize, but apparently it is not the case.

So, I think the next step in debugging this is to find the code path actually leads from dpy_gfx_switch to -[QemuCocoaView updateUIInfo]. Probably observing the change of QemuUIInfo is also a good idea.

By the way, I'm wondering if it really sets a refresh rate lower than 50 Hz. I think the refresh rate of a modern display is 60 Hz or higher, and CGDisplayModeGetRefreshRate, which ui/cocoa uses to get the refresh rate, just tells the number as is. I would like to know if the value of CGDisplayModeGetRefreshRate is stable at least. You can simply add printf after CGDisplayModeGetRefreshRate to see it.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 12, 2021

With a couple of experiments, I found setting display mode with "Settings" application may indeed cause a revert, but it depends on the version.

  • On Ubuntu focal, it does not revert with X11. it reverts with Wayland.
  • On Ubuntu groovy, it reverts with X11. It does not revert with Wayland.
  • On Ubuntu hirsute, it does not revert either with X11 or with Wayland.

So you may use Wayland with Ubuntu groovy or hirsute.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

In view.m you allocate a structure QemuUIInfo on a stack, and don't memset it to 0.
Then, later you try to assign refresh_rate from the result of CGDisplayModeGetRefreshRate if that's not zero. But in my case the result of CGDisplayModeGetRefreshRate is always zero, which leads to info.refresh_rate containing random stuff from the stack.

It's not clear why on my machine the result from CGDisplayModeGetRefreshRate is always zero...

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

Some people say that CGDisplayModeGetRefreshRate is unreliable, and it's better to use CVDisplayLinkGetActualOutputVideoRefreshPeriod. See this for example: glfw/glfw#137

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

Or alternatively, it should be possible to use iokit directly, which is what DisplayLink is doing under the hood.
Maybe this property will do: https://developer.apple.com/documentation/iokit/iodisplaymodeinformation

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 12, 2021

@kanazarov I pushed a fix. Please test it.

I choose Core Video, not I/O Kit, because CGDisplayIOServicePort is deprecated and apparently CVDisplayLink is designed for this kind of purpose, considering CVDisplayLinkCreateWithOpenGLDisplayMask is present. OpenGL rendering relies on CALayer and CADisplayLink seems the best fit for this case, but it was not available on macOS.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 12, 2021

@akihikodaki yeah, it works for me now.

And I figured out the last bit about the Xorg not accepting the mode. It turns out the current EDID settings hardcoded a maximum dot clock to 1200 MHz. I changed it to the appropriate maximum of 2550 MHz, and it was finally accepted. I'm now ready to submit a patch to qemu.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 14, 2021

@knazarov I reviewed the patch. Please check email. It looks fine for me, but did some style nitpicks.

I pushed new changes to GitHub. Briefly:

  • It was rebased to master of the upstream.
  • The fix of the deadlock of -[QemuCocoaView handleEvent:] and cocoa_display_init was incomplete so fixed it again.
@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 14, 2021

@akihikodaki I've just sent corrections to my previous patch according to your comments.

By the way, do you intend to make the qemu window resizable? I only have the ability to make it full-screen and back. In my case there are no active corners that I can drag.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 14, 2021

@knazarov It is now Reviewed-by: Akihiko Odaki <akihiko.odaki@gmail.com>.

Yes, it should be resizable but only if checking "Zoom To Fit" in "View" menu. Also use the top left or top right corner to drag because clicks on the bottom corners are often reacted by the guest.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 14, 2021

Regarding the hanging sendto, I've found a very interesting scenario.

The MacOS reserves .local domain for Bonjour mDNS (looking up nearby devices). On the other hand, in ubuntu there's Avahi that does the same thing. When starting up, Avahi tries to locate qemu.local, and that's when sometimes the sendto call will hang. I managed to workaround it by either disabling ipv6 in qemu, or by changing the default domain for Avahi from local to lan.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 15, 2021

@knazarov I applied a patch to enable the vmnet framework provided by macOS to my tree. You may try how Avahi behaves with it. It should perform better thanks to network stacks integrated into the kernel. See run.sh for the command line with vmnet.

Note that it requires root when initializing. You can drop root before starting the VM by specifying runas option.

@knazarov

This comment has been minimized.

Copy link

@knazarov knazarov commented Mar 15, 2021

@akihikodaki I can live without ipv6 or Avahi at the moment. The proper way to solve this problem will be to isolate it as a small piece of userspace C code and open a bug as an Apple "radar".

The bug happens when sending a broadcast ipv6 mDNS request, so it's very likely somewhere inside the kernel. The reason qemu hangs here is that slirp doesn't account for the fact that sendto may block for UDP, because almost always it doesn't. The completely proper way of course will be to teach slirp to operate UDP sockets with O_NONBLOCK. That wouldn't be trivial though, so I will delay it for later, until the most important things are complete.

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Mar 15, 2021

I see. slirp is the most convenient backend anyway and getting it proper working with IPv6 is still great.

@manu-romero-411

This comment has been minimized.

Copy link

@manu-romero-411 manu-romero-411 commented Apr 10, 2021

Hello @akihikodaki, I've tried your QEMU patch on an M1 MacBook Pro, but when trying to run some graphics demanding job (like the WebGL aquarium) my Debian guest system lags like hell, and I can't figure why. Is it a normal thing in this QEMU build? Thanks in advance

@akihikodaki

This comment has been minimized.

Copy link
Owner Author

@akihikodaki akihikodaki commented Apr 10, 2021

@manu-romero-411 You may use gl=off without running a demanding job as the baseline of performance measurements. On my M1 MacBook Air, Fedora 33 running the WebGL aquarium with gl=es is a bit less laggy than one with gl=off.
There are different factors contributing to worse performance. QEMU build configuration is an obvious suspect. Another possible cause is the graphics stack on the guest including the kernel driver and Mesa userspace library. Debian may have somewhat older software compared with Fedora and Ubuntu, which I tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment