Skip to content

Instantly share code, notes, and snippets.

@KhaosT
Last active July 26, 2024 05:17
Show Gist options
  • Save KhaosT/e9b60fc0fb99b9f4512759b953cbf38c to your computer and use it in GitHub Desktop.
Save KhaosT/e9b60fc0fb99b9f4512759b953cbf38c to your computer and use it in GitHub Desktop.
Guide for using Apple Vision Pro as HDMI display

Displaying HDMI sources on Apple Vision Pro

While it's possible to stream most content to Apple Vision Pro directly over the internet, having the ability to use Apple Vision Pro as an HDMI display can still be useful.

Since Apple Vision Pro does not support connecting to an HDMI input directly or using an HDMI capture card, we have to be a little creative to make this work. NDI provides the ability to stream HDMI content over a local network with really low latency, and it works great with Apple Vision Pro.

This page shows the setup I’m using.

What’s needed

  • HDMI NDI Encoder
    • Personally, I recommend the Kiloview N40, as it supports streaming up to 4K60 while being fan-less, and can use USB-C as a power input.
    • Kiloview N60 comes with more features, but it’s larger and has a fan.
  • Vxio app
    • I made this one, you can also use other NDI monitor apps as well.
    • Separately, Finn made Castaway that makes it possible to use a USB Capture Card + Mac/iPad instead of an NDI encoder. It's a more cost effective option for most people I imagine.

Things to consider

  • To ensure high video quality, an NDI stream typically consumes significantly more bandwidth than a typical H.264/H.265 stream. Before purchasing the encoder, make sure your network can support data transfers at that speed.
    • For 4K60 stream, the bandwidth required is typically around 250~300mbps. If you're having issue to stream smoothly, try go to the encoder's settings page and reduce the streaming quality.
    • You can test this with some free software NDI encoders.
  • Since this setup requires encode and decode the video content, it is not latency free. From my experience, I typically see 3~4 frames (~50ms, same as moonlight based on my measurement) of delay from the content source.

Setup

Setting up an NDI encoder is pretty straightforward. Just connect the device to your local network via Ethernet, supply power, and connect the HDMI input to the encoder.

If the visionOS shows the Local Network permission dialog, and after granting the permission the app still shows searching, you may need to force close the app, and reopen it.

After that, you should be able to see the encoder show up as a source in the Vxio app. Select the source, and you'll get your HDMI display 🎉

Screenshot

Misc

Adjusting audio queue size

Depending on the NDI encoder, the default audio queue size might be too small for some devices. In that situation, try increasing the audio queue size so that the audio no longer experiences abrupt stops between samples.

Disable Multi-TCP

If your Wi-Fi channel is congested, you can try force N40 to use UDP instead of Multi-TCP. In my experience, UDP performs a lot better in those network condition. You can do so from the N40's configuration UI, select NDI Connection -> Multi-TCP Disallowed.

UDP

Portable setup

I was able to make this setup portable by getting one of those mini router (GL.iNet Beryl AX) with a LAN port.

IMG_3978 IMG_3979

#AppleVisionPro #visionOS

@Yamhao
Copy link

Yamhao commented Jun 2, 2024

NDI 6 and HDR support is added in the latest update. Now we just need the actual hardware to support that 😅

I was told a few weeks ago that they were targeting June/July for the firmware update for the Kiloview N60

Only for N60? Any chance for N40? Thanks

@gjc9620
Copy link

gjc9620 commented Jun 3, 2024

@Yamhao Is there a cheaper option? 1080p is acceptable

@lightsailvr
Copy link

Trying to feed this an HDR signal from NDI 6 (Using Nobe Omniscope) but it doesn't seem to be decoding it properly. Is there some sort of setting for HDR?

@Jefe533
Copy link

Jefe533 commented Jun 20, 2024

Sending a PQ image (looks low contrast, desaturated) from DaVinci Resolve to Vxio via NDI 6. I think there needs to be some way to let Vxio know the image is PQ and to “trigger” Vxio to go into “HDR Mode” and invert the PQ curve, displaying properly super bright and saturated HDR imagery.

@KhaosT
Copy link
Author

KhaosT commented Jun 20, 2024

Are you sure the tools you’re using encodes HDR information correctly in NDI frame? The app consumes the metadata from the NDI frame to determine whether the source is streaming HDR and what transfer function it should use.

@Jefe533
Copy link

Jefe533 commented Jun 20, 2024

The tool I'm using to encode my PQ signal to NDI may not be storing the HDR information correctly in the NDI frame. I'll follow up with that developer. It could still be helpful in your app if there was a setting to "force" a transfer function for a case like mine.

@KhaosT
Copy link
Author

KhaosT commented Jun 20, 2024

I’m not sure that would work if the developer didn’t encode the stuff correctly since the underlying video data might be wrong in that case.

You can learn more about the metadata needed in at https://docs.ndi.video/docs/sdk/hdr

and use the test pattern app on Windows to verify HDR is working correctly.

@zixzaxlad
Copy link

I was wondering if it's possible to input a 3D HDMI signal and view it in Vision Pro as a stereo window?

@kda186
Copy link

kda186 commented Jul 11, 2024

Thanks KhaosT for the great tip!!!

I have also completed the ndi process for avp, I am adding ffmpeg and other features, if you have any good ideas please let me know and I will implement it.

At present, I have added various modes of 3D display. Yes, it is possible to use NID for 3D transmission, and it is also very good to play back at 4K 60 in VP.

For now, I have a few minor issues that need to be addressed:

  1. In ImmersionStyle, my expected decoding picture texture scaling does not seem to be successful, and the picture is always rendered at a fixed size.

  2. When I compare the decoded playback picture with the MAC full-screen picture after USB connection, the brightness will be one level lower. I plan to use shader graph in reality composer pro to solve the problem, but I can't always achieve good results.

I am new to apple development. You can say this is my first mac application development. If anyone can give me some advice, I will be happy to buy him coffee.

@woncheol-kim
Copy link

Is it possible to use the Vxio app as a client and the OBS-NDI app on a desktop computer as a server? I’ve tried this setup, and while the connection seems to work, the screen remains empty. The same issue occurs with the Castaway apps: the connection is established, but the screen is empty. However, the HDMI capture board works fine with OBS Studio.

@KhaosT
Copy link
Author

KhaosT commented Jul 19, 2024

Re kda186: you might want to try UnlitMaterial. PBR material reacts to light, and depending on the environment light it might behave differently.

Re woncheol-kim: Are you trying to display Vision Pro content in OBS, or displaying OBS content in Vxio? For Vision Pro -> OBS, the app does not act as a NDI source. For OBS -> Vision Pro, it should work as long as the NDI output is configured correctly on the OBS side.

@woncheol-kim
Copy link

Re woncheol-kim: Are you trying to display Vision Pro content in OBS, or displaying OBS content in Vxio? For Vision Pro -> OBS, the app does not act as a NDI source. For OBS -> Vision Pro, it should work as long as the NDI output is configured correctly on the OBS side.

Thank you. What I want to set up is OBS-to-VP, and since you have confirmed that it’s possible, I’ll check my OBS settings.

@woncheol-kim
Copy link

Now it works fine. However, I have a request: it seems that the Vxio app requires UPnP to be enabled in the router settings. I wish I could disable UPnP and manually input a server IP address in the Vxio app instead.

@haseebrabbani
Copy link

haseebrabbani commented Jul 19, 2024

hey @KhaosT loving your work on this app! been playing my PS5 on a super huge screen. I got the Kiloview N60 in the hopes of enabling HDR, but after tweaking its settings (on latest firmware which im not even sure has added support yet) im just not sure if the Vxio app is detecting HDR video. is there an indicator in the Vxio app that would tell me whether HDR is detected or not? if not, might be helpful to others as well 🙏

@KhaosT
Copy link
Author

KhaosT commented Jul 19, 2024

@haseebrabbani I don’t think the current firmware supports NDI 6, which is required for HDR. I can look into adding indicator support for this.

@BMJ9
Copy link

BMJ9 commented Jul 26, 2024

I’ve been loving your app on my AVP and iPad! On the iOS app, is it possible to have an option to remove the status bar at the top of the screen? Sometimes that gets in the way when I use your app on an external monitor.

@KhaosT
Copy link
Author

KhaosT commented Jul 26, 2024

@BMJ9 going to add it in the next update 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment