Skip to content

Instantly share code, notes, and snippets.

@KhaosT
Last active February 28, 2024 03:07
  • Star 34 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save KhaosT/a4d90479e3cb0aae959ad55b08cd8358 to your computer and use it in GitHub Desktop.
Guide for clouding gaming on Apple Vision Pro

Cloud Gaming on Apple Vision Pro

GeForce Now

Cloud Gaming is a great way to enjoy graphically demanding games on Apple Vision Pro.

Since Safari on visionOS does not support PWA mode, here is how you can access cloud gaming services on Apple Vision Pro.

What’s needed

  • A compatible game controller
    • visionOS supports game controller from Xbox, PlayStation, and Nintendo Switch.
    • visionOS is really opinionated on where to send game controller events so if your controller is not working with the service you're using, try tap the center of the view and that should inform visionOS that this app should receive the game controller events.
  • Nexus⁺ app
    • Disclosure - I made this because I wanted to access these services myself. Some other apps may also available.

Using GeForce Now

GFN

GeForce Now is the ideal cloud gaming service to use on Apple Vision Pro. With 4K resolution support (GeForce Now Ultimate membership required), the visual quality of the games is stunning.

GFN-Bookmark

After pairing the game controller with Apple Vision Pro, you can access GeForce Now from the bookmarks.

Before starting your first game session, make sure you go to GeForce Now settings and adjust the picture quality for higher resolution streaming, as the default option may not be optimal for a large canvas. (Due to the lack of H.265 support in Safari's WebRTC implementation, you may experience visual glitch streaming in 4K 120FPS configuration. In that case try switch back to 4K 60FPS instead.)

If you run into issue starting the game session, try turn off Turbo mode from the edit menu.

GFN-Edit

Using Xbox Cloud Gaming

Xbox

Xbox Cloud Gaming is also available and offers a wide range of console games that might be interesting to explore. The service currently only supports streaming up to 1080p.

Xbox-Bookmark

You can access Xbox Cloud Gaming from your bookmarks.

The app also includes an option to use Better xCloud with the website. Better xCloud enables Remote Play from your Xbox console and offers a wide range of options you can tweak. You can enable the script by long-tapping on the bookmark option to open the context menu, select Edit, and enable Better xCloud.

Xbox-Edit Xbox-Edit-Script

To update the Better xCloud script, long-tap on the row to show the context menu and select Update to download the latest script.

Xbox-Script-Update

#AppleVisionPro #visionOS

@KhaosT
Copy link
Author

KhaosT commented Feb 9, 2024

If it still doesn't work, can you try disable iCloud Private Relay from Settings app and see if that resolves the issue.

Will try this now. Just want to make sure that I should be able to login without a subscription right? A sub isn't required to login to a newly created account?

I never tried that so I'm not sure. Maybe you can try go to GeForce Now on a computer (with Chrome) and see if it shows you any alerts?

@KhaosT
Copy link
Author

KhaosT commented Feb 9, 2024

If it still doesn't work, can you try disable iCloud Private Relay from Settings app and see if that resolves the issue.

Will try this now. Just want to make sure that I should be able to login without a subscription right? A sub isn't required to login to a newly created account?

You might need to go to https://www.nvidia.com/en-us/geforce-now/memberships/ and explicitly select Join the free plan for your account if you never used the service in the past. Hard to tell if that's what happened here...

Screenshot 2024-02-08 at 7 34 38 PM

@NayrusLove
Copy link

That’s exactly what happened. I joined in the free plan and now everything is working!

Thanks so much for your help!!!

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

Feature requests:

Auto Hide Site Title Pill

  • Showing the site name above the window at all times is redundant and kills immersion
  • Either have it disappear after a few seconds and then reappear for a few seconds if user pinches inside the window or the handle bar
  • Or at minimum offer it as a user option (show/hide site title above window at all times)

Add default sites

  • Add YouTube
  • Add Netflix

@AvoidMyCode for Auto Hide Site Title Pill, any recommendation on how you would expect to get the pill back? I'd like to do that but since visionOS doesn't tell us about the window bar interaction, I'm not really sure what's a good way for the user to get it back after they hide it...

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

How about if the user pinches or touches inside the window frame (assuming that console controllers can be distinguished as separate interactions and excluded).

The user would be pinching or touching inside the window typically to interact with the webpage e.g. when selecting a show or game etc. This implies that the user is not passively viewing the window but is actually interacting with it, so I think it would be fine to have the additional UI element above the window reminding them what window they’re in.

After a few seconds of non-interaction, we can assume the user is leaning back and consuming the content in the window.

I did that for the HDMI streaming app since the user cannot interact with the content. However for this one, since Nexus⁺ is a browser, the gestures cannot interference with web browsing experience (since not everyone is using it for non-interactive content consumption) so pinch/touch/long press gestures cannot be used here... 😅

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

I think I got something, we'll just put an invisible button at the place where the menu should be and hope people can know to tap that...

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

It’ll be worth testing to see if people accidentally look at a game/video element inside the window toward the top middle whether it would mistakenly select/show the invisible button.

Since you'll have to look outside the window at the place where the menu button is at, it's probably less likely to trigger that unexpectedly. It's just a little weird that people need to look at the void and perform a tap gesture there... We'll see if it works 🙃

Juno and it seems like it’s possible to show/hide buttons around the edges of a window by when the user touches the window and then make them disappear after a few seconds.

Yeah for Juno it's mostly fine since when you're looking at the YouTube video, it can safely intercept all gestures, where for us here it's not really an option since we don't know what kind of website the user is looking at and whether it's safe to intercept a tap or not.

RPReplay_Final1707678566.mov

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

I'll submit this as v1.0.4 ^^ Should be available next week.

As for additional bookmarks, video streaming services don't really work that well in third party browsers right now since visionOS's web page full screen API is kinda broken so I'll probably skip those for now. If user are okay with the compromised experience, they can add those as bookmarks themselves 👍

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

This looks amazing to me. It’s literally perfect. However, I fear that anyone downloading the app without context may find it confusing switching to another bookmark. They’ll need to read an instruction upon running it off the first time that they need to look up. In my experience 50% of users don’t bother reading :)

Yeah since it's an option the user has to select themselves and we don't persist it across sessions, if somehow they got stuck they can probably get out by force quitting the app, and get back the controls. Hope that's sufficient to not get people mad 🤞

I exclusively only use them with Nexus+ and haven’t had any issues. Both work perfect. Right now, Nexus+ is the only way to experience Netflix like it’s a native app.

Interesting, I tried Netflix in Safari and it seems okay? What makes using Netflix in Nexus⁺ better for you?

@KhaosT
Copy link
Author

KhaosT commented Feb 11, 2024

What do you mean by “we don’t persist it across sessions”?

Oh it just means the user need to select that "Hide Menu" option to hide it for that specific window during that run. I didn't make it persistent (like a setting) since I worry that the user may forget how to get back to the menu and feel they are stuck inside the app 😅

Re Netflix

That's an interesting perspective. I guess it's really come down to Netflix not wanting to make an app for AVP right now... hope they change their mind in the future.

As for custom environment to dim the surrounding, the app provides the environment dimming option using the Apple's API (that auto dimming thing). Is it not strong enough for your use case? I have a prototype that makes the environment 100% black but that's just too much I feel 😅

For quickly switching websites, one tip is you can add shortcuts to open specific site in your bookmarks, and ask Siri to do that for you. Hope once Apple makes it possible to add shortcuts to Home Screen, people can make custom icons that go directly to specific site from Home Screen.

Screenshot 2024-02-11 at 1 47 02 PM

Simulator Screenshot - Apple Vision Pro - 2024-02-11 at 13 47 26

@KhaosT
Copy link
Author

KhaosT commented Feb 12, 2024

I don't understand why you can't know if a user is pinching/interacting with something inside the screen, and when they're not.

The main challenge is really around what's being shown in the window area is a web page, and the people is expected to interact with the web page the app has no idea what looks like... For video player, they can be certain the user's interaction with the window is intended to bring up the control since the video itself is not expect to be interactive. For a web browser I can't make that assumption 😅 Having the menu to show up randomly when user interact with web page can be super confusing/random/distracting (like imagine someone tries to read an article on a website and they are a slow reader) since the user's intent may have nothing to do with the menu.

@Cysmic
Copy link

Cysmic commented Feb 14, 2024

Do vibrations work with any type of controller currently @KhaosT ? If not do you have plans to add them or is it fundamentally not possible for some reason?

@KhaosT
Copy link
Author

KhaosT commented Feb 14, 2024

visionOS currently doesn't support the Gamepad.vibrationActuator API so it's not possible to make a controller to vibrate from a web page. Hope they change that in the future.

@Cysmic
Copy link

Cysmic commented Feb 14, 2024

That is disappointing. Hope that changes soon

@alvarengarichard
Copy link

This is amazing — GeForce now is a killer app for the headset to me, but encountering a bug.

Has anyone been able to bypass the sign in for GeForce now on Xbox Gamepass? Whenever I launch a game I get an Xbox Login Page prompting me for username / password, despite being logged in via the GeForce Now settings.

I can’t seem to get the AVP Onscreen Keyboard to pop when it’s asking me for credentials so can’t launch any Gamepass Games, any work arounds?

@KhaosT
Copy link
Author

KhaosT commented Feb 18, 2024

@alvarengarichard Yeah GFN doesn't auto login you to Xbox GamePass. You'll need to manually open the keyboard from the upper left corner to input text.

Simulator Screenshot - Apple Vision Pro - 2024-02-17 at 18 56 03

@alvarengarichard
Copy link

@KhaosT nice! Didn’t see that menu item. The input was still weird, but I ended up connecting a magic keyboard to make it work.

I ordered an Xbox controller to test as well as I can’t seem to get my ps5 controller working consistently, only via GeForce Now and not via Xbox Cloud.

@KhaosT
Copy link
Author

KhaosT commented Feb 18, 2024

visionOS is a little buggy when it comes to controller. If it doesn’t work, try force close the app (long press both side button and Digital Crown) and reopen it. Typically it works after that.

@Fofer
Copy link

Fofer commented Feb 20, 2024

The input was still weird, but I ended up connecting a magic keyboard to make it work.

Another method that works is Universal Control from a Mac. Connect to its screen to add it as a window (even just temporarily) and then you can use that Mac's keyboard to type in Nexus, GeForce Now, Microsoft's login page, etc. In fact Universal Control works in any app I've tried, even the iPad apps in the "Compatible Apps" folder. Sure, It's annoying that we have to log in each time to play some of these games, but at least this is a cool way to do so :)

And these are still early days, of the cutting edge. Thanks for all of the great tips and discussion here!

@KingNothing81
Copy link

KingNothing81 commented Feb 20, 2024

Now that Apple is disabling PWA support on iOS for EU users, I think it would make sense to also do an iOS version of your browser. So you can keep using cloud gaming services fullscreen on iPhones in the EU. And I think many EU users would be really thankful for that.

https://www.theverge.com/2024/2/15/24074182/apple-drops-support-iphone-web-apps-eu-dma

@Git-Axo
Copy link

Git-Axo commented Feb 22, 2024

Love what you did here! Cloud gaming has big potential for VR

@jbosse
Copy link

jbosse commented Feb 24, 2024

This is very cool. Have you been able to get a vscode tunnel to work? I can get to VSCode but when I connect to my tunnel I get:
Visual Studio Code
Fatal Error 🙈 Please check Developer Tools 🐱💻

Obviously visionOS doesn’t let you open Dev Tools.

@KhaosT
Copy link
Author

KhaosT commented Feb 24, 2024

Yeah service worker is currently not available due to OS limitation, I’m trying to work with Apple to get the special entitlement for this to work. Hope they grant it…

@jbosse
Copy link

jbosse commented Feb 24, 2024

🤞🏻

@bryangio
Copy link

Thanks for all your work on this. GeForce now at 4k works great. Question: Is there a way to get hdr working in GeForce now?

@KhaosT
Copy link
Author

KhaosT commented Feb 25, 2024

Thanks for all your work on this. GeForce now at 4k works great. Question: Is there a way to get hdr working in GeForce now?

I don't believe so, until visionOS supports proper H.265 WebRTC codec, we can't really do HDR...

@Vanders0
Copy link

Seems like it won’t let me launch any Geforce Now games on Nexus+ that don’t have controller support. Any workarounds to get it working with keyboard and trackpad?

Using GFN on my Mac with virtual display is fine, but I’d love to cut out the middle man and run solely on the AVP.

@KhaosT
Copy link
Author

KhaosT commented Feb 28, 2024

That’s a GFN decision, they decided on iOS variants, keyboard and mouse will not be supported 😔

@redphx
Copy link

redphx commented Feb 28, 2024

That’s a GFN decision, they decided on iOS variants, keyboard and mouse will not be supported 😔

I think it's because both WebKit and Chromium don't support PointerLock API on mobile.

You can test it here: https://mdn.github.io/dom-examples/pointer-lock/

@Vanders0
Copy link

Bummer. Thanks for the response. And for Nexus +. Super useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment