Desktop overlay in VR world
closed
octave crete
At the moment, we can switch to Desktop mode or VR mode on demand but it's not possible to see the Desktop while in the VR world.
It's possible to use OpenVR overlay app like Desktop+ and XSOverlay to do that in SteamVR game, but it does not work with other VR runtime and it adds unnecessary overhead.
Now that VD supports multiple monitor, it has become a viable alternative to Immersed to work in VR, instead of the desktop mode being mostly used to watch videos, launch VR apps and access the desktop occasionally.
Use case for this feature:
- Providing unlimited VR environment when using Desktop mode (ex: work in Half-Life Alyx)
- Easily make information accessible while using simulator (lookup an airport map while flying in MSFT)
- 3D/CAD design using a desktop app while seeing your design in VR.
- Help develop VR app: if you are working on a VR game, you could work in VR and edit your in-game plugins and assets in a desktop app while seeing the effect of your changes in VR. It would work for anything hot reloadable.
To avoid controller conflict, switching the controller to control the desktop or the game, could be modal like today (long press on the left menu button).
octave crete
In a native feature, I would have thought you would embed the desktop capture in the VR stream itself and let the PC client and PC GPU do the work. The headset would receive the same video flux, and no extra work would be needed there. It certainly complicates the feature I imagine though.
3
3nigma
closed
At the current time no standalone headset has the performance overhead to run both the VR stream and the Desktop stream simultaneously, this can be revisited in future when more powerful headsets release
Дмитрий Осипов
3nigma it works well when I use the new Quest's seamless multitasking feature to have some android remote desktop app showing me the desktop being in pcvr environment at the same time.
3
3nigma
Дмитрий Осипов you can use the Metric overlay to see the usage of our desktop stream and the headroom available in VR streaming.
The Multitasking system drops the quality of their overlays to compensate but in app that would mean compromising the purpose
octave crete
I still wish for this feature but I've found a workflow that's pretty good to get your desktop available while your are in a VR app with a Quest 3.
You need to be running the new v69 PTC version and activate the seamless multitasking feature: this allow having one flat application with you while in a VR app. It can be used to display your computer desktop using Meta Remote Desktop tool while in VR.
I initially tried to use Meta Remote Desktop with VD but it would not work at the same time: Remote Desktop would stop working as soon as you VD was in desktop mode. Also, Meta Remote Desktop is pretty terrible: it slows the headset down, has too much latency and does not look great.
The solution to get a good desktop experience in VR was to use Sunshine / Moonlight.
Workflow is as following:
- Sideload Moonlight Android Client using Sidequest
- Have your Desktop at 1920x1200 resolution*
- When opening Moonlight, resize it to the max
- In the setting, set the Video resolution to it's native resolution of 2000x1200
- Start streaming the desktop
* if your physical monitor does not support it, add a virtual monitor using VD. In that case, you need to use the dxgi-info.exe util to find the monitor id (ex: \\DISPLAY12)
Result is that you get your desktop, in VR, in good quality, usable with no noticeable latency.
Once it's setup, it works like a charm.
Only major drawback is that it only support a single monitor.
k
kanata
Likewise.
The Virtual Desktop image quality is crystal clear, how great would it be if this could be overlaid while using SteamVR.
lvsas
This is exactly the type of feature I've been eagerly waiting for to enhance my VR experience.
And yes using overlay apps like Desktop+ and XSOverlay adds unnecessary overhead. Integrating this feature directly would reduce friction and significantly improve user comfort, marking a major advancement.