Metal-based Apps on visionOS
With Unity, users can leverage familiar workflows to build Metal-based apps using Compositor Services for visionOS, allowing users to create fully immersive or mixed immersion experiences. Today, Unity provides a wide range of features and APIs that can be used to develop experiences for visionOS. Such packages include:
- visionOS plug-in
- XR Interaction Toolkit
- XR Core Utilities
- Input System
- VR project template
- Hand tracking
To get started, simply select Apple visionOS on the XR Plug-in Management window in Project Settings, then navigate to the Apple visionOS project setting below XR Plugin-in Management.
Set the App Mode to Metal Rendering with Compositor Services. If you are using ARKit features, you may also need to set a Hands Tracking Usage Description and/or a World Sensing Usage Description.
The Apple visionOS project settings menu has a few properties pertaining to the Metal Rendering with Compositor Services app mode.
| Property | Description |
|---|---|
| Initialize Hand Tracking On Startup | Initializes hand tracking when application starts up. |
| Metal Immersion Style | Defines the immersion style to be used in Metal-based apps. Immersion style determines whether an app will have pass-through. For Metal-based apps, only two immersion styles are valid - Mixed and Full. The Progressive immersion style is not supported with Metal-based apps. |
| Automatic | For Metal-based apps, choosing Automatic will cause the app to fallback to Mixed immersion style. |
| Full | With this immersion style, pass-through is disabled, and only the skybox will be shown. |
| Mixed | With this immersion style, apps will have pass-through video feed. This immersion style allows an app to switch at runtime between allowing pass-through and rendering the skybox. |
| RealityKit Immersion Style | Defines the immersion style to be used in RealityKit-based apps. Immersion style determines whether your application will have pass-through. Although all immersion styles are applicable to RealityKit-based apps, the effects of some immersion styles will differ depending on the VolumeCamera Mode. |
| Automatic | For RealityKit-based apps, choosing Automatic will cause the app to fallback to Mixed immersion style. |
| Full | If all VolumeCameras are set to Bounded Mode, content will still appear with pass-through, as if the app was in Mixed immersion style. If one VolumeCamera in the scene is set to Unbounded Mode, pass-through will be disabled and replaced with a black background, and only content will be visible. The Digital Crown will have no effect. |
| Mixed | With this immersion style, apps will have pass-through video feed. VolumeCamera Mode will have no effect on this immersion style. |
| Progressive | If all VolumeCameras are set to Bounded Mode, content will still appear with pass-through, as if the app was in Mixed immersion style. If one VolumeCamera in the scene is set to Unbounded Mode, content will appear inside a radial portal with a black background. See Progressive Immersion for more details. |
| Upper Limb Visibility | This option allows for controlling whether the hands are visible within the app. This option is ignored if the App Mode is Windowed, or if the App Mode is set to RealityKit but the scene consists of only Bounded VolumeCameras. |
| Automatic | Hands visibility will be determined by visionOS's default hands setting, which will depend on the context. |
| Visible | In Metal App Mode, the hands will always be visible in the app. Hands always render on top of virtual content. In RealityKit App Mode, the hands are blended with virtual content based on depth. |
| Hidden | In Metal App Mode, the hands will be hidden, which may be beneficial if you wish to present virtual hands instead. In RealityKit App Mode, the hands will be displayed behind virtual content. |
| Foveated Rendering | This option controls whether foveated rendering is enabled or disabled. This option only applies to Metal-based content, and requires the Universal Render Pipeline. |
| IL2CPP Large Exe Workaround | When building your project in Xcode, you may encounter an error such as ARM64 branch out of range, especially with larger projects. This option will patch the project to work around this error. |
After modifying the project settings, select and build for the visionOS platform in the Build Profiles menu. This will generate a Unity XCode Project. From here on, you'll continue the build process in the XCode project, debugging in XCode as needed.
Please note that Unity is still building towards feature parity with the Metal API on XCode, so you might observe warnings from Metal’s API validation layer. To work around this, you can turn off the Metal API Validation Layer via XCode’s scheme menu.
Pass-through Video in Metal Mode
With visionOS 2, apps set to Metal Rendering with Compositor Services can now show pass-through instead of just the skybox. The steps for enabling pass-through with Metal-based apps are the same as the steps for enabling pass-through with Hybrid apps.
- Set the Metal Immersion Style under the Apple visionOS project setting to Mixed or Automatic.
- Set your Unity Camera's Background Type to Solid Color.
- Under Background Type, click on Background. Set the RGBA values for the background color to 0. You can technically use any color here, but ensure the Alpha value is set to 0.
You may switch Background Type from Solid Color to Skybox in order to switch from showing pass-through to showing the skybox, and vice-versa.
Note
When you set Background Type to Skybox (both in the Inspector and via scripting API), Unity will change the alpha value of the background color to 1. When you switch back to pass-through, you must also set the background color to something transparent.
Note
You will need to disable HDR on your Universal Render Pipeline Asset to get pass-through to work.
Porting VR experiences to visionOS
For users who are looking to port existing VR titles looking to visionOS as a fully immersive experience there are a few things you can do to make the transition smoother, in addition to the build workflow elaborated in the above section.
1. Prepare your graphics
Use Universal Render Pipeline (URP) - visionOS supports Foveated Rendering, a technique that provides a higher-quality visual experience for users on the Vision Pro. To take advantage of the Foveated Rendering feature, we recommend URP, which allows for Foveated Rendering to be applied throughout the entire pipeline. We also plan to focus future improvements on URP specifically.
Leverage Single-Pass Instanced Rendering - Unity’s Single-Pass Instanced Rendering now supports the Metal Graphics API and will be enabled by default. This reduces the overhead of certain parts of the rendering pipeline like culling and shadows, and also helps to reduce CPU overhead when rendering your scenes in stereo.
Ensure depth-buffer for each pixel is non-zero - on visionOS, the depth buffer is used for reprojection. To ensure visual effects like skyboxes and shaders are displayed beautifully, ensure that some value is written to the depth for each pixel.
Check Project Validation - there are a number of settings and scene configuration details that you will need for your app to function properly. They will be listed under Project Settings > XR Plug-in Management > Project Validation under the visionOS platform tab. If you see any issues, you can click Fix or Fix All to automatically resolve them. Some specific issues to look out for:
- The
Apple visionOSloader must be enabled - Your scene needs an
ARSessionin addition to the normally requiredTrackedPoseDriverfor head tracking to work - All cameras need a depth texture (you should enable
Depth Texturein your URP settings) - The only supported
Depth Texture Mode(setting onUniversalRenderer) isAfterOpaques
2. Adapt your controller-based interactions to hands
Leverage the XR Interaction Toolkit (XRI) - With visionOS, people will use their hands and eyes to interact with content. XRI provides a high-level interaction that abstracts implementation of hand-tracking and makes it easy to implement interactions like hover, grab and select in 3D space and in the UI for 3D spatial worlds. With XRI, you can also implement visual feedback for input, and its abstraction of input also makes it possible to author once, for multiple platforms.
Dive into Unity Hands package - If you’re looking to access even more flexibility in your input actions, Unity’s Hands Subsystem provides access to all the raw hand joint data via the Unity Hands Package.You can also learn about leveraging the Unity Hands Package for immersive experiences above.
For more information on porting your VR experience to visionOS, we recommend watching this guide to “Bring your Unity VR app to a fully immersive space”.