Android XR Will Let Apps Access The Passthrough Camera View, Google Confirms
Android XR will let apps access the passthrough camera view to, for example, run custom computer vision models, Google confirmed.
Android XR will let apps access the passthrough camera view, Google confirmed.
While headsets like Quest 3 and Apple Vision Pro use cameras to let you see the real world, today only the system software gets raw access to these cameras. Third-party developers can use passthrough as a background, sure, but they don't actually get access to it. They instead get higher-level data derived by the system, such as hand and body skeletal coordinates, a 3D mesh of your environment with bounding boxes for furniture, and limited object tracking capabilities. That means they can't run their own computer vision models, which severely limits the augmentation capabilities of these headsets.
The exception is that on visionOS 2, Apple is now giving enterprise companies raw access to Vision Pro's passthrough cameras for non-public internal apps, but this requires a special licence from Apple and is restricted to "in a business setting only".
David Heaney
Now, Google confirmed to veteran VR developer Antony Vitillo that Android XR apps will get access to the front camera view by requesting the same Camera permission as regular Android apps and using the same APIs.
Here's what Google told Vitillo:
• A developer can request camera_id=0 for the main world-facing camera stream. This corresponds to the “rear camera” in standard Android parlance. Similarly, a developer can request camera_id=1 for the main selfie-camera stream. This corresponds to the “front camera.” Both streams are accessed through the standard Android Camera APIs, i.e. Camera2 and CameraX
• To get access to the world-facing camera feed (similar to the back camera on a smartphone), the app will have to request camera permissions (analogous to the permission requested for camera access on phones)
•When apps request the selfie-camera, they receive an image stream containing the user’s avatar. This avatar video stream is generated by Avatar provider apps/services on the headset. These apps generate the stream based on user tracking data from OpenXR APIs, e.g. head, hand, eye, face. This tracking data is obtained from the physical inward-facing cameras that monitor the user’s movements and facial expressions.
UploadVR has also reached out to Google for independent confirmation.
At Meta Connect 2024, back in September, Meta announced that it would give Quest developers access to passthrough via a new API coming in early 2025, suggesting that Quest too will get this functionality soon.
