Major Developers Are Already Using Quest's Passthrough API
Meta will host Niantic, Creature, and Resolution to "showcase real-world examples of developers who are already leveraging our latest Passthrough Camera API" at GDC next week.
Meta will host Niantic, Creature, and Resolution to "showcase real-world examples of developers who are already leveraging our latest Passthrough Camera API" at GDC next week.
The revelation comes from a listing for a GDC 2025 Meta Developer Summit called Merge Realities, Multiply Wonder: Expert Guidance on Mixed Reality Development. Here's its description:
Join our experts as they share their hands-on experience and insights from developing and launching Spaceship Home, a groundbreaking Mixed Reality experience on Meta Quest. You'll gain valuable guidance on designing immersive MR experiences that captivate and inspire.
Next, we'll give you an exclusive look at some of the exciting upcoming roadmap items that will revolutionize MR development. Discover how these new features will make it easier to create stunningly realistic and engaging MR experiences.
Finally, we'll showcase real-world examples of developers who are already leveraging our latest Passthrough Camera API to take their apps to the next level. See how they're using this powerful tool to create seamless and interactive MR experiences that blur the lines between reality and fantasy.
Don't miss this opportunity to learn from the experts and stay ahead of the curve in MR development. Join us for an informative and inspiring session that will help you unlock the full potential of Mixed Reality!
Takeaway
The developers will gain insights from developers that have deployed MR as to how to make their game more playable and engaging across a wide variety of rooms. They will be introduced to the new Passthrough API which support real-time use of customized computer-vision to be deployed in games.
David Heaney
The summit listing comes a week after a Meta support page went live describing a "headset camera" permission which "allows an app to access the real-time passthrough camera feed from the front of your headset".
Meta announced the Passthrough API at Connect 2024 back in September, saying that it will "enable all kinds of cutting-edge MR experiences".
The Meta support page that emerged last week gives three examples of how Quest apps could leverage the passthrough view:
• Object recognition. Developers can create apps that recognise and use specific objects within your real-environment. For example, a digital board game that incorporates physical game pieces or boards.
• Location recognition. Developers can create experiences that respond differently depending on where the camera feed shows you are located . For example, indoors or outdoors, at a famous landmark, or in a specific type of room.
• Other machine learning functionality. Developers are able to run custom machine learning models against data from the real-time camera feed. This could be used for retexturing/shading, games involving participants who are not wearing headsets, person/animal detection, or any number of custom industrial/training use-cases.
David Heaney
Meanwhile, Google confirmed last month that Android XR, its upcoming OS set to debut in Samsung's standalone headset, will let apps access the passthrough camera view at launch.
What Is Passthrough Camera Access?
While headsets like Quest 3 and Apple Vision Pro use cameras to let you see the real world, today only the system software gets raw access to these cameras. Third-party developers can use passthrough as a background, sure, but they don't actually get access to it. They instead get higher-level data derived by the system, such as hand and body skeletal coordinates, a 3D mesh of your environment with bounding boxes for furniture, and limited object tracking capabilities. That means they can't run their own computer vision models, which severely limits the augmentation capabilities of these headsets.
The exception is that on visionOS 2, Apple is now giving enterprise companies raw access to Vision Pro's passthrough cameras for non-public internal apps, but this requires a special licence from Apple and is restricted to "in a business setting only".
