In the buzzing world of XR, passthrough camera access is generating quite the conversation. We already know where companies like Meta, Apple, and Pico stand, but the big question is, what will Google do with Android XR? After a chat with them, it’s clear they’re planning to implement a method similar to what we see on smartphones. Let’s dive deeper into this development!
The Camera Access Dilemma
For those who might be scratching their heads, wondering what’s the big deal, here’s a quick rewind. Today’s advanced standalone VR headsets are essentially MR headsets, offering users a mixed reality experience by displaying an RGB passthrough of the surrounding environment through front-facing cameras. This technology powers those fantastic mixed reality apps like Cubism, Starship Home, and Pencil.
But here’s the crux: while the system uses those camera frames to create the passthrough view, developers are keen to access these frames too. Why? The potential is immense: using AI and computer vision, we could significantly enhance the user’s reality. Remember, I’ve mentioned before how camera access is crucial to unlocking mixed reality’s full potential. For example, on the Quest, using a nifty camera access trick, I managed to prototype an AI+MR application capable of assisting in home interior design – something that wouldn’t be possible without such access.
The flipside, however, is privacy. Give the wrong developer camera access, and they could quietly gather images from a user’s environment, potentially analyzing them for personal information—like snapshots of national IDs or credit cards inadvertently left in view. And let’s not forget the unsavory possibilities involving images of people.
So, the challenge lies in balancing privacy with the immense possibilities of mixed reality.
The Stance of XR Companies
Not too long ago, camera access was freely available without hassles. If you’ve been with me from the start, you might recall the experiments we ran with camera textures on the Vive Focus around 2019—diminished reality, Aruco marker tracking, sound reactivity, the whole gamut.
However, as mixed reality gained traction, companies played it safe by restricting camera frame access due to privacy fears. Meta, Pico, HTC, and Apple all blocked developers from tapping into these camera feeds.
This became the norm until the XR developer community started clamoring for camera access. Advocates like Cix Liv, Michael Gschwandtner, and myself rallied to have this feature available—transparently, of course—so users could give explicit consent, allowing apps to run object recognition or vision algorithms on what they see. We pushed the question: why treat XR devices differently when smartphone camera access is granted through a simple permission request?
This advocacy led XR companies to reconsider. Meta committed to releasing a “Passthrough API” early this year. Now, what about Google and Android XR?
Android XR Treating Headsets Like Phones
Android, being the backbone of most smartphones globally, has a straightforward approach to camera access: request permission, and if granted, you can use the camera just like that. Google’s vision for Android XR reportedly mirrors this simplicity. Though whispers surrounded their plans, I reached out for a confirmation and finally received a direct response from a Google spokesperson:
"Much like any Android app, a developer can request camera frames with user consent for XR. Detailed permissions are outlined in our developer documentation, indicating that developers can specify camera_id=0 for the primary world-facing camera stream. Similarly, camera_id=1 is for the selfie stream. These streams utilize standard Android Camera APIs, like Camera2 and CameraX."
This means Android developers can use familiar classes, like CameraX, to manage camera streams on XR headsets, making it possible to perform various media operations and ML analyses. This is exciting news!
Yet, while you can access the front camera, the rear camera provides a reconstructed avatar, akin to Apple’s Vision Pro. This keeps Android XR consistent with the smartphone experience: accessing the “rear camera” shows users the surroundings, while the “selfie camera” reveals the user’s face—reconstructed as an avatar.
By ensuring Android apps run seamlessly on Android XR, Google is making a smart move. Preserving consistency with permission requests across devices is a strategy I wholeheartedly support.
However, a burning question remains: what about raw camera streams?
"Currently, we’re not providing methods for apps to access non-standard sensors, such as forward-facing cameras or reconstructed inward cameras."
For now, these won’t be accessible, though one can hope that enterprise users may have options in the future.
Unity developers might wonder about their options: while Android Camera2 and CameraX are Android native, Unity developers could potentially use the WebcamTexture class. If issues arise, inventive developers might find ways to bridge Unity with native Android functions.
A Cautionary Note on Android XR
Keep in mind, Android XR is still in its preview phase, with no officially released headsets running it yet. While I doubt significant changes will occur before its release, it’s worth acknowledging the possibility.
The Door Opens on Camera Access
With both Google and Meta leaning towards camera access, it’s likely other companies will follow their lead. As we approach 2025, it seems to be shaping up as a landmark year for groundbreaking developments in mixed reality. Personally, I can’t wait to see the innovations that emerge from the developer community!
(Header image courtesy of Samsung)
Disclosure: This blog supports itself through affiliate links, providing a small commission if you make a purchase through them. Check out the full disclosure in the link provided.
