(originally published on the Mozilla MR Blog, May 2020. Just in case Mozilla shuts down the blog after shutting down the team, I'm preserving the posts I wrote for the blog.)
WebXR Device API: Past, Present, and Future
We developed the API used in the first version of the WebXR Viewer more than 3 years ago (as a proposal for how WebXR might combine AR and VR; see the WebXR API proposal here, if you are interested), and then updated it a year ago to match the evolving standard. While very similar to the official API, this early version of the official API is not compatible with the final standard, in some substantial ways. Now that WebXR is appearing in mainstream browsers, it's confusing to developers to have an old, incompatible API out in the world.
Over the past year, we rebuilt the API to conform to the official spec, and either updated our old API features to match current proposals (e.g., anchors, hit testing, and DOM overlay), marked them more explicitly as non-standard (e.g., by adding a nonStandard_ prefix to the method names), or removed them from the new version (e.g., camera access). Most WebXR AR examples on the web now work with the WebXR Viewer, such as the "galaxy" example from the WebXR Samples repository shown in the banner above.
In the near future, we're interested in continuing to experiment with more advanced AR capabilities for WebXR, and seeing what kinds of experimentation developers do with those capabilities. Most AR use cases need to integrate virtual content with meaningful things in the world; putting cute dinosaurs or superheros on flat surfaces in the world makes for fun demos that run anywhere, but genuinely useful consumer and business applications need to sense, track, and augment "people, places, and things" and have content that persists over time. Enhancing the Immersive Web with these abilities, especially in a "webby" manner that offers privacy and security to users, is a key area Mozilla will be working on next. We need to ensure that there is a standards-based solution that is secure and private, unlike the proprietary solutions currently in the market that are siloed to create new, closed markets controlled by single companies.
While purely vision-based AR applications (implemented inside web pages using direct access to camera video) are showing great engagement, failing to use the underlying platform technology limits their capabilities, as well as using so much CPU and GPU resources that they can only run for a few seconds to minutes before thermal throttling renders them unusable (or your battery dead). WebXR offers the possibility for the underlying vision-based sensing techniques to be implemented natively so they can take advantage of the underlying platform APIs (both to maximize performance and to minimize CPU, GPU, and battery use).
It is too early to standardize some of these capabilities and implement them in a open, cross-platform way (e.g., persistent anchors), but others could be implemented now (e.g., the face and image tracking examples shown below). In the newly announced Firefox Reality for Hololens2, we're experimenting with exposing hand tracking into WebXR for input, a key sort of sensing that will be vital for head-worn-display-based AR (Oculus is also experimenting with exposing hand tracking into VR in the Oculus Browser on Quest). APIs like ARKit and Vuforia let you detect and track faces, images, and objects in the world, capabilities that we explored early on with the WebXR Viewer. We've kept versions of the APIs we developed in the current WebXR Viewer, and are keen to see these capabilities standardized in the future.
Integrating with a Full-featured Web Browser
The second change will be immediately noticeable: when you launch the app, you'll be greeted by the familiar Firefox for iOS interface, and be able to take advantage of many of the features of its namesake (tabs, history, private browsing, and using your Firefox account to sync between devices, to name a few). While not all Firefox features work, such as send-to-device from Firefox, the experience of using the WebXR Viewer should be more enjoyable and productive.
Our goal for moving this code to the Firefox code-base wasn't just to create a better browsing experience for the WebXR Viewer, though. This is an experimental app, after all, aimed at developers hoping to explore web-based AR on iOS, and we don't plan on supporting it as a separate product over the long term. But Apple hasn't shown any sign of implementing WebXR, and it's critically important for the success of the immersive web that an implementation exists on all major platforms. Toward this end, we moved this implementation into the Firefox for iOS code-base to see how this approach to implementing WebXR would behave inside Firefox, with an eye towards (possibly) integrating these features into Firefox for iOS in the future. Would the WebXR implmentation work at all? (Yes.) Would it perform better or worse than in the old app? (Better, it turns out!) What UI and usability issues would arise? (Plenty.) While there is still plenty of UI work to do before moving this to a mainstream browser, we're quite happy with the performance; WebXR demos run better in this version of the app than they did in the previous one, and the impact on non-WebXR web pages seems minimal.
We'd love for you to download the new version of the app and try out your WebXR content on it. If you do, please let us know what your experience is.