Computer Vision and WebXR
I'm exciting to finally share a pointer to a blog post about some experiements I've been doing at Mozilla on how to do computer vision in WebXR. Most of the interesting part of this is how we exposed video frames, camera intrinsics and extrinsics, and how we simplify synchronizing poses from an asynchronous computer vision thread back to ARKit Anchors, rather than the particular computer vision algorithms.
){kind=link}
We mostly just use some pretty standard stuff from OpenCV to demonstrate the APIs. OpenCV can now be compiled into Web Assembly (a standard build target since December 2017) and run in a web page or Web Worker, opening the door to doing more sophisticated custom computer vision on the web that can integrate with a nice responsive 3D page even if the CV doesn't run synchronously.
Here's a demonstration where OpenCV's Aruco marker tracker is running at < ~25 fps, but the graphics are rendering at ~60 fps. I use ARKit anchors for the content attached to markers, so the content remains fixed in the world even if the video blurs or the markers are occluded. Fun stuff!
Here's another stock OpenCV demo, of their face detector.
Go read the article if you'd like to know more.
Thanks for this article.
I wonder if there's any feature about computer vision in webxr.
Would we have face or object detection capacity in webxr ?
There isn't anything yet. The initial release of WebXR (over on github.com/immersive-web/webxr ) will not support any sort of camera access. It will come eventually, though.
Hi Blair.
Do you think that webxr will support marker tracker in the future or will we be force to use external computer vision library like opencv ?
Is there any chance that anchors evolve to allow that ?
I think WebXR will eventually provide access to a lot of native sensing features. The big challenge is figuring out what to expose, and how, since the platforms are very diverse right now. For example, I would love to support some very simple things (QRCodes of various forms, simple images with a certain kind of feature set), but first folks are anxious to get the basic API done.