Computer Vision and WebXR
I'm exciting to finally share a pointer to a blog post about some experiements I've been doing at Mozilla on how to do computer vision in WebXR. Most of the interesting part of this is how we exposed video frames, camera intrinsics and extrinsics, and how we simplify synchronizing poses from an asynchronous computer vision thread back to ARKit Anchors, rather than the particular computer vision algorithms.
We mostly just use some pretty standard stuff from OpenCV to demonstrate the APIs. OpenCV can now be compiled into Web Assembly (a standard build target since December 2017) and run in a web page or Web Worker, opening the door to doing more sophisticated custom computer vision on the web that can integrate with a nice responsive 3D page even if the CV doesn't run synchronously.
Here's a demonstration where OpenCV's Aruco marker tracker is running at < ~25 fps, but the graphics are rendering at ~60 fps. I use ARKit anchors for the content attached to markers, so the content remains fixed in the world even if the video blurs or the markers are occluded. Fun stuff!
Here's another stock OpenCV demo, of their face detector.
Go read the article if you'd like to know more.