This year at WWDC 2019, Apple announced ARKit 3 and RealityKit which bring additional capabilities to the already growing augmented reality (AR) framework. Last year at WWDC 2018, Apple’s ARKit 2 introduced Quick Look features over the web for iOS 12, which allows 3D rendered models to be viewed and interacted with in the real world directly through Safari on iOS. This is achieved by making use of the native platform technologies for supported browsers — think PWA features, making use of available browser APIs. As of this year, Google has recently announced similar ARCore features to be made available over the web in Android Q, which is currently pending public release. Having already implemented ARkit over the web using ARKit 2, I felt it’s time to get the word out there on the true capabilities of these features. Now without further adieu, we’ll dive into the details on how we incorporated ARkit’s Quick Look features into our PWA.

Reference Experience

As of October 25 2018, the React PWA (Progressive Web App) Reference Storefront, incorporates ARKit2’s Quick Look functionality over the web to allow shoppers the ability to view 3D models of products they may browse in the storefront.

Let’s venture into how we achieved our ARKit implementation through our Reference Storefront PWA. Our intention was to give shoppers of our storefront the option of an AR experience of a product when they visit a product’s display page that supports the functionality.

Firstly, creation of augmented reality requires knowledge of 3D modelling or at least a tool to help with the creation of models. Models can be created from images, from scratch, or from templates available on the web. We used Vectary (https://vectary.com) for our implementation, which is a subscription based 3D modelling web application however you can use any modelling application of your choice.

Second, ARKit mandates usdz as the file format for rendering 3D models. This was easy, since Vectary can export our models to a variety of file types (.obj, .usdz, etc.). If you’re not using Vectary, Apple has created a tool to help with this generation/conversion here: https://developer.apple.com/augmented-reality/quick-look/.

Lastly, we had to host our usdz files somewhere so they can be picked up by our Reference Storefront application. ARKit usdz files are externalized through content URLs within the storefront application configuration. The default URLs are configured to reference the usdz files, which are located on Amazon S3 however, any other CMS (Content Management System) provider may be used. Note that depending on the content hosting system you’re using to deliver your usdz files, you may need to configure CORS to allow HEAD requests.

We added some configuration properties for our project to use:

"arKit": {
    "enable": true,
    "skuArImagesUrl": "https://s3.amazonaws.com/referenceexp/ar/%sku%.usdz"
},

arKit.enable: Enable ARKit’s Quick Look capability to load on a product display page.

• arKit.skuArImagesUrl: The path to the usdz files hosted on an external CMS. Set this parameter to the complete URL of the files by replacing the sku/file-name parameter with %sku%. This parameter is populated when the page is loaded with values retrieved by Cortex

We were able to create all the necessary implementations around this strictly within our React component responsible for loading the product information on the product details page (productdisplayitem.main.jsx).

To start, we have to determine if a corresponding usdz file for our product is hosted on our configured content delivery provider. We’ll add an initial check to see if the “arKit” configuration has been set, and that the ar tag is supported by our browser.

After validating that the ARkit can be supported on the current browser, we then check for the corresponding usdz file by initiating a HEAD request to the usdz file location, and update our components state to indicate whether or not the file exists. We execute this whenever our component mounts, or when it receives new properties. We surround our request in a function urlExists() to simplify our calls.

urlExists(url, callback) {
    this.funcName = 'UrlExists';
    fetch(url, {
        method: 'HEAD'
    }).then((res) => {
        callback(res.ok);
    });
}
...
if (Config.arKit.enable && document.createElement('a').relList.supports('ar')) {
    this.urlExists(Config.arKit.skuArImagesUrl.replace('%sku%', res._code[0].code), (exists) => {
        this.setState({
            productData: res,
            arFileExists: exists,
        });
    });
}
...

Next, we’ll need to render our product image to display the usdz file we fetched (if the file exists). Upon rendering our component we’ll wrap our product image in an anchor tag with rel=”ar” if the usdz file exists. This will position the ARKit Quick Look icon directly around our product image, so that tapping it will direct the shopper into Quick Look with our 3D object file.

if (arFileExists) {
    return (
        
             { e.target.src = imgPlaceholder; }} alt={intl.get('none-available')} className="itemdetail-main-img" />
        
    );
}

When the image is called to the storefront, the required usdz files are retrieved on a per-SKU basis as they are available from the CMS provider. The storefront only displays the required AR tags, if the file exists. Any SKUs without a corresponding usdz file will not have an AR tag displayed on the product display page.

Summary

We chose to implement augmented reality into our PWA using this solution as it was (and presently still is) the most viable to implement for a general practice. The implementation solely relies on features at the browser level, without any need for integrating with custom libraries/dependencies. This also makes the ARKit implementation a standard to conform to in order to ensure future features that can be added through this implementation are easily adapted. Google has proceeded to follow a similar pattern by introducing their own ARCore technologies over the web, which is expected to be made available with Android Q.

By following these general practices for making AR available over the web, we have ensured our PWA can accommodate features that may further enhance the experiences of augmented reality. This opens us up to consume some of the features in ARKit 3 and RealityKit such as people occlusion and facial tracking.

In the current state of this implementation, usdz files must be created on a case-by-case basis for each product in a catalog to which ARKit is desired to be supported for. Depending on the size of the catalog, modelling these products as 3D models may take quite a bit of effort and time. To alleviate this effort, we would further investigate dynamically rendering object files from product photos, or integrating with a 3D object modelling service. These would be common hurdles that need to be accomplished by any means, as generating the necessary 3D models still require some sort of 3D modelling experience — for now.