Visually Search using your Phone’s Camera with The Web Perception Toolkit

The Web Perception Toolkit is an open-source library that provides the tools for you to add visual search to your website. The toolkit works by taking a stream from the device camera, and passing it through a set of detectors. Any markers or targets that are identified by the detectors are mapped to structured data on your site, and the user is provided with customizable UI that offers them extended information.

This mapping is defined using Structured Data (JSON-LD). Here’s a barcode for example:

    "@context": "",
    "@type": "ARArtifact",
    "arTarget": {
      "@type": "Barcode",
      "text": "012345678912"
    "arContent": {
      "@type": "WebPage",
      "url": "http://localhost:8080/demo/artifact-map/products/product1.html",
      "name": "Product 1",
      "description": "This is a product with a barcode",
      "image": "http://localhost:8080/demo/artifact-map/products/product1.png"

When the user now scans an object with that barcode (as defined in arTarget), the description page (defined in arContent) will be shown on screen.

Next to BarCodes, other supported detectors include QR Codes, Geolocation, and 2D Images. ML Image Classification is not supported, but planned.

The Web Perception Toolkit: Getting Started →
Visual searching with the Web Perception Toolkit →
Web Perception Toolkit Repo (GitHub) →

Bringing newspapers to life with augmented reality

In true Harry Potter-style, Nathan Gitter has used ARKit to bring newspapers to life:


AR Experiments: Flight Paths – A visualization of flight data that floats around you.

Flight Paths is an experiment that transforms your room into a flight path visualization. Touch any horizontal surface and explore as flights take off from JFK or SFO and fly around your space.

AR Experiments: Flight Paths →

Parallel Parking with Augmented Reality

Clever use of Augmented Reality: parallel parking.

Using the AirMeasure app to simplify parallel parking.

Might be handy for drivers who are still learning how to park πŸ™‚

AirMeasure →

How to make an ARKit app in 5 minutes using React Native

Apple has made ARKit very easy to use, but it still requires quite a lot of efforts to properly set it up and run the first demo, especially for those who are not very familiar with 3D programming. What we are going to show you in this article is, with the help of React Native and react-native-arkit, you can skip the non-trivial setting-ups and will be able to write your AR app in just 5 minutes.

// index.ios.js
import React, { Component } from 'react';
import { AppRegistry, View } from 'react-native';
import { ARKit } from 'react-native-arkit';
export default class App extends Component {
  render() {
    return (
      <View style={{ flex: 1 }}>
          style={{ flex: 1 }}
          debug // debug mode will show feature points detected and 3D axis
          planeDetection // turn on plane detection
          lightEstimation // turn on light estimation
            pos={{ x: 0, y: 0, z: 0 }}
            shape={{ width: 0.1, height: 0.1, length: 0.1, chamfer: 0.01 }}
AppRegistry.registerComponent('MyFirstARKitApp', () => App);

How to make an ARKit app in 5 minutes using React Native →

AR.js: Efficient Augmented Reality for the Web

AR.js is Efficient Augmented Reality for the Web using ARToolKit. It is Fast! It runs efficiently even on phones. 60 fps on my 2 year-old phone! it is a pure javascript solution, fully opensource. It works on any phone with webgl and webrtc

AR.js: Efficient Augmented Reality for the Web →

Time Trial: Augmented Reality Climbing

Time Trial is a new interactive rock climbing video game produced by Randori, and hosted at Brooklyn Boulders Somerville twice each month. Players are given the opportunity to compete in a dynamic setting featuring augmented reality on a climbing wall.