Pose Animator – Animate SVG Illustrations using your Camera

This is crazy:

Pose Animator takes a 2D vector illustration and animates its containing curves in real-time based on the recognition result from PoseNet and FaceMesh. It borrows the idea of skeleton-based animation from computer graphics and applies it to vector characters.

Built on top of PoseNet to track the body’s pose, and the aforementioned FaceMesh to track the face.

Pose Animator →
Pose Animator Demos →

Realtime Face and Hand Tracking in the browser with TensorFlow

The MediaPipe and Tensorflow.js teams have released facemesh and handpose:

The facemesh package infers approximate 3D facial surface geometry from an image or video stream, requiring only a single camera input without the need for a depth sensor. This geometry locates features such as the eyes, nose, and lips within the face, including details such as lip contours and the facial silhouette.

The handpose package detects hands in an input image or video stream, and returns twenty-one 3-dimensional landmarks locating features within each hand. Such landmarks include the locations of each finger joint and the palm.

Once you have one of the packages installed, it’s really easy to use. Here’s an example using facemesh:

import * as facemesh from '@tensorflow-models/facemesh;

// Load the MediaPipe facemesh model assets.
const model = await facemesh.load();
 
// Pass in a video stream to the model to obtain 
// an array of detected faces from the MediaPipe graph.
const video = document.querySelector("video");
const faces = await model.estimateFaces(video);
 
// Each face object contains a `scaledMesh` property,
// which is an array of 468 landmarks.
faces.forEach(face => console.log(face.scaledMesh));

The output will be a prediction object:

{
    faceInViewConfidence: 1,
    boundingBox: {
        topLeft: [232.28, 145.26], // [x, y]
        bottomRight: [449.75, 308.36],
    },
    mesh: [
        [92.07, 119.49, -17.54], // [x, y, z]
        [91.97, 102.52, -30.54],
        ...
    ],
    scaledMesh: [
        [322.32, 297.58, -17.54],
        [322.18, 263.95, -30.54]
    ],
    annotations: {
        silhouette: [
            [326.19, 124.72, -3.82],
            [351.06, 126.30, -3.00],
            ...
        ],
        ...
    }
}

Both packages run entirely within the browser so data never leaves the user’s device.

Be sure to check the demos as they’re quite nice. I did notice that the handpose demo only shows one hand, even though the library can detect more than one.

Face and hand tracking in the browser with MediaPipe and TensorFlow.js →
facemesh Demo →
handpose Demo →

Let’s play with Chrome’s Face Detection API

Recently Wes Bos sent out this tweet:

Sparked by that tweet, João Miguel Cunha set out to play with it. The code itself is pretty simple: create an instance of FaceDetector and call .detect() on it. That promise eventually resolves with an array of detected faces.

var faceDetector = new FaceDetector();
faceDetector.detect(image)
  .then(faces => faces.forEach(face => console.log(face)))
  .catch(e => {
    console.error("Boo, Face Detection failed: " + e);
  });

Let’s play with Chrome’s Face Detection API →

Building an image processor on AWS Lambda using The Serverless Framework

Good writeup on setting up an image processor using The Serverless Framework, a thing comparable to the aforementioned apex (and with an awfully generic and confusing name imho 😉).

  1. When a user uploads a file an ObjectCreated event is produced and a Lambda function is invoked.
  2. The Lambda function calls Amazon Rekognition to detect the faces and emotion of each face in the uploaded image.
  3. The Lambda function processes the image and persists the image in Amazon S3

Here’s an example set of results:

The code of the Lambda function that calls Amazon Rekognition and processes uploaded images is available on GitHub.

How to build powerful back-ends easily with Serverless →

Your Face is Big Data

tsvetkov_02

tsvetkov_04

Interesting project by Russian photographer Egor Tsvetkov in which he took photos of random, anonymous, people riding the subway, and then running them through a face recognition app named FindFace. The result: 70% of those photographed could be linked to one or social network profiles of ‘m, thus un-anonymizing them.

End of anonymity: Identification of random passengers (Russian) →

Real-time Expression Transfer for Facial Reenactment

teaser

We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photo-realistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video.

Wow. On the Internet, nobody knows you’re a dog … or someone else with your facial expressions transferred onto.

Real-time Expression Transfer for Facial Reenactment →

Responsive Typography using Face Detection

responsive-typography-face-detection

Typesetting based upon your distance to your screen. Clever use of WebRTC’s getUserMedia and JavaScript Face Detection

In case you don’t have a capable browser, this is what it looks like:

Responsive Typography: Breakpoints Demo →
Responsive Typography: Realtime Demo →

(via Jeremy)

Related: headtrackr →

Pinokio

Pinokio is an exploration into the expressive and behavioural potentials of robotic computing. Customized computer code and electronic circuit design imbues Lamp with the ability to be aware of its environment, especially people, and to expresses a dynamic range of behaviour. As it negotiates its world, we the human audience can see that Lamp shares many traits possessed by animals, generating a range of emotional sympathies. In the end we may ask: Is Pinokio only a lamp? – a useful machine? Perhaps we should put the book aside and meet a new friend.

Inspired by Pixar. Built using Processing, Arduino, and OpenCV

(via )