Cleverly Cropping Images on Twitter using AI

To crop uploaded images, Twitter doesn’t simply cut them off starting from the center. After first having used Face Detection, they – in 2018 already – switched to AI to cleverly crop uploaded images.

Previously, we used face detection to focus the view on the most prominent face we could find. While this is not an unreasonable heuristic, the approach has obvious limitations since not all images contain faces.

A better way to crop is to focus on “salient” image regions. Academics have studied and measured saliency by using eye trackers, which record the pixels people fixated with their eyes. In general, people tend to pay more attention to faces, text, animals, but also other objects and regions of high contrast. This data can be used to train neural networks and other algorithms to predict what people might want to look at. The basic idea is to use these predictions to center a crop around the most interesting region.

💡 Note that depending on how many images you upload, Twitter will use a different aspect ratio.

What I find weird is that this clever cropping only works on their website, and not in embeds nor other clients. Take this tweet for example, embedded below:

When viewed on the Twitter website it does use the clever cropping:

Now, it wouldn’t surprise me that Twitter hides this extra information from 3rd party clients, given that they basically imposed a no-fly zone back in the day.

Speedy Neural Networks for Smart Auto-Cropping of Images →

Auto Reframe in Premiere Pro

Nice new feature coming to Adobe Premiere Pro:

Powered by Adobe Sensei machine learning technology, Auto Reframe seamlessly adjusts aspect ratios such as square, vertical or 16:9 video to help improve time-consuming manual video resizing.

Adobe Blog: Get to the Finish Line Faster →