Cleverly Cropping Images on Twitter using AI

To crop uploaded images, Twitter doesn’t simply cut them off starting from the center. After first having used Face Detection, they – in 2018 already – switched to AI to cleverly crop uploaded images.

Previously, we used face detection to focus the view on the most prominent face we could find. While this is not an unreasonable heuristic, the approach has obvious limitations since not all images contain faces.

A better way to crop is to focus on β€œsalient” image regions. Academics have studied and measured saliency by using eye trackers, which record the pixels people fixated with their eyes. In general, people tend to pay more attention to faces, text, animals, but also other objects and regions of high contrast. This data can be used to train neural networks and other algorithms to predict what people might want to look at. The basic idea is to use these predictions to center a crop around the most interesting region.

πŸ’‘ Note that depending on how many images you upload, Twitter will use a different aspect ratio.

What I find weird is that this clever cropping only works on their website, and not in embeds nor other clients. Take this tweet for example, embedded below:

When viewed on the Twitter website it does use the clever cropping:

Now, it wouldn’t surprise me that Twitter hides this extra information from 3rd party clients, given that they basically imposed a no-fly zone back in the day.

Speedy Neural Networks for Smart Auto-Cropping of Images →

About the Author

Bramus is a Freelance Web Developer from Belgium. From the moment he discovered view-source at the age of 14 (way back in 1997), he fell in love with the web and has been tinkering with it ever since (more …)

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.