To crop uploaded images, Twitter doesn’t simply cut them off starting from the center. After first having used Face Detection, they – in 2018 already – switched to AI to cleverly crop uploaded images.
Previously, we used face detection to focus the view on the most prominent face we could find. While this is not an unreasonable heuristic, the approach has obvious limitations since not all images contain faces.
A better way to crop is to focus on “salient” image regions. Academics have studied and measured saliency by using eye trackers, which record the pixels people fixated with their eyes. In general, people tend to pay more attention to faces, text, animals, but also other objects and regions of high contrast. This data can be used to train neural networks and other algorithms to predict what people might want to look at. The basic idea is to use these predictions to center a crop around the most interesting region.
💡 Note that depending on how many images you upload, Twitter will use a different aspect ratio.
What I find weird is that this clever cropping only works on their website, and not in embeds nor other clients. Take this tweet for example, embedded below:
— ᴀʟᴇxᴇʏ sᴇᴋᴀᴄʜᴏᴠ (@sekachov) May 11, 2020
When viewed on the Twitter website it does use the clever cropping:
Now, it wouldn’t surprise me that Twitter hides this extra information from 3rd party clients, given that they basically imposed a no-fly zone back in the day.