14 DAY TRIAL // Video is becoming ubiquitous online and the devices used to access it are diversifying. You can watch YouTube videos on widescreen TV just as well as watch them on your smartphone. Google has been working on a way to make the videos look the same and use all the space available on the number of screens out there.
"At Google Research, together with collaborators from Georgia Tech, we have developed an algorithm that resizes (or retargets) videos to fit the form factor of a given device without cropping, stretching or letterboxing," Google researchers Matthias Grundmann and Vivek Kwatra announced.
"Our approach uses all of the screen’s precious pixels, while striving to deliver as much video-content of the original as possible. The result is a video that adapts to your needs, so you don’t have to adapt to the video," they explained.
The problem with the different screens is not so much about their sizes, but rather their varying aspect ratios. Most content today is available in a widescreen format, with a 16:9 aspect ratio. However, for decades, TV content was shot in the 4:3 format and some haven't transitioned even now.
What's more, sometimes a 16:10 aspect ratio is used and some TV sets are now built with the 21:9 aspect ratios similar to the one used in cinemas. Because yes, movies come with yet another aspect ratio.
And it gets even worse on mobile phones. While many have a 3:4 aspect ratio, which translates into 4:3 if you flip the phone on its side, it's hardly standard.
This means that a video, shot in any of the standard formats, has to be stretched, cut, resized and otherwise made to fit the screen it's being watched on.
There are several approaches, but you either get unused screen space, a stretched video or cut content.
Google's approach aims to bring the best of all worlds, a video that fills the screen, doesn't cut content and isn't deformed. Google's technique is called "discontinuous seam carving."
The originality of it is that it separates the scenes into salient and non-salient content. The salient content is what people focus on, which can't be altered, the actors or important objects in the video. These elements are just resized but they maintain their aspect ratio.
The rest is the out-of-focus background or the things that aren't crucial. This can be stretched or squashed without the viewer noticing anything and without losing any of the important details.
The technique is suited for just some types of scenes and is not ready for mass use yet, but it shows promise. You can check out the technical details on the project's homepage or directly from the paper (PDF) the researchers published.