Letterbox vs Pan and Scan Again?

Film enthusiasts and video fans will recall the discussions around the transfer of movie material to #DVDs and video tapes. Yes, I’m talking about the famous letterbox vs 4:3 TV format debate.

Film enthusiasts wanted to see all the actions of a movie scene as intended by the director and would accept black stripes on top and bottom of the TV screen, while other viewers wanted a screen-filling experience without black strips, accepting that some sequences might no longer be as intended, as the pan and scan process essentially re-directed the scene.

Today, most #home-entertainment systems are in 16:9 widescreen format and cinematic 21:9 movies seem to be more accepted to still display some form of letterbox.

So, what’s the issue now?

Well, it’s called #4k #UHD content shot by users on their sophisticated high-quality mobile phones, in a vertical format. The ratio is now 9:16, which is fine playing back on the phone itself, if we hold it in vertical position.                                            

But what happens when we get to our wonderful 65-inch OLED screen at home and we stream the video to that?

Now we have an extreme sort of vertical letterbox, which is just not fun to watch.

Samsung demonstrated a solution at CES this year, their innovative Sero TV. It elegantly rotates the whole screen by 90 degree when playing back a video streamed from a mobile handset. Unfortunately, that is not a solution for my home, [JJ1] there might well be others having this issue as well.

So, what can we do? Back to Pan and Scan?

Image enhancement algorithms help is to account for zooming losses in resolution, we might even apply super resolutions techniques.

But how to select the region of interest? One possibility would of course be to have the user control or just default to the middle. But what if the main picture content wonders from one end of the screen to the other?

AI to the rescue! We could employ an algorithm that finds key content and tracks, i.e. scans it in an intelligent way. This way, we could achieve a better presentation of content in a more pleasant way whilst filling the screen we have – at least to a bigger extend than without it. 

 Calling #AI experts and programmers – modern TV sets already come along with sufficient processing power based on graphics processing units (#GPUs) and neural network (#NNA) hardware. Now we need the algorithms to help us enjoy the content from our phones on the big screens in our living rooms!