Encoder Complexity Hits the Wall!
Over the past decade, we have seen incredible innovations in video encoding. High dynamic range and 8K video. IMF. Highly optimized map-reduce encoding frameworks. New and innovative quality metrics. Convex hull encoding. We also have four new codecs with very high efficiency: HEVC, VP9, AV1, & VVC.
New codec technology is impressive. But each new generation has come with a 6-10x increase in computational complexity. The complexity vs. efficiency graph is troubling. The graph below illustrates the problem. The complexity and efficiency are referenced to H264 where complexity and efficiency == 1. I assume 50% efficiency gain and 6X increase in computational complexity with each new codec including the next two generations from AOM and MPEG. We're already struggling with the encode time of AV1. It is hard to see the value of a new codec that puts another 6-10x increase on the computational complexity. The encode time and compute cost would both be prohibitive.
Figure: New Codec complexity and efficiency.
Adding to the encoding complexity growth is the computational complexity of convex hull encoding, which puts another multiplier on computational complexity. We’ve long counted on Moore’s law to manage the growth in computational complexity, but for the near future, encoder complexity is outpacing Moore’s law.
It's All About The Green
With this growth in computational load, we must also consider the environmental cost. The power requirements of compute is also on an exponential curve. In 2019, data centers will consume ~3% of world’s power. In 2025, that number will grow to as much as 20% of global power, at which point data centers will be one of the single largest sources of pollution in the world. The longer-term answer to video encoding can not be to simply add more CPU capacity. This is an unsustainable model; both financially, and environmentally.
It’s always fun to embarrass people by reminding them of their old quotes. Here are a couple of zingers.
- "640K is more memory than anyone will ever need on a computer" (attributed to Bill Gates, 1980’s, though he denies saying it)
- “Two years from now, spam will be solved” (Bill Gates, 2004)
- “I predict the internet will … in 1996 catastrophically collapse” (Robert Metcalf, One of the Ethernet inventors in 1995)
- “It’s like infinite, free, compute capacity!” (David Ronca, Video@Scale 2015)
- “Computational complexity is the currency we use to pay for encoding efficiency” (David Ronca, some time in 2016)
The last two quotes are mine. Suffice it to say that they have not stood up to the test of time. We know that compute is neither free nor infinite, it only seemed that way for a while. And while true that complexity is currency that buys codec efficiency, I'll paraphrase my mom's frequent reminder “compute capacity does not grow on trees!”
We must explore ways to meet the explosive growth of video while bending the compute and power requirements down. Future codec research must emphasize compression efficiency AND computational efficiency. Here's a thought exercise: If the next video encoder maintained the efficiency of VVC/AV1 but reduced computational complexity by 50%, would we consider that a successful new codec?
We also need to find fast and cheap models to determine the convex hull. Finally, augmenting CPU encoding with low-power ASICs will help to meet the growing demands of video while bending the compute and power requirements down. These are really big problems. And they are not easy. The real work and real fun of high-scale video has just begun.
Co-Founder & Division Lead (R&D) at Block Digital Corporation, R&D Sindric Solutions, Partner Chainrisk
5 年You should take a look at what emerging platforms like what Halsey Minor & Devadutta Ghat are doing over at VideoCoin. It needs to start with abandoning the data center model (a concentrator of CAPEX/OPEX, and with it complexity) and decentralize. This will create an overall more nimble network architecture from which to derive novel approaches to sustainability. Competition, rather than compute aggregation, is where the solution will arise. We at Block Digital Corporation are a startup working specifically to address this problem.
Video Practitioner, SW Engineer, Pythonist, Machine Learning
5 年very expected paper. There is a theoretical limit of the bitrate which can be achieved given a distortion level. i am afraid we are close to the limit - exponential boosting in complexity to get several percents in bitrate savings. Perhaps, the video compression field is getting to a saturation.?
Managing Director
5 年David, I would like to challenge your thoughts. Yes new codecs increase the cpu cycle but new encoding techniques esp those AI based reduce the cpu cycles as we do not have for instance to encode all profiles. I understand Netflix has developed a brute force approach for vod transcoding, for live we have to be lean, so use completely different techniques, very CPU efficient. If you add to that the new CAE techniques where profile might not be encoded If not watched, I see hope to keep cpu cycles under check.
Engineering Manager - Video Streaming / OTT
5 年Computational Efficiency should also mean - better and efficient ways of doing computation. How far do you think are the Quantum Computers?
Hey David, you meant AV1 instead of AVC in the first paragraph, right?