Piece of Cake - or Data?

Those that are familiar with my doctoral dissertation thesis, also know my adopted metaphor of “piece of cake”. In my diss, the context relates to the jungle of national laws to be applied in multinational inventions. When a company has received a joint invention where there are co-inventors originating from different jurisdictions, rights to the “cake” needs to be taken in pieces, according to the respective laws and rules. However, when patenting the invention where multiple national security provisions apply, the situation is more complex as you cannot patent the invention in pieces, yet filing a patent application in one country could violate laws of the other relevant countries involved. In other words, the filing could be illegal or invalid.

Lately, I’ve been applying the same metaphor to coding data. Sometimes – and quite often – coding input data is most efficient to be performed in pieces, namely by splitting the data to portions. This is especially true with input data containing variety of content, such as video. When input data is divided into portions with most similar content, encoding of the data portions can be optimized. Literally interpreting, my cake metaphor would mean that the cake would be separated to different ingredients but let’s just take the idea of “slicing to pieces” here and consider applying it to coding, and consider the pieces of cake differ from each other.

“In general, many known video codecs are not able to code efficiently extensive areas of images with substantially constant parameters whilst concurrently being able to encode highly spatially detailed areas of the images. It is customary practice to employ motion compensation in a form of prediction and prediction error coding methods based upon use of transformations such as discrete cosine transform (DCT) and wavelet transformations. These transformations employ a process wherein portions of a given image, e.g. a still image or an image forming a part of a video sequence, are divided into blocks which are then subject to encoding processes.” – This quote is from a patent application (since then granted) filed by Gurulogic Microsystems already back in 2012. The patent family “Block Coder” was a (r)evolutionary finding from company’s past work with developing video surveillance systems, and eventually led to development of company’s own codec: GMVC? (Gurulogic Multi-Variate Codec).

When data is split into smaller portions, coding of each portion can be optimized. “The encoder can choose the best division of the Coding Tree Unit (CTU) based on the content of the block. In a rather uniform area, bigger blocks are more efficient whereas in areas with edges or more detail, smaller blocks are typically chosen.” (https://bitmovin.com/compression-standards-vvc-2020) This is among others what “Multi-Variate” in GMVC? refers to – data can be split to variety of different-sized and/or shaped portions. But it can also be applied to variety of different data types, not only images and video but also e.g. to genomic data. (Similarly, in VVC, “versatile” indicates that it is meant for wide variety of applications).

Encoder can then choose the best possible coding method for each portion, based on the content of the data. “With high flexibility of block sizes and shapes that a CTU can be split into, an encoder can flexibly adapt to wide range of video characteristics that result in better coding performance.” (https://bitmovin.com/compression-standards-vvc-2020). In fact, this is the adopted ideology also in MPEG-G, namely the standard designed for genomic information representation: “…each Block is compressed using the entropy coding techniques most suitable to the measured statistical properties.” (https://mpeg-g.org/details-about-mpeg-g-part-1-iso-iec-23092-1/)

Should the encoding quality for a block not yet be sufficient, the block(s) can be further splitted, i.e. data can be split in a recursive manner.? The below illustration is from VVC:

There are also other codecs applying recursive splitting of data:

?“It will be appreciated that the actual splitting can take place in various forms and directions, such as diagonally, horizontally, vertically etc. The sub-blocks can be of the same or different size. Furthermore, the sub-blocks can be of the same shape or different shape.” This excerpt is again from Gurulogic US patent application, thereafter granted patent. Indeed, you don’t always need to slice the cake to equally sized, or even shaped, pieces. VVC uses the term “geometric partitioning” to reflect this: “Unfortunately, natural video content typically contains more curved edges that can only poorly be approximated using rectangular blocks. In this case, Geometric Partitioning allows the non-horizontal splitting of a block into two parts.” (https://bitmovin.com/compression-standards-vvc-2020)

In addition to having different sizes, also different coding methods can be applied to the blocks, depending on which method is best suitable for the content of the individual data portion. This is also essential aspect in Gurulogic block coder – optimizing not only the block size but also selection of a coding method from multiple options to be applied. Using cake metaphor, you might have different icing for different pieces. Also VVC introduces concepts such as “Multiple Transform Selection” and “Sub-Block Transforms”.

But there are also other aspects in coding, affecting coding efficiency, than the block size and the applied encoding method. Quantization is mechanism to control the size of an encoded video stream to meet the bandwidth requirements of a particular video application. However, it can be challenging to control the bit-rate of an encoded video stream while also providing an acceptable picture quality. “For video domains such as surveillance, conference, and autonomous driving, a large amount of visual redundancy exists due to high target and background differentiation. By dynamically adjusting the QP values of the target and background regions, the coding efficiency can be greatly improved.” (Saliency-Enabled Coding Unit Partitioning and Quantization Control for Versatile Video Coding, https://www.mdpi.com/2078-2489/13/8/394) Also Gurulogic Microsystems has successfully patented their method “Adaptive Quantization” – a cost-efficient method of changing quantization strength portion-by portion when using various encoding methods in coding of the data. In other words, quantization can be dynamically adjusted in coding, depending on the content of data portion. In VVC, “[a] mechanism called dependent quantization is used to adaptively switch between two different quantizers such that the quantization error can be reduced.” (https://www.researchgate.net/publication/334023991_Recent_advances_in_video_coding_beyond_the_HEVC_standard) ?

In coding data, one always needs to think also about data confidentiality and security, namely encrypting the data. But encryption does not come without cost, as the amount of data in the world is increasing. Therefore, it is necessary to provide more efficient solutions for securing privacy, storage, and transmission of video data. Gurulogic Microsystems Oy has patented their solution also to this problem, and provide a cost-efficient way to encrypt data by encrypting only essential parts of the data, yet so that non-encrypted data cannot be decoded without decrypting the encrypted data first. The savings achieved on computing resources and processing energy is very beneficial e.g. in smart phones, enabling smaller batteries to be employed, and extending operating times before recharging. Also VVC has a concept called “Selective encryption”: “Selective encryption can lower computation complexity and also is a suitable choice for situations where the video stream is going into other untrusted blocks (for applications such as transcoding, watermarking, and cutting) while being transmitted through the network.”(https://www.researchgate.net/publication/350484628_Selective_Encryption_of_VVC_Encoded_Video_Streams_for_the_Internet_of_Video_Things ?

There would be so much more to tell about innovative GMVC? coding methods but as with cakes, there is always much more inside that you can see. Should you be interested in studying our patents more, they can be found from public patent databases and company’s web pages:

https://gurulogic.com/patent-categories/