Life inside MPEG

Introduction

In 30 years of MPEG, and counting? I brought evidence of what MPEG has done in the last 30 years to create the broadcast, recording, web and mobile digital media world that we know. This document adds more details to the picture by looking at the current main activities and deliveries in the next few months/years.

MPEG at a glance

The figure below shows the main standards developed or under development by MPEG in the 2017-2023 period organised in 3 main sections:

• Media Coding (e.g. MP3 and AVC)
• Systems and Tools (e.g. MPEG-2 TS and File Format)
• Beyond Media (currently Genome Compression and Neural Network Compression).

Video coding

In the Video coding area MPEG is currently handling 4 standards (MPEG-4, -H, -I and -CICP) and several Explorations.

MPEG-I ISO/IEC 23090 Coded representation of immersive media is the suite of standards designed for a variety of immersive media devices, applications and services.

MPEG is currently working on Part 3 Versatile Video Coding, the new video compression standard after HEVC. VVC is developed jointly with VCEG and is expected to reach FDIS stage in October 2020.

Explorations

1. An exploration on a new video coding standard that combines coding efficiency (similar to that of HEVC), complexity (suitable to real time encoding and decoding) and usability (timely availability of licensing terms). In October 2018 a Call for Proposals was issued. Submissions are due in January 2019 and FDIS stage for the new standard is expected to be reached in January 2020.
2. An exploration on a future standard that defines a data stream structure composed by two streams: a base stream decodable by a hardware decoder, and an enhancement stream suitable for software processing implementation with sustainable power consumption. This activity is supported by more than 30 major international players in the video distribution business. A Call for Proposal has been issued in October 2018. Submissions are due in March 2019 and FDIS stage is expected to be reached in April 2020.
3. Several explorations on Immersive Video Coding

• 3DoF+ Visual: a Call for Proposal will be issued in January 2019. Submissions are due in March 2019 and the FDIS is planned for July 2020. The result of this activity is not meant to be a video coding standard but a set of metadata that can be used to provide a more realistic user experience in OMAF v2. Indeed, 3DoF+ Visual will be a part of MPEG-I part 7 Immersive Media Metadata. Note that 3 Degrees of Freedom (3DoF) means that a user can only make yaw, pitch, roll movements, but 3DoF+ means that the user can also have a limited displacement of the head.
• Several longer term explorations on compression of 6DoF visual (Windowed-6DoF and Omnidirectional 6DoF) and Compression of Dense Representation of Light Fields. No firm timeline for standards in these areas has been set.

Audio coding

In the Audio coding area MPEG is handling 3 standards (MPEG-4, -D, and -I). Of particular relevance is the MPEG-I part 3 Immersive Audio activity. This is built upon MPEG-H 3D Audio – which already supports a 3DoF user experience – and will provide a 6DoF immersive audio VR experience. A Call for Proposal will be issued in March 2019. Submissions are expected in January 2020 and FDIS stage is expected to be reached in April 2021. As for Immersive Video, 3DoF+ this standard will not be about compression, but about metadata.

3D Graphics coding

In the 3D Graphics coding area MPEG is handling two parts of MPEG-I.

• Video-based Point Cloud Compression (V-PCC) for which FDIS stage is planned to be reached in October 2019. It must be noted that in July 2018 an activity was initiated to develop standard technology for integration of a 360 video and V-PCC objects.
• Geometry-based Point Cloud Compression (G-PCC) for which FDIS stage is planned to be reached in January 2020.

The two PCC standards employ different technologies and target different application areas: entertainment and automotive/unmanned aerial vehicles, respectively.

Font coding

In the Font coding area MPEG is working on MPEG-4 part 22.

MPEG-4 ISO/IEC 14496 Coding of audio-visual objects is a 34-part standard that made possible large scale use of media on the fixed and mobile web.

Amendment 1 to Open Font Format will support complex layouts and new layout features. FDAM stage will be reached in April 2020.

Genome coding

MPEG-G ISO/IEC 23092 Genomic Information Representation is the standard developed in collaboration with TC 276 Biotechnology to compress files containing DNA reads from high speed sequencing machines.

In the Genome coding area MPEG plans to achieve FDIS stage for Part 2 Genomic Information Representation in January 2019. MPEG has already started investigating additional genome coding areas that would benefit from standardisation.

Neural network coding

Neural network compression is an exploration motivated by the increasing use of neural networks in many applications that require the deployment of a particular trained network instance potentially to a large number of devices, which may have limited processing power and memory.

In the Neural network coding area MPEG has issued a Call for Evidence in July 2018, assessed the responses received in October 2018 and collected evidence that justified the Call for Proposals issued at the same October 2018 meeting. The goal of the Call is to make MPEG aware of technologies that reduce the size of trained neural networks. Responses are due in March 2019. As it is likely that more Calls will be issued in the future for additional functionality (e.g., incremental representation), an expected time for FDIS has not been identified yet.

Media description

Media description is the goal of the 16-part MPEG-7 standard which contains technologies for describing media, e.g. for the purpose of searching media.

In the Media description area MPEG has completed Part 15 Compact descriptors for video analysis (CDVA) in October 2018. By exploiting the temporal redundancy of video, CDVA extracts a single compact descriptor that represents a video clip rather than individual frames, which was the goal of Compact Descriptors for Visual search (CDVS).

Work in this area continues to complete reference software and conformance.

System support

In the System support area MPEG is working on MPEG-4, -B and -I. In MPEG-I MPEG is developing

• Part 6 - Immersive Media Metrics which specifies the metrics and measurement framework to enhance the immersive media quality and experiences. 3DoF+ Visual metadata will be one component of this standard
• Part 7 - Immersive Media Metadata which specifies common immersive media metadata focusing on immersive video (including 360° video), images, audio, and timed text.

Both parts are planned to reach FDIS stage in July 2020.

Intellectual Property Management and Protection

In the IPMP area MPEG is developing an amendment to support multiple keys per sample. FDAM stage is planned to be reached in March 2019. Note that IPMP is not about _defining_ but _employing_ security technologies to digital media.

Transport

In the Transport area MPEG is working on MPEG-2, -4, -B, -H, -DASH, -I, -G and Explorations.

MPEG-2 ISO/IEC 13818 Generic coding of moving pictures and associated audio information is the standard that enabled digital television.

Part 2 Systems continues to be an extremely lively area of work. After producing Edition 7, MPEG is working on two amendments to carry two different types of content

• JPEG XS (a JPEG standard for low latency applications)
• CMAF (an MPEG Application Format).

MPEG-4

MPEG-4 part 12 ISO Based Media File Format is another extremely lively area of work. Worth mentioning are two amendments

• Compact Sample-to-Group, new capabilities for tracks, and other improvements – has reached FDAM stage in October 2018
• Box relative data addressing – is expected to reach FDAM in March 2019.

The 7th Edition of the MP4 file format is awaiting publication.

MPEG-B ISO/IEC 23001 MPEG systems technologies is a 15-part suite of systems standards that are not specific to a given standard.

In MPEG-B MPEG is working on two new standards

• Part 14 Partial File Format will provide a standard mechanism to store HTTP entities and the partial file in broadcast applications for later cache population. The standard is planned to reach FDIS stage in April 2020.
• Part 15 Carriage of Web Resources in ISOBMFF will make it possible to enrich audio/video content, as well as audio-only content, with synchronised, animated, interactive web data, including overlays. The standard is planned to reach FDIS stage in January 2019.

MPEG-H ISO/IEC 23008 High efficiency coding and media delivery in heterogeneous environments is a 15-part standard for audio-visual compression and heterogeneous delivery.

Part 10 MPEG Media Transport FEC Codes is being enhanced by the window-based FEC code amendment. FDAM is expected to be reached in January 2020.

MPEG-DASH ISO/IEC 23009 Dynamic adaptive streaming over HTTP (DASH) is the standard for media delivery on unpredictable-bitrate delivery channels.

In MPEG-DASH MPEG is working on

• Part 1 Media presentation description and segment formats is being enhanced by the Device information and other extensions amendment. FDAM is planned to be reached in July 2019.
• Part 7 Delivery of CMAF content with DASH contains guidelines recommending some of the most popular delivery schemes for CMAF content using DASH. TR stage is planned to be reached in March 2019

MPEG-I

Part 2 Omnidirectional Media Format (OMAF) released in October 2017 is the first standard format for delivery of omnidirectional content. With OMAF 2nd  Edition  Interactivity support for OMAF, planned to reach FDIS in January 2020, MPEG is substantially extending the functionalities of OMAF. Future releases may add 3DoF+ functionalities.

MPEG-G

MPEG plans to achieve FDIS stage for Part 1 Transport and Storage of Genomic Information in January 2019.

Application Formats

MPEG-A ISO/IEC 23000 Multimedia Application Formats is a suite of standards for combinations of MPEG and non-MPEG standards (when there are no suitable MPEG standard for the purpose).

MPEG is working on two new standards

• Part 21 Visual Identity Management Application Format will provide a framework for managing privacy of users appearing on pictures or videos shared among users. FDIS is expected to be reached in January 2019.
• Part 22 Multi-Image Application Format (MIAF) will enable precise interoperability points when creating, reading, parsing and decoding images embedded in HEIF (MPEG-H part 12).

Application Programming Interfaces

The Application Programming Interfaces area comprises standards developed in order to enable effective use of some MPEG standards.

In the API area MPEG is working on MPEG-I, MPEG-G and MPEG-IoMT.

MPEG-I

MPEG is working on Part 8 Network-based Media Processing (NBMP), a framework that will allow users to describe media processing operations whose execution is requested to a network service. The standard is expected to reach FDIS stage in January 2020.

MPEG-G

MPEG is working on Part 3 Genomic Information Metadata and Application Programming Interfaces (APIs). The standard is expected to reach FDIS stage in March 2019.

MPEG-IoMT ISO/IEC 23093 Internet of Media Things is a suite of standards supporting the notion of Media Thing (MThing). Tiis is a thing able to sense and/or act on physical or virtual objects, interacting between themselves and humans, and possibly connected to form complex distributed systems called Internet of Media Things (IoMT).

In MPEG-IoMT MPEG is working on

• Part 2 IoMT Discovery and Communication API
• Part 3 IoMT Media Data Formats and API.

Both are expected to reach FDIS stage in March 2019.

Media Systems

Media Systems are standards targeting architectures and frameworks.

In Media Systems MPEG is working on Part 1 IoMT Architecture, expected to reach FDIS stage in March 2019. The architecture used in this standard is compatible with the IoT architecture developed by JTC 1/SC 41.

Reference implementation

MPEG is working on the development of 10 standards for reference software of MPEG-4, -7, -B, -V, -H, -DASH, -G, -IoMT

Conformance

MPEG is working on the development of 8 standards for conformance of MPEG-4, -7, -B, -V, -H, -DASH, -G, -IoMT.

Conclusions

Even this superficial overview should make evident to all the complexity of interactions in the MPEG ecosystem that has been ongoing for 30 years and has produced 180 standards (note that the above only represents a part of what happened at the last meeting in Macau).

In 30 years of MPEG, and counting? I wrote “Another thirty years await MPEG, if some mindless industry elements will not get in the way”.

It might well have been a prophecy.