Knowledge Graphs are transforming the Video industry from Glass to Glass

Human responses to scenes in videos vary, and moviemakers have experimented with styles of video making, mostly using intuitions so far. There's a statement which does rounds in movie circles - the audience in a large auditorium does not even cough, or sneeze when the movie is in an interesting segment.

Certain actors have the ability to work with certain props and delight audiences.

But what is the causality to holding the attention and delighting the audience?

Studying how creativity expresses itself, and how audiences react needs us to document the knowledge and insights at each stage. Knowledge graphs - the ideal data structures to capture this - are transforming the way the Entertainment Video industry works - from glass to glass.

From the glass that captures the video, to the glass where the video is watched.

At the camera, the creator captures the intent and context behind the scene. At the television, the TV captures the reaction of the audience. They will be shared with each other for better insights and experiences. Today, the glasses on either end of the video distribution chain are connected and facilitate this. [yeah, everything will be done preserving copyright, content ownership laws, and the privacy rights of individuals]

Capturing Knowledge behind a video:

The intent and context of the video is captured in a Knowledge Graph of plot concepts, scene props, actors, directors, cameramen, crew, state of the video in the screenplay and plot, background scenery, and perhaps even the economic aspects involved with the studio or the location.

Let's say a movie involves the first encounter between the male and female leads in a crowded town bus, a traffic signal or on a foreign trip in a posh restaurant - with a peppy music playing in the background.

Notice that this one scene has introduced several knowledge concepts.

• "First encounter" indicates a cardinality in meetings between people.
• "Leads" indicate that they are meant to be central characters in the story.
• "Male" and "female" indicate genders.
• Depending on the people being cast in the roles - their history and character perceptions come into play.
• Crowded town bus, a traffic signal or even the restaurant are props to set context.
• Peppy music sets the tone of the scene.

Props matter a lot in movies, and give shape to scenes. They may just fill up backgrounds or represent a metaphor that carry along the entire movie. Props are often used in a subliminal way - without sound of dialogue - but do their job. Why props matter in movies?

Every frame is a work of art, frames make up shots, and shots make up scenes. A scene is meant to drive a core plot concept through.

Movies are a timed sequence of plot concepts - sometimes interspersed just to add to suspense and confusion. A movie goes from plot concepts like: A new king, A place is bombed, a volcano erupts, collecting gems, breaking rules, monkeys taking over the city, a spaceship losing control, sister taking care of a brother when parent are away - potentially 1000s of them. Certain genres tend to use (or overuse) certain plot concepts. You'll realize why those superhero and Bond movies became a big bore - and why they started experimenting with alternative plot concepts.

Christopher Booker wrote an excellent book on the 7 basic plots . These surprisingly underline the themes taken by most movies.

Almost all fiction goes through stages of anticipation, dream, frustration, nightmare and then onto resolution in the climax. These are called meta-plots.

The actual plots involve :

• Overcoming the monster
• Rags to riches
• The Quest
• Voyage and return
• Comedy
• Tragedy
• Rebirth

At a more granular level, titles are then organized around metadata that involve the genres, sub-genres and micro-genres. A movie like Interstellar, is organised around:

Genres: Science Fiction, Adventure, Drama & Mystery

Micro-Genres: Time travel, NASA (Space Agencies), Father-Daughter relationships, Apocalyptic future, Weather hazards

At Sensara , we've set out on a journey to capture all these details in every video and organising every movie and show as a path in a multi-dimension graph of concepts in the fiction universe. Much of them automatically understood using AI - but also organised and captured with precision using expert guidance. Capturing what is happening at unprecedented depth allows for any downstream analysis where we know how the audience responds.

Some of the "about"-ness of content is also captured by efforts by IAB - but not at the level of detail expected by the video content industry.

Capturing Knowledge behind a pair of eyes:

Move over to the audience side.

Did people have a smile on their face during a scene? Were their eyes glued? Was this scene subject to memes on social media later? And who watched the scene? Someone looking out for a car to buy this year?

Attention can be measured and organised into: Attention (Gazing, Distracted, Inattentive); Responses can be measured and organized into: Emotional response (Smile, Frown, Fear, Neutral, etc), Sound (Unrelated cross talk, Speech, etc.), Activity on the Remote (or Video player) - like Switch Outs, Pauses, Replays. Social watch parties are also recording social responses at different stages of a movie or show.

There has been much research on figuring the correlation between narrative and attention - like this paper by Hinde et. al , and there is a lot to learn from this.

Audience types have been well understood for quite sometime now. IAB has an extremely well defined taxonomy for audiences - grouped around Demographics, Interests and Purchase Intents.

X marks the spot. (The Cross product yields a treasure)

Two deep knowledge graphs - one from the content's side, and another from the audience's side - when mixed - yields an unprecedented level of knowledge of how people respond to video.

Plot concepts are inferred using frame/scene analysis using structural and semantic information embedded in the video.

Overlaying audience attention information and analysing upswings and downswings in attention yields viewer preferences.

When the content universe is indexed into the knowledge graph, and the audience universe is mapped into an audience knowledge graph - we end up with an encyclopedia of what catches people's attention and what does not. Let's take some practical examples from there on.

Imagine an intended audience for a new OTT original movie being planned. An upscale, educated home - aware of global warming, the ills of over-industrialisation, that feels a certain responsibility to nature and ecology. Do the movie's script writer, and director know all the genres, micro-genres, plot concepts and props this audience may have been already exposed to? Can a new plot be concocted that appeals to this audience - in bring on the same relevant topic, but in a way they wont feel bored?

Or let's say, a producer is willing to fund a movie that can cater to a captive audience - on a relevant topic they've never seen a movie on. Something like steganographic terrorism?

Because no one has ever made a movie that can cater to the large Deep learning-aware audience. What kind of plots, topics and props to pick up?

With Connected TV capturing the world, fine-grained viewer behaviour is now available for analysis at an unprecedented scale. Similarly, progress in Video AI has allowed for the content industry to invest in automated scene indexing of videos. Knowledge Graphs - organising all of the available knowledge into ontologies and concepts helps make connections across dimensions. We are certainly staring at a new world of possibilities in data-driven movie making. This is only the beginning, as we see the first applications roll out soon - and make the world of TV more "knowledgeable".