Managing media assets in an immersive content future

VR Fish
Neal Romanek
Acquisition
August 15th 2017 at 12:00PM : By Neal Romanek

VR and 360 content presents an especially thorny issue for asset management. David Schleifer, COO at Primestream, explains why

We are surrounded by cues that make us aware of how we fit into a 3D space. Many of these senses and reactions are based on survival instincts - the ability to hear someone walk up behind us, to see movement in our peripheral vision, a sense of how we are oriented when we move our bodies or swivel our head. All of this creates an overwhelming amount of data that we seamlessly interpret as we go around our day getting our morning coffee, driving, walking through crowds, diving into swimming pools, and everything that we as people do in our lives.

As movies and television have evolved, we’ve learnt a new language, a new way to interpret the world around us based on a window into another world, the screen. What we’ve now become accustomed to via that ‘screen’ are jumps in location, perspective time and space. The screen is sometimes supplemented with surround sound, but usually we’re just left to our imagination to weave the data we get into human stories.

Close ups, wide shots, panning, zooming, and many other devices are all an established part of the storyteller’s toolkit.

Then along comes Virtual Reality (VR) and 360, and suddenly we have a tendency to fall over reaching out for things that aren’t there. We lose perspective, and even though this world, like ours, is not flat, we are lost!

There are differences between VR and 360-degree images, but what they both share is an immersive environment and the ability to represent, like reality does, what is going on all around you. Despite both VR and 360-degree images being more like reality, we find ourselves struggling to know how to use it.

Let’s get physical

There are several disconnects for us as viewers. I think a major one is that we do not physically navigate our way through the world in the viewer, we can spin around and look up or down, but we don’t have all of the normal cues leading up to this. We didn’t walk into a town square or into the theatre, so we immediately look around to get our bearings as if just teleported in. Then we can’t really move around in 3D space, we are restricted to rotating to see what is next to us. In VR environments you can navigate through space, but generally not in the normal way like walking over to something you see. So basically, we’re a little lost and as a result it takes us time to orient ourselves into what is a static scene with action someplace that we have to find.

One of the challenges we face is that the formats for capture and for delivery are in flux, and how those streams are delivered is also evolving

Throughput the whole VR/360-degree experience, the director can’t guide you through the story in the same way as in a 2D film. They no longer have the fast cuts, the zooming, the pacing they are accustomed to using to move you through the story. As a result, there is a focus on the novelty as opposed to an improvement in how we can tell the story.

There is a lot of experimentation going on today, and a lot of it is on the capture side. How to we grab better immersive images and stitch them together. Primestream is dedicated to managing those files. Images are captured, metadata is then added and managed around those files, and ultimately, the workflow is organised around them. This puts us right in the middle of the efforts to better understand how to tell stories with these new assets, pulling us into discussions around the creative process. As creative tools like Premier evolve to meet the needs of editors using them, we too are evolving how to manage the data and metadata that adds value to those captured images in the process.

360 MAM

MAM providers and NLE manufacturers have been chasing ever-changing formats for the past few years, so our workflows have evolved to include format independence, but there is always a limit to the assumptions manufacturers have made about the files they are processing.

A first step to handling 360 material was understanding equirectangular files. A second step is understanding where there might be VR or 360 specific information that has been added to a standard format like MPEG. Finally, in order to facilitate an end to end workflow it is important to find developing partnerships and standards that are gaining support with both Capture toolsets and finishing toolsets.

One of the challenges we face is that the formats for capture and for delivery are in flux, and how those streams are delivered is also evolving. Gone are the days when a SMPTE standard for SDI controlled your infrastructure rollout. We are seeing file-based workflows and IP-based delivery dominate across any new innovative introduction.

We are no longer engaged in tracking content segments with time-based markers, but rather content that needs to be marked up in time and space with directional data

 

In one sense getting started is simple because manufacturers are leveraging existing formats like MPEG-4 or H.264, but in order to create more and more immersive content cameras are pushing the limits of what they can capture because if you were happy with 4K, you now need that resolution all around you. Each lens, each eye on the camera is capturing 4K that needs to be stitched into a seamless sphere into which the viewer can be inserted. Can 8K per lens be far behind? You also want 3D spatial sound to create an even more connected and immersive experience and all of this requires spatial data and other markers.

We can’t force the market in a specific direction, but will need to be there for them when it evolves. We can participate in partnerships that seem to show directional leadership, like when a camera manufacturer and a codec supplier team up to create an end-to-end workflow, but ultimately we need to be reactive to changes in the marketplace that can come from anywhere. Will there be a shift from equirectangular to cube maps, to pyramid maps, to something else? We will need to be there if and when it does.

We are no longer engaged in tracking content segments with time-based markers, but rather content that needs to be marked up in time and space with directional data. Not only when, but also where the interesting content is.

We fully expect the requirements for VR/360 asset management to continue to evolve as people figure out how to use it to perfect their storytelling techniques over time, and we know that the creative tools will need to move quickly to keep up with that innovative re-imagining of how we tell stories. We are not there yet, but you can already see the early signs that we are creating an exciting new creative narrative.