VR veteran: 360 degree interview with Dr. Ralf Schäfer
Dr. Ralf Schafer, division director of video at the Fraunhofer Heinrich Hertz Institut in Berlin, was working on VR long before the recent craze for the format. He tells TV Tech Europe about his own research and what VR might look like in the future.
How did you begin working on VR?
I'm head of video division of Fraunhofer HHI. We have about about 90 people working in this area. One of our main topics is video encoding - we substantially contributed to the standardisation of H.264 and H.265. And our other major focus has been 3D and immersive imaging technologies, and VR is a part of that. A long time before VR became hyped, we had already developed 180 and 360 degree cameras and display systems.
The idea we started with was to use more than one camera to show the content on a head-mounted display. So we started with three HD cameras stitched together and fed it to a head-mounted display, so by turning the head, you could see the section that you were looking at. It’s very similar to what VR is today, but we had only three HD cameras.
Then we thought it would be nice to build a projection system for that. So we built a 180 degree projection system, but realised we would need cameras to create content for that - we didn’t want only static pictures or computer-generated pictures. Now we are already at the fourth generation of those cameras. And now that the whole VR thing has popped up, we are in the game.
The advantage of a broadcast service is you have enough bandwidth to transmit the full panorama in high resolution
Is the current mania for VR different from the 3D push of a few years ago?
3D obviously didn’t work at home. Production was much too expensive. In the cinema it’s still stable - the number of 3D films is still stable or even growing. It’s now shot in 2D and then converted to 3D, and those conversion technologies have improved quite a lot, but it’s still an expensive process. And it would be much too expensive for TV production.
People also seem unwilling to wear glasses at home. If you watch TV, you also do different things - eat your dinner or read the news. In the cinema you stay in one place for 1/1/2 hours and no one has an objection to wearing glasses, but at home it doesn’t work.
So the consumer electronic industry had to look for something else, and the other thing that came up was UHD – which will be a big success.
But then Facebook bought Oculus Rift for two billion dollars. That has created a lot of fantasy in peoples heads: “Wow, Facebook spends two billion. This must be a big thing.” And I think that’s how things started.
Meanwhile, the technology is also ready, and the whole mobile industry is behind it. Every phone with a high resolution display is potentially a VR device. So a lot of people have jumped on it.
But it’s still unclear what the killer application will be. Games is obviously one application. But the other ones aren’t clear yet.
Sky, in its collaboration with Jaunt, shot some VR news footage last year. Do you have an idea what kind of mainstream applications VR might have? Or will it remain in a fairly niche market?
It’s not something where you will have to only use VR glasses. It’s also something you could use on your TV set or iPad where you can navigate around scenes. That might be interesting for applications like sports or live events like music shows. You could put 360 degree cameras in the crowd or onstage and then the user could navigate, even using an iPad or TV.
It’s still unclear what the killer application will be. Games is obviously one application. But the other ones aren’t clear yet
How would VR work in a broadcast infrastructure?
You could present it as an OTT service, but it would be better to broadcast it. The advantage of a broadcast service is you have enough bandwidth to transmit the full panorama in high resolution. You might need 40 or 50 megabits. It’s still quite expensive because you might need one complete satellite channel or complete cable channel, but it’s possible.
But if you stream that, the bandwidth is likely to be limited. You’d have to down sample the panorama, which is not a big deal now, since the displays are still quite bad, but they will become better. The HTC Vive is already quite nice, and you will see the lack of resolution with that downscaling.
It might be downscaled to HD or even 4K, but that’s still not enough if you want to see that even on your TV set at home. If you could transmit a much higher resolution – maybe even 10K – it would look much nicer.
How far away is that from happening?
We will be setting up a demo at IBC.
A lot of people suddenly want to get into VR. What is the entry into shooting and distributing VR?
It depends on the quality you want to provide. If you use these GoPro rigs, something comes out of that, but the quality is not so nice.
At Fraunhofer, we built the OmniCam-360. At NAB we presented two new versions, one of which is a 3D version (pictured above). In previous versions we used mirrors to avoid parallax. The other systems had a star-like arrangement of cameras and you had some serious parallax, which is okay if objects are far away, but if they are close to the camera, they don’t work when it comes to stitching together the images.
We are also the only ones who can do stitching in real time. For stitching, we bring all camera feeds into a single pc and we use some powerful graphics boards. What comes out is a panorama of 10K by 2K in 360 degrees.
But the price for one system is about 300,000 euros, so most people continue to play with GoPros - but the result looks like the cost.
Fortunately, the cameras will become smaller and cheaper. The problem is you also need good optics in order to get the quality out of the cameras. And it’s always a compromise between the size of the camera system and the quality.
Three years ago we built a 180 degree rig with six Alexa cameras (pictured below). It was very good quality, but very bulky and not very practical. We shot in the Royal Albert Hall in London and a Chelsea football match and also a concert with the Berlin Philharmonic Orchestra.
Based on your experience, what special considerations are there in shooting VR?
We are just the providers of technology, so we always work together with creative people. We don’t do the shooting ourselves. There are certain rules you have to obey.
The biggest problem is the lighting. Especially in a system where you use the mirrors, and you get a flare on one segment, it’s very disturbing.
And sensitivity of cameras and lighting is a huge problem. Normally, if you point camera at a dark area you can open the iris more than you would in a bright area. But in a 360 environment you have everything at one time, you have the dark and the bright. And you have to use the same iris for every camera, otherwise it’s difficult to stitch together.
That’s why we were so happy about being able to use the Alexa rig, because the Arri’s have such a wide dynamic range.
What new skills are required with VR and how are people learning them?
The general problem is the optical systems of GoPros. They have wide angle optics, which come out looking very unnatural. Then the synchronisation is a problem too with the GoPros, which mean you can’t shoot live, you have to rely on an offline process. I don’t think the use of GoPros will lead to satisfactory results.
On the other hand there are much better cameras that will be two or three thousand euros. And then it becomes more affordable.