Animation In A VR World: How Is It Different and How Is It The Same?

Whether you believe the hype or not, virtual reality is here. Billions of dollars are being spent on vr headset development, vr content creation, and getting users to adopt the technology.

The content side of vr has opened up new opportunities for filmmakers (note: ‘vr’ is sometimes used interchangeably with 360 degree video). Indeed, already major directors and animators have tried their hand in the medium, from Jon Favreau with Gnomes & Goblins to Patrick Obsborne with Pearl. And animation and vfx studios have also made a major effort to become vr content producers by jumping into immersive and interactive projects.

But how different is that from making content for the linear narratives of old, and what lessons have creators learned so far in producing animation for vr? Cartoon Brew talked to studios Industrial Light & Magic, Framestore, and Reel FX, and with artists from projects including Gnomes & Goblins and Oculus Story Studios’ Dear Angelica, to find out the key things that anyone wanting to animate for – and in – vr should consider.

How is animation for vr different (or is it different at all)?

It’s clear that vr offers a new way to experience content, but for the content-makers, does that change the way they need to approach animated vr content? In general, yes, say the experts Cartoon Brew spoke to.

While linear mediums tend to involve locked off shots or camera angles from which to animate, vr experiences are akin to real world interactions, with multiple vantage points that are always changing. That means you have to start ‘thinking’ about vr animation in new ways.

“The biggest difference between vr and linear projects is that you are putting yourself inside the experience and looking at the animation with this new perspective,” said Jake Rowell, a creative director on some recent projects for vr studio Wevr, including Gnomes & Goblins. “This participation gives you a lot to think about and react to with regards to scale and speed of movement relative to the players’ size.”

That kind of thinking is certainly appropriate to room scale (or larger) vr experiences since at any time the user can look around and would need to see animation looking good at all times. However, animating for smaller scale vr experiences, say where the user is seated, require an animation approach similar to linear projects, according to Tim Alexander, a visual effects supervisor from Industrial Light & Magic, whose related studio ILMxLAB is forging ahead in vr and immersive experiences. “You only care about a single camera angle and although perhaps you have parallax, you can’t orbit all the way around a character so you can get away without polishing animation for all angles,” Alexander said.

Animating for vr is changing, quickly

Right now, there’s a big mix in vr content – from experimental pieces, to promotional film and product tie-ins, to games and animated shorts. Some involve live action, some have photoreal cg, some cartoon-y, and many have a combination between all of these. That means there’s a mix of mediums in vr, and a mix of methods in how the animation is being done.

Among the more interesting vr projects of late is Saschka Unseld’s Dear Angelica from Oculus Story Studio. It is a narrative short with a painterly aesthetic. Although not heavily immersive, the revelation in Dear Angelica is that it was made possible with a new tool called Quill that allowed for the painterly brushstrokes to be realized in a vr environment.

Wesley Allsbrook, who began as an illustrator on the project and then segued into an art director role, said Quill was in constant development during production (and still is). “We had everything that I wanted for basic drawing by February of 2016. I needed pressure sensitive, calligraphic – wider on one axis than another – strokes with transparent and opaque taper, scaling on every axis, a world with an optional ground plane and absolutely no sense of relative scale, layering for independent scaling and placement of assets, opacity settings, and the ability to re-color assets. And rotational opacity. Inigo Quilez had that in the very first test of Quill. Unfortunately we didn’t get to use rotational opacity in Dear Angelica. That would have been a real game changer.”

Directing that performance – and the user

Since the user can look around at just about all angles in vr, the usual cinematography and editing of linear narratives is gone. The user is now the camera operator. So, depending on the experience and the story being told, animating for vr can also involve the requirement to direct the viewer where to look.

“You really need to account for a variety of audience members and anticipate what they might look at and when they might look there and try to stage your animation in such a way that allows them to progress through the story,” said Bill Haller, an animation supervisor at Reel FX, which has produced a range of vr shorts, as well as product and film tie-ins (most recently one for Power Rangers). The studio is also working on a Google Spotlight Stories project with director Jorge Gutierrez, having also been the studio behind his animated feature.

One of Haller’s colleagues at Reel FX, animator Steve Cummings, suggests, too, that this side of animating in vr is crucial as the user can otherwise “easily become disinterested or just plain confused as to what they should be looking at. In order for this to all work the animator really needs to work closely with not just the other animators on the project but with layout, sound, and lighting as well.”

Framestore senior animator Shayne Ryan calls this ‘earning the user’s gaze’, and also says that part of animating for vr is simply being conscious that a user might miss some content (of course this is one selling point of vr – multiple viewings). But, as Ryan told Cartoon Brew, “the flip side to someone missing content, and probably the most daunting aspect of VR, is that there is absolutely nowhere for our animation to hide. As soon as we introduce something we need to keep it alive until the choreography of the scene naturally removes it.”

Animating in vr: things to think about

How do you animate for vr? In most cases, it involves the same technical approach as linear projects. One major difference is the process of reviewing that animation, made difficult by the need to wear a vr headset. Of course, the idea of animating directly in vr is where Quill and Oculus are headed (as well as other tools out there), but so far most projects have utilized the same or similar pipeline used on linear jobs.

A recent Framestore vr animation project tie-in with Fantastic Beasts and Where to Find Them was realized, for example, by viewing everything during animation through an ‘unappealing’ wide angle lens. “This totally distorts the image and ruins any sense of scale in the scene,” said Framestore’s Shayne Ryan. “It quickly became really difficult to judge how close or far something should be from the viewer. The recurring theme of the production was ‘I need to see it in headset!’ which we could only do through rendering out a stereo 360 video, encoding it, and installing it to a device.”

That’s something ILM’s Tim Alexander is also after, as well as some kind of collaboration toolset. “It’s very difficult to have a single person in a headset giving notes, thoughts, and inspiration while everyone else in the room is watching a TV screen,” Alexander said. “Deep, simultaneous, interactive collaboration will bring new insights and creativity into the process.”

Alexander continued: “Currently, I think two areas of collaboration would help immensely. First being able to be in vr together — understand where your collaborators are looking and what they are talking about. Second, having markup tools, navigation tools, and simple editing tools. Collaboration in vr will help ideas flow and will help teams to work together more effectively.”

One of the challenges of animating for vr arises out of the continuous shots with no camera cuts. How do you hook up to another animator’s work so that it is seamless? Reel FX’s Steve Cummings suggests using two methods. One is pose hook up frames, where the animators between two ‘shots’ seamlessly match their posing on a very specific frame, and the other is hiding a character behind an object for a switch frame.

“Another example was figuring out how to lead the audience with our action into the next story point, in the vr world,” said Cummings. “One way we did a vr short is that for the start of the shot we worked with layout so that when the viewer arrives in the beginning of the story, they are already looking where they need to be, that became the ‘front.’ Then, throughout the rest of the piece we would try to lead their attention through our characters’ movements or use of props.”

The future of vr and animation

The take-aways from these experts in vr animation seem to be that current animation tools and techniques certainly still work in the vr world, but with new considerations required. When animating in vr there’s more to think about in terms of continuous shots and leading the user/player on a journey through the experience.

The experts also have a wishlist of vr animation tools, mainly in the areas of previewing and reviewing work, as well as for collaboration. These tools are certainly coming – or here already. Just look at Quill, Unreal Engine’s VR Editor, Animvr, Limitless, or the recent launch of Storyboard VR, for example.

There’s also an increasing number of places to experience vr content, from at home and online, to specialized vr plazas, and film festivals incorporating vr screenings (SIGGRAPH’s computer animation festival is one of the latest to do this). Just as animators might seek out linear content to watch and learn from, the same can now be done more and more with vr.