How Wallace And Gromit Transitioned To Augmented Reality For ‘The Big Fix Up’ (Interview)

Wallace & Gromit, the endlessly malleable British franchise, has been moulded into yet another new form: augmented reality (ar). Today sees the launch of Wallace & Gromit: The Big Fix Up, an interactive narrative experience that promises to bring the emphatically British inventor-and-dog duo to life “in an entirely new way.” It’s a bold claim, backed by strong talent and big money.

The premise is simple: Wallace and Gromit have launched a new business, Spick & Spanners, and have been commissioned to fix up the British city of Bristol. When they find that they aren’t quite up to the job, the user is invited to lend a helping hand.

The project is a joint initiative from Bristol’s Aardman Animations, the producers of the classic Wallace & Gromit series of stop-motion films, and a consortium of British digital media companies known as Fictioneers. The characters have appeared in video games before, but this is their first outing in ar (and the best they have ever looked in cgi). In contrast to elaborate vr set-ups, The Big Fix Up requires only an ar-compatible smart phone to work. Crucially, the app is free to download.

The Big Fix Up was originally supposed to include a live location-based component, but the pandemic put an end to that plan. After a few fix-ups of its own, the project was reimagined as a lockdown-friendly experience that can play out entirely at home or wherever the user happens to be, the scale of the visual world adapting to the surroundings.

Fictioneers, which was formed just ahead of this project, is composed of Potato, Sugar Creative, and Tiny Rebel Games. The consortium worked closely with Unity, whose AR Foundation framework and new MARS (Mixed and Augmented Reality Studio) tool were integral to the production. The script was written at Aardman, which was involved throughout the creative process. The University of South Wales provided research support, and the project is backed by a multi-million-dollar grant from public body Innovate U.K.

Ahead of the launch, Cartoon Brew caught up with two key members of Fictioneers: Susan Cummings, co-founder of Tiny Rebel Games, and Lee Bowditch, 3d artist for Sugar Creative, who served as the lead 3d artist on The Big Fix Up. Cummings came to the project with 25 years’ experience in game design, production, and publishing, while Bowditch had existing ties to Aardman: he created digital doubles on Early Man and the upcoming Chicken Run 2.

The pair told us how the project came about, how it pivoted when Covid struck, and why Wallace & Gromit lends itself so well to ar…

Cartoon Brew: When did the consortium come together?

Susan Cummings: In mid-2018, we became aware of Innovate U.K.’s Audience of the Future program. Around £30 million was put aside for immersive entertainment R&D projects.

Tiny Rebel Games brought together Sugar Creative and Potato — many of us had worked together in the past on other things, though not collectively as companies. We bid for the Moving Image grant. It was a multi-million-pound grant that we were given for a vision of how storytelling might change going forward — how stories could start to blend different types of experiences, using mixed and augmented reality for example.

Our notion was that we could tell a story in real time. We built a platform for that, which we called the Must platform. The Big Fix Up is basically our first proof of concept. When it launches, the story is going to be told in real time over the course of about 25 days. You’ll install the app, the framing of that being that you get to join Wallace and Gromit’s company, Spick & Spanners. They’ve set up this company to do odd jobs, they’ve got in over their heads, they need help, and that’s where you come in.

Will users have to have the app from day one in order to follow the full arc of the story?

Cummings: No, because we know that people like to binge, and that not everyone is going to join on day one. One of the things about our platform is that it allows you to catch up … There’s about 160 different pieces of content, so you can totally take your time, or you could frantically get through it and catch up with your friends, so that you’re on the leading edge of the story.

So you had the consortium, you had the grant — how did the partnership with Aardman come about?

Cummings: Going into the beginning of the project, we were tasked with finding an IP partner, and Aardman was someone I had a relationship with, in terms of conversations in the past. When we told them what we were and what our goals were, they jumped onboard almost immediately. We had a series of conversations and settled on the fact that it should be Wallace & Gromit. They totally shared the vision and wanted to help.

We didn’t want a licensing partner who’d just tell us to go off and do our thing. We really wanted a collaborator, and Aardman has really been involved every step of the way. Which is why it’s so great having people like Lee on the team; our product manager as well came over from Aardman. There’s been a lot of crossover like that, which made for a better process. It was like a writers’-room collaborative process between their creative team and ours.

What is it about Wallace & Gromit in particular that lends itself to an augmented-reality experience?

Lee Bowditch: I think it lends itself well to ar because of the inventiveness of Wallace. The app lets you get involved with that — to construct different contraptions and send them off on jobs. In terms of other IPs that could have happened, I think Wallace & Gromit was a really good one to start off the platform.

The characters appear here in cgi, but the aesthetic preserves something of the stop-motion look. Did keeping that look present any challenges, specifically in the context of ar production?

Bowditch: Absolutely. Aardman are very particular about their style. It’s so iconic that if we hadn’t nailed that style, it would have been very obvious to users that it wasn’t authentic. We strived to get the plasticine look — we had production sculpts which were then scanned using a 3d scanner at Aardman. Those meshes that were outputted were given back to us, and we used them as our base for all the 3d models.

We also had the limitation that it had to be in-game: we had to use these [characters] in Unity, it had to perform well, and obviously it had to be animated. In the past, with cg adaptations [i.e. digital doubles of stop-motion characters], it’s been on a film level, which means you can have thousands of polygons and 4k textures. But we were limited to what the engine can cope with.

Were there any past projects that you used as reference for how to port this kind of aesthetic into this kind of medium?

Bowditch: We kind of winged it. We had Early Man stuff to look back on, where I’d had a lot of back and forth with [Wallace & Gromit creator] Nick Park about getting the feel right — the mouth shapes, everything that makes it quintessentially Aardman. We had that wealth of knowledge, and we went from there.

The Big Fix Up uses Unity’s new MARS platform. Can you explain what this platform is — what it brings that’s new to ar production?

Bowditch: What we’ve used MARS for, specifically, is something in the app called Fix Up Your Space, which allows users to scan their room. MARS gives us a smart layer on top of conventional ar scanning so we can generate a navigation mesh. It identifies vertical and horizontal planes we can use for the navigation — it registers surfaces and floors and stuff like that. We have a library of assets we’ve included in Fix Up Your Space that allows users to place portraits of characters on the wall, or the contraptions in real space.

MARS does a really good job of reading surfaces. It also adds a feature in Unity where, without having to build every time, you can make a 3d space and test that all your assets are working perfectly. Then you know that when you build, they’ll be perfect on your phone.

Since you started on this project, the coronavirus happened. You’d conceived it as an experience that integrates the city of Bristol, but you pivoted in response to the pandemic. What exactly did you change about the project? Did the team change?

Cummings: The finale of the story was meant to take place in Bristol. We used a company called Fantasmo to LIDAR-scan key areas of Bristol. [But then] asking people to come to Bristol didn’t seem like a very responsible thing to do anymore! We did bring on some more Unity resources to work with Fantasmo and figure out the best way to replicate that experience for the home user.

We took that LIDAR-scan data, and we had to make it look prettier, more presentable to the user. It was never meant to be seen — it was just data for the development environment.

Bowditch: The scans that we got back from Fantasmo were these hugely dense decimated meshes of Bristol, really nicely done: you can see crazy detail of all the buildings. For the main areas of the finale, you’ve got Queen Square and College Green, so we had to build those surrounding areas based on that LIDAR scan.

For [City Hall] in College Green, which is a huge, massively grand building, we had to go in and make sure it looked nice enough that when you’re playing at home, you don’t question that it’s a nicely finished mesh with nice textures.

Cummings: We treated it a bit like scalable dioramas. For someone who’s trapped in their house right now: you find an available surface, then you can scale it to be really small or quite large. In my neighborhood, I’ve walked around testing this stuff, and I placed it in a local park.

We did look at other ways of doing it. We talked about an idea that was more like a 360-panorama, which would sort of fake the experience — you’d pretend you were in Bristol. But it suddenly didn’t feel like ar anymore. We really wanted users to be cognizant of the fact that this was happening as an overlay to the world around you.

What’s really cool is that [our approach] allowed us to sort of film-direct these moments: to zoom in, zoom out, and change the camera for you, so you really get the full feel of the moment, and the scale of it.

We still do plan to do the city-scale experience. The government has provided additional funding to spend another six months working on that. We’ll be announcing in the future what our plans are there.

“Wallace & Gromit: The Big Fix Up” can be downloaded from the Apple Store and Google Play in the U.K., U.S., and Canada. A global roll-out is planned.