[This story from The Guardian describes a new application of presence-evoking deepfake technologies that may provide entertainment but also raises ethical issues; see the original version of the story for three different images and a short video teaser. For a related development, see the We Got This Covered story “‘Family Guy’ imagined by AI as a live-action ’80s sitcom is giving off strong uncanny valley vibes.” –Matthew]
[Image: Source: ITVX on Twitter; follow the link for another video teaser]
Behind the scenes of TV’s first deep fake comedy: ‘None of it is illegal. Everything is silly’
Is that Harry Kane and Stormzy arguing over a broken patio tile? This new, CGI-assisted comedy is astonishing to watch – but is it ethically OK?
By Stuart Heritage
January 9, 2023
Spencer Jones beckons me into a Soho editing suite. “Do you want to see some of the stuff we’ve done so far?” he asks, readying a clip of his new ITV sketch show. It’s funny enough; a young impressionist does an impersonation of Tom Holland griping about something in his flat.
But then Jones stands up and walks me through to a different room. He closes the door and opens up a laptop. “Now watch this,” he says, grinning. It’s the same clip: same script, same flat, same line delivery. But there’s one small difference – this time, incredibly, the sketch is being performed by Tom Holland. Except it isn’t. My jaw drops open, and my eyes start flitting around wildly, unable to compute what I’m seeing. In other words, I have just had my first experience of Deep Fake Neighbour Wars.
The show, in a nutshell, is this: we meet a bunch of celebrities, who all live like normal people in normal homes, bickering with their neighbours, who also happen to be celebrities. It is Stella Street, basically, except the faces of the performers are obscured by photorealistic images of the celebs. The effect is eerily seamless. Flick on to an episode by accident, and you’ll be doomed to spend a minute or two wondering why the hell multimillionaire movie superhero Tom Holland has decided to slum it on an ITV sketch show.
Joining Jones (the show’s head writer, most recently seen on the BBC’s charmingly handmade comedy The Mind of Herbert Clunkerdunk) in the laptop room is Barney Francis. He’s the head of StudioNeural, the company that provides the deepfake technology for the series. Next to him, on a separate computer, is a photo of Harry Kane. Except it isn’t a photo at all; it is an AI-generated image of him that has at this point been refined more than a million times. “This is an earlier iteration,” Francis says, leaning in to point out various imperceptible flaws in Kane’s face. “You can see that the eyebrows are still coming and the beard looks quite plasticky, but we’re really getting to the stage where the pores are coming through.”
The whole thing is astonishing to witness. Until now, for most people, deepfakes have been an area of largely theoretical concern – the technology has been used to any number of malicious ends, from fake news to fraud to revenge porn – so Deep Fake Neighbour Wars is likely to go down as the moment when a huge chunk of the population will experience it for the first time.
“It’s an order of magnitude bigger than anything that’s been tried with this technology globally so far,” says Francis. His involvement in the show began when he started using deepfake technology (he prefers the term “synthetic media”) for the Sheffield international documentary festival. “At the festival, we used it for narrative storytelling and video art, basically,” he says. “One piece we did of Mark Zuckerberg blew up, and there were two weeks when it all got quite intense. And then, off the back of that, production companies like Tiger Aspect came to us asking if we could use the technology for longform. We have always wanted to have a philosophical exploration of this technology, and then look for interesting ways to apply it.”
It’s a strange but happy marriage, this relationship between comedian and tech wizard. One of the things that Jones and Francis bonded over, for example, was the desire to muck around with the established order by putting famous faces on to less than realistic bodies. “We loved the idea of Mark Wahlberg not being a fitness freak,” Jones smiles. “He’d just been on a cruise and he’s really bowling around. We gave him deep-vein thrombosis and asthma.”
“It totally blew our minds,” continues Francis, “because me and my team are normally working to achieve total realism.” He still seems slightly thrown that someone would want to use deepfakes for something other than perfect photorealism. “It’s the first time we’ve sent back a bad test result and someone said it was perfect.”
How the show will go down with viewers is a different matter. It’s one thing to watch celebrity impressions on Spitting Image, for example, because it’s instantly evident to everyone that the grotesque puppets on the screen are not the actual politicians the show is lampooning. But here the likeness (not to mention the voices and mannerisms) are so spot on that there isn’t that comedic remove. There is no established deepfake etiquette yet, either, so the show could, potentially, go trampling through all kinds of ethical grey areas in search of laughs. That said, it doesn’t seem like something Jones is particularly worried about. “None of our heroes in our show are doing anything illegal,” he says. “Everything is silly. If you turn us on halfway through, and think that the real Harry Kane has really had his patio tile cracked by Stormzy, you might need to have a little look at yourself.”
You will notice Jones used the word “heroes” there. Throughout my hour with him, this was uniformly the term used to describe the various celebrity subjects of the show. It sticks out a little, not least because hero worship doesn’t traditionally tend to lend itself well to comedy. My assumption was that this was his way of keeping the more nefarious aspects of deepfake’s reputation at bay, as well as any accusations that using a celebrity’s exact likeness could be exploitative, but Jones maintains that the whole thing is simple affection.
“They really are our heroes,” he says. “This isn’t ‘they’ve fallen from where they were and then now they’re in working-class environments’. We’ve just reimagined them with everyday problems.” There is nothing more universal than horrible neighbours, he points out.
The decision has also been made to begin each episode with an onscreen disclaimer, to make perfectly sure that everyone knows these are not real celebrities. They are, however, being played by real people – a range of up-and-coming performers, some of whom are so green that they don’t have agents – which raises another question. This is their big shot at being on TV. Doesn’t it hurt to be filmed, only to have their faces obscured the whole time?
Katia Kvinge plays a number of characters on the show, from Greta Thunberg to Phoebe Waller-Bridge, and she doesn’t seem to care very much. “I don’t mind at all,” she says. Although she regularly appears on camera as herself, she says: “I do voiceover work, so I’m more than happy to just do things with my voice. And in a way it’s better. One morning before work, I was so tired. But then I was like: ‘Oh, it’s fine, my face isn’t on camera today.’”
The concession to this potential lack of recognition comes at the end of each episode, where the deepfake masks fall away and the performers get to smile and wave at the camera. Not that it’s all that unusual for the masks to fall away anyway, because the technology is still fairly restrictive. To maintain perfect deepfake contact, Francis explains that the performers have to keep their faces to camera as much as possible. If they spin around, for example, the deepfake won’t have anything recognisable to hang on to and will simply disappear.
“You’re there with your scene partner, and naturally you’d turn to face them, to agree or just, like, make eye contact,” says Kvinge. “But the whole time you’re constantly trying to keep still for the technology. I auditioned for something after we finished filming, and I was like, ‘Oh! I can turn my head!’”
There are other things to consider. If a character turns around, covers their face or pulls an exaggerated facial expression, it can disrupt the deepfake. “One of the gutting things for me was that we couldn’t have people behind windows looking out, right?” Jones says. “Because it’s an occlusion. Can’t have it raining, either. But I think, by the end of it, we all got good at knowing what the limitations were.” The nature of the technology means that deepfakes work better if the performers share a basic facial layout with the subjects, too. “I was a 95% facial match with Olivia Colman,” says Kvinge with pride.
What’s thrilling about Deep Fake Neighbour Wars – apart from the fact that it’s funny, something that tends to get shoved to one side in all the technological excitement – is that deepfake is still an emerging technology. Compared with two years ago, the things that synthetic media can do are astounding. So, I ask Jones and Francis as I prepare to leave, where will we be in another two years?
“We’re developing new automated tools,” replies Francis. “But, boy, it’s a long way off from being a fully automated process. It still needs human intelligence just as much as artificial intelligence. You still have to squeeze as much detail in texture and all those things, and that takes years of experience and human intelligence to do. It’s going to be years before there are automated processes that can achieve 4K [ultra HD] like this.”
Jones grimaces. “Hopefully we’ll be able to film the back of people’s heads by then.”
—
All episodes of Deep Fake Neighbour Wars are on ITVX from 26 January.