I’m standing near a gas station in the American countryside. Small triangular flags, hung from the ceiling, flutter. The alphabet “E” in the glowing sign “Golden” flickers on the rooftop. The tube lights below it turn on and off. Across the highway, tall cacti dot the landscape. A mountain lounges afar, soaking the magic hour. It is serene, inviting, beautiful—and eternal.

Because this magic hour will remain forever. So will anything (and everything) else. I can shrink the mountain, brighten the sun, whip the wind. I can make the cacti eucalyptus, the golden sky blue, and the triangular flags rectangular. My fingers can tame time and modify nature. How? Two words: virtual production. Because I’m neither in America nor at a gas station. Heck, I’m not even outdoors. I’m in Annapurna Studios, Hyderabad, marvelling at a curved 60-by-20-feet LED wall.

Even though this technology hasn’t dominated Indian cinema yet, it’s become a potent force in Hollywood, marking such films and web series as The Mandalorian (2019), Avengers: Endgame (2019), The Fabelmans (2022), and many others. A study by Grand View Research, excerpted in Bloomberg in March 2023, estimates the global virtual production market swelling from $2.1 billion in 2023 to $6.78 billion by 2030. Grand View Research’s October 2023 report stated that India’s virtual production market will grow at 22%, reaching $507.1 million by 2030.

COVID-19 accelerated that demand as this technology, capturing any setting on an LED wall, circumvents the hassles of shooting on locations. It’s evident in the virtual production studios popping up in the country in the last few years, such as Chennai-based Stage Unreal and DB Productions. In Mumbai, K Sera Sera’s virtual production portfolio includes such Bollywood dramas as Judaa Hoke Bhi (2022), Bholaa (2023), and Ganapath (2023). Annapurna Studios and Qube Cinema, a firm providing digital cinema technology and solutions, partnered to open a virtual production studio in June 2023, where a Telugu feature film, Miss Shetty Mr Polishetty (2023), and some ads, including one for Rado starring Katrina Kaif, have been shot.

But is the essence of virtual production—blending pre-filmed footage in the background with the physical set in the foreground—new to cinema? No. It stretches as far back as the 1930s. By projecting a desired backdrop on a screen via a projector behind it, the rear projection technique enabled directors to shoot scenes featuring cars, flights, and monsters, resolving the roadblocks of recording dialogues in noisy settings and executing risky sequences. Remember the iconic shot from North by Northwest (1959), where Cary Grant ducks on the road to evade a crop duster plane? Rear projection.

But it also produced a synthetic-looking background that contradicted the foreground. Then came front projection where a projector, in front of the background screen and performers, projected footage on a two-way mirror, tilted at 45°, which reflected it to the screen behind the actor(s), producing a much sharper and saturated image. It was followed by chroma keying, whose popularity exploded in the ‘80s with affordable computer graphics. As the technology took rapid leaps, enabling realistic and complex visuals, chroma keying became the dominant visual-effects technique. Unlike the front or rear projections, it required actors to perform against a green (or a blue) screen which, through Computer-Generated Imagery (CGI), transformed into an appropriate backdrop—say, flying weapons in a thunderstruck-sky in a superhero spectacle.

In virtual production, though, the actors don’t need to imagine the background, as they’re literally in front of it. It’s often said that “acting is reacting”—a quote so popular it’s attributed to multiple sources—which reveals an immediate advantage of virtual production over the green screen. Virtual production’s photorealistic background helps cinematographers, too, in framing and lighting because, like actors, they don’t have to imagine anything. The LED walls also emit realistic colours and reflections, mitigating the ‘green screen spill’ (the green light reflecting on the actors that must be removed in post-production to make the scene look natural).

By matching the pre-recorded footage on the LED walls with the real-world foreground, virtual production completes the illusion of a continuous image. It can slash travel costs, logistical issues, and unexpected delays. It can also nullify the inconveniences of filming a star in remote or crowded locations or in settings where permissions are difficult to procure. So for a song shot in virtual production where Shah Rukh Khan extends his arms in the Swiss Alps, the makers will just have to ensure that everything captured by the camera outside the LED screen—or on the physical stage—must be covered in snow as well. Besides, the readymade backgrounds—edited and replaced with ease—enable astounding globe-trotting possibilities. 8 am, Kashmir; 4 pm, Eiffel Tower; 10 pm, Antarctica. Lockdown—what lockdown?

In chroma keying, most visual effects are done in post-production, where the final scene bears little resemblance to the one being shot. But in virtual production, where most visual effects are captured while filming, the post-production work largely shifts to the pre-production stage. As a result, it compels filmmakers to prepare and plan for weeks, if not months, in advance. Let’s consider a director who, wanting to shoot a four-minute scene in virtual production, has prepared a script and a storyboard. The next step? Previs (or pre-visualisation): an animated rendition of the scene capturing the setting, actors, and actions. “When you’re doing previs you suddenly think of shots that you had otherwise not conceived—if you came directly to the studio,” says Jayendra Panchapakesan, Qube Cinema’s co-founder. “So I can say, ‘I want to do this crazy camera movement, is it possible?’ And it will be possible. But how to make it happen?”

The answer: techvis, or tech visualisation, the animated version of the same scene incorporating varied technical details, such as the lens type, the camera movement, and the dimensions of the real and the virtual set. The definition of a camera lens in a virtual production scene, much like in a regular movie, largely remains the same, adds Panchapakesan—for instance, a tight close-up for a dramatic scene, a wide shot for a vast terrain. But virtual production also imposes a crucial restriction. “Since the wall’s length is fixed,” he says, “depending on your lens, how much of the wall you see will change. Let’s say, I place the actor 100 feet in front [of the screen], and I’m using a 24 [mm] lens—my lens will see beyond the wall. So I need to know how much of the physical world I can see.” Sometimes the director may want more of the physical world in a scene, which needs to be scanned and put into the virtual world. “So the techviz is essentially placing the camera, deciding the lens, seeing the world, and fixing the wall. All of it will finally tell me which elements will stay in the real world and which in the virtual.”

Virtual production also changes the composition of the film crew. A cinematographer who has only shot on real location will most likely not know the intricacies of this technology. That’s where a virtual cinematographer comes in—a nodal point between the real and the virtual world. Jagadeesh Bommisetti at Annapurna Studios is one such professional who, before working on virtual production, had shot a few Telugu films. “As a cinematographer,” he says, “you need an additional step here: matching the background digital lighting and the foreground physical lighting.” While building “digital assets”—all the elements on the LED wall, including, if any, meta-humans—the cinematographer can inform the “VAD [Virtual and Art Department]” on how he wants to “light the scene”, which does “not happen in regular shoots”.

A director used to working with a green screen, remembers Annapurna Studio’s Chief Technical Officer, C V Rao, was floored by virtual production’s possibilities: “Sir, it feels like I’m flying. I can see my final output. I can adjust my lighting. This is really crazy.” But the “main problem,” he adds, is that “many directors whom we meet—and I’m not blaming them; that’s how we’ve worked—don’t prepare well before the shoot, and they’re not ready to prepare.” Besides, the LED screen can sometimes interact with another device, the movie camera, in unpleasant ways. So, if a cinematographer focuses her camera directly on the LED wall, it produces an interference pattern—comprising repetitive lines, dots, and colour—that ruins the final image. A problem so pervasive it even has a name: the Moiré pattern. Want to see it live? Take a phone camera and record something on TV. “It’s a virtually insurmountable problem,” says Qube’s co-founder, Senthil Kumar, “that will never go away.” It does have a simple solution though, he adds, “the camera just needs to be slightly off-focus.” But it also compromises the depth of field, which “cannot be infinite, only slight, so that the foreground is more focused and the background is slightly off.” Which means it’s difficult to capture a vast landscape in deep focus—unlike a scene shot on location.

The Moiré pattern can also appear when the camera is too close to the LED wall. What determines the ideal distance? The pixel pitch—the distance between the two pixels on the LED panels. The lower its value, the higher the screen resolution, and the closer the camera can be to the screen. “Sometimes clients ask, ‘No, sir, we want to keep the subject a little closer,’” says Rao, ‘to get the proper scaling. If it is far from the wall, my perspective is not correct.’ But because of the Moiré pattern you need to keep your subject at least 10 or 15 feet away from the wall.”

Virtual production also runs into another problem: this time, natural. The walls excel at emitting soft, diffused light, but struggle to replicate the direct sunlight and its attendant shadows and contrasts. How did the skies, then, look like in the first two seasons of The Mandalorian? Overcast. Or, as its director, Jon Faverou, put it: “Hard daylight is best done in hard daylight.” And even though virtual production can capture sunset, there’s nuance within nuance. The inverse square law says that the intensity from a light source is inversely proportional to the square of the distance from it. But unlike the (natural) sun, the LED wall is not an infinite distance away. “So the [light] fall-off from the LED screen will be different than it’d be if the sunset were, you know, how many million miles away the sun is,” said Academy Award-winning cinematographer Roger Deakins to Greig Fraser, who shot The Mandalorian, on a podcast.

The LED wall is not infinitely long either. “Let’s say you want an extra-wide shot of a castle, something like Baahubali,” says Panchapakesan. “We obviously can’t do that. We’ve to do a green screen of that on the wall itself and then do a set extension in post [production].” Deakins highlights another limitation. “If you’re shooting a film totally on stage with sets, you still can’t have a guy run 100 yards. There’s no way of tracking behind them.” To which Panchapakesan has at least a partial, if not the whole, solution: “Use a treadmill.”

***

Across cultures and countries and decades, we’ve watched movies carrying an implicit assumption of an intricate relationship between the real world, the filmmaking crew, and the audiences. If the makers saw an awe-inducing visual while shooting then, we assumed, it awed them first. Or they designed elaborate sets to create meanings and elicit feelings—a real, textured, tactile world.

When Madhubala lip-syncs Chhupna sakega ishq humaara/ Chaaron taraf hai unka nazaara in Mughal-E-Azam (1960), and the camera cuts to her defiant dance form split into a dozen square mirrors on the ceiling, isn’t her rebellion heightened by the precise, lived-in set design? “I’ll give you another example: the song Thare Rahiyo from Pakeezah,” says art critic and academic Karthik Kalyanaraman who curated the world’s first AI art exhibition in Delhi in 2018. “At one point, she goes to a pond, and the moon is shining on it—it’s a set—and there’s a split moment where she turns, and there’s just a flicker of disgust on her face. If that would have been in virtual [production], it’d have completely destroyed it.” Or take Shah Rukh Khan, doused in manic energy, dancing against the backdrop of majestic mountains in Satrangi Re, as the camera swirls around him. An actor in front of nature, an actor subsumed by nature, unfurling his dual passion—real yet mystical, tangible yet invisible—an example of what we risk to lose, the visceral vitality of shooting on location, if virtual production vanquishes the very thought of makers and actors being humbled by the real world?

“I don’t agree in the slightest that AI will reduce the charm of filmmaking,” says filmmaker Anand Gandhi who has increasingly relied on virtual production for his next ambitious project. “We could just go back and make cave paintings, then, because that’s extremely charming. Human aspiration has been to constantly replicate and record reality and simulate potential futures and allegories and metaphors to a high degree of realism and fidelity so that they become immersive, true, and believable.” It’s why, he adds, viewers prefer a medium of “higher fidelity”—a sharper technology—“so we go from VHS to DVD to BluRay. You try showing VHS to people today.” AI hasn’t just changed the possibilities of moviemaking but, given its humongous scope and merits, also encouraged directors to redefine it in the manner that feels most natural, most true, to them. Moving beyond the realms of tech, it’s acquired an ideological and philosophical tenor. “Art is not just about recording our environment or simulating futures or dreams or nightmares in high fidelity,” says Gandhi. “It’s also about capturing the invisible and simulating the translucent—unclear ideas simmering beneath the zeitgeist. And art doesn’t disappear just because fidelity increases. In fact, it challenges the artist who was busy getting lost in creating the legwork of fidelity.”

“Imagine this,” says Panchapakesan, “if in [a movie on the scale of] Lawrence of Arabia, I only do the master and the extra-wide shots on location and come back. So instead of 10 days, I spend one day [on location], reduce my production costs, and do the rest here. Like, you use jimmy jib”—an affordable and versatile camera crane—“for a particular scene, likewise, you pick and choose which part of the film you want to shoot in virtual production. People won’t shoot their entire films on it.” The technology, however, says Kalyanaraman, “might become so cheap and effective”—say, “one-tenth of its current cost in the next two years”—that it may lure more directors to shoot a large portion of their movies on virtual production. “Two years ago, nobody could have predicted AI would reach where it is today. Not even the AI experts.”

And this could reshape, if not irreversibly alter, directors’ attitude towards the medium itself. “I’m worried that those filmmakers”—especially the young, independent ones—“who could have explored real space and locations will be tempted to go this route,” says Kalyanaraman. Some big budget films have “already gone this route”, but if the indies also start to emerge (primarily) from the sea of pixels then, he fears, “this increasingly unreal world” will lose another “sense of reality”. Just consider the world we inhabit: we Zomato our food, Dunzo our groceries, Netflix our films. Even before the COVID-induced lockdown, we had been living in digital prisons for long. Tech overtaking remnants of real-life in cinema could elongate our digital incarcerations—or make them permanent.

“Virtual production would first affect [Indian cinema] bit by bit and then it’d grow very fast,” says production designer Aradhana Seth, who has worked on such movies as The Sky is Pink (2018), Angry Indian Goddesses (2015), and Don (2006). “Earlier only the big budget films were using it but now even the smaller ones can use it once the tech is in place.” The major action sequences in the Netflix series Trial by Fire (2023)—recreating the 1997 Uphaar cinema tragedy—were first conceptualised on a green screen. “But as the idea kept evolving,” says its production designer, Angelica Monica Bhowmick, “and as they wanted the fluidity and flexibility, and since we had so many actors on set, it kept leaning more and more towards real locations.” Virtual production demands precise planning at the pre-production stage, she adds, “which reduces the level of flexibility. It’s something that can work in your favour, or not, depending on your working environment.” Admiring the way tech has evolved, Seth admits that it also encourages a “standardised way of thinking”—a sameness of process—“so it’s got both good and bad”.

“What’s essential to cinema?” asks actor-director Rajat Kapoor. “It’s the sum total of all the accidents that happen in front of the camera, such as sunlight, clouds, rain, snowfall, or something that happened three kilometres away from the camera, but you can still see it.” It’s those accidents, he says, that make the shot. Such an image—“which you can’t manufacture”—“happens to you” while “shooting on a given location” on a particular day. “It’s not about filming Paris at 10 am and Trivandrum at 10 pm. It holds no excitement for me. I’d rather go to Paris and spend a week there, looking at the light before I take my camera out.”

***

By reproducing a desired setting—which, earlier, either needed to be visited or created—virtual production also prompts a crucial concern. “People who build the sets will obviously be used less,” says Kumar. In the long run, he adds, there will be “less and less jobs for any [filmmaking] department” because of AI. “So it’s a much bigger debate about how we compensate people. Maybe a universal basic income is the only solution, so that people can have the comfort of a base to work on things they love or excel in areas where they make more money.”

After a virtual production shoot at Annapurna Studios, an assistant set director, remembers Rao, asked him, “What is our future, sir? We did small work in the foreground, and you extended it on the wall. We won’t have any work.” Rao disagrees, saying virtual production’s use will only be “minimal”, restricted to a few scenes. “Has VFX replaced 100% of filmmaking? It has not.” He does agree, though, that there will be “some deviation” in the future. But once the genie is out of the bottle—broadcasting the technology’s benefits— it’ll serve not one but many masters. A process already underway with an entire Bollywood movie, Judaa Hoke Bhi—made by a mainstream director, Vikram Bhatt, for a big production house—made on virtual production. Nothing rings as loud in the Indian film industries as a successful trend—or actors’ diktats. So if a few star vehicles, featuring extensive use of virtual production, can hit the box-office jackpot, then this technological tsunami can drown out conventional filmmaking.

“Production designers will also evolve,” says Bhowmick, “something that’s already begun to happen. Basically, we are designing. So we’ll continue to design whether it’s for a physical or a virtual set, although with slightly different skill sets.” Virtual production, however, will certainly have a much bigger impact on the lower echelon workers, she adds, “the construction supervisors, the carpenters, the painters”. Seth echoes that point, saying the “[production] dadas”—the below-the-line workers—“have already dwindled”. The Indian filmmaking economics, says Bhowmick, also “works in such a way that it’s still cheaper to hire those professionals as compared to churning out productions on computers.” So in this country “both will co-exist”—the traditional and the modern—“like everything else, like you’ve a bail-gaadi [bullock cart] and a Chandrayaan”.

Bhowmick isn’t antagonistic to virtual production, much like Seth (and Kalyanaraman who, at best, remains “ambiguous” about it), but she does cite a few examples that add more nuance to this debate. “In Oye Lucky! Lucky Oye!, Paresh Rawal plays three different characters”—symbolising the hero’s father figures—“and he had said in a behind the scenes interview that, ‘When I’d turn up for the shoot, all the three sets were so different that I used to feel like a different person.’ Now how would an actor do that if you put him on a green screen?” These elements add an extra “rasa [flavour],” she adds, complemented by an additional factor, which comes with its own “incalculable variety” and “indiscipline” and “madness”: our country itself.

While shooting Aadhaar (2019) in Jamua, a village in Jharkhand, she encountered “many potent and nice surprises” that deserved to be a part of the film. “So we’d see one guy selling jhaalmudi [spicy puffed rice], and it’d be like everyone is doing their stuff and there’s so much chaos, but this guy is just busy making his jhaalmudi. And we thought, you know, he should be in the background. So how do you plan these things?”

During the election campaign sequence in Newton (2017), she says, the team “pulled so many things” from the actual settings, “the special gaadis they campaigned in, for example, because our locations are so rich.” A man in a Chhattisgarh village “roamed around in a weird cowboy outfit”—lost in “his own trip”—leaving her amused. “So we put him among the campaigning bunch. It’s these little things that make our films so memorable. Kaise, matlab [I mean...] How can you capture and encapsulate the whole of India into pre-production?”

(This appeared in the print as 'Digitial Dreamscapes')