The integration of AI in rotoscoping, particularly for the visual effects in Dune: Part 2, has caught the attention of the film industry. Perfection42, one of the industry leaders in predictive AI for visual applications, says visual storytelling is transforming itself because of AI helping filmmakers to cut their production time, and old techniques, such as rotoscoping in movies, are experiencing a second life.
“We’re experiencing a rebirth of rotoscoping. The smart use of AI for the blue eyes effect in Dune: Part 2 demonstrates how technology is streamlining once labor-intensive tasks. This is a clear indicator of the film industry’s move towards more creative and efficient processes,” said Simona Vasyte, CEO of Perfection42.
The production team behind Dune: Part 2 provided insights into their approach: “We came up with a different technique, using what we’d learned before from the hundreds of blue eye shots in the first movie and creating a machine learning model, an algorithm trained from those ‘Dune’ shots to find human eyes in an image, which would then give us a matte for the different parts of the eye. We then used this multi-part matte to tint the eyes blue.”
AI-driven rotoscoping extends its utility beyond changing eye colors to facilitating the creation of complex animations, erasing unwanted elements, integrating CGI with live action seamlessly, and performing intricate color grading. These uses exemplify AI’s transformative potential in elevating visual effects and narrative depth.
“The variety of tasks now achievable through AI, from enhancing animations to modifying backgrounds, also reduces the workload. We have seen cases where the time required for some tasks was reduced even further, from several months to just a few weeks,” added Vasyte. “We predict that AI will reshape the landscape of filmmaking and more filmmakers will follow the example of Dune, allowing creators to explore and execute their visions with unprecedented freedom. We will see a boom of movies that use AI to enhance their footage, even this year.”
For instance, by using the style transfer technique (applying the visual style of one image or video onto another) with rotoscoping filmmakers can create stunning visual effects, such as enhancing the appearance of characters with supernatural abilities or adding otherworldly elements like magical spells. By tracing live-action footage of stunt performers or actors, filmmakers can create dynamic and realistic action sequences that captivate audiences.
As the film industry continues to explore the possibilities of AI, the dialogue around Dune: Part 2’s achievements underscores the evolving relationship between technology and creative expression. With the growing use of AI, people fear that hyper-realistic movies will be created solely by AI, but experts highlight that the main goal is far from that.
“Our aim is not to achieve hyperrealism but to enhance and expedite the creative process. This technology offers tools that expand what’s possible, opening up new avenues for creativity rather than merely replicating the real world. It’s about empowering creators to realize their artistic visions more efficiently and with greater flexibility,” Vasytė concluded.
