Poor audio quality is a constant challenge for video creators: Hobbyists and professional filmmakers alike need to work around audio issues caused by wind interference, sub-optimal microphone locations, crowd noise and other sound nuisances, any of which can render videos unusable. Although this challenge historically hasn’t been easy to fix, exciting new solutions using AI advancements will revolutionise audio processing — and make high-quality creation easier for everyone.
Today, Adobe unveiled Project Sound Lift, an AI-powered technology that separates speech recordings into distinct tracks of voices, non-speech sounds and other background noise in a video. Project Sound Lift is a one-click solution that helps users effortlessly manipulate audio recordings across a range of scenarios, leveraging AI to independently enhance, transform, and control speech and sound independently. Adobe’s Enhance Speech technology — now available in Adobe applications like Premiere Pro — is integrated within Project Sound Lift to further transform the way creators produce and control studio-quality audio content.
Developed by speech AI researchers at Adobe Research, Project Sound Lift was announced today on stage at MAX in Japan as part of Adobe’s “Sneaks” showcase, where Adobe engineers and research scientists offer sneak peeks at prototype ideas and technologies, each showing future potential to become important elements of Adobe products that are trusted by millions of users across the world.
While prior audio AI models often require clean, distinct input sounds — such as a single speaker or sound event without background noise or echoes — real-world recordings rarely meet these conditions: They can contain noise, reverb, multiple speakers and other sound events that are often impossible to control for. This limitation has hindered audio AI’s application in everyday recordings and made it challenging for non-experts to utilise often complex audio tools.
Project Sound Lift can now separate voices and ambient sounds from daily life scenarios, including splitting speech, applause, laughter, music, and other various noises into distinct tracks. Each track can be individually controlled to enhance the quality and content of the video.
Check out some examples of Project Sound Life in action here.
Project Sound Lift can separate a video’s overlapping sounds and enhance the most important parts. In these examples, Project Sound Lift isolates Adobe CEO Shantanu Narayen’s introduction voice and keynote speech from background music and separates Adobe Creative Cloud Evangelist Paul Trani’s voice from applause—all with just a single click.
Project Sound Lift can separate overlapping speakers simultaneously and isolate them from background noise. This can be incredibly useful when capturing footage at a public event—in this case, on the floor of the busy Adobe MAX conference. By reducing the volume of background speakers while maintaining the ambient sounds from the background, the voice of each on-camera speaker can be heard clearly without losing sense of the environment.
Project Sound Lift can go beyond enhancing audio quality: It can also offer a creative canvas for fun and inspiration. In this example, Project Sound Lift separates the speaker’s audio track from background street noise, and applies a voice modulation technique, transforming a human voice into a whimsical robot-like sound.
This “sneak peek” of Project Sound Lift offers exciting examples of how innovation is unleashing a new era of creativity — where AI is expanding access to powerful new workflows and unleashing our most imaginative ideas. Learn more about other Adobe Sneaks from MAX 2023 here.
