Movies have always been pictures, but music makes them come to life. The sound of footsteps in a hallway, the low-frequency rumble of thunder far away, and the metallic sound of a blade being pulled in a hall that looks like a cathedral are not just for show. They are parts of stories that help them stay together. The sound effects connect the movie's physical and emotional parts. They make fantasy seem real and give size, movement, and mood a sense of importance. For a long time, generating sound for movies has depended on a strong but time-consuming mix of Foley performance, field recording, chosen sound libraries, and deterministic digital signal processing. These methods have created some of the most famous cinematic soundscapes of all time. But as projects get bigger and deadlines get shorter, the industry is starting to look for a new creative partner: generative music that leverages deep learning. The Craft Comes Before the Code The Craft Comes Before the Code To make good sound for movies, you need to work together, edit carefully, and act out the scenes. Foley artists look at scenes on a screen and make sounds like footsteps, cloth swishes, things hitting each other, and people hitting each other. Every movement is recorded, synced up, and molded to fit the pulse of the music. Field recordists go to genuine places, such towns, forests, and industrial areas, to get the world's unpredictable textures. After that, editors go through huge sound banks and change the pitch, stretch, layer, and shape the sound to fit the mood and tone. Digital signal processing can change the form of the sound, add reverb, filter it, and create granular textures. You have a lot of creative freedom with this process, but it takes a long time and will always be a work in progress. When a director asks for twenty different versions of a mechanical impact or a small alteration in the surroundings, the manual workflow can get stuck. These methods are still useful and not old-fashioned. But they are become more and more useful when combined with machine learning algorithms that can quickly and on a large scale make good modifications. The Growth of Generative Audio The Growth of Generative Audio As natural language processing and computer vision have gotten better, so has generative audio. In the beginning, neural audio systems were mostly employed to generate music or talk. In contrast, modern systems have included sound effects and background sounds that can be used for many things. Representation is a big problem in generative audio. When it comes to movies, sound effects need to be very accurate, have a wide dynamic range, and seem authentic. Raw waveform models showed that neural networks could make sample-level audio, but they had trouble with long-duration cinematic textures because they were too expensive to compute. Spectrogram-based systems made things more efficient by making the data less challenging. They did this by modeling time-frequency representations and keeping perceptual properties. Neural audio codecs have been a game-changing discovery in the previous few years. These systems divide sound into groups of single tokens. This makes it easier for generative models, which are normally based on transformers, to work in token space. This difference between learning how to model information and learning how to represent things helps you make extended sound effects that can be used in a number of different ways. It also works well with text conditioning, which lets models make sounds from words. Models that use diffusion have made quality and stability even better. These systems create sound in compressed latent areas to find a middle ground between how real it sounds and how easy it is to compute. The end effect is sound that is more natural and less mechanical, which is great for movies. Making Sound Happen Making Sound Happen Text conditioning is one of the most interesting new things in generative audio. Sound designers can use everyday language to talk about the soundscape they want, such "distant thunder with heavy low frequencies" or "metallic impact in a huge stone hall." Then they will get made-up versions that match that description. Language-guided generation systems like AudioGen, AudioLDM, and Make-An-Audio show how they can assist people come up with new ideas more quickly. Diffusion-based architectures are great for producing backgrounds that shift, echoing textures, and abstract design aspects that are popular in movies. These models are not a replacement for Foley or recording sessions. Instead, they give designers quick drafts and textures to work with and make their work better. This change makes generative audio a way for people to work together. It turns into a drawing tool that allows you fast explore timbral space before you start making music. Putting it into real-life movie pipelines Putting it into real-life movie pipelines In the world of professional filmmaking, generative systems don't usually work on their own. Instead, they are built into digital audio workstations like Nuendo or Pro Tools, where sounds that were made are modified, layered, put in space, and mixed with sounds that were recorded. The hybrid method gives the sound creator control over their own creation. Sounds that are made are not finished things; they are used as raw materials. Designers change the timing, dynamics, spatialization, and psychoacoustic impact to make sure they fit with the story's goal. Automation makes doing the same thing over and over again easier, but it's the knowledge of people that gives stories their emotional depth and helps them make sense. This synergy is part of a bigger trend in the arts: AI can make new things conceivable, but people decide what they mean. What happens next: evaluation and ethics What happens next: evaluation and ethics When judging movie audio, you can't just look at objective signal metrics. The sound in a movie has to match the picture perfectly, make viewers feel something, and make the environment feel bigger. A waveform that is technically correct but doesn't help the story come to life isn't good enough. Adoption is also becoming more important when it comes to moral and legal issues. Being honest about where the training data came from, who owns it, and who receives credit for it is really important. Studios need to know if the sounds they make are based on the training data or if they are fresh works of art. Multimodal conditioning, which means systems that make music directly from video frames, and conversational refinement interfaces, which let you tune things over and over again through conversation, are two of the most exciting areas for the future. Generative learning and physics based on simulations could work together in hybrid physical-data models. In the end, end-to-end cinematic pipelines could put together production, editing, spatial mixing, and rendering that makes you feel like you're there. A New Age in Movie Sound A New Age in Movie Sound Generative audio technology are changing the way people think about and make sound effects for movies. Neural audio codecs, diffusion modeling, and representation learning have all improved. This means that professional settings can now use high-quality, scalable sound synthesis. But the foundations of sound design for movies are still the same. The sound is based on the story. What happens is based on how you feel. Artists have additional tools to work with thanks to technology, but it doesn't replace them. In the next ten years, developing movies will probably include generative audio as a standard feature. It won't take the role of human ingenuity; instead, it will help sound designers focus more on the story and the minor variations in tone.

From DSP to AI: Evolving Approaches in Audio Processing

Generative Audio and the Evolution of Cinematic Sound Design

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

From DSP to AI: Evolving Approaches in Audio Processing

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

From DSP to AI: Evolving Approaches in Audio Processing

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps