Key Points
- NVIDIA’s new Fugatto model is a generative AI tool designed to create and modify audio using text prompts.
- The model generates audio across various languages and accents. Fugatto is tailored for music production, language learning tools, and dynamic video game audio.
- The model can perform tasks not part of its pre-training, such as combining specific accents with emotional tones or generating evolving soundscapes.
- Fugatto joins Meta’s sound generation AI and Google’s MusicLM in the growing generative audio landscape.
NVIDIA has launched an experimental generative AI model, the Foundational Generative Audio Transformer Opus 1, or Fugatto, which it likens to a “Swiss Army knife for sound.” Fugatto leverages text prompts to generate or modify audio, including music, voice, and other sound files. Developed by an international team of AI researchers, the model boasts robust multi-accent and multilingual capabilities, offering unprecedented versatility in audio creation.
Rafael Valle, an applied audio research manager at NVIDIA and one of Fugatto’s developers, shared the vision behind the project: “We wanted to create a model that understands and generates sound like humans do.” Fugatto’s applications span industries such as music production, language learning, and video game development.
Music producers, for example, can utilize Fugatto to quickly prototype song ideas, experiment with various styles, and adjust elements like instruments and vocals. Similarly, language learning tools could benefit from the model’s ability to generate instructional audio in user-selected voices, while game developers can create dynamic soundscapes that evolve with player choices.
One standout feature of Fugatto is its capacity for emergent behaviors, enabling it to perform tasks beyond its initial training. With minor fine-tuning, the model can combine separate instructions, such as generating speech with specific emotional tones and accents or simulating complex environmental sounds like birds singing during thunderstorms. Fugatto can also create dynamic audio that evolves over time, such as the changing intensity of a rainstorm moving across a landscape.
While NVIDIA has not announced plans to release Fugatto to the public, its debut underscores the growing capabilities of generative AI in audio applications. It joins similar tools, such as Meta’s open-source AI kit for sound generation and Google’s MusicLM, which users can access through its AI Test Kitchen website.
NVIDIA’s Fugatto sets a new benchmark in audio AI and offers a glimpse into how future tools may redefine creativity and interaction in various domains.
