AudioGPT is a tool for processing audio with the help of large language models (LLM).

AudioGPT uses ChatGPT for task analysis when receiving a user request, selects a model according to the functional description available in the speech base model, executes the user instruction with the selected speech base model, and summarizes the response based on the execution result. With ChatGPT’s powerful language capabilities and numerous basic speech models, AudioGPT can complete almost all tasks in the field of speech.

Specifically, the AudioGPT running process can be divided into 4 stages: modality transformation, task analysis, model assignment and reply generation.

AudioGPT core functions

• generate music
• background sound
• Generate subtitles from audio
• text to audio
• Text generates audio and simulates sound
• Generate audio from pictures
• Inpaint the audio (partial masking)
• Synthesize video based on audio and face photos
• Detect events in audio, along with start and end times
• Mono to Dual
• Detect when a specific sound occurs with a textual description
• extract a sound
• remove background noise
• Multi-person mixed voice separates single voice
• voice translation