NEWSLETTER
The new Text2Speech model, Bark, has just been introduced and has restrictions on voice cloning and allows prompts to ensure user safety. However, the scientists decoded the audio samples, released the constraint instructions, and made them available in an accessible Jupyter notebook. Now, using just 5-10 seconds of audio/text samples, it is possible to clone an entire audio file.
What is the crust?
Suno’s innovative Bark text-to-audio model is based on GPT-style models and can produce natural-sounding speech in multiple languages, plus music, noise, and basic sound effects. Suno developed the Bark text-to-audio paradigm using a transformer. In addition to making natural-sounding speech in multiple languages, Bark can also create music, ambient noise, and basic sound effects. The model can also generate facial expressions, such as smiling, frowning, and sobbing.
Bark uses GPT-style models to create voice with minimal adjustment, resulting in voices with a wide range of expression and emotion that accurately reflect the subtleties of pitch, pitch, and rhythm. It’s an amazing experience that makes you question if you’re talking to real people or not. Bark has impressively clear and precise speech generation capabilities in multiple languages, including Mandarin, French, Italian, and Spanish.
How does it work?
Bark employs GPT-style models to produce audio from the ground up, just like Vall-E and other amazing work in the area. Unlike Vall-E, high-level semantic tokens incorporate the first text message instead of phonemes. Thus, it can be generalized to non-speech sounds, such as music lyrics or sound effects in the training data, in addition to speech. The entire waveform is then created by converting the semantic tokens to audio codec tokens using a second model.
Characteristics
- Bark has built-in support for multiple languages and can automatically detect the user’s input language. While English currently has the highest quality, other languages will improve on only one scale. Therefore, Bark will use the natural accent for the corresponding languages when presented with code-switched text.
- Bark is capable of producing any form of sound imaginable, including music. There is no fundamental distinction between speech and music in Bark’s mind. However, on occasion, Bark will create music based on words.
- Bark can replicate all the nuances of a human voice, including timbre, pitch, inflection, and prosody. The model also works for saving ambient sounds, music, and other inputs. Due to Bark’s automated language recognition, you can use a German history indicator with English content, for example. As a result, the resulting audio usually has a German accent.
- Users can specify the voice of a certain character by providing indications such as NARRATOR, MALE, FEMALE, etc. These instructions are only followed sometimes, especially if another audio history address is provided that conflicts with the first one.
Performance
Bark CPU and GPU implementations (pytorch 2.0+, CUDA 11.7 and CUDA 12.0) have been validated. Bark can produce near real-time audio on today’s GPUs using PyTorch every night. Bark demands to run transformer models with over a hundred million parameters. Inference times can be 10-100 times slower on older GPUs, default collaboration, or a CPU
review the repo and Blog. Don’t forget to join our 20k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at asif@marktechpost.com
Check out 100 AI tools at AI Tools Club
Dhanshree Shenwai is a Computer Engineer and has good experience in FinTech companies covering Finance, Cards & Payments and Banking domain with strong interest in AI applications. She is enthusiastic about exploring new technologies and advancements in today’s changing world, making everyone’s life easier.
