How I used an AI tool to play violin with my guitar

Written by Sunny K. Tuladhar , Software Engineer (AI)

A quote from the kits.ai website reads,

Imagine playing a note on your guitar and having it instantly translated into a deep, resonating cello.

For musicians, these AI-based instrument “converters” are all the hype right now.

I didn't actually use kits.ai though, because… well, it's not free. This entire blog is based on me using a single AI-based tool Tone Transfer by Google— which, by the way, is totally free.

But is it worth it? Let’s figure it out!

AI tools for music

AI tools enable you to change the tone and sound of your instrument. So if you can play “Twinkle Twinkle Little Star” on your guitar, and are wondering how it would sound on a piano, well look no further.?

Basically, the notes stay the same while the sound changes to another instrument.

Amazing, right?

Do we really need AI music tools?

I had a Casio keyboard as a kid which cost around Rs 3,500 ($26 approx) back then, circa 2005. It boasted 100 instruments— including drums, violin, guitar, and more. So, why would I ever need to learn these actual instruments if I could play them all with my keyboard?

Well, because the sound was terrible.?

Even the piano, supposedly the main instrument in the unit, did not sound like a piano. The guitar sounded like something I had never heard of. It was all subjectively bad!

In digital keyboards, the voices are being produced in two ways. One is sampling, where a recorded sound is replayed and the other, synthesizing where the user can tweak a signal to get the sound they?want.?

When thinking of sampling, think of an expensive digital piano that sounds like an actual grand piano. When it comes to synthesizers, think electronic, futuristic sounds. Think Daftpunk.

These are the “normal” ways.

Enter AI: Recreating sounds the cool way

Fast forward to 2021 when I tried a cool new method to create music, the AI way.?

To recreate the sounds of a particular instrument, an AI model is trained on thousands of sample audios of that instrument. When it learns from these, it also learns the little nuances and variations of the instrument since the audio it is trained on is from actual humans playing.?

So AI creations feel more realistic. At least much more real than regular keyboards. And these definitely do not sound “bad" — well, most of the time.

Sometimes the AI struggles with precise timing and dynamics, resulting in peculiar or unusual sounds as it tries to accurately replicate the variations. So, AI-generated music can sound incredibly authentic but also a little weird at times.

Another thing about AI music tools is that they will take in any sound as input. It does not have to be another musical instrument. There is no such thing as an invalid input in an AI model as long as it is a legit audio file.?

So the AI will also convert the sound of pots and pans clanging into the violin. The sound of a beatbox to guitar? Sure! It will sound weird, but who gets to decide what is weird? Anything can be used artistically.

So what is going on underneath?

Tone Transfer falls under many musical projects Google has been running under Magenta.

Magenta is “An open-source research project exploring the role of machine learning as a tool in the creative process.” Magenta’s Tone Transfer is built upon Differentiable Digital Signal Processing (DDSP) and Self-Supervised Pitch Estimation (SPICE). Here's the paper on it and the GitHub link.

The key concept here is to carefully manage the various characteristics of simpler signals to produce complicated realistic signals.?

For instance, if the frequencies and responses are precisely calibrated, a group of linear filters and sinusoidal oscillators which are DSP components (let’s call them simple sound stuff) can produce a violin-like sound.?

However, it is challenging to manually adjust this dynamically, which is why synthesizers with straightforward controls frequently sound artificial and "synthetic."

The problem with the DSP (without the extra D) is they aren’t differential, hence backpropagation is not possible. That's where the Differential DSPs come in. These can be used to train a neural network such that it can learn on its own how to recreate sounds.

In a nutshell, music could be split into three things:

• the volume
• the notes (pitch) being played
• the timbre (voice) of the instrument

The AI model takes the notes and volume of what we played and outputs it in a different timbre (instrument).

How does Tone Transfer work?

1. Capturing the audio: To train the model, it needs a 10-minute audio recording of a sound but has to be monophonic (only one note playing at once).
2. Training the machine learning model: At the start of training, the sounds the model creates are just “bad” sounds. However, after a couple of hours into the training, the model quickly learns to produce audio that sounds like the real instrument.
3. Fully trained model with digital signal processors: After training, the ML model learns how to control a collection of DSPs (simple sound stuff) that each makes up a portion of the final sound. These include the pitched components (notes), non-pitched components (like bowing noises, slapping sounds, breath noises, string scrapes, etc.), and reverberations of the room the original instrument data was recorded in.
4. Synthesize audio using pitch and volume: The ML model is now able to synthesize audio that sounds like the original violin given any new combination of pitch and volume. This includes pitches that were not heard in the original training data.

Tone Transfer has a plugin that works as a VST in Ableton. Since I don’t use this DAW, I haven't gotten around to using it.

I tried it myself as you can see in the video below. I did not train the model, I just used the sounds already available by default in the Tone Transfer UI. Below are the steps I followed for this.

• Recorded my raw guitar in FL Studio with no effects and played only single notes since the model is monophonic.
• Chopped the recorded audio into 15-second segments and sent it through Tone Transfer.
• The model recreated the same notes but with different instruments.
• Downloaded all of these sounds and mixed them together to create the following solo.

What do musicians think of this?

Considering everything mentioned above, what do real musicians think about this? I asked the three Sunnys in the Nepali music scene. Do they represent the entire music scene in Nepal? Probably not, but I already asked them so here’s what they think.

Will AI tools replace the more uncommon instruments and will everyone just learn the keyboard?

Sunny Tuladhar (ST) (not me) (from Jindabaad)

Dimaag kharaab. No, it can’t be replaced. I think it is to get the job done but it's very cool at the moment. But it depends upon how realistic it could sound in the future. It also reduces the hassle of hiring people for simple recordings of different instruments.

Sunny Manandhar (Suncha Vox from Albatross)

AI will not be taking over the world soon, so we don't need to worry too much about it.

Sunny Mahat (from The Midnight Riders)

I think the world's become really fast and expensive..so if AI gives people access to music instruments which they'd previously never had time or money to get, it's a boon. And I would also add, the audience of the new generation cannot tell the difference between AI or the 'human element'. See the number of popular singers who can't actually sing without digital support. And even historically, a lot of instruments like the bass, and drums, have been played through keyboards or programmed. I think as long as music is made for TikTok, the audience won't be able to tell the difference.

What do I think?

It's not replacing humans anytime soon but it might get close. Regardless of the apocalyptic AI ideas, it's really cool. Since it's already here, we can all give it a shot. See how it fits with your music.

About the Author

Sunny K Tuladhar is an AI engineer at Leapfrog and a passionate musician. Read more about his work on Medium.