Python and Music: Building a Music Tutor - Lakshya Gupta, Anant Gupta

Again welcome everyone. So so our talk first talk for today for this track is Python and music uh building a music tutor by Lakshia and Anand. So over to you. >> So good morning everyone. My name is Anand. Uh we work at JP Morgan and uh we work uh in the field of uh using AI in finance. But this talk is something different you know and we just wanted to give that uh disclaimer early on and uh welcome everyone uh on the session on piano play. We definitely promise to keep it interesting. Uh so before we start uh how many of you have tried to learn a musical instrument in the past few years? Yeah, a good show of hands which is interesting right? But now, how many of you have continued to this day on that particular instrument? Oh man, that is pathetic, man. I'm [laughter] I didn't expect that bad a response. I'm not prepared for this stats. Yeah. Okay. Yeah. Yeah. So, yeah, but that is the sad reality. I mean uh and we don't have to have studies but there are studies which show that almost 50% of people they quit learning within the first one year and naturally that number is raise rising higher and higher because of a variety of reasons right so let us do one thing as part of today's 20 25 minutes talk right let us try to find a solution to this problem with me guys okay so we'll start with some interesting stuff right so some famous piano songs that we have. I think uh you all know Beethoven, right? I mean he has been made very famous but we see here he's not playing the piano. He's writing something on this particular sheet which we'll come to later. Then we I I think everybody knows this guy and uh obviously it's a personal choice but I find uh his piano notes interesting and for the people in Bollywood right inclined towards it I mean we will not leave anybody you know out uh we have Aishman Kurana he's a blindfolded man playing piano supposedly giving him some higher powers right yeah but now this was all the good part let us get to the brass tracks If we now look at the sheet music of these songs that are played, right? It looks like this. Not exactly enticing, right? Your entire interest veins away if somebody tells you that we need to get into this, we need to study this. Yeah. And so coming back to the reasons why people are leaving, right? We tried to break it down into three major points. The first one is complex theory. Let's face it, right? In today's 10-second real world, okay? If someone wants you to study this entire notation, you're going to have a headache. We have to accept the fact we don't have patience. We don't have the time or energy to go through the music theory, notations, the tempo, etc. of the songs like we do in a music school. We just want to play some songs, man. That's it. That's our agenda. We don't have any grand plans. But if let's say you have to go through all this music theory as a mandatory requirement to start your journey that's a hard stop. Most of the people say that boss this is not something that we can do and we leave it there. So we have to get something you know to replace this part. The second is lack of guidance. On one hand we have thousands oh cancel thousands we have lacks of videos on YouTube which will teach you any instrument and on the other hand we have a tutor who will teach you a 1 hour per week or maybe a 4 hours per month class which will teach you a particular instrument. But both of these are not the sweet spots that we want. We want something in the middle because our problem is scheduling. Scheduling is the bottleneck in our lives. Either we are free at 10 p.m. at night when we have when we are done with our work or we are free at 8:00 a.m. in the morning when we decide not to go to the gym or maybe it's a day off at our kids school, right? We are on holiday for days at end and then when we come back we have taken a 3 days off and we want to spend the entire day learning an instrument. So all these you know odd hours we need some kind of personal guidance to continue our music journey and sadly we'll not get with either of these two options. We need to get better at that. The third is slow progress. Oh yeah we hear a song we want to play it and now what we want to do is we have purchased an instrument. The more optimistic guys have purchased expensive one. The less ones have purchased a cheap one. The cheapest maybe that is available. Okay. We have earmarked the YouTube videos. Okay. Send a note to our WhatsApp. We have also purchased that comfortable chair so that we can practice hours on end. Right? All thoughts and you know fancy castles in the air. But it's been a month. You have not reached anywhere. You have just barely reached five notes. Dreams come crashing. You finally realize boss maybe I'm not made for this. There are some other uh you know DNA that is required for this. And that's it. Goodbye instrument. Goodbye piano. It's tucked away. You can see what happens to the piano. You know, most of it most of you are aware with this particular imagery, right? Yeah. There are people who are talking about singularity and general artificial intelligence, right? And when those words are thrown around, we should definitely do a better job at making this experience a little bit more fruitful learning an instrument. And that is what we want to do. Okay. So let us break down the problem, right? I mean you will say that boss this is all we know. We all aware of it. What is the solution to this? We want to drive the solution as an inference engine would do. Break it down into baby steps. So the first thing is embed software into each and every element of the ecosystem. We want software to be embedded into the music theory part. We want software to be embedded into the instrument part so that all of them become interconnected because as soon as you have software you'll be able to get feedback from it and then you will be able to create a wholesome product around it. The second is music to human action. Imagine turning each and every element of music into a actionable human element. For example, if somebody tells you that you need to press these three keys at t equal to t0 and then these two keys at t equal to t1. Okay, this is a instruction that we can follow very easily. But if you overlap that with additional information, you know, we have to break through that and that becomes a barrier for us. So we need to transform each and every part of music into very simple instructions. The third is [snorts] custom learning base. Today we'll have four hours to spare, right? But tomorrow we might have just wait 15 minutes to spare. There will be some people who will start from the beginning. There'll be some people who are already experts and there'll be some people who will start from negative because they have learned things the wrong way. It is quite possible, right? We have to come up with a solution that will cater to each and every one of our consumers. We have to think of the solution in that fashion. Which means that we have to make sure that our solution is able to come up with a custom feedback for each and every action that is taken. If a person is progressing in a fast manner, he should be able to move faster. If a person is progressing in a slow manner, we should be able to tone it down and give him or her simple instructions to follow. Okay. Uh the fourth is constraint learning. There should not be any spatial constraints. Tilt the phone this way, keep the video that way. None of that matter. You have an instrument and it should be as easy as just plugging in and start playing. So that is the uh I mean the wholesome point is to turn each and every song into a sweet learnable lesson. That is the mantra that we are trying to follow. So now that was the background of the problem and you know the the way we are trying to solution right let us get in the technical details around it. So the first element is MIDI that I would like everyone to know. MIDI is nothing but musical instrument digital interface. As you can already hear right it is nothing but a software representation of the analog waves. Imagine that you have analog waves which is the sound file. Okay. converted into something that is a digital artifact. Now this becomes very powerful because then it can be transferred, it can be worked upon, it can be stored, all of those things can be done. The two key components of this particular module is the MIDI controller. This is the part that actually converts your soundwave to the MIDI file. And the second one is the sound module which converts the MIDI file to the sound waves. So we have kept it very simple. These are the two elements. One is the conversion to MIDI and the one is the conversion from MIDI. I think for the rest of the flow uh Lakshia will be introducing himself and then taking you through. Uh thank you Anan. So hi everyone I am Laksha. Now uh now that we have discussed the what and the why that uh for the pitch that we have in front of you, let's uh get a little technical into the how aspect of it. So there are four main components that we're dealing with over here. The first is the song processing, the lesson creation, the UI part, and the hardware part. We'll uh dive deeper into each section. An can you so first of all the song processing. Now when I was learning the keyboard, when I was learning the piano, right? Uh the motivation does vary. I mean I feel if anyone has taken tutorships under some teacher then there'll be certain time where you know there'll be a little more classical pieces that they'd be uh you know you'd be asked to learn now for example when I was asked to learn green sleeves or the Turkish march by Mozart it felt very boring to me because those are not songs that I hear personally a lot right I want to play let it go I want to play numb I want to play some KK songs or kishokumar songs because those are the songs that I listen to more uh you know on a more daily basis so how do we solve all that. Now imagine an app where you just upload a song whatever app whatever song you are currently hearing you are like okay this is a beautiful song I want to try it out I want to play you upload that song as an MP3 or or a wave file and if you're able to create lessons from the song itself so that is where the song processing part comes in you'll be uploading the song or you'll be recording it uh then we'll do an audio separation now I see that uh we saw that okay there were many people who tried to learn music now anyone someone who you know uh has a little information about you know song produ production of music they'll understand that in a commercial song there are many layers even if it sounds very simple it's not simple at all you'll have the main melody vocals you'll have multiple uh vocal harmony layers over it you'll have multiple uh melody instruments like say strings multiple guitars are there there are keys then there are multiple percussion instruments on top of it becomes a tedious task there's a visualization will show you in which how the layers look even if we just try a direct uh MIDI conversion. So the first step becomes just the audio layer separation and once we have the audio layer separated then we do a MIDI conversion because then it becomes a little more granular to be able to handle and process. Next slide. Now as for the text tag I mean this is uh Pyon this is Python conference I think many people would be interested in how we are leveraging Python for these particular tasks. So uh okay so for the audio separation right now basic pitch is a python library that handles that pretty well. Pretty m is another python library that is very very good for uh media manipulation and for the model that we are using for uh audio separation. Now demox is one model which is Facebook created by Facebook. The other is UVR which is uh universal vocal removal. Now again many people I saw here have tried to play the piano right on the right hand you just try to play the melody part and that melody part is usually just the vocal part of it. So that's why we wanted uh a far cleaner vocal removal as part of this as well and UVR is the state of the art for that. So yeah the second is a listen creation. Now that we have independent midies at our hand now it becomes a little easier for us to handle and to create more customized listens for the user. Now when we say custom made lessons what what do we mean with that? So uh firstly we want different difficulties in that. Now you would have uh okay so firstly you know any piano player would first try to learn just right hand part of it. Then there'll be certain more complications that add through the left hand. Uh I'll probably show you a few few demos and like plays and then you'll see that you know if you just playing a single note that is a little easier with left hand. But you might have seen certain professional pianists who play in such a way that it seems like both of their hands are moving independently as if there's magic in that. Right now we want to have that gradual learning experience for the user. So we have uh different difficulty levels so that the user can adapt at their own pace. Secondly, we want to customize the learning of each independent song as well. How do we do that? Firstly, the user should have the access to uh the ability to firstly select the tempo and even the song segment that they want to practice where you know this particular line I want to practice more because say I have a little more difficulty in this or this is a little more complicated piece compared to some other section of the song. So those kinds of customizations is what we strive to provide. So yeah for this we are using again uh pretty mid and mido for MIDI manipulation and as for uh now and music 21 and librosa uh these are python libraries which have a little more music specific implementations for you'll have chords you'll have tempo time signature uh key analysis those kinds of uh basic music theory components okay perfect now so this is what I mean by there are multiple layers now this is a very The volume is low. Uh is now this is a very simple piano interlude that comes right after the chorus of uh Tumiho. It's a very simple piano interlude. But if you look at the notes, if you just do a direct conversion into MIDI, you'll see that multiple same notes that are being played over different octaves itself. Just to like this this the point of this is just to show you how many layers there are in a single song. We can go to the next one. Oh, okay. Now for the UI visualization. Now, as I said, now that we have the lessons at our hand, now we want to create a little more intuitive uh UI for the user. Here, first of all, as I mentioned, the user should have the capability of of customizing their own lessons through uh tempo, through segment selection. Apart from that, we want to create a gamified uh version of it. Now I imagine many of you may have at least heard of the game uh Beat Saber the VR game in which you hit notes right and then probably Dance Dance Revolution where you have to hit your foot on particular notes. Now imagine that kind of an interface for your piano where you're playing the you have the that sort of a MIDI visualization uh hopefully a little more simpler than that but apart but uh now that is playing and then you're playing the piano as well and you get live feedback on top of it with scores on top of that. Now ideally if if we are able to you know make it more uh how should I put it competitive with your friends where you can compare scores and all uh that kind of a gamification experience we do strive to uh you know aim uh aim for we'll show you how it it's working out right now. So yellow will be the one that you are supposed to play if you do hit it right. you're getting it on green below. Uh the melody extraction is not perfect. So just bear with that. We tried our best till this particular point of time. You get the scores at the top and if you are anywhere off then yeah you get immediate feedback out of it. Now for the hardware part, I'll uh I'll uh hand it over to Anel so that he can explain it. Now signals that we need to we have transformed the original uh sound file into. What we need to do is we now need to transfer it first to the device which is our piano and the second is the feedback from the piano back to our software layer. That is very important to be able to transfer that we first convert the uh the MIDI signals into something that the Arduino can understand. So for that we came up with a handshake uh payload. Next slide. We came up with a handshake payload that will determine the metadata. So for example, I want to have some kind of delay and these are the groups of keys that need to be pressed with these many delays. So this is a simple payload that we have come up with and then the PI serial is used to in integrate with Arduino. The second is that the Arduino also needs to send that signal to the LED strip. So that is a separate protocol and we have used Adafruit NeoPixel to do that conversion. The third is that we have again taking the feedback back from the piano to our software level and that is done through pi serial and middle a simple schematic of the Arduino uh flow that we have. Uh we have the 5V 2 amp power supply. The reason for this is that you know we have too many LEDs and a USB power signal was not sufficient to light it all up. So you'll have to you know amp it up with an additional power source. And then the usual we have the output coming from pin six in Arduino that is connected to the LED strip that is in line with the keys that we have. And then we have the power signal connected to Arduino as well as a power supply. [music] Heat. Heat. [bell] [music] >> [music] [music] >> Yeah. See the idea is that uh we will have a onetoone key mapping of the LEDs and so you will have two observations right one is the LED lighting up and then we will also have the feedback coming in form of the UI. So this UI can then be ported to anything it can be ported to a Android TV device because once you have the visualization in front of you it can be ported to any other layer and the hardware part is something that will be very close to the person. I mean the hardware part is something that if a person is traveling outside he or she might not be able to carry it. But if you you are within your home and you know in a comfortable position you can use the hardware layer.