Building an Audio-Reactive 3D Brain Visualizer with AI

Guess everything we do is available on this website.com. Go check it out. Provide your feedback. Yeah, we had that brain. If you go on the side, we had that 3D brain which we want to turn into a musical. So, we had the TMS doesn't fit the screen and you can't rotate it for whatever reason. That's a real well hyper realistic model of the brain. It's actually made of uh many different parts. I was thinking we could be there should be a way of actually grabbing it. Yeah, this TMS transcranial magnetic stimulation simulation stimulation simulation. Yeah, just when you do the mouse a business, it's just moving the coil, the TMS coil, it's not moving the brain. Yeah, we had another version. Yeah. Oh, cuz Right. So, we have a seizure zone. So, this this is using the same uh brain models. It's actually telling you which parts of the brain uh yeah. So that's how it's loading. There's multiple parts to it. I don't think it's uh loading all of them actually to make it a bit quicker to make it a bit more responsive. And in this case you can select your essentially symptoms auditory hallucinations and head turning or muscle stiffening and arm jerks and it will show you which part of the brain is likely to be affected. So this model, we want to use this model again. Hopefully we don't have multiple copies of it everywhere. Hopefully we just developable in one place and then we could be using it multiple times as well and want to turn it into like a music generation. So we're talking to perplexity about the syncupation which is not just changing of rhythm right so essentially one of the things we're considering is taking this model turning it into um a music uh generation because well guess what we also have an EG brain waves into music. Yeah. So if you have brain waves into music then you have this uh 3D brain model. We wanted to map music onto it. Probably have two options, one scientific and one artistic. And you can probably imagine what the difference would be. Uh yeah, so we have the 3D model. We have the EG2 music that you can go check on my website, scroll through it. There's no play button at the moment. There should be a play button. So we're kind of going to be combining them all somehow. A music generator. I've So we we also had a synthetic music generator. Yes. So we're doing neurology, cardio, biomedical imaging, vision, hearing, EMG. So muscle activity, mental health. Mental health should be probably gone. We're not really doing it. Well, everything is about mental health, including the stream. Then we have simulations and modeling. Yeah, that's the main thing. So have some games, data analysis, AI, obviously, education, signal processing, blah blah. [snorts] So music. Yeah, we have a bunch of Yes, we have, for example, this uh music generator. Does have a start button. Yes, it does. Uh, this is how it sounds like. [music] [music] >> [music] >> Yeah, it's very basic. Pretty basic. But what we were thinking is in the same way [music] we turn brain waves, pre-recorded brain waves actually from humans in this case into music. In the same way we could be turning music into brain waves or activation of the brain. Activation of the [music] brain. [music] So essentially we'll have music playing and the brain will get activated and there will be a toggle somewhere between artistic and scientific mode. Now the question is uh where would it be getting the music from? Is it from uh generator the music generator or microphone? microphone will be easier and we have to start somewhere. So we have obviously we have code for all of it. That makes things a bit easier. And then we can change the tempo or note duration over there and see what difference it makes uh when looking at the brain. [music] Uh that's the general idea. Oh, for each each one of those you have the randomness. Yeah. So this tempo when set to 100% randomness uh the tempo should be varying. It's not varying at the moment. It's not varying at the moment. [music] Should we fix start fixing [music] this or move to the brain? The brain is more exciting. This brain model has about 100 parts to it. Parts of the brain. So we could be activating each one like one at a time. Two at a time over there. Yeah, this is [music] called this Driox Atlas of the brain. It's one of uh two or three opensource atlases. You can just like download, use it for whatever you want. [music] >> [music] >> I have to think about how exactly will it look like. Yeah, the other thing is that if you want to make it scientifically accurate, it's probably the resolution on the auditory areas of the brain would not be high enough because it will essentially have the whole whatever auditory cortex one auditor cortex 2 as a one single thing. So would be able to only do one single color as opposed to a more detailed uh a visualization simulation. Let's do a new notebook. [music] Okay, we have a bunch of papers. the rhythm of perception, neural entrainment and auditory oscillation. Well, there's a few questions that we would like to add include. Yeah, I want [music] to include the epilepsy [music] a music channel MG. [music] There's also auditory auditory hallucinations uh related to epilepsy. We found a bunch [music] of papers. We're obviously not going to read them. Not going to do anything silly like that. We'll be doing a video overview and longer audio and it's already [music] generated a quiz. Yes. So, generally we're doing Yeah, we have an EG to music converter. That's a real EG and real music. Real musical notes. >> [music] >> We'll be adding a play button to it at some stage. Go check it out. Uh, we have a jazz music generator that we made a while back. And more interesting, we have this model of the 3D model of the brain which we are planning to do more things with. So, one of the things would be mapping music onto it. Okay. Should have some infographic. So yeah, it's this is the text correct. Yeah, the text is correct. I didn't quite get uh Yes. So that was notebookm notebook. Yes. So everything we do is eventually available on biocs.com. Go check it out. Provide your feedback. Yeah, we wanted to map. So what we I have an idea of a web application where we map music all your sound coming through the microphone or um synthetic uh music like this map it to the brain cuz this brain has about 100 parts. So we can light up each part individually in some sort of way and we can try to make it scientifically accurate as well. Maybe maybe I don't think there'll be enough resolution. Yes, it is like hundreds of parts to this brain but just the auditory auditory pathways are the resolution for them would be not high enough. But yeah, we'll see how we go. >> [music] >> bunch of sources. Turn them into resources into new resources. Hopefully not copyrighted or anything. I don't think yeah this 7inut video the beat brain and breakdown breakdown not sure. Let's listen to it. Today we are diving into the absolutely incredible and sometimes kind of weird relationship between music and our brains. We're going to look at how a simple beat can actually organize our minds and how for some people it can do the exact opposite. All right, let's get into it. You know the feeling, right? A great song comes on and suddenly your head is nodding, your foot is tapping. It happens almost automatically. But why? What's the deal with a beat that just kind of hijacks our bodies like that? Well, it turns out the answer is literally hardwired deep inside our brains. So to really understand this, we first need to look at this fundamental link in our neurology. This deep connection between what we hear and our impulse to move. What you're looking at here is actually pretty profound. When you listen to a rhythm, it's not just the hearing parts of your brain that fire up. Research shows that your motor related areas also activate. Even if you're sitting perfectly still, your brain is basically rehearsing the movement before you even make it. And that's what creates that spontaneous urge to dance. And this right here pinpoints the exact network that's involved. You see these areas, the preoter cortex, the cerebellum, the basil ganglia. These are not your hearing centers. No, these are the parts of your brain that are crucial for planning and coordinating movement. The fact that they light up just from hearing a sound, it confirms we have this built-in auditory motor circuit pretty much just waiting for a good beat to drop. Okay, so our brain is wired to connect sound and movement. That's clear. But that doesn't really explain why certain rhythms feel so good, right? The answer to that lies in this really fascinating theory about how our brain is basically predicting the future. It's an idea called predictive coding. And you can think of your brain as this super advanced prediction engine. It's not just sitting back and passively taking in information. It's actively building these little models to anticipate what's going to happen next. And music with all its patterns and structures is like the perfect playground for this predictive system. Now, to really get this, check out the difference between rhythm and meter. The rhythm is the complex pattern of notes you're actually hearing. Think of a cool drum solo, but the meter, that's the simple, steady 1 2 3 4 that you tap your foot to. The meter is your brain's prediction of where the beat should land. And the real fun starts when the rhythm starts to play with that prediction. So, here's where the magic happens. The pleasure we get from music comes from that little dance between what we hear and what our brain expects to hear. If a rhythm is too simple, it's boring. If it's totally chaotic, your brain just gives up. But these moderately syncopated rhythms, they hit the sweet spot. They mess with your expectations just enough to be surprising. And your brain gets this little reward, a little dopamine hit for successfully figuring it out. And that is the feeling of groove. This powerful link between external rhythms and our own internal brain states raises a massive question. If music can sync up with our brains for pleasure, could we actually use it for therapy? Well, a recent study explored exactly that. looking at how personalized music could help regulate brain rhythms for people with epilepsy. How they do it? They use technology like this, the Neurosity Crown. It's an EEG headset that measures the brain's electrical activity, you know, brain waves. This lets them see in real time exactly how different musical instruments and styles were affecting the neural rhythms of each person. And the process they developed is pretty brilliant. It starts by measuring a person's brain wave response to find which instruments are most calming for them specifically. That personalized data is then used to train an AI which can then predict the best therapeutic music for new people based on their unique brain activity. It's truly personalized medicine all driven by data. And this chart from the study shows you exactly why that personalization is so critical. Now, I know it looks complicated, but just focus on the colored bars. They show different brain wave frequencies for three different people. For the person on the left, you can see the violin, that's the blue bar, gives a more balanced response. But for the person in the middle, the violin actually causes high activation, while the piano, the red bar, is way more calming. A one-sizefits-all approach wouldn't just be ineffective. It could actually lead to overstimulation, which is the very thing you're trying to avoid. But this brings us to a really stark and fascinating paradox. Because while music can be an amazing tool for regulating the brain, for a small number of people, it can be the very thing that disrupts it. This condition is called muscogenic epilepsy. And yeah, you heard that right. It's a rare type of reflex epilepsy where seizures aren't random. They are consistently set off by a specific trigger. And in this case, that trigger is music. The very same patterns that can bring order and pleasure can, for these individuals, actually trigger a seizure. So, how can music cause a seizure? Well, the theory points to these hyperexitable neurons in the brain's auditory networks, which you see highlighted here in the temporal loes. When a very specific sound is heard and it can be as precise as a single note from one instrument, it triggers this abnormal synchronized storm of electrical activity. And that cascade is what becomes a seizure. Wildly enough, even the emotion a song brings up can play a role. This double-edged sword of music as both healer and trigger just goes to show how much we still have to learn about the brain's rhythms. But what if instead of just analyzing these rhythms on a chart, we could actually listen to them? This is the incredible idea behind a technique called sonification. It's the process of translating complex data, in this case, EEG, brainwave data, into sound. So, instead of needing a neurologist to read squiggly lines on a chart, this could make seizure activity understandable to pretty much anyone. And you have to hear this incredible quote from a project at Stanford that describes it perfectly. They translated a patient's EEG into a song with two voices, one for each half of the brain. So when the seizure begins, one of those voices just goes haywire. It becomes an unmistakable audio cue. You don't need years of medical training to hear that something has gone wrong. And this just perfectly captures the real world potential. I mean, this is not just some cool science experiment for caregivers or for patients themselves. A simple intuitive audio alert could be an absolute gamecher for identifying seizures in real time, making care faster and so much more accessible. And that brings us full circle. From our basic innate desire to move to a beat to music being both a precise therapy and a specific trigger, it all comes down to the brain's incredibly complex rhythms. The idea of translating those internal rhythms into a language we can all hear opens up a whole new universe of possibilities, not just for epilepsy, but for understanding things like mood, focus, even consciousness itself. So, if we can truly start to listen to the brain, what other secrets is it waiting to tell us? >> Yeah. So, it's missing a a voice model. Notebook is missing a voice model. I mean, yeah, just until a couple days ago, the audio you could have had it in interactive mode, but that seemed to be not available anymore, unfortunately. Or where it went. Did I reach some sort of quarter? Not sure. Did we do a mind map? Now, the mind map really helps. Really, really helps. It was alo really quick generating it. Music, rhythm, and the brain. Better title. Oops. Yeah, when you click on it on the box instead of the arrow, it's actually sends it into the Okay. Okay. So we have beat and med processing, pre-attentive detection, predictive coding model mechanism, the groove experience, social interaction, interpersonal synchronization, social bonding, metrical models. Why did they come up with like names like that? So not a music therapy. Yeah, obviously personalized instrument prediction, neural network analysis disease, clinical and therapeutic applications, rhythmical auditory stimulation complexity shift in preference agit music conversion. Yeah, that's what we're doing. Auditory pattern recognition, neurological phenomena, musical hallucinations which is can be the case in some uh some types of epilepsy. Yeah, the seizure zone. Can we just map it quickly to a microphone? Oh, this one has um JSON files and things. Not sure why. It's more complicated. Oh, because yeah, it has the So, this is actually where the brain model is saved. But I probably need to save it in the in a central location so multiple uh applications can access it as well. That's only Yeah. 26 megabytes. So, that's workable. Yeah, we have the simology examples. Uh, what should we call it? Uh, audio brain something. Let's call it mic mic brain because we will uh connect it to the mic real quick. Okay. So, can we can we change the code? Uh so instead of uh keep everything as is for now but we want to map the audio to the brain uh whatever color activation. So essentially essentially we want to be able to turn on the microphone and when we do the areas of the brain that light light up will uh directly correlate to the frequency and the amplitude uh coming through the microphone. agent mode. I found mic brain folder. Yes, it's it's looking at uh other files in the folder. I don't know why. No, that's okay. That's okay. As long as it's we just let it do its thing. Hopefully, it doesn't break things too much. I mean, we have the original thing saved. Replacing 34 lines. verified existing structure. Okay, I'm dating. I've added audio mappic mapping logic. Now I'm checking for any remaining references or syntax issues before finalizing. Okay, start microphone. Hello allow 1212. Testing testing. Yeah, it's actually it's actually pretty cool. Well, it is nice. It is kind of what we meant. The previous thing still working. No, this shouldn't be working. That's okay. I like the different colors, whatever they mean. We have only three areas of the brain currently being activated. The brain model will update in real time with activation colors driven by frequency and amplitude. One, one, one. That's kind of nice. It's actually working. Okay. Can you explain why only So, we Okay. So, first of all, we I assume there's a reason we have a symmetric uh brain activation. I mean, it didn't have to be that way, did it? Can we actually activate like all um more parts of the brain instead of just the three regions? And can we also update the HTML as well? So, uh we just essentially have the the canvas and the description at the bottom of the page. But yeah, we want more areas of the brain activated. But yeah, it's it's it's nice that it's actually responding to amplitude and uh frequency as well. But uh yeah, we need to be explained how it works, educational purpose, uh technology and data sources uh so on so forth and and also like an explanation how scientifically accurate it is. Uh right. Let's see. Yeah. So it's updating. Yeah. So we have this brain. I can describe. It's hard to describe in words what what we are looking at. Uh yeah, you should be able to see it as well. We have this brain model. It's hyper accurate. It has about like 100 different parts. Okay. So we have this mic brain audio driven brain activation visual visualization. We start the brain. Whatever. Yeah, it's still working as before. I don't know which agent is doing the coding, but it's doing great so far. We only have three regions of the brain being activated. I like the colors. They're different colors. But yeah, this brain model has about 100 uh different uh parts to to it. Yes. Yeah. So low frequencies activate the motor and frontal region regions mid frequencies temporal auditory. High frequencies activate occipital and parietal visual spatial regions. Well, which is not scientifically accurate. Is it? Yeah. This visualization is intentionally approximate and educational and does not does not represent real neural activation. Yeah, the atlas, the brain atlas we're using is available on brain.org. There's a human in the loop. If you have any questions, also if you want to check out the everything we do is eventually available on bicos.com. Go check it out. Provide your feedback. Okay. The linting error went away to start the microphone. Yeah, we need ideally more areas of the brain activated. The simulation is intentionally approximate education does not represent real neural activity measured from the brain nor does it in true EG source localization from microphone audio. The left right and frequency regions mapping are chosen for visual demonstration only and a not medically validated model of brain activation. It does work though. It does work though. We could have more regions. I like how the left and right is different now. But there's only one, two, three, four, five, six. Six uh regions are being activated. How heavy is it on the It's only 107 megabytes. We are continuously working on advancing mic brain. Well, we just we just literally made it today. Whatever. with plans for more detail regional information and additional inter interactive features in the future. Yes. So, currently it's mapping your microphone. It could be potentially mapping music. Okay. Currently, we only have six uh regions of the brain being activated. Can we increase that and uh update the text description? Uh we want a section of uh yeah we want a whole section on um how this uh web application simulation can be improved in the future as well. But yes, we want more. Maybe a slider. Maybe add a slider to to choose uh how many regions are activated like from make the slider go from two to all. One, two, one, two. Yeah, you can play can play some music into it. Okay, I'll find that music generator thing on my phone. Can't believe I had to make a manual manual [snorts] change to the code. Yes. So, we have six six regions can go up to 150. Yeah, the activated regions is weird. Supposed it is 150. I don't think they are all being activated, but it is working. I mean, it's doing the job. We should add the canvas in full screen uh option. There's also some sort of uh filter on the microphone. Let me turn it on for a sec. 181 megabytes in the browser. That should be fine. >> Now applying the UI slider, dynamic region selection, and updated content in one pass. verifying the new slider logic and dynamic band grouping to catch any edge cases before finishing. >> Hey, can you explain? Uh, so first of all, not all the brain regions are being activated even if even when I select the whole 150 on the slider. Uh, not all of the brain regions are activated when the sound is going through. So maybe can explain why or if you can just like fix fix the code uh straight away. Also, can you explain uh can you update uh and explain in detail the part in the text that u talks about how the mapping is uh happening exactly and the difference between volume and frequency. We will see if that works. But yeah, currently I don't know if there's more than 150 regions or not. I didn't actually read it response. Okay, so that's six regions. Now it's 150. Okay, I actually fixed it. There's still a hole there. Yeah, one of them is not being activated. I don't know why. Let me turn the filters off. Yeah, you can actually hear the Yes, you can see the brain getting activated, which is nice. Still some holes in it. Some of the if we can do control. Can't zoom in. Oh, one one. It's kind of nice. It's like um what's called like a music visual visualization of your branch. There's six regions, a 150 regions. It's actually whatever regions are not currently being um activated by the sound. >> Yeah. So they gray grayed out. >> Why not all regions were activating? The original code had a bug in the update audio highlights function. When you selected more than the regions already classified into the three frequency bands, low, mid, high. It would add extra regions from the unassigned list. Regions that didn't match the keyword filters for any band. However, it only colored the regions from the three main bands, it didn't apply any color to the added unassigned regions. That's why even at 150 regions, some parts of the brain stayed white. The fix. I updated the code to color every selected region individually. Regions in the low band get orange red based on low frequency energy. Midband gets yellow based on mid-frequency energy. High band gets blue based on high frequency energy. Four unassigned regions added. When the slider goes high, they get assigned to whichever band currently has the highest energy level, so they still light up dynamically. Now, with the slider at maximum, all 150 regions should activate and change color based on the audio. Updated text explanation. I revised the how it works and scientific accuracy sections to explain the mapping in more detail. Frequency determines which regions light up. Eg. Low pitch is motor/frontal areas. Volume amplitude determines how bright the colors are. The louder the sound in a frequency band, the more intense the activation. The text now specifies the exact frequency ranges. >> 50 to 250 hertz for low, etc., and clarifies that the mappings are simplified analogies for education, not real neuroscience. The slider's role in distributing regions across bands is also explained. Made changes. >> Made changes. Made changes. We like the made changes. And that was Gro code fast. Thank you, Grock. I'm not sure. Yeah, we'll try to improve it some more. Let me know if you have any ideas. I'll make it. Everything we do is eventually available on bingos.com. Go check it out. Provide your feedback. So this is a synthetic uh music generator which is also also available on the website just using it off the phone. So it goes because currently this uh it's called mic brain because it's currently only using your microphone. So it kind of works. Yeah. I don't quite understand the mapping. The mapping is a bit odd. Yeah. Let's think about it. How do we improve the mapping? Because currently, yeah, there's three frequencies. Low, medium, high. Amplitude is the intensity of the color. So, it's kind of nice. It's probably probably taking too much uh too much is given to the red frequencies, whatever they are. Yeah. But there's an explanation of what the of how the mapping uh low mid high. Can we add like uh mid low and then this will be high low? No. So you have low a medium high. Classify brain models into low mid high frequency bands based on their names using reax buttons. Yeah, that one is a bit okay. Okay. So, we have low, high, a very high. Where is it taking those um names? This parietal. Can we least all the names first and then add the very low very high? Yeah. Instead of three colors, we want uh uh four colors. So, we have very low, low, medium, high, very high. Yeah. Yeah. So, instead of three levels, instead of three colors, we want uh more colors. Uh there will be five. Anyway, the thing is working. I was just saying we'll probably have too much dedicated to one, two, one, two. No, it's still making changes. Okay. So, it's actually doing it. Classify brain models into very brain models. No, it's the regions. very low, low, mid, high, and very high frequency bands. Okay, so it's actually did there. Keep it. Keep it. One, two, one, two, one, two. What's happened? What happened to that one? Meant to be five colors instead of just three. Hey, some of the names I think uh might be wrong. One, two, one, two, one, two. Okay. It's still making [snorts] That's interesting. How many colors are Yeah, they're meant to be. There's that purple now. One, two, one, two. But yeah, too much is going towards that red. Very low. What are the colors for each? How many colors are there? Yeah, there's this purple. There's that purple one, two, three, four. Not sure. [snorts] One, one. One. Yeah, that color is too similar to the orange. That red. Red is too similar to the orange. One, one. Wait, how many one one? Well, kind of works. Yes, we have red, orange, purple, blue, yellow. That's nice. >> The brain models are now classified into five frequency bands. Very low, low, mid, high, very high. using regex patterns based on region names. The visualization has been updated to handle five bands with corresponding frequency ranges, colors, and region mappings. The changes include classification logic patterns now categorize regions into five bands, eg, very low for medial frontal areas, very high for occipital/visual areas. Audio processing. Five frequency bands are analyzed. 0 to 50 Hz, 50 to 250 Hz, 250 to 1,000 htz, 1,00 to 4,000 htz, 4,000 to 8,000 htz. Visualization. Each band has a distinct color. Red for very low, orange for low, yellow for mid, blue for high, purple for very high. UI descriptions. Updated text to reflect the five band system. The application should now provide more granular brain activation mapping based on audio frequencies. You can test it by running the local server and accessing the mic brain page. >> Yeah, that's fine. Um, yeah. So, it kind of works. There's like missing beats there. Like there's holes in the brain. Holes in the brain. Holes in the brain. [laughter] Yeah, it would be fun uh playing music to this one. I don't know what's up with the with the zoom. There's something about the zoom uh zooming functionality with control scroll with constraints on zoom level. Uh uh let's make it 1,000 for a sec. Right. Okay. And the default is six. We can Yeah. I don't know what the default uh setting should be. If you have any ideas, do let me know. Yeah. So, yeah. Nice, nice, nice. So, we have a brain. We have a brain. It's an accurate brain model. It's open source. You can uh grab it for your own projects. We mapped the activation the colors to to different frequencies and amplitudes or coming through the microphone. How good is that? Should play music to this. Ah, if music is copyrighted, [laughter] we can just mute uh we cannot we can just not send the music to the stream. So, I'll mute my microphone that goes to the stream, but I'll just uh play the music to the thing and looking at whatever the brain, my brain, you could try and determine what song is playing. How about that? Can you do that? Can you do that? Let's choose a song. And yeah, I won't be playing it on the stream into the stream for copyright. a copyright reasons. But yeah, something actually it actually looks pretty pretty cool. So, I'll actually choose a song that I don't think is uh copyrighted, but it has a lot of like written to it. And u yeah, let's put it uh yeah, Mike Brain uh performance for you. But yeah, you get you get the drift. It's also important like Yeah, it gets saturated. Like if I'm too close to the microphone level, you need some sort of um Yeah, we probably need another controller for sensitivity. Sensitivity. Sensitivity. Sensitivity. Yeah. And obviously the bot can do all these things. How amazing that is that. Yeah, it did pretty much everything we asked for. Thank you very much. And let me Well, currently you can, you know, adjust the sensitivity just by moving your mouth away from the microphone. That should be fine. Yeah. Let me know if you have any questions. Everything, this model is open source. Everything we do is open source as well. Just go F12 on binding.com and yeah, there will be some salt that will [laughter] be better better better. Not sure what's happening with the noise suppression. Noise suppression. Yes. So, don't forget to like, subscribe, unsubscribe. Uh, you know what to do. Mainly comments. Your we are mainly here after feedback uh for the website. So, if you have any of those, if you have any any feedback, do let us know. Do let us know.