Living with a Blackhole Quietbox - T2sday Update 3173

Hey everybody, it's T Tuesday. If you're new here, let's start with a little bit of a recap. So, here we go. Oh, okay. Blue light. We've got bubbles in the reservoir. Is that what this is called? I don't even know. So, that's probably pretty confusing, but for the last 9 months I've been working with this thing called a quiet box, a black hole quiet box. But, that's not where the T Tuesday update, the project to build a whole new computational stack with all kinds of bells and whistles, that started back in 2018. But, the reason I wanted to do a little bit of a recent review is that over the last couple of months, one of the older videos, one of the T Tuesday updates from several months ago, suddenly started getting a few more views. Now, again, the total views is like 187. So, you know, if you do ever decide to subscribe to this channel, I bet it'll be the smallest channel you subscribe to. But, you know, I noticed that there was this little pulse and I went and looked at it and [clears throat] it was this update. Hands-on the black hole quiet box 3165. We're at 3173 now, so that's eight updates later. Eight months ago, it all makes sense. Um, and the reason for this is YouTube search. People have started searching for black hole quiet box Teensy Torrent and they're finding that video. So, today, uh uh I I I thought I would try to just do a little update, uh a little cat you know, 9 months with a 10's torrent black hole quiet box. That's been my uh much life for the last 9 months. So, I want to try to talk about it as sort of first first item of business and then let's talk try to talk about progress and everything else. So, the goals for this time uh and my results I did pretty well. Spike in her hub is still delaying. That's going to be on the docket for next month for July. Big friend time, I'm not going to talk about it a lot, but I'll have a picture later and so on. So, life with a black hole quiet box, the good, the bad, and we'll talk about the project progress. I'm not going to be able to talk about these discussion topics perhaps in the after party at the live stream if anybody wants to talk about them. So, what appears to be going on, and this is just my intuition, this is just my feeling, that a lot of the people who are finding 3165 and a lot of the folks who are finding the 10's torrent discord, they're there because they want local AI. They want to be able to experiment with AI large language models, all kinds of the stuff of the day, but they don't want to do it on the cloud. They want to have it under their own control, and I think that's good. I think that's great. I think that's the destiny of AI. Why should we ever trust AI unless it's actually ours and close to us and loyal to us, you know? Couldn't that be something that we could be going for? Um so, I understand it. However, the black hole quiet box, you know, is not quite there yet. So, it seems like a lot of the folks who are showing up in the Discord are kind of looking like, "Can I already run 120 [clears throat] billion parameter models? Can I just, you know, run Python [clears throat] and start and the answer is, you know, mostly not. There there's still a lot of work to do. There are a lot of models that have been ported and a lot of people are porting models. So, if you want to get in on that kind of fun, if that works for you, then um you know, you should think about going for it. The important point for purposes of people watching this video, and this is where the viewers are going to go that is I do not care about machine learning at the moment. I am not using the quiet box for machine learning. I'm using it for this crazy kind of cellular automata that I probably won't get to talk about much today, but there's plenty of other videos, papers, and so forth available. So, on the hardware uh uh it's called the quiet box, you know, which is is a [clears throat] you know, it's a it's a clever name. Once you start living with it, you realize it could just as well be called the a hot box cuz, you know, it's got a liquid cooling system. So, it is absolutely very, very quiet, but it's got this giant radiator on top that looks like a car radiator with all the little fins and, you know, just tons of heat coming off it. And I don't know if you could possibly see at the beginning at the little opening previously on video uh that at the end we we saw that the water level on the thing started out way up near the top and now here 9 months later, it's down a couple of inches. Now, my understanding, I've never had a liquid-cooled machine before, is that it needs it needs to go way, way, way further down before I have to deal with topping it back up. But, this is a lot of machine for me. I mean, I've used some kind of supercomputers over the years and so forth, but this is a lot of machine just for me to own. The AI clock, the rate that the chip works at, you know, I'm using it at its lowest rate, which is 800 MHz. It'll go 1.35 GHz, but I'm saving that for later. The NOC, that stands for network on chip, is super fast. Each, you know, every clock tick, every AI clock, it can move 64 bytes across one of its links hopping across the the the chip for a aggregate bandwidth, which you're never going to see, of 30 TB a second. And the reason you're never going to see it is because, you know, it's the same network for everybody. And if I, for example, wanted to send a packet from one particular little T6 tile, Tens Torrent, Tensix tile to the host, say, uh to be part of a display, for example, it has to go hop hop hop hop hop hop hop over the same paths that all the other packets are going. And if there's something already going, it's going super fast, but if it's busy, the thing has to wait. Furthermore, the memory of the Black Hole, there is essentially no attempt to hide the fact that memory is many different types. There's super fast RAM that you can use, but there's a tiny amount of it for each of the little hearts, each of the little hardware threads inside each Tensix tile. Each Tensix tile, in addition to having a whole big AI coprocessor, has five regular CPUs. And each of them has their own little bit of fast RAM. They all share this megabyte and a half of L1 RAM. Then, there's a tons of DRAM off on the edge of the chip, and then there's the host RAM that you connect to via PCIe and so forth. There are, you know, the software is young, the documentation is young. This is still I mean, they're selling it, but it's still a research grade thing. At least that's the way it feels to me. Knock addressing, I got screwed by the knock one addressing. The documentation talks about how knock zero and knock one use different coordinate systems, but it turns out that out out of the box, the firmware hides that so that knock one actually uses the same addressing as knock zero, and I lost a lot of time. I'm not even going to talk about the L0 data cache. I lost days from that. On the software side, you know, there's all these different projects going on. There's all these different I mean, it's a whole new stack. They're trying to go from risk five assembly and all of these routers and so forth all the way up to machine learning. You know, there've been a number of grand visions that people had thought of with you know, there would be a great overall vision of how you could do it. You know, you could view it as a as a graph reduction. You could view it as a a whatever. I don't know. But you those have mostly blown up, and now it's TT metal, TT metallum, where the metal is signifying that it's it's a relatively low-level model. Now, for me, it turned out that TT metallum was way too high. So, I ended up in the choosing my own adventure side of things, building from very, very low. And it's been a challenge, and this is where the nine months mostly have gone. You know, all of these new acronyms that I didn't know. PCIE, TLB, cache bar, and risk V, knock goes also fast for uni and for multicast. It's fun, but it's slow. I want it to be further along. but I want it to be significantly further along by now. The overall ecosystem of the quiet box has some gaps. Here Here's my joke. I said, "Doc, is it safe for me to do this?" And the doc said, "404 not found." Because the black hole documentation on the one hand, what's there of it is really pretty good, very detailed, stuff that you can code to. On the other hand, there's big chunks of it missing. So, I, you know, got on the Discord. I asked a question in January, something like that. And so, you know, I kind of went back three months later and and kind of shamed people into actually answering the question. I apologize for shouting and so forth. And and one of the people who I'd spoken with previously at Tenstorrent, Nick King, actually, you know, went to own it and got some answers and so forth. I still have some more questions, but I'm I'm not pushing it right now. So, you know, there's a lot of time lost living with that kind of thing. Lesson so far for me, computation is easy, communication is easy, coordination is hard. Cuz you've got to make sure things happen when they're supposed to happen, and do guys wait, do other guys not wait, do they just keep going? Also, Tenstorrent seems like a chip company, but I think it may live or die based on its software stack, because it really is a much more complicated. So, you know, the elephant in the room, Nvidia, uses I mean, it's it's taken many steps away, but it uses a fundamentally simple model that you have thousands of processors that are all doing the same thing, star star star, with a bunch of footnotes, but basically, it's that makes it easier to think about. Whereas, Tenstorrent is is trying to be the hardware is much more general, which means the task of using it effectively much more challenging. At the same time, I think the future is in here somewhere. So, it's exciting. Okay. That's >> [laughter] >> lived 9 months with a black hole. We should have had a black hole baby by now, but it's stillbirth. So, in the project, the MFMX project, which is to build an implementation on the black hole quiet box of this model, the movable feast machine that I've been working on for pushing 20 years, I don't know, a long time. Made some progress, not enough to have a demo, which is actually why this episode update is what it is today. So, you know, this was my to-do list way back when. Hardware, firmware, software systems. I don't see systems. Make more to-do. And so forth. You know, this when you're building a whole stack, you know, you got to go from the hardware to the systems, which don't even appear. Over time, I gradually filled some of these in. They turned green. I also added some. I also declared some red, saying, "I'm never going to get to this. So, let's just admit it." And then gradually things sort of spread out. And And stuff got accomplished. And And this is more or less the current picture. I hadn't been updating my to-do list for a while, so I did it for this. The big thing really that wasn't there at all is this idea of multiple images that we've got hundreds of individual little processing units, these TenSix tiles I mentioned, each of which have a megabyte and a half that they share of memory that can have a program in there. And the TenSix tile here could have a different program than the TenSix tile over here. You'd have to download them separately, and that slows things down a little bit, but you can do it. And we are doing it. And so, we have several different the event window processor, the regional hub, and a few debugging software things and so forth that are running completely different code. And this so from last October to yesterday, day before yesterday, these are the sizes of the different images and EWP and hub are the two that are really meant to be implementing MFMx. EWP is to perform little events to do cellular automata transmission transitions from a before state to an after state. Hub is dealing out local transitions, gathering them back up, applying them, and then eventually sending them up to the hosts where they can be displayed somehow. Okay. So, that's the that's what we've got. So, they're you know, the logs so we redo the packet system again. Uh >> [laughter] >> Uh but it's it's not bad now, I think. I I think I can live with this for the rest of this round of the project. Now, the log transport that is to be able to write general debugging messages inside the TenSix tile and ship them up to the host reasonably reliably where the host will keep them all separated. The stuff that came from this tile goes here, the stuff came from that tile goes there, and so forth. That's running. Event window transitions are running. The ability for the regional hubs to talk to each other has been demonstrated, but they're not saying anything useful yet. The remaining thing that I want to talk about is the calling the T6 viz challenge. You know, if we're talking about T6, which the Ten's torrent people use as an abbreviation for TenSix 10 6. Uh um this channel is called the T2 tile channel, not the T6 tile channel. And the T2s were the uh hardware that, you know, I built from, you know, little tiny little boards that we made custom little setups for. Plug them all together and so forth. There's plenty of background material. I suggest starting at the pilot episode about the T2s. But the T2s have one secret advantage that the T6s do not have. And this illustrates it. Now, this also illustrates a second thing. The month of May uh the was assigned to me one of the goals was to have big friend fun. And this is my friend Dick. This is my oldest best friend uh in the universe. Uh Dick and I have been best friends for almost 60 years. So, you know, two extremely old farts. Uh and Dick came out for a week during May and he, you know, isolated so that, you know, all of the stuff that I'm still doing as if there's a pandemic going on. Don't tell me if everybody else thinks it's over. So, that he could actually come in and join us with the house and and we had dinners together and so on and so forth. And it was wonderful. It was great. Uh we worked on some stuff that might lead to things coming down the road. We'll find out about that later. But the secret advantage. This is the T2 matrix that's right behind us. And in fact, down here, this corner of the T2 matrix is running right now. And what is the secret advantage? The secret advantage of the T2 matrix is output is parallel. You you you get the camera. We I use this Fujifilm XT4 uh with, you know, like a a 8K image sensor or something, you know, I don't remember. Uh uh to take shots of the entire grid at once, uh uh like that. And that meant the communications ability between the tiles never had to worry about exfiltrating the data in order for it to be recorded. And that's a big advantage that the T6s don't have. So, that means if if I want to make a video, and I not only want to make a video, the point is for the for the quiet box black hole, I want to make a a live stream where you can see what's happening on the quiet box right now, you know, plus or minus a minute, uh uh and maybe even down the road interact with it somehow, uh um to to really get the flavor, to really get the feel uh of what the Moofus machine could feel like if we could run it fast enough to have it be in our own space-time volume of interest. But, in order to do that, all of the T6 tiles, all of the hundreds of T6 tiles on the uh quiet box have to send the data to the host so that it can get uh uh ramped up and eventually um get displayed. And that's going to happen on the same NOCs, on the same networks on chip that all the other communication is supposed to happen, including the event windows talking back and forth to regional hubs and so forth. That is the T6 viz challenge. How are we going to get the display out so that we can see stuff without completely creaming the network so that it can't do events? Skip the rest of this stuff, and so that's going to be it for this time. Uh uh um The goals for July get inter hub working for real. I mean, like I said, I'm sending dummy packets back and forth, but let's send some kind of events somehow. I still have design questions about that. And let's have real host viz running. Let's bite the bullet and find out how much it's going to cost us in simulation speed if we have to send out the data for even for something like, you know, 10 frames per second. Forget about 60 frames per second. I don't want to spend the bandwidth to send out that much data. Could we compress the atoms? Could we save stuff? That's all on the table for June to report out in July. Have morbid fun. Enjoy shaving shacks. [laughter] And that is it for today. Thank you so much for taking a look. I'm I'm excited that that people are finding the the black hole even though I'm not doing probably what most of the people who may have come to find this want. The more the merrier. And I hope to see you next time.