MIMD and More on the Blackhole Quietbox - T2sday Update 3174

Everybody, it's T Tuesday. Perhaps you are someone new. The last T Tuesday update from June is in the tiny little sense of the uh um T Tuesday project channel is number one against its own previous 10 episodes and it's got 500 views which is already more than well, more than most. So so thank you if you're if you're coming to check back in, thank you again. This is going to be another kind of a little bit nerdy but you know, hey, if you're interested in what life is like living with a Tense Torrent Quiet Box Black Hole, well, this is the place to be. So, the goals for today, the long and short of it is I did not do very well. Spiky Interhub for real, don't even have to explain what that is because I failed utterly. Real host viz running, also. And the fact that there was two reals in those things was that that was my claim that it was supposed to be sure that I'd be able to do it for real this time and I didn't achieve either of them. I gave myself a possibility of trying to do compress atoms to save bandwidth when sending it up for it to be displayed on some video stream at some point. I I did okay on that except for the part about sending it up to be displayed. And yeah, I had big fun. So let's go. For today, I want to talk a little bit more about the architecture uh of the Quiet Box well, of the Black Holes and the Tense chips the Tense Torrents chips. These ideas of hearts hardware threads. Um and, you know, calling it a hardware thread is kind of deceptively simple. It's like, "Oh, I've used threads. I've used software threads. How different could hardware threads be?" Well, it kind of turns out they're quite different. Feels quite different. Uh um and in particular, it shows up in the init seek, the initialization sequence. Now, I've been through this uh story before. Uh uh back uh in the 1990s, I worked on a sort of distributed social network, sort of like a multiplayer, not massive, online role-playing game, blah blah blah, called CCR. And it became quite complicated. It had many layers. And the getting CCR started up it was like booting an operating system in a lot of ways. It was like booting an operating system. It had a tremendous number of stages and a tremendous amount of code that needed to run while not all the facilities of the operating system were working yet. So, there were these kind of levels where, you know, here you're really in the stone age. All you can do is this. You can't even allocate memory. And then now you can do a little more, and so on. And that has started to happen in the code that I'm running through the writing for the T6 tiles. And it's causing no end of bugs and lockups and mysterious behavior, which is why I I think uh I largely failed at my goals uh this month. Uh in addition, I I actually got to do some kind of old-fashioned uh you know, computer science algorithms, like you know, linked lists and stuff. I want to try to talk about that a little bit. We'll see how much time we've got. Okay. So, I talked last time about the the fundamental Tenstorrent uh value proposition. It is that they're going to provide MIMD CPUs that have local memory, that have instructions that are being fetched from local memory. So, this little CPU over here and this little CPU over here do not have to be doing the same instruction at the same time. Versus SIMD versus Nvidia, the traditional GPUs that are fundamentally based on everybody doing the same instruction at the same time just on different data on different parts of the screen. And as I said last time, there's lots of sub asterisks in that and and Nvidia can do much more than just simple SIMD and so forth. But, it is a represents a fundamental alternative on the attitude about how we should do synchronization. In SIMD, single instruction multiple data, we're essentially saying we want everybody to be sync to be synchronized all the time. And if you don't want to do that, like if you want to have an if statement that some folks some proces- sors are are going to do and some processes are not going to do, well, that's going to cost extra in the model. And in SIMD, you have to do it by running both alternatives and telling these guys to be quiet while we do this one and then tell those guys to be quiet while we do that one. In MIMD, you have synchronization nowhere. Everything is running independently, at least in principle. And if you want any synchronization, that's going to cost you. So, I asked for MIMD. The movable phase machine cellular automata architecture that I'm building up here is fundamentally a MIMD notion. The the tran- cellular transitions are complex wads of code and different areas of the cellular automata are running different wads of code at literally the same time. So, I want I got it and now I'm choking on it. Um so, on the black hole quiet box, uh and actually on Wormhole and and pretty much all of the stuff that done so far, uh um the uh the little teeny cores, the Ten six cores, uh have atomic instructions, which means they can read and write memory in a single operation, and that allows you to build uh synchronization systems. It allows you to build mutual exclusion primitives. And we don't need to understand how that stuff really works, uh but we we it's important to understand that there are ways of doing it. Uh uh So, in principle, this L1 memory, this megabyte and a half that five uh hearts all share, any location in that, in principle, could be a place that they would synchronize, that that different hearts would synchronize. Uh um and so, okay, there's, you know, lots of options, but you got to decide where you want to do it. And every time you decide to synchronize, it's going to cost, and there's a bunch of little gotchas uh about the this heart and that heart have different views on L1 memory. One of them may see a location having been changed while the other one does not. That's what the atomics are supposed to deal with, but, you know, yeah yeah yeah yeah. The thing that got into my head was that I thought I understood software threads. I thought I understood threading. You spun one up, you let it listen to uh internet connection, it services it, it shuts down, whatever it is. Uh uh hardware threads are different. Five fixed threads means they're all there, they're all running. You're bank um a cycle saved is a cycle wasted. If you can't think of something for all five of those hearts to be doing, you're leaving performance on the table. So, you know, when you look at the uh, the system diagrams from TensTorrent, this is a sort of a typical picture. You got a floating point unit, you got a general purpose, blah, blah, blah. And then you've got these 1 2 3 4 5 risk processors. And in this rendering, they all look the same. So, you think, "Oh, you know, you just pick an arbitrary one and then go do something." But that's not really the case. Here's another picture that, you know, now we've got the, uh, the five, uh, hearts uh, across the top in this particular rendering, but look at this, only three of them connect to this TensX engine. And so forth. In fact, the different hearts on a TensX tile, uh, are differentiated. This is This might be hard to see. This is a picture by Marty 1885, who was, uh, used to work for TensTorrent, but something, I don't know, he, uh, doesn't work for TensTorrent anymore. I hope, uh, uh, I hope it there was no bad feeling there, cuz I I thought he was great. He was on the Discord and he and he was Johnny-on-the-spot, uh, getting rid of spammers and so forth, which the TensTorrent Discord is kind of having a lot of trouble with now, and they don't seem to have a Discord spam bot. I don't know what they're doing. Anyway, the point is, in this, you know, here's a core, 1 2 3 4 5. They're all intended for a special purpose. One to bring the the the, uh, data in from outside, one to unpack it, uh, uh, one to do math on it, one to pack it back up again, and one to send it back out. And if you could do it that way, then that would be great. But that's of course not what I need. So, what I came up with in the last month is this idea of a task. Now, you know, I mean, there's millions of notions of task, so calling it a new abstraction. But for me, a task is this quite complicated thing. In particular, it's not the same as a thread. It's not the same as a hardware thread, because a single task may and typically will involve multiple hardware threads. And while it's doing that, it may use both the heart private fast RAM as well as the L1 shared memory. And if it's going to do anything that involves some significant work, it wants to do that using the heart private RAM if it can. And again, it it also may require case-dependent sequencing. So, in this is another one where, you know, so SIMD doing the same instruction all the time is going to be too tight. My old init >> [laughter] >> that I've been using for 9 months and so forth was essentially like the Arduino for folks that are familiar with that, which is actually based on processing, which is based on wiring, you know, this whole history over there. But the concept is you have a setup function that runs once at the beginning, and then you have a loop function that goes over and over and over again. And I just brought that in and set it up on the T6 tiles. Except there are five hearts, so the whole thing is times five. Here's setup for heart B, T0123 and C. Live is do the loop, step is one time around the loop, all times five. The trick is, of course, this has nothing to this has nothing about tasks. This has nothing about what's supposed to happen in all those different things. It's just hardware oriented. And in the reality that, you know, the initialization of some things depend on other things. For example, an initialization of some complex thing might want to send a message to the host saying I did it or I couldn't do it. And yet, the very ability to send a message to the host is being set up by other inits like that. So, >> [sighs] >> the Arduino style times five was insufficient. And so, in it it it killed me this past month. So now I have the task epoch framework. So the idea is the mental idea is tasks are functional goals, things that we wish to do. They may involve multiple hearts and bits of code and time and so forth. And epochs are stages in initialization, stages in pregnancy, stages in building up a new life. And so what we actually want is to specify what happens at a combination of a task and an epoch like that. And so there's I now have code that you know has a list of epochs. The first one is begin which is like very absolutely nothing and then born grow live and reserve, you know, you think that might have been die but I don't want these things to die. I just want them to keep on going. And then there's this. There's the task table and and this is my giant accomplishment for the month. Like that. So it's got a row for each task. So there's a boot task, there's a task that is involved running the clock. It also involves calling handlers if you can you can hang work on the clock to run every millisecond for example. And the clock takes care of that. Here's that log thing that is for sending messages back to the host about what's going on in the tiles. Here's the random number process which is really important for the MFM architecture so much so that I'm dedicating an entire heart to be in charge of random number generation and feeding it out to everybody else. And then there's the network on chip, the communications. And then in each of these spots there are which hearts are going to be active at this combination of boot and begin at this combination at so in the live epoch, at the in the log functional task, all hearts need to run. Versus underscore means none of them. BO1 means heart B, heart zero, and heart one all need to run, but no others. Heart two and heart two here, heart two is the one that's doing random number generation. And so, the weird thing is and I guess this still kind of blows my mind, but it it's working kind of good. It's not working well enough that I can show you a demo. I'm sorry. I'm a failure. But so, we have so, there's now handlers. So, the little bits of code that go with a given task and a given epoch, a given task, and a given epoch and a given heart heart num are written up in little functions like this. And so, here is the epoch. In the grow epoch, only the T0 tile has something to do. And down here, all of the tiles have something to do, and so forth. Um the random number one has, you know, the random number generator needs to init itself. All the other ones, except for the T2, because it's the one that's in charge of random numbers, and the NC, cuz it doesn't have access to heart two for this particular purpose, they all have to do something and so on. So, this is one function that this is run on one heart. This is run on several hearts and not all of them. And so forth for these at different times. And yet, because it's task-related, it's all about the PRNG, the random number generator, it's all in the same place. And this makes me much happier. So, that's what tasks and the init sequence is about. I hope it's going to take me all the way through to getting all this stuff working and moving on. In addition, I tried to do atomic uh compression to save on space. And you know, so there's you know, the Lempel-Ziv LZ compression algorithm that goes all the way back to the '70s. Um and it's been worked on by zillions of people that, you know, Lempel-Ziv LZSSW, you know, and so forth. Uh I went looking around on the web. Uh this was uh one by Matt Seabrook who very kindly put it in the public domain. Yeah, but the a lot of these all go back to Haruhiko's Haruhiko's Okamura's 1989 implementation of Lempel-Ziv, which in turn goes back to like the '70s. So, I looked at that and I learned a lot from it. I looked at uh Ari Dyat's uh one, but it was scary full of code that was way too optimized and so on. And I had my own special needs. In particular, I needed uh compression that did not require a power of two buffer size. And they all used power to buffer sizes cuz they're a little bit faster. Uh but for me, I needed to make it as big as I could while still fitting it into the fast memory of Heart 2, which only got 4K of uh fast memory. So, I needed to rewrite it so that it could not necessarily be a power of two. And I made it as big as I could pretty much, so 348 elements. Uh um In addition, now there's this challenge. Okay, so we have a grid uh and a grid which has about 15,000 atoms in it. And and in a hub, there's, you know, dozens and dozens of them uh all over the place. The hub is not the top-level entire grid. It's a little substructure that has about 15,000 atoms in it. And as events occur, those atoms get changed, and the changed atoms need to be reported to the host so that they can be displayed. Uh um Um so I wanted to collect a list of modified atoms so that when it was time to send some packets off, I'd know where to find them. But I also needed to be able to access the middle of elements in that list, not just at the tail end, the beginning and end of the list, but into the middle as well, in case atom that was modified got modified again before I ended up sending it up to the host. So the idea was use coordinates instead of pointers. And then once you have, you know, so U8C is two 8-bit coordinates, X and Y. So I'm saying, you know, 107 3, that atom the atom located there has been modified. So then you come through this and do regular stuff with it. Um and here's the declaration, doubly linked 2D spatial list. So we have There's the whole grid of these little things and then there it is. Uh uh the next and prev of a doubly linked list that that absolutely standard except they are both 2D coordinates where each coordinate axis is one byte. So the entire thing takes 32 bits, one word to have two pointers. In exchange we have to we have to allocate one of those pointer one of those 32-bit words for every element of the array whether or not they're actually in the list, but that's what allows us to pull it out of the middle. Okay, there it is. There was a lot of fun there. It was fun doing some traditional hacking. I'd never implemented compression before. That was fun. And so forth. So going forward, I want to get this task table idea firmed up and I want to get it out there. I want to do a round of cleaning on the software base. It is a mess of half-done ideas. I want to say once we get the task table working well enough that we can send atoms up to the host, say, I want to do a cleanup and then go for host vis again. And, you know, you got to push the axe in order to go on but pop them as much as possible and have good fun. That is it. Thank you so much for stopping in. I hope to see you next time.