Video summary
In this episode of T Tuesday, the host presents the final boss challenge of running a stochastic cellular automaton known as the Mughal Feast Machine on Black Hole Quiet Box hardware using TensorTorrent architecture. The primary objective was to visualize events occurring across a grid of Tensix tiles without relying on screens embedded in the chips themselves, which are absent compared to older T2 tiles. To achieve this visualization from an external host with internet connectivity, the developer implemented a regional hub concept where groups of nine tiles operate together; however, due to routing inefficiencies within the Network-on-Chip that forced data packets to travel long distances around the chip, the architecture was adjusted from 3x3 hubs down to more efficient 2x2 configurations. This setup allows specific "hub" processors in each region to manage event windows and coordinate with external memory before transmitting results back to the host for rendering.
The live demonstration showcased a successful data stream where various cellular automaton elements, such as fork bombs of different growth speeds and rare resource atoms like Drag and Rez, were visualized on screen after being compressed by approximately 50% during transmission via PCIe connections. Despite the success in displaying isolated hubs running independently for hours without crashing, the system currently lacks inter-hub communication capabilities, meaning regions cannot yet share events or migrate entities between them seamlessly. The host explained that future updates will focus on enabling these cross-region interactions first within a single card and eventually across different cards using high-speed cables, while also refining the codebase which had become cluttered with bug fixes during development.
Looking ahead to September 1st, the project aims to benchmark the Average Event Rate (AER) for an HD-sized grid of cellular automata cells rather than relying on synchronous frame rates typical of traditional computing systems. This metric is crucial because events in this asynchronous machine occur at different times across sites, making standard FPS measurements inaccurate; instead, the goal is to measure how many individual events each cell processes per second over time. The broader mission behind these technical hurdles is to demonstrate a scalable form of computation that differs fundamentally from current digital deterministic hardware used for AI training, which consumes massive energy and prioritizes raw gigahertz speeds. By developing technology capable of slower but more reflective processing loops, the project hopes to create systems better suited for simulating living systems rather than just maximizing data throughput.
Read the full video transcript
Hey everybody, it's T Tuesday. I hope
you had a good month. Uh, guess what
happened to me on Sunday?
Focus is terrible. Is it Is it This
might be working. Looks a little better.
[ __ ] Hey. Sunday evening. Uh, heading
for the, uh, T Tuesday update. It's a
start. It's been a long ways. All right.
Now we're going to try again with uh
Drake.
Whoa.
Yeah, that's more like it.
Okay, cool. Step by step.
So, that's all brand new. We got tons to
talk about. I have no idea how long this
is going to take, so I'm just going to
go into it. If you're just joining us,
and we have a bunch of new subscribers,
welcome. Thank you. uh uh and I think a
lot of it is because there's building
interest out in the nerd world in the
tens torrent architecture chips machine
systems and so forth. Uh uh and so the
what I've been doing for the last year
or so uh um almost a year since last
summer is trying to uh build pretty much
from bare metal pretty lowlevel uh start
uh to build an implementation of the
Mughal feast machine stocastic cellular
automa to run on the black hole tends to
quiet box which is sitting outside uh uh
blasting off its heat and with the
progress that we've got to report today.
We're going to have a live demo in just
a few minutes. Now, if all goes well, uh
I think I can probably start to say,
you know, probably
we shall see. All right. Uh uh so these
were the goals that I was supposed to
get to. I was really hoping to have uh
all green, but I do not. uh the cleaning
up the code cuz the code's been a
terrible mess. And in fact, the code is
in much huger mess right now cuz I was
fighting bugs, nasty, nasty bugs that I
had no freaking clue uh for an awful
long time until the very last minute.
But yes, I got the new thing that I
talked about last time, the t the tag
the task tag table. I got enough
experience with it that I could start to
figure it out and so forth. Real host
viz running. That's the main event. Pop
yaks. Yeah, it was nice. It was nice.
So, here's what I'm going to try to talk
to you today. I have no idea how much
time it's going to take. I'm going to
try to keep it to 20. We'll see how it
goes. Uh uh. All right. So, the 10's
torrent architecture is a spatial grid
of these little tensics tiles. Each
thing that has a T at the beginning, T0,
T39, that represents a Tensix tile,
which has a significant amount of uh
hardware. In each Tensix tile, there are
five hardware threads. Uh there are 140
Tensix tiles per black hole card, at
least if you have the old T black hole
cards like I do. The new ones uh all
have just 120. And then the box that
I've got has four of these whole things
in one box. And each of those tiles is
connected by two knocks, two network on
chips. NOC knock zero routes uh west and
south and knock one routes east and
north. So you can pick which direction
you want to go depending on your
destination. So months ago I made the
decision that I wanted to cuz the the
tensics tiles, you know, the for what
I'm using them for once again, you know,
we'll see
the um I wanted to to batch up batch
them up together. So I came up with this
regional hub concept where I was going
to take a 3x3 grid of Tensix tiles and
have them work together to all so eight
of them would do little cellular automa
event transitions and one of them would
be the hub that they would all send
things to. And so here is a case where
this northwest uh tile has sent back an
event window which is going to this
little memory up here that remembers
them all. And then the T6 viz challenge
is how do we get them off of the roof of
this building and send them someplace
that we can actually see them. Now uh
there turned out to be a slight problem
with this architecture. Uh uh so what
about say this northeast tile here,
right? Well, so the NOX zero goes west
and south. Well, we don't want to go
west. Uh I'm sorry, it goes east and
south. Uh, and we don't want to go east
because we want to go west and we don't
want to go north. So for the northeast
tile, any way you slice it, whether you
go on knock zero or knock one, you're
going to be routing all the way around
the chip in order to get to the thing
that's right next to you. Uh, so same
thing for the southwest tile. you it's
just, you know, you can do it, but you
have to deliver the packet and then turn
it around and switch it on to the other
knock. So, it costs a lot. Uh um so what
am I going to do about this? Well, uh
this is what I did about it. [laughter]
I went from a 3x3 regional hub to a 2x
two regional hub. So, the problematic
ones, the northwest, the northeast, and
the southwest, they don't come up. So to
get from here to any of these three
event window processors and back is all
on knock zero or on knock one depending
on which way we're going. Now here's a
little puzzle for the folks in the in
the chat room right now. Uh uh you know
how could we get the 3x3 regions back
again without having this problem of
having to send packets all the way
around the world. So this is how it
looks in the uh in the declaration file.
It's a 2x two cell now uh that is filled
with EWP that's event window processors
and then at 1 one which is you know 0 0
up here and one one here uh is a hub
image and off we go. So that's the
architecture that we're working with.
The T6 viz challenge is how do we get
the events once they've happened they're
only stored in these regional hubs all
over the place. There's no screen on
that thing. So, yes, the problem is we
have to take the data off of the 106
tile, off the ones that are being the
hub, and get it to the host somewhere
which has an internet [clears throat]
connection. Uh uh the T6 tiles have no
screens. Unlike our beloved T2 tiles,
the purpose, the name of this channel,
the T2 tile project, was we were
building these. Each one of these has a
screen on it. So once it decided what
things were going to happen, it could
display it right there and we took
pictures of it to figure out what the
entire grid was doing.
So [snorts]
a little more about the architecture.
Each black hole chip, the quiet box got
four of them, uh, uh, has a PCIe tile.
There's two of them, PCIe0 and PCI1. On
the ones I've got, the only one that's
used is PCI0. As far as I can tell, no
black hole chip uses both of them. It
just depends on I don't know what. So
every bite that wants to go from
anywhere on this entire array of 106
tiles has to go through PCIe0.
So for example, here is a little 2x two
hub uh uh that we just like we saw and
in the lower right is the hub part of
it. The other three are the event window
processors and it has to route straight
up to get to the PCIe. So, it's pretty
easy as far as that guy is concerned.
But it's really like this. There's this
incredible hodgepodge of all of these 2x
two regions. Each one of them has a hub
that has to send data a significant pile
of data to the host so that it can be
rendered into an image and sent out to
wherever as well as being saved. So the
plan is, and I'm not going to go through
this, you know, make a 2D spatial list
of the atoms that's changed. So we don't
have to send the whole uh grid, the
whole regional hub grid. We're just
going to send the atoms that have
changed. We're going to compress them.
Uh dedicating an entire one of these
little hearts, of which you have five on
each tile. We're dedicating one on the
hub just to compressing the atom changes
down. The compression is not great, but
it's about 50%. And at this point, I'll
take it. And once they've been
compressed, send them through the knot
along with everything else that's going
there uh to the host. And the host then
has a software thread uh dedicating to
dedicated to decompressing the changes
it's getting from each hub. And it then
figures out well given that it's an atom
of drag given it's an atom of Facebook
atom of fork bomb FB um figure out what
color it should be and draw it directly
on a array uh that represents an image
and then just ship that image regularly
regardless of how many changes manage to
come in. There's no big synchronization
stuff here at all.
So let's uh what button do I want? I've
completely forgotten. Huh? Right. We
want uh that this one. Okay. So, this is
the windows that I've got that are
connected to the black hole. The what I
want to do first is just uh Whoops. Um
All right. There we go. I'm going to
touch the built not new file. Uh uh
which is the trigger that I'm now using.
uh because the to build the system takes
quite a while uh um and then I have to
come and start the thing and then for it
to set up it's it's setting up right now
but it takes a minute and something uh
uh in order to get all the code
downloaded and get everything all
running. So I used to have to uh do the
build and that would take a couple
minutes so I would go away and then I
would have to come back and start it and
that would take a couple of minutes and
I go away and come back and it was
terrible. So I made it so that when the
build se succeeds successfully
it it touches it makes a little file
that this other script is looking for
and whenever the script sees that file
it starts up the program. So, I touched
it here uh uh to get it going. And now
we can go over to the uh Okay,
if all goes well, in maybe 20 seconds,
something like that, the video stream
will pick up. At the moment, it's just
routing within the local network here,
but you know that there's it's it's
internet. It's TCP. Uh um the the live
stream will pick up and we'll see what
we can see.
Uh
it works. Uh there it is. Uh uh okay. So
that is a fork bomb. Uh so what this
current demo is doing is for each of the
little hubs, each of the regions uh in
the thing in the in the first uh in the
upper left in the northwest because
that's what corresponds to black hole
card number zero. Uh let's make it a
little bigger. Uh okay. Uh um and we we
can zoom in on it.
All right. So these things that you see
here that are look like rectangles,
those are fork bombs. The maximum fork
bomb that spread as fast as they can.
The ones that still look like uh blobs,
circles, those are the minimal fork
bombs that just pick one random neighbor
and try to change it to a fork bomb. So
they spread much more slowly, but just
as relentlessly. And you may be able to
see it's a little bit difficult. I try
to get the pointer in, but we'll go back
to it. uh a couple of things that are
just looking like a little sprinkles
like we saw in the opening uh little
clip. Uh those is Dreg and Rez. One of
the earliest earliest uh elements that I
ever explored when developing the uh
mobile fee machine. And now um
uh Whoops. Yow. Uh okay. And so now what
has happened is now the second put me
back in the uh second black hole card
which over here has started initialized.
The third black hole card is
initializing and finally uh they're
going to start picking up over here. And
the only reason it's taking that long is
because this initialization sequence
took a big chunk of time to do black
hole card zero and then it went on to
black hole card one and two and three.
And so these delays in things appearing
are just corresponding to the delays
that are happening there. Uh uh but you
know we can we can zoom in.
Oops. Only if we There we go. Okay. So
just to pick a little bit in the center.
Uh um can I actually do this?
I guess I cannot. Okay. No. Um
the uh there's one popping. So, we've
got the drag, which is the brown
background with the yellow. There's
that's all that or the greenish stuff.
And all those little green sprinkles,
those are actually resource atoms. Rez
the drag. They're very rare and they're
uh they're red. So, you might or may not
might or may not be able to see them.
It's not clear. Let's go back to the uh
full view. Okay. So, at this point, uh,
pretty much all of the tiles, and this
shows you
all the tiles have been seated with
something, either FB1 for the slow
growth, FB4 for the super fast growth,
or drag and res, and they're all just
crumping along.
I've had these things running now for
many hours, and it it's all working. So
again, is this what the cellular automa
is supposed to look like in the end? No.
Uh what's going on here is that each of
those little regions, those 2x two
groups of tensics tiles, uh uh cannot uh
talk to any of their other neighbors.
They have events within it, but the
interhub communication is still missing.
So that's where we are now. Okay. Well,
so
display isolated hubs. This was a long
time coming. I thought we were going to
see this demo three months ago.
We're seeing it today. Uh better late
than never. Okay. So, what's next? I
mean, this has been a 100 days or so uh
trying to do it. So, what's coming up
next is InterHub Communications, and
that has a whole rat's nest of
possibilities. There's at least two
major parts to it. Number one is having
the little uh regions on the same black
hole card on the same black hole chip be
able to share with each other. So
something that was an atom that was born
on one hub might be able to migrate to
another hub and have a exciting life
there or whatever. Uh um but then
there's the next level which is uh a
thing on the entire card from something
in the northwest quadrant that's all on
black hole card one and it might want to
migrate to the east and end up on black
hole card one or it might want to
migrate to the southwest and end up on
black hole card three and that's going
to have to take an entirely different uh
communications path because the knock
can't do that. we have to use these uh
these crazy 800 gigabit per second
cables that are connecting the stuff all
the back that all has to be designed
also just going forward with this I've
got to gain some experience with all
this infrastructure cuz you know there's
a lot of new stuff and that was one of
the reasons why it took so long is that
I really built out a lot of stuff
without
slowing down and getting some experience
with each piece and shining it up a
little bit and so forth cuz once you do
that. Once you actually go through it
and and use it for a while, it it starts
to make more sense in my brain. It's
it's very strange phenomenon. Even
though I wrote the code originally, when
I was originally writing it, I wasn't
exactly sure what I wanted. But once it
starts to work, then in my mind, I say,
"Oh, this does this X does Y." And I
don't think of it in terms of the code.
I think in terms of the tasks that I
think it accomplishes. And that in turn
changes my view of the code. when I look
at it, I say, "Why did I do that? I
don't need to do that at the time. I
didn't know I didn't need to do that."
And so on. And the next big goal that I
really, really want, AER subh. I want to
explain what that is cuz it's well, for
one thing, it's kind of a science of
this or it's a piece of the science
anyway. AER stands for average event
rate. And normally when you have a
cellular automa, typically they're all
synchronous. They all go kachchunk
chunk. So you can talk about the
frequency of updates and it's just like
one number. This thing is updating at 30
frames per second or whatever it is and
that's great but that doesn't apply to
the mule feast machine cuz each
individual event that uh happens in one
of those event processors is
asynchronous. It it doesn't happen at
the same time as all the other ones.
There's all this coordination that goes
on and the things get resolved. So we
can't just say 30 frames per second.
What we actually want to say is how many
events does each site in the cellular
automa each cell of the cellular automa
get on average over the course of a
second and that is the average event
rate. It's the number of events per
second per site on average averaged over
an indefinitely large machine. The
machine getting bigger and bigger and
bigger and bigger. Now in this case the
the T2 tiles that we were working with
they were indefinitely scalable. So you
could actually get an AER indefinitely
scalable number out of them. And what we
got was very very low like 8 millair
something like that like a thousand
seconds for uh the sites to all get one
event something like that. Here we're
talking about AER subh
because we're limiting it to the s the
size of our grid is HD 1920 x 1080 at
least it is in the moment at the moment.
If I need to scale it down I will. All
right. So that's it. And going forward
uh uh so for September 1st that's not
very far away.
I've
try to get myself to clean up uh the
code now that at least the communication
to the hub is running. Have some kind of
inner hub spikes running. I've got
little pieces of that, but I have to
make a decision. And I would like to uh
come up with at least a path towards uh
benchmarking AER HD, get the average
event rate over an HD sized cellular
automa. Maybe expose the live stream to
the net. I don't know. Uh maybe we'll,
you know, just have a secret on the T2
tile Discord saying, you know, go to
this Earl in the middle of the night and
have a lot of fun. So that is it. Uh I
went a little bit long, but not too
badly. Uh uh so
I hope this made some sense to you and
it did not. It helped convince you that
you know
uh uh there is a story here. uh um and
you know the path to best effort is part
of the bigger story uh um that why I'm
doing all this is to try to come up with
uh technology that fits in with living
systems uh better than the technology
the digital deterministic computing
technology that we have now and you know
the AI that's been developed in the last
few years uh is all built on top of this
deterministic
digital hardware and the quiet box is
the same thing. Even though the
technology, the deep learning neural
networks don't really need that and in
fact they kind of cook up different
answers even if you try to hold things
as constant as possible.
So this whole project, the T2 tile
project is let's demonstrate a scalable
kind of computation that would be
different and would not take as much
energy and would not be focused on
maximum gigahertz and pumping tons of
gigabytes of data around at incredibly
high rates, which is what AI is all
about right now. And so instead say, uh
uh we can go slow because everybody's
[clears throat]
going at once. And as long as we can
have little reflex loops that get from,
you know, seeing something to doing
something quickly when necessary, we can
then take big long detours and think
about the beauty of life and so on and
so forth as well. and the T2 tile
project and MFMX, the project to get the
MFM architecture running on the black
hole quiet box from Tensor torrent is a
part of that bigger picture. So, I hope
you'd be interested in coming along for
the ride, you know, once a month. It
doesn't hurt too much. In any event,
thank you for being here. I hope to see
you next time.