Video summary
The video presents a chaotic yet insightful update on the development of the MFMX project, highlighting a deliberate shift in philosophy from fixing every bug to embracing them as features that drive innovation. The presenter explains that after months of intense debugging and encountering numerous issues, he realized that striving for perfect hardware scalability was not the only path forward. Instead, he proposes leveraging "shitty hardware" and inherent software bugs to force the system to adapt, effectively using instability as a mechanism to test robustness. This approach involves creating a new architecture where a central host computer coordinates groups of tiles into "supercells," allowing specific processors to take charge and process events while others stand by, rather than relying on every single site to handle events independently at all times.
To demonstrate this concept, the presenter showcases a live demo featuring a modified version of a "fork bomb" he calls a "time bomb." Unlike traditional fork bombs that cause immediate crashes, these entities are designed with a built-in mutation mechanism where they reproduce and spread across neighboring cells but eventually die out after a certain number of generations. The visual result is a dynamic landscape where areas of activity expand and then vanish in black pulses as the mutants expire, illustrating how the system can handle non-trivial physics even when individual events are not strictly independent. This method allows for high-definition simulations at 1920x1080 resolution by distributing the computational load unevenly but effectively, proving that a starvation-free mechanism can exist even with erratic event distributions.
The core argument of the update is that software correctness is often a myth and that relying on flawless hardware or perfect code distribution is unnecessary if the system can self-correct through flexible design. By allowing the software to carry more of the load and accepting that events might cluster or spread in weird ways, the developers can build a more resilient simulation engine. The presenter admits that many planned features, such as fully implementing the neighbor update protocols after event bursts, are still incomplete, but he feels confident that this iterative, bug-embracing approach is the right direction. Ultimately, the video concludes with a sense of cautious optimism, suggesting that by leaning into these imperfections and letting the "time bombs" run their course, the project can evolve into a more powerful and adaptable simulation platform despite its current messy state.
Read the full video transcript
Hey folks, it's T Tuesday. I've got all
kinds of results. Some that are green,
some that are red. And then the main
event, Interhypes, InterHub Spikes. Is
it red or is it green for most of the
month? I thought it was going to look
like this.
Let's take a look at this.
So, it's putting up a bunch of debug
information, which uh helped me a lot.
Yeah. Um and uh so now we've got uh some
of the tiles, the hubs are using Dragon
Res. Some of them are using this new
thing called time bomb, which is a
variation on fork bomb. We'll talk about
it later. Watch this.
Boom.
Ah.
>> [laughter]
>> Um,
see, I was trying not to do time lapse,
but this is obviously sped up. Um,
this is a fork bomb that's taking over
Dreg and Rez. Uh, um, and it's a weird
fork bomb because it has these little
diebacks. Those black pulses are um
uh places where a bunch of the stuff
died. So
h I'm calling that progress. Uh I felt
like um
you know how am I going to spin this? I
felt like you know I'm not I'm not going
to be able to show anything. And it was
only you know a few hours ago. It was
Tuesday UTC before this thing started
working. So, uh it's it's not fraud at
the polls. Well, maybe it is fraud at
the polls. I'll explain a little bit
going forward. So, now I'm claiming uh I
cleaned up some uh code debris. I made
some more debris, but I think it's a
little bit net cleaner. Uh uh inner hub
spike running. Yes. Uh there's a few
footnotes. [laughter]
Uh uh I did not manage to do this other
stuff and I'll post some of that through
for the uh next time. Uh uh but let's
talk about what we did get to. So what I
want to do today is just dip our toe
into some of the stuff that I was
working on. Uh it was a short month cuz
we started on August 4th and here it is
September 1st. I was going full blast a
lot. Um, and it was just bugs and bugs
and bugs. Uh, and eventually I started
getting to the point of maybe I
shouldn't just fix them. Maybe I
shouldn't like the bugs. And so I want
to talk about that briefly and maybe try
to make a little bit of a bigger picture
uh conclusion out of that or take on it
and then uh have time to actually check
in with a live demo and see what's what.
So this is completely unreadable. This
starts uh August 4th with the last T-
Tuesday update and it ends with August
31 uh uh several hours ago. Um I can't
even mention most of this stuff uh about
the things, but I did pick out a couple
of them. And so, you know, so here was
the thing. We're doing uh we're using
the 10ens torn quietbox uh black hole
quiet box as a demo to try to implement
the movable feast and to get it to run
fast enough that it could do non-trivial
physics. We'll see what that means going
forward um fast enough that you could
look at it and and see things moving.
And that was not true of the T2 tiles
that I made hardware and the software
and everything myself. They were
incredibly incredibly slow. But they
were indefinitely scalable if we know
what that means and they worked. Uh uh
so the question here was how are we
going to go fast? And one of the things
that was happening in the T2 tile is
that for every single event, a little
teeny patch, uh there was a process if
you were near the edge of a tile, there
was a process of trying to get locks and
waiting to see what's going on and then
when you're done sending updates out to
everybody so everybody knew and seen
what it was like. And you did all of
that for one event and then you did it
all all over again for the next event.
And so the idea slow event window
clocking is what if we could do that but
do a whole bunch of events even though
that meant that the individual events
would not be independent of each other.
And I'll talk about that going forward.
And I had this horrible, horrible idea
that is completely anti
indefinite scalability. But that's the
point of this exercise. We're trying to
use the hardware for what it provides.
We're not trying to be indefinitely
scalable. I mean, we're not trying to be
gratuitously
uh uh finitely scalable, but it's not
required that we actually show how to be
able to stomp out more and more 10 storm
quiet boxes and go till the cows come
home. We are doing high def. We're doing
1920 x 1080 sites and we want that to go
fast enough with complex physics that we
can see stuff going on.
So, what if uh we had the host M
computer, the one in the center that
there's only one of. It's x86. It's
nothing fancy. It's just a server. Uh
what if it sent commands out to uh say
what the whole thing is supposed to do?
Like for example, uh every fourth
regional hub should be in charge and do
a bunch of slow event window clocking
and everybody else should stay back. And
then after that the the center would
chase say something else. Now the the
third one in the each little group of 2
by twos would be in charge and so forth.
So the idea was we already have tens
tiles. We in the tens tiles they have
hearts. The tensix tiles are now bunched
up into 2x two groups and each of those
is called a cell at the moment. Um, and
in those 2x two tensics tiles, there are
three event window processors, the
little minions, and then one grid hub
that stores all the stuff and
coordinates all the work of the minions.
And now we have a supercell, which is a
2x two grid of 2x two tensix tiles. And
what we're going to do is we're going to
take for those what is that 16 106 tiles
2 by two by two by two uh um we're going
to pick a leader and let it go poach
into the other space the space of the
neighboring tiles and they will know
because it's being controlled from the
center that they're supposed to stay
back and let someone else do events. And
then when it's it's done, they're g have
to have some way to get the uh the
updated atoms that the leader picked and
did events on even though it wasn't
strictly in its space back to the
followers and then change the leader. So
we we would do that much more slowly.
Take a bunch of events even though that
made stuff come out a little bit weird
in terms of the distribution of events.
Um
I'm going to come back to that uh just
as an example. So, you know, I I spent
quite a chunk of time uh to try to see
that, you know, I'd already had cases
where the individual little processors
in a single Tensics tile were passing
packets back and forth amongst
themselves and using little uh circular
ring buffers to pass control around. But
I was starting to realize that I need a
more general mechanism. I don't need to
just pass packets. I need to pass
signals to say, you know, now you're in
charge of doing whatever it is. So, I
try to, you know, I made a task manager.
I did a whole bunch of stuff that it's
not working and it's very complicated
and I really should have waited. Uh, so
actually what we just saw, what we'll
see again hopefully uh um is uh
barely using the task abstraction. So,
that's one of the things I still do
that. I try to bite off more than I can
chew when I should just do a little bit
and so forth. I'm going to skip the rest
of this stuff. Uh um so uh bugs and
and stuff. So yeah. Um so
the hack that we just saw uh um it's
doing that slow thing. So the idea is
the center tells everybody to stop doing
events. That's new ability to do that.
Didn't have that uh in uh July. Uh um
and once everybody has stopped the
center says okay for each supercell now
it's 0 1 2 3 it's two that's the leader
of the supercell and now it's start and
now we start everybody back up again and
they everybody knows who the super
leaders are and who the followers are
they do tons and tons of events until
the center says stop and so on there's
millions of bugs in it and part of it is
not even implemented stuff was only I
only implemented this stuff hours ago
and like a whole chunk of the finishing
thing where we're supposed to after
we've done a whole bunch of events,
we're supposed to send it back to all
the neighbors. That's not even there.
But what is there is enough to uh get
Interhub starting to happen and what we
see is what we will always see in this
kind of physics that the fork bombs are
taking over from the lesser uh
abilities. So, I got to thinking, what
is the case? What are the actual FM MFM
rules about, you know, is that fair? Is
it am I allowed to do uh a whole bunch
of events with these guys in charge and
then stop and then say, "Okay, now that
guy's in charge and he could do a whole
bunch of events and you can sort of
imagine like a whole army going and then
stopping and going as it gets pushed
back the other way."
So I out of curiosity, I feel guilty
about this, but it's also it's like, you
know, googling oneself. Now it's AIing
oneself. So I I asked the AI
uh what does this this bozo Dave Aley
say about the rules for uh getting event
windows? And it took a little help, but
it it eventually found stuff. Uh and I I
I mean it had been a while. I didn't
actually remember what paper it was in.
a center site holding an active at
random or by any starvationfree
mechanism. Okay, that's very general.
That just means in the long run I have
to eventually get around to giving an
event to every site. Uh and in the
meantime, if I want to do a whole bunch
of them going this way and then a whole
bunch of them going that way, that's
completely fine. That's a starvationfree
mechanism.
So why don't I just use
the looseness the flexibility that the
mobile feast machine architecture gives
us and that's what this is all about. So
any distribution is starvation free what
starvation free why did MFMS not take
advantage of that because I was trying
to be as flat as possible. I was trying
to make every uh uh atom site just about
like as likely to have an event even
though we only uh cared about it on
average. A and
now I'm saying well you know this is
interesting because I'm doing a a
different implementation. I'm doing a
second or this is really like a third or
a fourth or maybe more implementation.
And I'm saying well okay you know having
a flat distribution of probability is
nice. It's but you can't re rely on it.
It's not guaranteed. And if it's not
guaranteed, which means for our code to
be robust, it has to work. It has to
lock in itself. So even if the events
come in some weird way, it'll figure it
out. Well, then why don't we lean on
that? And that was the big picture view.
What if shitty hardware is just better?
What if it's better? Because it forces
software to carry some of the load. It
forces software to say, "Well, I can't
just sit here and demand that uh
hardware does the exact right thing."
What about correctness? Yeah, no, you
can't have correctness. Software
correctness, it's a myth. And therefore,
h um [clears throat]
shitty hardware just kind of drives that
home. And I'm taking this in a sense of
shitty and buggy, right? [laughter] Uh
because you know I've got all these bugs
and you know fork bomb as one example
will roll all over. You give it a tiny
little hole and it'll blast through
there if it can. And so you know can we
shape the power of a fork bomb to handle
variations in hardware? Yes. Yes we can.
I've done it in the MFMs. I'm going to
do it here in MFMX and so on.
So
lean into the
That's my feeling today. Uh all right,
let's take a look um at the demo. All
right,
this is running live right now. Uh I
don't know why the uh that little
display line down there. All right,
there we get rid of it. Good. Um, and
you know, it's can we, if I go here, if
we take a look at the whole thing. If we
zoom in, uh, um, you know, there's all
this text being drawn on top. This is
debugging stuff that I wrote in the last
month. And so, the fact that there's two
sets of numbers and parentheses there
that are both the same number, the same
pair of numbers, uh, shows that I've
actually debugged, uh, what it was. cuz
when I started out they were not the
same numbers because I always get
confused especially when each individual
little um uh hub each individual little
region um has both the stuff in the
center which is like its sites and then
it has cache copies of the neighboring
sites. So these are all these things
that are green are all full of dragonres
physics. This one here uh and this one
over here uh is also drag and res but uh
uh while it was starting up, it starts
with one drag, which makes res and more
uh uh drag and so forth. And if the one
and only drag manages to get itself all
the way to the edge uh making another
drag, it can get stuck. So that that's
what happened. this one and this one the
the one and only drag went to the gutter
and in this uh individual thing the
there are no events that are done at the
typical edge because those are supposed
to be sites that are shared with the
neighbors but we didn't have inner hub
and then these guys this one this one
and that one down there are running this
new modified kind of fork bomb that I
call time bomb and it's the same as a
fork same as a fork just reproduce into
the neighboring cell and increment a
counter which determines the kind of
colors that it gets drawn with. Except
uh time bomb has a small chance of
having a mutation and when it has a
mutation it picks a number which is
number how many generations it can go
how many times it can split and after
that number of generations which gets
passed on to its kids. The the numbers
keep counting down and the kids have one
less and so forth. Uh uh once it once
the count starts to reach zero these
things all die. the mutants all die and
they take out the entire event window
with them. So that's why we're seeing
these areas of blackness appear. It's
when a mutation is dying. So let's do
it. Let's turn on the the InterHub uh uh
process right now. All right. Boom.
There it is. Uh let's go full.
So, it takes a minute or two. Um, and we
Oh, let's turn off the grid. That's uh a
pain. And it's also kind of fun to watch
it go away. Uh cuz I I don't actually
erase the grid. I just let the atoms
update themselves over it. So, here we
go. So, right. So, down in the lower
right there, that uh fork bomb uh that
time bomb is now spreading into the
neighboring thing. Same with the uh top
right. Uh uh the bottom right can't
really tell if it has yet and so forth.
Uh um but now we're off to the races and
this is going to turn into a time bomb
world completely end to end. Uh um over
the course of
I don't know the next half hour, 45
minutes or something like that. I don't
I don't remember. I mean, so I think the
video we watched at the beginning was
about uh three and a half hours, three
hours, something.
So there it is. It's a beginning. It's a
beginning. It's been a a beginning. It's
been a long time coming.
Whoa. Uhoh. What the heck is that doing
there? Uhoh. We are into slides that
we're not supposed to be into. Well, oh,
that's interesting. So, let's see. Uh, I
could go over here. I could riff until I
get to where we were supposed to be. Uh,
um, oh man, this is terrible. Uh, uh, so
I guess I can't even tell you. Uh, so it
was really, it was very, very bad that I
uh um had
gotten so late to start this thing out.
Um,
and wow, where did it go?
Uh, uh, all right. Well, I guess that'll
be different. Uh, we will not have a
list of the things that I'm going to do.
I mean, I suppose I could, uh, try to
imagine. Uh well so what it is is uh we
want to have local inter hub events
going um not be still be could still be
sequenced by phaser from the hub but
they have to right now what's happening
is all those cache updates are going
through the host everything's getting
sucked up to the host and then sent back
again. No, let's get that working
and have lots of fun and so forth. Um,
and
have more time to get the slides in
order. That's it. Uh, uh, thanks for
coming to take a look uh at the progress
such as it is. I feel good about this.
Step by step. That's it.