Video summary
The speaker introduces the project as their first reverse engineering endeavor, focusing on the 2002 first-person shooter *Nightfire* for the PlayStation 2. Although the speaker's primary career involves hardware and electronics design rather than game development, they were motivated to undertake this project by a desire to understand how games from that era functioned, particularly given the nostalgia associated with the title. The initiative was not a solitary effort but a collaborative community project involving a Discord group, highlighting the necessity of teamwork for such complex tasks. The goal was to extract assets and understand the engine structure to enable modding, level creation for virtual reality environments, and speedrunning analysis, all while appreciating the unique constraints of early 2000s consoles like the PS2's limited RAM and specialized co-processors that made development feel akin to embedded systems work.
To achieve these goals, the team utilized Ghidra, a reverse engineering tool originally developed by the NSA, to analyze the game's binary files. A significant challenge was that the game utilized two different engines for its action and driving segments, which were seamlessly switched during loading screens. Fortunately, the developers had not stripped symbol information from the executable, allowing the team to map function names and structures effectively. By comparing the PS2 version with a GameCube version of a similar title in the series where symbols were preserved, they could transfer knowledge across different architectures. This process revealed that assets were heavily optimized for specific hardware, often baked directly into binary formats rather than using standard files, requiring custom Python tools to extract level data, audio, and models into modern software like Blender.
Once the internal structure was understood, the team explored methods to run the game on modern hardware, comparing emulators like PCSX2 with a technique called high-level emulation. This approach involves running the original object code on a compatible processor, such as an x86 CPU, while injecting custom code to handle platform-specific APIs like DirectX or audio subsystems. The speaker demonstrated how this injection allows users to modify game behavior, add logging, or even replace entire graphical and audio components without rewriting the source code. This method proved particularly useful for overcoming limitations of emulators, such as compressed audio on GameCube discs or missing features on the PS2 due to memory constraints, effectively allowing the game to run with higher fidelity or modified gameplay on contemporary PCs using tools like Wine and CXBX Reloaded.
The presentation concluded with a deep dive into specific bugs found within the game's code, illustrating the intricate nature of reverse engineering. The team discovered a logic error in a key generation routine that had gone unnoticed for over two decades, as well as issues where placeholder values caused out-of-bounds memory reads that behaved differently depending on the memory mapping of the host system. These findings underscored that while the project was legally complex due to intellectual property concerns and ethically nuanced regarding the use of AI tools, it offered invaluable insights into game architecture. The speaker emphasized that such a massive undertaking, involving hundreds of thousands of lines of code, is impossible for a single individual and serves as an invitation for others to join the community to continue preserving and understanding these classic titles.
Read the full video transcript
[applause]
>> Hello everyone.
So, I'm start off with a bit of context
about myself. Um my day job actually has
nothing to do with games or writing
games. I generally design and build
hardware and electronics.
So, I do stuff from single units to
100,000.
Um so, this is actually my first time
doing a reverse engineering project. Um
although I do have some background in C,
which is what uh at least half of the
game was written in.
Um so, also for context, I'm not solo on
this project. There's a Discord
community with a whole bunch of people
in it. So, uh when I say I and we, that
is who I'm referring to. It's not just
me.
Um that's also if you're considering
your own projects, finding communities
and finding people to work with is
absolutely invaluable. This isn't
something that you could really be solo.
So, um I imagine a few of you recognize
the name already, but for context, it's
a first-person shooter that was released
in 2002. So, right around the period
that a lot of people started playing
games of that nature.
Uh so, it's very nostalgic for a lot of
people, myself included.
I have a lot of hours in Skyrail. Um
I'm sure a lot of you do as well.
But, there are actually three three
games with the same name. So, there is
the console version, which is definitely
the best version. I'm going to just say
that now.
Um there was also a PC version, which
was written by Gearbox, which used a a
version of the Gold Source engine as
well. Uh and there was also a Gameboy
Advance version. I'm going to be talking
about the console version specifically.
So, why? Uh nostalgia is clearly a thing
that a lot of people have, but this was
also something I started around the
start of the pandemic when a lot of
people suddenly had a lot of free time.
Um and a lot of other projects came out
of at a similar time as well. So, if
anyone knows the Jack and Daxter series,
there's Open Goal, which is a complete
source um remake of all three Jack and
Daxter games. If you haven't seen that,
really worth checking out as well.
Um and there are people who've done the
same with the GTA games as well. So,
full source rewrites of that as well.
Um it's really interesting problem. So,
I've learned a whole bunch of skills
along the way. Um
going from never reverse engineering
anything to being like comfortable with
reverse engineering, understanding the
principles, understanding what goes
wrong, what well, what goes right when
the compiler is doing it, but annoying
when you then try and look at what the
compiler has outputted.
Um there are other benefits as well. So,
this is how modding can be done um on
games of this era. Um
it's much harder when you don't have
access to the source code um and when it
uses a completely fresh engine, which is
what a lot of the game uses.
Um it also allows us to extract
resources. So,
um it's incredibly popular to create uh
video game levels and put them into VR
chats. I don't know if anyone uses that
as well, but um that's been a thing that
a lot of other people have actually done
with different games and this just makes
it possible to pull out all of the
assets
and put them in places they were never
meant to be.
Um it's also weird to look at the
consoles of that era because they are
pretty resource constrained. So, the PS2
is 32 megs of RAM, the Xbox is 64 megs,
and it's one of the last generations
that allowed you to use 100% of the
hardware. So, it actually feels
surprisingly close to embedded
development, which is what my my day job
really is. So, um some of the techniques
there are surprisingly more transferable
than I thought they'd be.
Um it's also really useful for
speedrunners as well. People are still
speedrunning the game 24 years later.
So,
um in fact, within the last year, people
discovered a glitch they just didn't
know about before.
One of the levels you can just drive
through the wall in a particular spot
and cut out about 30 seconds worth of
time. So, yeah, things like that should
be possible when you understand the
collision data, which we still haven't
done yet. There might be other holes in
the level somewhere or other glitches of
that nature.
But, that's a bit of context about why
you might want to do this kind of thing.
But, where do you actually get started
on a project?
In my case, it was getting the disc into
a computer and looking at the files that
are actually on the disc. So, we can see
there are two interesting elf files
there. There's action and driving. So,
one of the first things that you realize
is there's not one game engine, there
are two completely different game
engines running this thing, and they
just hand off between each other in the
loading screens.
We also see that there's one big file
there, right? files.bin is a single
package containing every single asset
there.
But, yeah, we have no idea how to
interpret that data yet.
So, the next step that I took was get it
into Ghidra.
If anyone's not aware of it, Ghidra is a
reverse engineering tool that was
released by the NSA for entirely
non-nefarious purposes, I'm sure. But,
yeah, it's really good. There are
plugins for pretty much any platform
under the sun, including PS2, Xbox,
GameCube. Um, there are plugins, you
drop them in, you drop in the
executable, and you get whatever
whatever the compiler outputted.
Um, in this case, we were actually
really lucky. The
The developers of the game didn't remove
the symbols, so we can see every
function name in the game.
That gives us a lot of context about
what each function might be doing and
how the overall structure works.
Um, at least for the action engine. For
the driving engine, they did strip out
the symbols, but in a previous version
in Agent Under Fire, which is another
game in the same series with very
similar context, they didn't strip out
the symbols from the GameCube version.
Um and Ghidra actually has some really
powerful tools that takes the overall
look of a function, so what um signature
it has, how it is called by other
functions, and it builds up a tree, and
it can compare the two side by side and
map likely this function here is the
same as this function here in the other
thing. And that even works across
different architectures as well. So, you
can take the GameCube Agent on the Fire
uh
put it on top of the NightFire PS2
version, and just transfer the two
symbols across pretty effectively.
It's not perfect, but it's kind of
mind-blowing that it works as well as it
does.
Um so, with that, we got a reasonable
understanding of how the engine was
structured, um what the data looks like,
um
and we were able to write some Python
tools that would allow us to extract a
bunch of the assets. So,
um yeah, we found what within that
files.bin, what the levels were,
how they were identified, um
and everything that goes into those
levels as well.
Um the other thing to note is that
because they were very performance
constrained, they like hyper-optimized
for every platform. So, the GameCube
version has textures that are in a very
GameCube specific format, um same with
the audio playback. It's not like the
engine is receiving a a common file and
interpreting it. They are baked into
exactly what the platform and hardware
is capable of doing.
Um same with the PS2 models as well, for
example.
So, the PS2 has a co-processor that
takes
uh a bunch of data, but also tells you
how to mangle and draw that data, um
and that's done by a co-processor, but
the instructions for that co-processor
are actually mixed in with the model
data. So, you get a binary blob that is
just handed over in one go, um
and actually getting the geometry out of
that is a bit fiddly, but it's very
doable.
So, at the end of it, we have full level
data extracted into Blender. Um
Yeah, this works for PS2. Uh it works
for all of the levels and it also works
for all of the assets that are used to
make up the game as well.
The only thing that we haven't got
working at this point in terms of
extracting models is stuff that uses
skinning and um
uh face uh face blending as well. But
all of the other all of the rigid stuff
is extracted now.
It also means um when we've got the the
executable and we understand what all
the functions are, we can start
extracting the embedded stats. So here's
some of the audio data, but we've got
the same for for weapons, for everything
else that goes into actually making the
gameplay.
Um and we've also been able to extract
all of the audio as well in original
format.
Um so one of the other things you might
want to do with these projects is bring
up a game that was, you know, written 20
years ago for hardware that is starting
to fail and get it running on modern
stuff. If that's your only goal,
emulators work really well. So PCSX2,
Dolphin all run the game perfectly. Um
but there are limitations of those two
platforms. So the GameCube audio quality
has been crushed to fit on the much
smaller discs. So I think a GameCube
disc is 1 and 1/2 gigs maybe. Um they're
just physically smaller. They don't have
the room for all of the full quality
assets.
Um same with PCSX2. Um
the game looks pretty good on that
platform, but yeah, some of the textures
are not as high quality as on the Xbox
version as well. So the canonical best
version of the game I'd say would be the
Xbox version.
Um there are also other limitations of
the platform. So PS2 32 megs of RAM and
that they had to cut some features from
the multiplayer to get it running. Uh
the Xbox had I think twice that. So um
that was the best version to play
overall if you want. I think it was
14 bots or 16 bots.
Um
but yeah, um the Xbox version is the one
that you really want. Um and the Xbox is
basically just an old PC. Um,
it was cut down a bit in terms of the
operating system, but functionally it's
an x86 processor in a box, and
basically that means that the object
code that the game is will run on on any
other uh x86 processor.
Um, with some compatibility stuff in the
in between the two.
So, you need to map from the Xbox
functions, which are similar but not
exactly identical to DirectX. Um, but it
turns out someone's already done that.
Um, in CXBX Reloaded here.
So, that this process of taking code
that was written for one platform, for
one set of APIs, and running it on
something similar but not identical is
called high-level emulation.
Um, and surprisingly you can then stack
that with another emulation layer, which
is what Wine kind of is, but except the
name tells you it's not an emulator. Um,
but that's debatable, that's semantics.
Um, so yeah, um, you can play the game
on Linux, on a modern PC, through Wine
and through CXBX Reloaded if you want
to.
Uh, the compatibility of that process is
a bit janky.
Um,
but that leaves us with another thing
that you can do is if you're running it
on a modern PC, you can inject your own
code into that process as well.
Um, Windows just lets you do this as a
thing.
So, if you want to work out how the game
is working, you can take a function, you
can write over the top of that function,
you can jump into your own code at that
point instead.
Um, you can then essentially do what you
like. You you own that process, you own
that function. So, you can make it do
nothing at all, see what it was doing in
the original context,
or you can write your own version with
logging and inject it into the game.
Um, in in the future you could also
inject the entire graphics subsystem or
the audio subsystem. You could replace
the whole thing, so you don't have to
have that compatibility layer anymore.
You can effectively swap out the um the
front end of the game, so to speak,
without needing to rewrite the whole
thing from source.
But, you could also repeat that process
for every single function in the game,
and eventually have a full copy of the
code that you've been able to test
against the original code or against the
rest of your code.
And eventually, with enough effort, you
could end up with a complete copy of the
game.
Testing at every single step along the
way.
Uh and that isn't a concept that I came
up with. This was first something um
documented by Mark KC. I don't know if
anyone has heard of him. Yep. Few people
there.
Um so, yeah, that technique is something
that he demonstrated and we've built on
top of.
But now, let's have a a bit of trivia.
Here's a snippet of code. I will tell
you that there is a bug in there
somewhere.
I will stand out of the way and say,
"Show us if you can spot the bug, given
that this is meant to generate key
codes."
Anyone seeing it?
Well, you are all Oh, go on.
>> The loop you've got forward is
reinitialising it.
>> It is initialised. Sorry, I've just cut
that part of the snippet off. It is
initialised to zero. Sorry, that was my
bad. Um
there is a bug in there, but you are all
thinking exactly the same way the game
developers did.
I will point you
there.
Does anyone see it now?
Yeah. So, for context, if you're trying
to generate numbers between 0000 and
9999, you've got to do modulo 10, not
modulo 9.
And that bug has been undiscovered for
22 years. So,
there we go. I shared that on the
Discord.
And yeah, people agreed.
Until I got that message.
So,
it turns out
>> [laughter]
>> it is actually
>> There we go.
So, what's actually going on there is
surprisingly, there is another bug.
But, yeah, this was my my thought
process when seeing that, and it took me
quite a long time to work it out in the
end. Um
but it turns out the scrambling
animation that is being done by that
tool is just generating a random digit
and displaying it for a few frames.
Um and it only locks that digit in on
the next time the animation process is
run.
But, when you get the code completed,
that last the the the scrambling process
stops. The animation just remains on the
last frame, and it never runs that last
step of updating that last digit. So,
the actual code was not ending in nine.
It was just that happened to be the last
random digit that was scrambled. So,
the code was correct, but there was
another bug that compounded that.
Um so, yeah, there are some other bugs
in the game as well. So, quite a lot of
people thought that the GameCube version
didn't have some effects for performance
reasons. It turns out those assets
actually do exist. They're just pointing
the wrong way, and the GameCube just
culls them out.
Um there's another bug as well, which
was even more confusing. So, I was
re-implementing part of the the menu,
and it turns out they used a placeholder
of I think it's 999 or -999 as a
placeholder for an empty value.
So,
when the original code did that, it was
reading out side of the bounds of an
array, and because it owns the entire
processor, and that was still a valid
bit of memory, it just read it anyway.
Um
but, when I injected a new version, that
new version was in a new DLL, which had
different memory um mapping. And so, it
ended up reading out of the bounds of
the new array, and crashing, even though
it was a completely faithful
re-implementation of the original.
Um so, that took a while. There were
lots of head scratchers with this.
Um but, yeah, like I said at the
beginning, this is not something that I
really think is possible solo.
Um, it's probably half a million lines
of code.
And it's taken us, you know, 3 years on
and off to get 7% of those rewritten.
So, not something you can do solo, for
sure.
But that's partly why I'm giving this
talk is to show people how this goes. I
want to make sure more people aware of
it, and I want to make sure that people
know how they could get involved if they
are interested.
Um, obviously there is a legal question
here. So, the game is still EA's IP,
even though the actual rights to that
are uncertain. So, it's not clear who's
allowed to distribute it, if anyone.
It's not clear whether the Bond
trademark is still
permitted to be distributed in this
context. Um, so we don't think there's
any commercial value to
uh, to the IP anymore, but it is still
their IP. So, there's a risk that if we
do open source it, it would be uh,
seized or uh,
What's the word? Cease and desisted.
Um, so yeah, I've talked a little bit
about this, but um, yeah, we think it's
not possible to do.
And there's the double-edged sword of AI
here as well. So,
in some context it could be useful. Um,
it does definitely work identifying
mangled functions. It can very quickly
take a snippet of code,
say this looks similar to some other
code.
But it does feel like you're de-
defeating the point a little bit of a
passion project. Um, inspired by humans,
inspired by nostalgia, to then farm your
hobby off to an AI, just feels like
you're really defeating the point there.
So,
uh, I'm kind of in two minds about that
one.
Um, so yeah, there's a a whole community
of us. If anyone wants to join, I'll
leave this up at the end, but
I will switch over now to a quick demo
and actually show that in action.
>> Some of you were at Dark Room on Friday
when we first
>> 45 minutes later
>> So it does take a while. This is a
terrible laptop.
Still doing something. So these are all
logging that has been injected into the
game as well. So we can see what the
menu system is doing.
By the end of the talk
>> [music]
[music]
>> This is all fixed resolution.
But this is modified.
And when it
>> You're on a fool's errand, Mr. Bond.
I've just won.
I'll credit you with persistence, Mr.
Bond. Persistence and failure.
There are NATO launch sites on Earth
which could challenge the arsenal on my
island.
Those bases will be incinerated
by turning these defensive missiles into
massive defense.
>> So yeah, thank you very much. If
anyone's got any questions
>> [applause]
>> So two things. If anyone wants to join
the Discord and find out more, then the
QR code will appear on screen in a
second. And if anyone's got any
questions, I'm going to be going over to
the QR question and A and answers tent.
So yeah, thank you very much.
>> [applause]