MRMCD2026 - FACEB13D exploit: Liberating the A100 beast inside Nvidia’s CMP 170HX e-waste

Yeah. Hello. Okay. So, Lenos tech tips tech tips channel made a made a video over the same topic our work yesterday and it received a million almost a million views by this morning. So maybe people decided that this is not interesting at all. Okay. Hello, I'm Patrick K. I come from Helsingi, Finland. I'm currently studying at the Alto University, formerly the Helsing University of Technology trying to do a doctorate on quantum computing. But uh for the last one and a half years I've been very much distracted by the huge development of artificial intelligence. But anyway, so what I am going to talk about today is an Nvidia crypto mining processor and I have one copy here for the for display. Sorry. So, this almost looks like the A 100. The A100 is a very powerful artificial intelligence processor. So powerful but that that the sale of it to China has been forbidden by the by the US government. On the other side you have the [snorts] on Ali Alibaba you have hundreds or thousands or tens of thousands maybe 100 thousand of these and these these have been used for crypto mining until maybe three or four years ago. So in 20 20 21 Nvidia released a set of cryptomining processors. They were these were somehow nerfed versions of their gaming gaming GPUs with limited limited functionality. And later that year they released two two other two more the 188x and the 178x which is this one. And the special thing feature about this um these GPUs is that they have very high they have very fast high bandwidth memory meaning okay and this is important because the Ethereum algorithm it's not uh it doesn't use much compute and the hash rate is dependent on the on the memory bandwidth how fast it can read the Doug or directed async click graph on that is stored in the stored in the VRAM. The VRAMm requirement was quite small. It was a bit over four 5 GB when Ethereum mining ended in 2022. And all of these cards became practically e-waste. They were sold online for under $200. The original price was maybe 500 5,000 $5,000. The price for the A100 was nearly or over $10,000. Okay. So if we compare this that there are several things that have been cut down. First of all, Nvidia limited the number of CUDA cores to only a bit over 4,000 4,400. But most importantly they limited the num amount of accessible accessible memory to one uh 1/4 or in the in the case of this one of the physical memory that was on board the chip and on available on the on the A100. But previously this has this card has uh attracted some interest in in the AI artificial intelligence use and there are some features that are still unlocked in the in the locked original version of the card. All of the tensor cores for doing matrix matrix matrix multiplication have been limited to about one 64th of the original per performance. But the uh scolar uh performance for 16 bit floating points was not not uh limited. And uh it was found out that uh if you did not do fused multiply addition and instead did uh 32bit um compute matrix compute in two separate steps, you could get half of the original original performance. Okay, here's a photo of the PCB. It's almost the same as on on the A 100, Nvidia has left out a few of the power faces. But even though it's limited, it can still do 300 watts with with an BIOS with a certain BIOS version. And they also removed the capacitors for the PCI bus. Instead of 16 lanes, there are only four four lanes available, but you can add capacitors by soldering them on. Okay, so this is a photo of the GA 100 packets and as you can see it has six of these uh high bandwidth memory version two stacks. And what actually actually happens is that when they when they are produced every one of these packages has a full 96 GB of this high bandwidth memory. In the newer version or or in the older version they had at least 40 48 GB. But Nvidia for all most of the cards available uh they some of the uh parts of the memory has have been fused off and we assume that uh the bending of the H uh B M stacks to the silicon interposer is a very risky process. So they add used uh six stacks anyway and count on having at least five stacks worth of available memory and uh there are set of fuses. We can read a fuses and we see that uh the stacks or the half stacks are either defective or disabled. And when we have analyzed several of the several of the dies, what what turns out is that uh most of them are not defected. They have been intentionally disabled by Nvidia. And the way they are disabled is that you don't Nvidia doesn't disable a whole stack. A stack is uh eight silicon dice on top of each other. They don't disable the whole stack, but they they can be disabled at half taxs. Nvidia caller calls them FPBs. So in practice, all of these half stacks are somehow active. Usually only uh two are fully active and uh four are half active half of the me memory available. Okay. So uh many people have been trying to make some use of these cards previously. The first one published blog on this topic was by somebody called Nikon Nikoni and uh he used it for making physic simulation simulations of physics and he noticed that he could get this faster uh 32bit floatingpoint performance by avoiding the FMA instruction. Then there's also a Chinese guy Kangwi who has published a few papers on the topic and he discovered that actually all the cores and the tensor cores are there but somehow the SM streaming multiprocessors they have an issue rate limiter that limits the amount of work to something like 116th of what is physically available on the processor. And then the latest development what happened this summer uh imp uh implemented a bypass which uh by avoided nerfed limited um dot product uh 8bit integral dot product instructions and instead use it used uh 16 bit dot products and uh the benchmark benchmark results for running local LMLs. This is a typical uh typical benchmark uh benchmark is running the llama to 7b or 7 billion parameters model and uh the results were almost 150 tokens in tokens a second in token generation which is about the same as an Nvidia uh ATX for 4090. Okay, this was before. So I started studying this uh this card in March this year and this uh and this started a whole pro uh project or in trying to unlock it. Originally it happened on the on the GitHub uh git github thread related to another batch driver and one of the first uh claims or observations I made is that it's it's impossible to have only 8 GB of memory on on one of these because nobody produced these HBM HBM2 stacks at uh density of 2 GB, they were all at at least eight or 16. So it would it would have been impossible for Nvidia to somehow buy these small capacity cap scap capacity chips for for exclusive use on these cryptomining processors. So I argue that there must be uh 40 or at least 40 GB or 32 GB of memory on the card. And uh the first experiment we did is somebody on the on the discord server not sorry not on GitHub at that time had an engineering sample of the A 100 card and the engineering samples are less limited. They allow for uh using any of the available V bios versions. and uh he did flash the card with the cryptomining v biosis and what happened is that uh the end result what the what was that the memory was limited and the compute was limit limited. So we could uh deduce that most of the limitations on the card are caused by the uh by the firmware the vb bios and not by any physical limitation on the card card itself. So there are basically two kinds of nerves meaning that limitations on the on on performance. Some of them are implemented in the firmware and this is the compute per performance and addressable addressable memory. But there are also some some other limitations that are fused into the silicon die like the number of these streaming multi multiprocessors and the uh number of active active HBM2 stacks and PCIe speed. this well of course uh error current correcting code for the memory and the in NV link interface are also fused off. Okay. So the next thing we figured out is that most of these nerves the limitations are implemented or accessible or readable in the bar zero address space and somehow somebody figured out the names of uh registers in the address space. So we could read these register values and compare them to the uh unlimited unlocked values for uh on a on an A 100 and on other uh amper generation GPUs. And we noticed that most DPUs actually most consumer DPUs are somehow somehow limited in performance. And the crypto mining GPUs especially were limited. And the limitation was were related to specific specific register values. And among the res register values are these uh privilege level masks special registers that limit the access to other registers. So the normal registers can be read but they cannot be modified by the driver or the host any software running on the computer the on the host system unless you are first able to modify the privilege level level mass the PLM registers but this of course is impossible for in any normal piece of piece of software. And the only the only thing that could uh modify the PLMS is firmware running on the multiple different uh uh processor processor cores that I implemented on the silicon die. Okay. And the Okay. So there's a medieval castle there. So actually this is security. It's not securitying anyone. What what what Nvidia has done they have built a corporate uh corporate casual a corporate fortress on the GPU that pre prevents the owner and the user from fully using the fully using the GPU. And we could count that there are at least eight levels of security. Security by obscurity. This was largely broken by a leak a few years ago called Lapsus leak and some many of the Nvidia secret became well not not public but widely known. Then they also have something called stack canaries which is uh well a software method of uh checking that the code has not been modified. They have uh different sec security levels for the processors. They have an immutable boot that can is most likely maybe in the silicon mask itself. They have then they have these falcon processors which have a security co-ress processor that does a advanced encryption system a uh cryptography. They have a special chips on some of the some of the cars that protects the SMI BIOS uh ROM chip. Okay. And then they all the code that they distribute is encrypted with strong strong cry uh encryption and even signed by a private RSI signing signing key. So very difficult to crack anything. But now I will go through how we managed to how we managed to write right registered in uh level high security level L3. Okay. So there's a uh there is a drawing of the Falcon uh controller core on the chip. This comes from an uh earlier crack on the Nvidia or it's actually it's the Nintend which used an Nvidia GPU and the team in 201 17 18 and later they managed to uh bypass the security features of this these uh chips and this uh Nvidia gaming consoles. But the important thing thing is here that they have a secure co-processor in the upper right hand corner which does all all the all the encryption. And there are how how many are there? Eight different falcon cores on the Z A 100 die. Not all of them have this this secure co-processor. Okay. So what we uh started doing is is analyzing the different V vios files that were available on the internet and comparing them and trying to understand that is there anything that can be modified and how do you modify them and uh what parts are encrypted and what kind of encryption is used for each part of this Uh the kind of the plan was that maybe maybe we could take some part of the A100 code and splice it together with the with the code for for the 173 HX and get some uh overcome some of the limit limitations. But uh a few observation we made observations we made. First of all, we noticed that there were to uh many airs signatures on in the VBOS file, [clears throat] but the signatures always came they came with came in pairs and we figured out that one one of the signatures is a production signature and one is for for debugg or test code. meaning that the Va BIOS that was distributed always had two sets of two sets of code. One for production use and one for uh debugging and development. And I also noticed that uh actually the code for the A100 and uh 178X was identical, meaning that uh the same code was signed by an ASA key and uh and well encrypted in ASA and signed by ASI. They were identical. The difference where we some someplace else in the in the VB bios and this is the the memory the DM DM part of the V bios that that is tied to [snorts] each individual uh firmware blob in the in the code and well we never never figured out how these DM sections were actually signed was is there an airsa signature for hold for the whole VB bios and exactly what parts of it it are signed? This is still well still to be fig figured out. Okay. So this is the breakthrough breakthrough. I started analyzing some of the comparing the code and when comparing two different versions of of the V bios one one earlier and one later where the where some code had been undergone some some changes. I noticed that at first they started the same the encrypted versions of them and then there was 16 [snorts] 16 bytes of difference and then they were might be the same again and then 16 bytes difference and then again until they were became totally different and uh this this is an indication that encryption It's not the uh AES CBS CBC which is cipher blockchaining meaning but the but the weaker version of the encryption meaning that every 16 byt section of code or whatever produces the same the same um clear text always produces the same cipher text. And then the most interesting part is that each uh firmware section uh had a padding at the end to make it an even um number of bytes. And this by this padding was all zeros and it always produces produced the same cipher text. And uh cipher text we we noticed was something something 717D149 94. This uh repeated many times over and over and in many parts of the uh parts of the parts of the code and it turned out that this in fact is the debug uh debug code. there was another uh other other recurring recuring sequence in the in the production code. Anyway, we had a clear text cipher text pair and now the problem is to challenge is to find out what key would encrypt 000 padding to this this number value. And the thing that makes this easier is that we know that this is this is the debug code. Okay. So if we ask Google that what kind of uh test and debug sign uh keys do they use in the industry hardware and software industry. Google says that 00 0 is quite popular and 1 0 1 0 which is in fact a a if you look at it as a text decimal and then there's one one more string that is quite popular which is 0 1 2 3 a b c d okay so what does nvidia use for their for their testing keys we looked at an Nvidia example and this is They kind of demonstrate that you could use something like this. And again we find that this AB CD string in different per permutations. This code happens to be this key happens to have be 256 uh bits but we only the AES uh encryption on on these cards only uses 128. So we don't need that many num numbers. Maybe maybe we could select some of the numbers from this some of the numbers of this uh from this uh from this example. And then another observation of the of the Nvidia keys that the secrets or the keys there are 256 different different keys for uh production and and also for debug but it's unlikely that they had fuses or fuses for each one of these secret keys. uh it would take uh uh 65,000 different different fuses or if they have implemented in the in the silicon mask would take a lot lot of silicon real estate real estate. So we assume that they only have a 120 bit uh secret and then I they add an 8 bit uh key number someplace in the someplace in into the key. And we we have no idea what would be the human readable form for these keys. But we could assume that uh they have they have something in the beginning which is secret and then they add uh hexadesimal uh key number to the end. If it was key 37 you would add add hexadimal 22 and five into the end of the end of the key. Okay. But uh this is one possible uh possible example of the human readable form for the key form of the key. We don't know how it's actually implemented. Is it big Indian and in what order do they use what order does Nvidia use the 32bit long words? Anyway, I wrote the piece of code here that we take what NVDIA published on their on their website. There are eight numbers. We choose four four of these and then we decide that to in which order to use these. They are used as the round keys in the in the ring algorithm. And uh of course we have to decide figure out are they big Indian and or little Indian or what? Well if you want to try you have to figure figure that out yourself. So now we have the debug key for the uh for Nvidia's Falcon microode. So what does it give us? Is there any is there any use for this key? If somebody can guess the right hand or gets one of these mugs, what would it what would be what would the debug key uh enable us to do? Can you guess? Why do we go after the deboki? Does it allow us to do do anything useful? >> Yes. Wasn't he flipping some relationship? >> If the card was in the debug mode, if if it was an in engineering sample that had that did not have the uh debug mode, engineering sample fused off, then we could use the debug mode to do do useful stuff. >> Yes, you can go reverse the framework. How? >> By decryting it. >> Yes, but we only get the deu de code. Is it useful for anything? >> It'll probably be similar to the production code. >> Exactly. You get the price. And then Okay. Yes. So there are two two sets of encrypted code. One of well heavy secure encryption. One for production and for one for debug debugging except that the debug key is something like a b cde e f. So what we get out of this that we can Nvidia always distributed debug versions of their of their uh software. So we can look at the debug version of the code and it's identical to the identical to the production version. Okay. So this the production is just random numbers. Okay. Okay. Okay. So now the interesting stuff there was there is a bug in a certain piece of firmware. Uh not actually not firmware but uh but this Falcon microode or Nvidia calls it u code. Um Nvidia has uh implemented part of the part of the drivers the resource management uh implementing a risk five chip or uh core on the G100 chip but it doesn't have the C uh the the security security core processor instead it has to use the Falcon core uh to do the uh do the verification of the code and this uh piece of code the booter load is distributed as part of the part of the firmware. No, sorry, sorry, not the firmware but part of the Nvidia opensource drivers. So, and it happens to have a bug. It doesn't check uh the length of the signature. Meaning that if we give it an oversized uh oversized uh signature, it will overflow all of the all of the memory including the stack and the stack canary. So this we gives us the we can uh implement a payload that overrides uh the overrides the stack and this enables us to do uh a rock chain. Okay. So originally this work happened in the uh discord chain. We started uh uh sorry originally happened on on a GitHub uh GitHub uh chain of discussion and then we started a discord chain. There were at at one time 200 people people do doing this. But when we finally found figured figured out how to do the rob chain, we kind of split off into a smaller private group and John Prive was one of the one of the Persian people and he and others implemented an an unlocker uh a driver patch. It was all implemented by modifying the drivers and uh what the end end result of this work was that uh two driver modification guides a published paper this is in this is available on internet and an email of to video warning of the vulnerability but most of these people here were North Americans Americans uh they did not want to uh meet Nvidia's lawyers. So they decided that they will not publish anything. So we they nuked the uh discord servers and decided that everything will remain secret and never never will be published. Somebody else has to do has to do the work again. I was not heavily involved in this uh the rope development at this time. I did not have a test system. So, I was kind of a bit bit outside, but that was good because I never really really looked at the driver modification guides. But what we instead decided when this was uh made uh nuked, we decided or some other people, not these people, I and some other people decided that we will do a reimple re-implement implementation of this but do it in a clean room. meaning that um in a clean room meaning that we will avoid using any of secrets and only uh a material available publicly on the internet. So we started from from from scratch and at first the first challenge was extracting the extracting the code for this bootter load from the Nvidia public uh open source drivers. Luckily uh Novo drivers had a special tool for this. Then we decrypted it using the deboo key. Uh disassembled it. And in the final version uh what we have have on the left uh on the right side is uh there was an a artificial intelligence intelligence that gave each function a name and uh okay. So the [snorts] the Falcon core has only 64 uh kilobytes of me of memory a limited address space and it is to write to the right to the bar zero register pace and there's a special instruction for you do doing this EO write something and when we had the uh uh disassembled code. We looked for this and found that okay there's at least one function which writes to the about bar bar zero added space and this is a typical typical kind of a typical function this one it has sets the stack canary and before it enter uh exit it's well it sets the stack canary checks the uh canary if if it hasn't been changed And if it if it fails the checks, it goes to an inf infinite loop. There's also some member error handling. Okay, but of course the payload over already overrote the stack canary. So it doesn't the the canary check is in inefficient. Okay. So uh this is what I wrote just manually the first version of the rob chain [sighs] in a certain position in in the stack. we've uh write the uh a canary value that is duplicated in the global global address and uh the other values here are the uh register values that are popped from the stack and uh then there is an address for the next function next address to return it return to and uh we used an address 10B9 which happens to go inside an other function. They register right indirect and uh the G gadget we jump into the into the middle of the code so that we don't have to provide the parameters and register 10 and 10 and 11 and instead can use zero and one. Okay, I need to go ahead. Okay, so this was the first version of the clean room unlocker. We published this I think it was July 12th. This was the first public publicly available unlocker version and uh it was published on GitHub and immediately the price of these cards started rising from $200 to $500 in a few days. This the first version only uh allow only unlock the compute performance and uh but did not unlock the memory and the problem is that I didn't know the know the address of the of the PLM for the memory unlock. We could write we could write the new me memory values into the bar bar zero address space but uh our code was so so crappy that it needed a uh function level reset and uh this would overwrite or restore all the other values except the the PLMS but the PLM for the compute uh this was known so it would uh survive the survive the the reset. And then the uh the host could write these registers that limit the compute from with the uh with no privileges as the uh privilege level privilege level mask had made them writable from the host. [sighs] [gasps] Okay. The the difference great difference between the uh clean room unlocker and the other other other unlocker is that the original unlocker was that this did not include any driver modifications. We loaded the payload directly from a Puton script and it could you use any any driver versions even even Windows or and even a version that uh does this from from an F unified extendable firmware uh piece of code. Okay, the next version we try to make our uh rock chain reentrant meaning that meaning that we could uh uh the first version was crappy and it prevented locked the locked the falcon core the sec 2 falcon core and the boots could not load without the without the reset. we try to make it uh clean up after itself. Anyway, in this one of the interesting things that we found is that there's if you if you jump to a misaligned address which is not an instruction but in middle of an instruction, you could find find a uh something that translated to a mop add return with a with a value minus 4,000. And this would grow the stack by 4,000 bytes, giving us more space for doing our uh doing our whatever harm or liberation work we wanted to do. Okay. But uh at this point something something stupid happened. The Chinese Okay. There was much talk on the Russian telegram channels about some secret bios and and stuff like that. Nobody believed that there was anything coming and they were quite a surprise that when we we first published our first unlocker. But then 3 days later there was this uh link to some Chinese unlock which was a zip file of 500 megabytes including driver sources and and um in including driver sources and the blob of the uh of the uh compiled driver. But when you did the diff on the driver sources, it turned out to be the original uh proof of concept that was that we did in the in the original original group and this has kind of taken over taken over the the market for unlockers. Unfortunately, it relies heavily on on driver modifications. Okay. So, [sighs] you could run this unlocker on anything from touring and amper in including other uh other crypto mining cards. This has has already been done. Okay. So, let's see. The work was well what I I was doing in a clean room. I did not use any artificial intelligence and did most of the coding by hand. But there were others there who were heavily using artificial intelligence Quinn and FA fable models. But basically the med the message is that it could it could have been done without artificial intelligence. But then on other on the other hand uh some people were heavily using it and after this Chinese unlocker came out. So people kind of forgot about the the clean room ethics and uh started analyzing the lapsus leak with artificial intelligence and okay there's a new new discord very private discord server discord server and some of the one of the members had has implemented a bot using a different uh different AI models let's call it red bot And it's interesting that you can ask questions that how do how does this thing work and it gives gives very good answers about every about everything including work that is still under progress and has not been published but if you look closely you might uh you might find might find something interesting there. Okay. So one more thing, the core team was US and Russia. Very good friendship between us and some Americans. Europeans were kind of left out. And uh special thanks to one one guy for Indi Indonesia who was the first one to actually have it working have have the system working. uh Claudia and Quen Quen uh different Quen 27 and 35 bits models. Somebody uh uh somebody in Russia running it locally was the was the one who wrote the uh Putin script to load the models load the payload to the Falcon core. Yeah. and the RTX 390 3090 cannot go into 80 GB. Okay. The price history I don't know it's maybe $3,000 yesterday 5,000 tomorrow after Linux tech tip tech tips published their video yesterday. Okay. Message to Nvidia do not create e-waste. It will only motivate hackers and give them the hardware resources to trash your security. Okay, so there's only only some references and questions. Yeah, if somebody wants to have a look at the Look at the GPU here. So, okay. Yes. So, HP memory was cheap at some point. >> [clears throat] [snorts] >> Maybe it was not not not that cheap. But Nvidia was the prices for these things were well quite extra extraordinary. But of course 80 GB or 64 GB of memory today is well 10 times the price that it was that it was last year. Okay. So if no more questions I will try to make the slides available somehow. Unfortunately I could not load them to the event website. Okay. Thank you.