Confidential Computing with OpenBSD - Hans-Jörg Höxer - EuroBSDCon 2024

okay thank you very much for the kind introduction I to be honest it's the first time in years that I'm doing a speak a talk like this so bear with me when I'm a little bit nervous maybe and if you don't understand me let me know if I can improve it so what I want to talk about today is confidential Computing with open BSD and maybe yesterday I changed my mind a little bit and maybe we are talking about uh confidential Computing with wmd actually but let me introduce myself shortly uh in the mid 2000s I have been actively developing with and for open BSD mainly in the IPC area IPC CTL I think was written by me Isa kbd I was in involved in stuff like this uh my first experience with openbsd is actually openbsd do5 I think this was 19 96 something like this uh and since then I'm working for a DRM company Genoa uh that is mainly building firewalls and VPN applications based on open BSD so my daily work includes mainly developing on open BSD so confidential Compu what is this all about it's to me it was a buzz word to be honest and some time ago maybe last year beginning of last year at our company at my day job this the term confidential Computing keeps kept popping up people talking about it so I got interested and was wondering what what they are talking about and I was hearing some claims like uh from customers that they want to work with confidential confidential data in a public Cloud so public Cloud what does this mean so to me this means mainly something with we VMS virtal machines and the problem that these people seem to have is that they want to work with sensitive data in an untrusted environment you have a public cloud provider and you sort of have to trust the cloud provider and then on the other hand there are people that provide VM service to customers to or to to to to projects to host their project on a virtual machine and and and these people have a similar problem if you want to use your SSH private key in that virtual machine you sort of have to trust the hypervisor you have to trust your your VM hosting environment and obviously in that area there's a supposed solution that is called confidential computer Computing and some slogan I read some time ago was something like this turn public Cloud into a private Cloud so this is a bold statement and I started thinking about this what do they mean what what does this mean what do they want actually to achieve so let's look at an an untrusted environment everybody think I think knows this we have our Hardware we have an operating system running on top of our hardware and I do we see the mouse pointer yes here it is and we have some userland process running on top of the kernel and we all know sort of the kernel has all the Privileges the kernel controls the resources including the memory used by the userland application therefore we have to trust the kernel because the kernel can Peak into our userland application and similar the kernel provides some isolation between other userland processes so we sort of hope we we we should hope that we can be sure that one process is isolate from the isolated from the other process but nonetheless both both processes have to trust the kernel and when things go SE severely wrong and the kernel has a bug and evil application of course can trick the kernel into exposing information of the other process including the memory of the process including for example if this would be an SSH agent holding a private key and if I can trick due to Bucks the kernel into exposing these private Keys then the keys are gone now with virtualization we have sort of a similar situation now we have our Hardware on top of the hardware a kernel is running a hypervisor is running and that is providing an environment where we can run uh our virtual machine and now the the situ situation is quite similar the hypervisor is privileged he controls all the hardware he controls all the resources of the machine and the hypervisor can Peak into the memory into the uh memory of the virtual machines running on top of the hyper again the hypervisor provides some isolation between virtual machines so they are separated similar like the processes and as we learned yesterday uh open BSD and vmd using uh privilege separation to really make the chances really low that a bug in the hypervisor might allow uh uh does not allow or reduces the risks of some guest exploiting bucks in the hypervisor to gain information about some other guest so this would be the one scenario that WMS explo exploit each other and the other thing is still we have to trust the hypervisor to not leak information about the virtual machines and now in in these entrusted en untrusted environments confidential Computing seems to provide the claim that it's possible to uh to protect the address uh the memory used by the virtual machines by for example encryption and therefore that the hypervisor does not need to be fully trusted anymore because the hypervisor now only sees encrypted memory of the guest and each guest has its own encryption keys so even if one guest is able to look into whatever reason gains U access to a page of an other guest he only will see the encrypted State the encrypted memory of that other guest and this obviously needs um so this needs some um support oops oops this needs some support by the hardware where because to get the hypervisor out of the loop that the hypervisor is not trusted all this encryption of the memory has to be done somehow has to be done uh done somehow by the hardware so to summarize confidential Computing makes these claims that it provides techniques um to keep data confidential uh oh this is a difficult world to uh difficult word to keep data confidential I think that's what I want to say here and that it also keeps uh the the Integrity of data the data cannot be modified and it also uh claims uh that it provide me provides mechanisms to guaranteed uh some Integrity of code code that is working on the data shall not be modified somehow by some attacker by the hypervisor for example so there are several levels possible levels of isolation involved uh the the the most fine grained level might be something like isolation on on function or Library level a little bit more coar grained it is uh isolation on the level of of a full application and what we are looking or what I started to look into is this most corar grain level of isolation isolation of a full virtual machine so for example um with the application isolation you might know that the signal app on some iOS phones and Android I think also they they use Enclave technology uh to store the the private Keys used for the communication with signal encrypted in in that enclave and only inside The Enclave the the the keys are visible outside they are not so this is maybe on this function or application Level isolation and this is more about enclaves and stuff like this what I am I started to look in is on the level of a virtual machine which is a much larger thing than just an enclave so as I mentioned before this is not possible without support of the hardware and Hardware can provide many mechanisms to support this one mechanism is runtime encryption and that is the one what what I I am looking in into in the context of the virtual machines other me mechanisms are stuff like attestation this is about to to to verify or to guarantee that your virtual machine and the code you're executing is an A Defined state that you're actually running that operating system and that application you want to run and not something else that got Modified by someone maybe if if you're familiar familiar with Concepts like secure boot or measured boot this is something similar like the attestation uh mechanisms uh uh that are supported by confidential Computing and then there are um more mechanisms to guarantee stronger isolation and but I think this is mainly regarding Integrity of memory for example Integrity of binaries but I'm mainly looking in the runtime encryption and actually only in a specific aspect of the runtime encryption so the hardware has to provide some support for this and what I started look into is actually the secure encrypted virtualization provided by AMD CPUs uh Intel has something similar something different they call it TDX I think this is trusted device execution extensions something like this uh arm also supports or upcoming arm architectures are supporting something they call arm CCA that's confidential compute architecture and this is all focus focus basically on uh virtual machine uh confidential Computing for virtual machines some other thing you might know is enl s sgx I think this was software guard extensions something like this and this is more an enclave thingy meant uh to to run a library or functions of a library with in a safe compartment and I think the original idea was more from the DRM World digital Rights Management so that you can run your DVD decoder inside that compartment without leaking leaking the secret key sorry exactly I will have a slide about that so to summarize what I got interested in is building something like this with open BSD yeah using amdv technology and I only focus on the encryption of the memory what I cannot encrypt right now what I'm working on is uh to encrypt the CPU State this right now we cannot do that it's only about memory encryption it's not about integrity so there are limitations but my main focus is actually to learn about this just by doing it trying it out and then we will see if things work or don't work and then we can see what we can do with it with it so here is a very simple sketch of a modern CPU modern AMD CPU uh what we have the gray box is the die the chip the thing that represents your CPU and within the CPU we we have multiple cores and at somewhere on on the chip on the die of the CPU is a memory controller that allows the cores to access the DM of of your machine and AMD into this AMD hooked their framework their technology for secure encrypted virtualization what they did they introduced a new component on the CPU chip the so-called platform platform security processor PSP maybe they call it now security processor S I don't know I'm using the the the the name platform security processor and and the a short form PSP and the PSP is actually a small arm five CPU sitting next to to the CPU course and it's running a little how do they call it I think they call it uh trust o a little operating system that is basically doing Key Management Key Management what for are they doing Key Management okay here's the mouse we have the memory controller and they included in the memory controller an as engine for doing as encryption with basically line speed so they say it's a super fast fast this means uh when for example the Blue Core is using a memory encryption and when the Blue Core is storing data to the DRM the memory controller will encrypt will will encrypt the data that is coming from from the Blue Core with a specific key specific to the Blue Core and will it store encrypted in the memory and for the other cores they they and each core is is actually using its own own key not not only each core each VM that is running on the course has its own dedicated key so you can run multiple VMS and all VMS will have their different keys in that picture I think it's two VMS a blue one and a red one and they have that blue and red key and the this the other way around if if the the Blue Core is accessing physical memory the memory will be uh read back will be read back from dram decrypted and loaded into the caches of the CPU so and the key idea here is that the actual guest is controlling the encryption what does this mean the page tables used by a guest the guest uh guest page tables now uh use a additional permission bit within the page table entry the so-called Crypt bit C bit and with this bit the the guest can control whether uh data mapped by the guest page table is to be encrypted or not if the data is encrypted the data is private it goes to the DRM and only the the guest VM itself can read it back and decrypt it if the cbit is not set then the data is public it will not be encrypted and will go as plain text to the main memory to the DRM and with this it's sharable between the guest and the hypervisor yeah and and this is is a sort of a departure from the original yeah x86 security model uh where where now the the Privileges of the hypervisor that is or the operating system is diminished and the guest gains more privilege this means uh meaning the guest has a control and and can decide whether to share data or not to share data to encrypt data or keep it in plain text and this is I think this is this is a New Concept they came up with and how does this work now I need the mouse pointer again or maybe not here it is okay what do we have here consider this example we have our CPU just for Simplicity there's only a core uh the the other cores we do not care right now and that core is executing a guest the the the red guest that is identified in this table uh we um AMD uses the term asit it's an address space identifier and each virtual machine guest is assigned a unique ASD and each guest is identified by this Aid so in right now we are executing the virtual machine the guest with the asid to in its memory there is there's some text there's some he Heap there's some stack and that stuff is all encrypted now when the guest is accessing a virtual address because the guest is just running a normal open BSD open DS use virtual memory virtual addresses so when the guest accesses a virtual address this address needs to be translated from the virtual address to the guest physical address AMD and I think Intel also they use something they call nested paging this means the guest is maintaining his own its own page table and this page table is also encrypted within the memory of the guest so it's not visible to to to the hypervisor and what the guest is doing he's translating or the CPU is is is by uh reading the the guest page table translating the virtual address to a guest physical address and this guest physical address again is transl translated by a second page table the nested page table that is maintained by the hypervisor and and this uh hypervisor maintained uh page table is actually yielding the actual physical address the actual physical address with addressing some memory in our dam the new thing now is about the script bit when uh the when in this example the page table entry is Mark that this virtual address uh that this translation should yield an an encrypted translation uh the CPU will remember the cryp bit and say okay this will be an encrypted memory exess and then the physical address that was yielded by traversing the nested page table is modified AMD decided to reduce the physical address space when you're using memory encryption so actually we always have 64 bits addresses but AMD decided 20 years ago uh 10 years ago that there will only be 52 I think the upper 12 bits of the physical address are always zero they will not set so we only have up to 52 bit of physical address space in Practical implementations I think it's if you buy a mainboard then I think it's 44 bits 45 bits something like this so you can put I know I don't know many terabyte DRM into that machine now when you're running with with memory encryption some of these bits are repurposed of the physical address address bits they are taken away and are now rep repurposed one for setting the cryp bit and in that picture we we have the cryp bit at the position 51 counting from zero and this bit is now set because the page table the guest page table had the bit set so now we for basically it's forwarded into the actual physical address and then here and then also the asid the address bace I identifier that is identifying that particular VM is also patched into the physical address in my example in my example here I cannot do this my example I reduce the physical address basically by eight bits here and here so I reduced it from 52 to 44 or something this in reality it's actually more like four or five bits but then the number two wouldn't have shifted and the picture wouldn't be so good so now from the core we get this actual this physical address forwarded to the memory controller so and the memory controller now realizes okay we're doing memory encryption here so the bit 50 one is set this means this and this access must go through the encryption path you have to if it's a right to the memory we have to encrypt the memory if it's a read from memory we have to decrypt it uh and he also knows okay we are using the A2 so that's a virtual machine number two and here we have a table and that table is actually built by the by this PSP by the platform security processor that provides the that generates the keys and stores such a table in the memory controller and the memory controller look up okay for the ac2 we use that particular key and now the memory uh controller can do the memory access and encrypt or decrypt the data so to make sure within the core itself the data is in plain text and the in the cache in the in the memory and in the level two level one cache all the data is in plain text and is sent in plain text to the memory controller and and from the memory controller leaving the CPU leaving the the component CPU to the main board to the DRM the data is encrypted so on on this path we always see the encrypted data within the CPU itself it's not another thing that is not encrypted with this Tech technology is the actual CPU State somewhere in the CPU we have the registers the instruction pointer St pointer the flex and these are also not encrypted so at some point the hypervisor when the guest stops and is exited and the hypervisor takes over the hypervisor sees the actual CPU state so the hypervisor knows the instruction pointer the stack pointer the flags and so on and if if you consider using as and I for example Hardware encryption instructions of the CPU uh the a key uh AES key used for the encryption is stored in the floating Point Unit in the FPU because the AES and E is implemented basically by the FPU by the floating Point Unit so when the guest exits and the hypervisor takes over the hypervisor all also can see the FPU State this means if you you're doing asni then the hypervisor still can see your as keys that you are using and if you're doing IPC for example then then I'd like to take questions afterwards okay so and and then I think this this needs to be remembered the CPU state is is is not encrypted right now with this basic SCV technology so that are some of the oops limitations I came along the CPU state is not encrypted right now there's also no Integrity protection and this attestation also has some limitations AMD provides further levels of this technology SCV that's encrypted state that is all about encrypting the CPU state but this is the next level I haven't I'm I'm working on this but it still needs to done and then there's also a Next Level that is called SV SNP secure nested paging that provides more uh that provides um more integrity and we had a comment about the security so there is a bunch of I think these cves are quite reason so if you don't update your firmware your VMS will leak memory and you can decrypt stuff then there are people that were able to to hack the the platform security processor to extract the firmware to to load a different firmware uh on that PSP the PSP is doing the key management so this is not so cool but right now I I I I do not care about this because I want to figure the things out and I hope in 10 years all these bucks are fixed by AMD and my goal is to build a confidential VM on open BSD uh and basically learn from it why open BSD okay I'm using open BSD since 20 years with vmd vmm we have a great virtualization platform I read the code I sort of understood it and I think okay I will use this to implement confidential Computing using these AMD Technologies with open BSD on open BSD for openbsd and my main goal was in the beginning to keep it as simple as possible so the big picture what needs to be done to actually launch a virtual machine using uh you using SCV and memory encryption we have the vmd vmd is the component that initially loads where's the mouse tiger here that initially loads the the kernel image into the guests into the guest address space along with some stack along with a page table then vmd builds the CPU register State whoops for example the instruction pointer needs to point at the first instruction of the kernel so when you start the guest that is start execut starts executing uh starts executing uh instructions from the first instruction of the kernel so and all this is written to memory and with a non-encrypted virtual machine that's basically all that I'm I'm hugely simplifying here uh that's all that need to be done to start the virtual machine there's a special instru there's a set of special instructions in the a in the AMD instruction set something like launch VM then all that state is loaded and and the guest is executed if we want to use amdv we have to do an additional step we first have to encrypt all the the image that we want to execute this is why something I didn't mention yet that when you enable SCV on the virtual machine all instruction fetches go through the decryption path regardless uh regardless if the page with code code instruction is mapped as with the cpit or without if it's an instruction fetch it always goes to the decryption path this means the initial image that we are running needs to be encrypted by vmd and this is done with vmd talking to the platform security processor instructing the processor to generate a new key to associate um to associate that virtual machine guest the asit of that guest with that particular key and then the PSP has to read all memory or all the pages that are used by the guest has to read it has to encrypt it and write it back to memory and then when the guest is launched it can start executing fetching the first instruction and this gets decrypted again and the instruction can be executed so to get things going what do we have to do we have to do we have to implement a driver for for this platform security processor we have to modify vmd to talk to the platform processor to encrypt the kernel image to build the page tables using the cbit stuff like this then we have to modify the vmm uh driver that's a in kernal part of of of the hypervisor to actually enable SUV when when he launches is a virtual machine there's a virtual machine control block and there you have to say Okay this guest should run withv enabled and then we also have to modify the guest kernel that is running because the guest kernel should use the cryp bit to encrypt all its memory the guest konel is building page tables and he has to to use the cryp pit to make sure that the guest is encrypted so my plan was sort of make it as simple as possible just focus on on booting the the ram dis kernel I do not care really about devices a ram dis kernel does not need a dis it's just running out of a ram disk I will fully encrypted that guest not not caring about dma or something like this because with dma we have to talk to the hypervisor and we have to share memory with the hypervisor and therefore we have think about do we map this page encrypted or not I do not care right now I encrypt everything thing and the sort of works because in the be uh because the main uh Hardware components that need to be emulated by a PC platform that is emulated emulated by vmd is a handful of chips timer chip realtime clock in uh interrupt controller and the serial console that one is interesting for me because I want to see something so if we can access the serial console then I see something and I'm not blind and this is all done with inout instructions with programmed IO and not with dma so my idea was so this actually should work if I encrypt everything if I use everywhere the hard bit uh C bit in the guest if I hard Cod it then it should work and this was a plan in last year December so I started with implementing minimal support for this platform security processor there's a bunch of commands you you need to use to encrypt to initially encrypt the the the guest memory that's a so-called launch protocol and this is doing this red arrow here and I needed some more commands so we can skip this then I had to modify a little bit with vmd um I only use direct kernel XX so I'm not loading a bios I'm not going through real mode or something like this vmd allows to directly execute the kernel image you want to run so it's basically elf loading and executing and vmd also has to modify has to use the pcpt uh the the the c c bit in in the page tables so the memory is actually encrypted and then here vmd has to talk to the PSP please um encrypt everything the modifications for vmm were quite simple basically in the configuration block you have to say I there's a bit you have to set and stuff is sort of working so in the guest kernel I also defined a cryp bit and on my machine I know it's bit 51 but AMD says H this is implementation specific so on some CPU might be different so you should actually figure out what bit you are using and I just hardcoded it and then of course in the pmap module where where where you're setting up the actual mappings I was also hardcoding the cbit and with this it was actually possible to put Ram dis kernel it actually also saw some stuff of the word iio devices but yeah but and this took about two months too earlier this year so in the round two I was working but because my goal my second goal was then I want to use the same kernel as guest and host before I mod I had a modified host that could could run uh encrypted guest and I had a modified guest kernel that was able to be to be run uh encrypted so to just use a single kernel for both purposes the kernel has to to detect am I a guest or am I a host am I using encryption or am I not using encryption and depending on this he has to be more flexible about defining the script bit so I cannot hardcode it anymore and that's what I did so the next thing that I that we should do we want to do dma we want to do word IO and this is is only po this in my first approach I just yeah did a hack to map everything unencrypted that is using the budm a m API so all memory used by for example embus would be unencrypted however this is not really working it was working but this is not what you want to do for some reasons so therefore we need to uh I needed to implement something like bounce buffers this is an thing that is quite old it's from if you have some old Isa Hardware that can only address below 16 megabyte physical memory then you also need bounce buffering copying stuff from high memory into the low memory and back and forth and the AP API is there so we I modified this that uh by by using these bounce buffers that the bounce buffers itself are mapped unencrypted so they can be shared with a hypervisor they show up unencrypted in the hypervisor and the Boost dma API provides a function that is called boost map sync or something like this that is actually doing the copying on the original PC platform it it need doesn't need to do anything this boo uh this sync function on for example on some other platforms like spark it needs it is needed to implement flushing of caches and stuff like this so the Hardware is dma into memory and before accessing that memory caches needs to be flushed or the other way around when the CPU writes stuff uh to the memory and before the hardware can access this caches needs to be flushed and this is implemented by the sync AP and I am using it in a similar way just for copying back and forth between the encrypted and non-encrypted memory and with this uh we can skip this we can skip this yeah and and with this is it was basically possible to just use word IO and then there are some optimizations I want to skip so now earlier this year in May I got this sort of finished and now I I could use the same kernel as guest as host and the confidential VM was sort of working almost because there were bugs in the Vio five minutes thank you and in in the vo block I got crashes when doing a build nonetheless I I mailed a diff out because we are doing open source so it's important to get people involved I showed the diff around and a colleague of mine Stefan frit did some debugging in the v IO and we got these crashes fixed um Alexander Bloom was doing testing and debugging of this dma bounce buffer stuff that I did then I also got input from Dave vuler thank you and from Mike Ling kenis and so on and we got the stuff committed and right now we have sort of um a stable VM that survives a make build and the memory is encrypted is it so to really figure this out I now we can skip this and look at this I decided I I need something like a heat map I want to examine the physical the guest physical memory and see if that memory is actually encrypted and from this I modified the VMT to get a dump of the physical memory and then I plotted this heat map so here here's the lowest address I think that's a one gigabyte machine so here we have we have address zero there is somewhere 1 gab and it looks like I would expect here in the lower part we we have the initial kernel we have the uh kernel text image above it are the initial page tables very on the top we have stuff like the DK buffer and in the upper range there that's that's memory this is actually used by by the guest running so this is when you are using a guest without encryption that's the picture I got now with SE enabled what I did I was compressing the pages P dumped the memory compressed the pages and if the page was easy to compress or good compress then it's not encrypted then it's compressible data and with memory encryption I got something like this we see that this uh the the the used pages are hard to compress that's a darker green the lighter green or white means easy to compress so the memory used by the encrypted machine was hard to compress so I think I was confident that we are actually running encrypted and then I did two minutes left I did a little heck there's a so-called zero thread in the kernel that is nulling pages on the free list so I and usually uh This Thread just come uh is maintaining 20% % of the free memory being nulled being written zeros to so I modified it that it actually SS out everything that is on the free list and so I booted the machine waited a minute shut it down dumped the memory and I see that basically all pages got touched by the or most of the free Pages got touched by the Sero thread so I assume they were encrypted so what have we accomplished we have the simplest version ofv enabled as an open open BSD guest on a host the stuff is committed next steps would be the state encryption fixing all the bugs I guess there are plenty of bugs left optimize dma there can stuff that can be done we haven't talked about performance there are stuff that needs to be done you haven't talked about this attestation about this secure booting stuff so there's stuff that needs to be done and there's even more stuff to be done so that's what I have to tell I just want to say thank you to some of my colleagues that helped me with the slides Mia Tes hour and yam clemco he's also working in open BSD and Alexander Bloom and Stefan frit for debugging and committing committing and I like to thank the people at tech for their input thank you question [Applause] there's room for a couple of questions so so the latest amd64 CPUs can have a 50 seven bit uh virtual address base so what prevents them from using traditional Shadow paging in the hypervisor to fake SCV uh the answer uh the question was uh it it's recorded okay then I don't have to repeat it uh I cannot answer this I do not know I'm still in the beginning of figuring things out so I do not know yes there was a question by Rene I think earlier already does the uh is the cash memory of the CPUs also encrypted no the the the question was if the cach memory is encrypted no in in Cache we have plain text and only going out to the DRM leaving the CPU the the die that stuff is encrypted so I hope Al if if you think about stuff like um Spectre meltdown I don't think this will help with encryption um is this uh the first open source hypervisor that no no no with with with uh Linux you have KVM and qo that they can do this they they already do this they already do this thank so what happens when you reconfigure cach uh set RS to use a scratch pet memory do they get invalidated uh before you get to inspect them with the because this is needed during early bring up of a memory controller for training I cannot answer this question again a specific detail I do not know I'm sorry okay last question yeah you touched on performance at the last point in the slides uh so what is the overhead of encryption that's a good question so AMD says it's one% or something like this regarding the memory exess and as far as I know they're encrypting anyway the memory if you do not configure anything then they put in a random key and so it's easy to say there's no penalty when you use encryption because you're using I I might be wrong here if someone from AMD has some here in the room has details let me know uh and and with with regarding Alexander Bloom is doing some tests and the performance is degraded I would say is it or is it Horrible No but it's not good it's mainly bounce yeah I don't any bounce and that of course ah thank you so I I I haven't tested uh I didn't put the the encryption machine in my automatic test setup yet but I have added a thing that I can just set a flag to force bounce buffering even on a non-encrypted thing so that helps a lot to find bugs in in the drivers if there's some uh Boosty Ami sync missing and with that alone I get a decent performance degradation like 30% 50% and there there's room for optimization but we we'll see next year thank you very much thanks [Applause] again e for