DEF CON SG1 - CSIT Village - Daryl Lim, Zhi Kai - Rolling Api Tokens A Thought Experiment

And now to wrap up day two, Darrell Liam explores a world without long-lived API tokens using concepts inspired by the double ratchet protocol to continuously rotate secrets. It's about reducing risk by design. If you're thinking about authentication and future-proofing systems, this is one to catch. Let's welcome our final speaker for today. >> Hello. You all can hear me at the back, right? All right, great. All right, yeah. So, today's talk is about a thought experiment on rolling API tokens, right? And it's like mentioned earlier, it's inspired by the double ratchet protocol. And I was really thinking about it like how can we like the fact that how how can we apply this to like API tokens, right? For machine-to-machine authentication, right? So, this talk will be going through about that. And just a little disclaimer is that this is not a finished product, right? It's just a design idea that I think might be worth a conversation. All right, so just to give some background, APIs are the glue to modern software, right? Like microservices, CI/CD, mobile applications, and all of them are just talking through APIs constantly, right? And non-human identities outnumber humans 50 to 1. Your organization might have 200 engineers and 10,000 service accounts. Most of them authenticate using static API tokens. Now, these tokens are usually created once, stored somewhere, and used forever, probably, right? And in March 2025 last year, uh one of these secrets one of these leaked secrets, right? Actually cascaded into 23,000 compromised CI workflows, which I'll go through a little more later. And this is not really a misconfiguration. This is more of a design flaw. So, how can we design leak-resistant systems via ephemeral just-in-time credentials. Right. So, currently how is standard API secrets are used nowadays is that CICD pipelines need highly privileged credentials for legitimate reasons. They such as pushing Docker images, deploying the broad, running Terraform, all require powerful keys. And the problem is not that these keys exist, it's more that they are static and of high privilege. So, that combination is what makes leaks catastrophic. Now, if someone got the CI runner's environment variables right now, what could they do with those environment variables? So, here's an example workflow that you might see. Developer commits to a Git repo and the GitHub action builds it and then pushes it to Docker image. So, this Docker token has to live somewhere accessible to the runner and during execution that secret is in memory. So, what could these rolling tokens be used for? Uh we use in M2M and NHI API interactions, not really used as like a OAuth replacement. And runtime memory should not be guaranteed to be private, right? Because you could always have a compromised runner, container escape, or misconfigured systems. And most security protects data at rest and in transit. So, these rolling tokens focuses more on data in use. And what it does is that it minimizes the window of exposure exposure to seconds. So, why does this matter? Some of these uh scenarios has happened based on the OWASP NHI top 10. For example, The New York Times had exposed the GitHub token and 270 GB of source code was stolen in January 2024. And nobody noticed it until it showed up on 4chan in June, like 5 months later. Right. Uh for Cloudflare, if Cloudflare failed to rotate its stolen credentials after Octa got breached. Uh for hugging face, they had their spaces secret exposed. They did manage to remove the odd level tokens entirely after before anything actually happened, but it kind of tells you like how bad it could have been. The US Treasury Department affected by one compromised beyond trust API key allowed attackers to have remote access to some of the agency's workstations. And finally, Microsoft had an OAuth application hijack and then it just had full exchange access. So, uh going to the TJ actions incident that I talked about for the March 2025 incident uh yeah, March 2025 incident. Uh TJ actions changed files is really is an action that is widely used on GitHub actions and it was compromised by a stolen GitHub personal access token. And the payload was elegantly simple. What it did was that it downloaded memory scripts, it grabbed for secrets, base64 encoded them, and then print to the workflow log. And for those who used the this action, you did nothing wrong. You kind of just used a trusted dependency and it was weaponized. So, this is the attack flow and this image is provided by Palo Alto Networks. It's uh what happens is that it the attacker pushes a malicious commit and then base64 the encoded payload. Right, and they updated all existing Git tags to put to point to the malicious commit. So, if you set a tag at V1, basically you were still vulnerable and most people pin to tags rather than the commit chars. And we've also documented the entire full flow and the workflow log is basically the exfiltration channel, which shows that the attack surface is the entire execution environment and only the only the execution environment. All right, and how did he manage to even get control of this TJ actions? They actually This was actually caused by another personal access token leak. Right, and review dog action setup was infected at first and gave attackers the personal access token used to compromise TJ actions. So, we're kind of like trusting everyone our depend uh our dependencies trust and trusting them with that they had to use the API keys uh that they secure the API keys securely. All right, so what can we do about this? All right, so introducing like the rolling API token specifications. For those who are familiar with the signal double ratchet protocol, this might look a little familiar to y'all. Yeah, but basically for those that don't know, uh there's two ratchets. Uh the two the two ratchets work together. The Diffie-Hellman ratchet governs key agreement between client and server and rotates periodically. While the token ratchet governs token generation within each key period. You can really think about it as the Diffie-Hellman ratchet changes the lock while the token ratchet generates a new key per door opened. All right, so for the initialization, step one, you can use an existing API uh use an existing API token such as the Docker personal access token and that becomes your root key. Step two is that both sides will perform some form of Diffie-Hellman key exchange. You can use something like ECDH or something better. And we can use the root key for authentication using a HMAC on the DH pub and uh root key as the secret. And once this is done, the client and server has the secret DH uh and is authenticated. So, the root key is kind of like a uh security deposit. You just kind of need to use it once and that's it. So, what happens in a Diffie-Hellman ratchet is that every 60 seconds, the client checks its DH pair is due for rotation. And the client generates a new DH pair, sends the new public key to the server. The server does the same, generates its own DH pair and sends its own public key back to the client. And both clients basically both sides independently compute new shared secret and the old secret is discarded. So for on the token ratchet KDF and basically you will create a you will run the KDF on the DH out and the root key to generate the first chain key and each subsequent chain key is derived from the previous one. So for the full token generation flow we have two terms epoch and chain. So the epoch is a period or a period given where the DH out is valid and is bounded by the 60 second DH rotation mentioned earlier. While the chain is a sequence of tokens within one epoch. So it's like two layers of freshness epoch level from the DH ratchet and the request level from the chain key. All right, so here are just some metadata in the token. It's not too important. So here is a sample a program that is uh implementing the protocol. Let me just pause it. All right, so as you can see over here during the initialization uh section the left side is the client and the right side is the response on the server. And notice that the client sends the the DH pub over and the server sends its DH pub over to the client. And there's also the authentication that I mentioned earlier which uses the KDF with the root key and the DH pub to make sure that they are not being man in the middle. So we're going to run a few pings and I want you to notice that the tokens are actually changing every single request. All right, so what we'll do now is I'll run an intercept and I will s- I'll intercept one of the requests and send it to a repeater. And notice that the first request is okay. But subsequent requests uh basically it fails. Uh so basically the request fails immediately and once it's consumed, it's basically gone. Finally, there is one more thing I want to show you, which I'll go through a little bit later, which is the out of order request. So most likely and more likely than not, you are probably not come in a oops, sorry. Yeah. Yeah, so notice over here, I sent the request in randomized order and it's still able to recover itself. Uh I'll be going through this a little bit more how it handles the out of order like shortly. All right. So for the out of the order request, in the real world, requests don't usually arrive in sequence. It could be due to network conditions, retries, asynchronous jobs. And it kind of makes like sequence maybe it makes like sequence five might land before sequence four. So the solution for this is look ahead generation. So if a token arrives with a higher sequence number, uh gener- basically you'll generate the intermediate chain keys in between them, store them and accept matching ones. And once the token is used, it's discarded immediately and is never ever reused. And the look ahead is also bounded, which I will cover next in the security considerations. All right. So there are a couple of guardrails to make this safe in practice. Number one is that key must be device bound. So the root key and the Diffie-Hellman private keys should live somewhere in the TPM so that because if you put it inside the memory, if they can dump the root key and the chain key, you're kind of back to square one. And operations must happen inside the chip, right? Keys can never touch at memory. Chain keys must also be short-lived. And previous epoch 30 second uh sorry, for the previous epochs, we are setting the time to live to 30 seconds and the epoch before that to be zero immediately. This is so that it can handle some form of out of order when the when the epoch is being changed. And there is the maximum chain key look ahead cap at 100 like I mentioned earlier. Right, so without this, the look ahead can become a denial-of-service vector. Right, so if the attacker sends like a sequence number of like 1 million, all right, if without the cap, the server will just generate 1 million keys, right? And the cap basically kills that. And the force epoch change every 60 seconds kind of just make sure that the attacker who steals a chain key cannot write on it indefinitely. Okay, so before I go through some of the comparisons, let me address a couple of things that this design does not solve. Right, so for one, TPMs aren't really everywhere, right? Most cloud runners don't really have a TPM, all right? But I think about GitHub hosted runners or AWS Lambdas. And this is a real deployment limitation. All right, so we can have some partial substitutes. I believe there's some software TPM that you can implement by virtualization. And without this isolation primitives, the chain key sits in process memory. Another thing is about stateful authentication. Right, the static keys are basically stateless in stateless in design. And stateless bearer tokens scale trivially, right? And this doesn't really does that. And if we scale it like horizontally, let's say for example, if we have 50 back-end instances, they need to share chain key state, right? So they may need like sticky sessions or shared cache like Redis. And this shared store now needs to be on the same level of protection as the TPM, right? So it's kind of like a a situation where there's no free lunch, right? Security kind of gain a security gain costs operational simplicity. And finally, talking about maturities that this is not a just a thought experiment and a working prototype. It's not really formally verified kind of and also not production deployed at scale. All right. So, going on to the current technologies, how does this compare to stuff that we already have? So, OAuth 2.0 is great for human authentication. Tokens are static bearers and vulnerable to replay until expiry. And JWTs are generally great for verification without DB lookup, but it's also nearly impossible to revoke before expiry. And GitHub secret manager protects keys at rest. Runtime injection makes them static bearers again. And mTLS is also very strong, but the thing is that PKI is operationally very heavy. All right, with cert life cycles, revocation, and intermediate CAs, and etc. So, what rolling tokens hope to achieve is that is an application layer uh security without a need for a PKI. And it's this is designed to designed for assumed compromise runtimes. So, where could this be used? Well, for starters, we can use it in air-gapped environments where tokens can be issued and validated in fully isolated environments and continue to operate without external network access. So, there's no need to phone home. Another thing is that in zero-trust proxy-heavy environments, application uh basically is a application layer security where infrastructure is not part of the trust boundary anymore. And I think more interestingly about agentic AIs and MCPs nowadays, it kind of prevents the agent from remembering long-lived credentials in context windows because this context can be exfiltrated by a prompt injection or by logging. All right. So, this rolling tokens ensure the agent never holds the credentials useful for more than a few seconds. So, thinking about this, we could go a little further. We could implement fine-grained permissions. So, we could bind permissions cryptographically into the token, mix permissions into uh sorry. We can mix the permissions that is written in JSON into the KDF. And also, it kind of moves us from a situation where we have a valid session to an authorized for exact action. All right. So in conclusion, static secrets are a single point of failure and there is no such thing as a truly trusted environment and human detection is way too slow and just have because you possess a token doesn't mean you're authorized to use it just like the TJ sorry the just like the TJ action situation. And basically for rolling tokens, a static secret is no longer a golden ticket and it kind of moves from a state basically it moves from a static to a dynamic secret. Right. And it's secure even in compromised environments and it kind of shrinks basically shrinks the exposure surface dramatically. And it's also self-contained so you don't really need a third party to phone home something. Right. So static secrets assume the environment is trustworthy while rolling tokens assume it isn't. Yeah and with that I come to my end of my presentation. Thank you very much. Hope you're enjoying the rest of the Def Con.