Don't Drive Blind: Crashing Testing Model Deployments with NVIDIA Garak

You wouldn't buy a car with a horrible crash test rating, much less one that has never been crash tested at all. You want to know how it handles a collision before you're the one in the driver's seat. Your AI models deserve the same treatment before they hit production. You need to know what happens when someone tries to break them. Today, I'm going to show you how to crash test an AI model using Garrick. First, let's see what happens when a model goes to production without any testing. I just sent a jailbreak prompt to GPT-2, a model with no safety fine-tuning, and it did exactly what I asked. No pushback, no refusal, it just complied. A model with proper safety training should refuse this outright. GPT-2 doesn't even try. This is one of the four ways an untested model can fail. The article covers all of them: hallucination, prompt injection, jailbreaks, and toxic output. But the point is the same for all four. Any one of these is cheaper to catch before launch than after. So, how do you catch these failures before they reach production? You crash test the model. Garrick is an open-source LLM vulnerability scanner built by Nvidia. It runs adversarial probes against your model, the same kinds of attacks a real user or attacker would try, and reports back what broke. I'm pointing Garrick at GPT-2 on Hugging Face and telling it to run DAN 11.0. It's a well-known jailbreak technique that tries to trick the model into safety guidelines. DAN stands for Do Anything Now. It's a long elaborate prompt that basically tells the model to pretend it has no rules. You can see it downloading and loading the model locally. This is all running on my machine. Nothing is being sent to an external API. Now, it's queuing up the probe and sending the attack prompts. Garrick sends the same prompt multiple times, five by default, to see how consistently the model responds. That matters because a model may refuse once and comply the next time with the exact same input. You don't want to know if it can fail. You want to know how often it fails. Once all five attempts come back, Garret runs detectors against each response. Think of probes as the attacks and detectors as the judges. The probes throw the punches and the detectors score whether the model stayed standing or went down. Each detector is looking for something specific, like whether the model adopted a jailbreak persona or whether it failed to refuse a harmful request at all. And DAN 11.0 is just one probe. Garret has probes for prompt injection, toxic output, PII leakage, hallucination. You can run as many as you need to get a full picture of where your model is vulnerable. And here the results. Two detectors ran against the model's responses. The first one, the DAN detector, checks whether the model actually adopted the jailbreak persona. You know, whether it started responding as if it had no rules. Two out of the five responses were clean. Three weren't. That's a 60% attack success rate. So, three out of five times the model played along with the jailbreak. The second one, mitigation bypass, is asking a different question. It's not checking whether the model adopted the persona. It's checking whether the model ever pushed back at all. Did it ever say that it can't do that or give any kind of refusal? Zero out of five. Not once. The model never even attempted to refuse the request. That's a 100% attack success rate and Garret flags that as immediate risk. So, to put that together, the model went along with the jailbreak persona more often than not. And it always failed to refuse the request in the first place. It didn't even recognize that it should say no. That's the difference between a model that gets tricked and a model that has no defenses to begin with. Garret also has an HTML report that you can share with your team for more visual look at the results. That's why Garrick gets wired into your CI/CD pipeline through Eval Hub, the evaluation orchestration service for models on Red Hat OpenShift AI. You define a benchmark collection that includes Garrick as a provider, set your pass/fail threshold, and call Eval Hub's post such evaluations endpoint from your pipeline. Garrick runs as a Kubernetes job against your model's live endpoint, right alongside any other benchmarks in that same collection, instead of as a separate manual step. Run that scan as a gate in your deployment pipeline, not as a one-time launch task. Every time a model change, prompt change, or retrieval source change is about to go into production, the pipeline runs Garrick first. If a small tweak quietly breaks something that was previously safe, you find out in CI, not from a customer. The second layer is runtime protection. There's a reason Safety Your Ford doesn't test that your SUV would avoid a UFO falling from above. Some incidents can't be predicted. Testing before deployment catches known failure modes, but it doesn't stop something new from happening in production. That's where the NeMo Guardrails Orchestrator comes in. It's built on the open source NVIDIA NeMo Guardrails project and included with Red Hat OpenShift AI. It sits in front of your deployed model and screens inputs and outputs as they pass through, using detectors you can configure and tune for your use case without retraining the model itself. It exposes endpoints that can validate a message against your configured rails without even generating a response, or run input rails on the incoming message, generate the response, and check it through output rails before it ever reaches the user. Pre-deployment scanning with Garrick catches what you already know to test for. The NeMo Guardrails Orchestrator catches what happens live. So, here's the full picture. Scan for known attack patterns with Garrick, gate every production push on the result through Eval Hub, and add the NeMo Guardrails Orchestrator as a runtime layer that watches live traffic. Not because every car will crash, but because you'd rather know the rating before you're the one behind the wheel. Testing before deployment, automated scanning in a pipeline, and guardrails watching live traffic. Three layers, each one catching what the others can't, and it all starts with that first scan, the one you just saw. The full article, Garrick, and more are all linked in the description. Go try Garrick against one of your own models and see what it finds. Thank you so much.