AI Is Breaking Your Systems Faster Than You Can Fix Them | Gremlin, Dynatrace

Now, when it comes to AI, the general perception is that adding AI, adding AI tools, AI agents make your system smarter. It's true, it does make it smarter, but it also make them more fragile. More AI only deepens the blind spots, and we all know AI hallucinate, it can even lie. So, you don't even know what is going on. And of course, when it comes to observability, you do need to see what is breaking and prove that your systems can survive it, but AI kind of blurs the line. Now, Gremlin and Dynatrace are partnering to bring resilience testing and reliability scoring into one dashboard to make life easier for sysadmins, DevOps teams. And today we have with us two guests, once again, Kolton Andrus, CEO and founder of Gremlin, and Philip de Blois, VP of Solutions Engineering at Dynatrace. Philip, Kolton, it's great to have you both on the show. >> Thank you so much. Glad to be here. >> Always a pleasure. >> It's my pleasure, actually. Of course, we are going to talk about what you folks are doing, but before that, can we also talk about how is AI changing the way companies need to think about resilience? At one end, you can see all these tools can make you more resilient, but at the same time, it can also be part of the problem. >> I think what we're seeing is, you know, anytime we go through an innovation cycle, there's a lot of inflated expectations, there's a lot of great gains to be had, and we're kind of in that honeymoon phase, where everything is is seems amazing. Everything's moving quickly, we're able to just start having bots write code for us, we're able to fully automate our deployment pipelines, we're able to do this world of agentic development, where they're able to get things done quickly and out the door. Uh, but what is the trade-off there? What are the side effects? And I think one of those are is that you some of a lot of that code that's making it out the door hasn't been tested as fully. And there's likely to be vulnerabilities, there's likely to be misses in that code making it out the door that need to be tested. And so I think there's, you know, a benefit to all of the opportunity we have, but there's a trade-off that we need to be able to go through and to be able to validate that the systems really behave the way we expect. And not just in like our test or integration environment, but in the turbulent environment of production. >> Colton is right on, right? We've got uh uh a great opportunity with AI, and I think companies are are really leveraging that and are doing a lot of experimentation today. Uh the next frontier though is as we start seeing these things in production, and how are they going to behave in production? Uh we're shipping code really fast, uh the complexity is increasing, uh but how can we make sure that that code is going to be reliable? How can we make sure that it's secure? How can we govern all of that? Uh so I think it's it's uh it's changing quite a bit of the landscape, and it's also adding some questions uh that we didn't have before. So if from an observability standpoint, we can say things like, "Okay, is there is it running? Is it uh uh you know, performing in the way that we expect?" But with AI workloads now, uh we have additional questions. Is it accurate? Is it uh is it working in the way that we expect it to? Um is it providing answers that are accurate? And so uh from an observability standpoint, we need to add some of that layer to really understand how these systems are are behaving in a production environment. >> What kind of governance, what kind of guardrails you are also putting there from observability point of view? Because we trust observability tools, you know, that hey, this is the metrics, I mean, data never lies. But here the risk is that can we even trust that? >> Yeah, I think that's one of the the the the big differences in the approach that that take here at Dynatrace is that, of course, we leverage LLMs and we leverage GenAI in the platform to be able to help bring answers to to the data, but we also leverage deterministic AI and I think that's a key component is that you can you can trust deterministic answers much more and from there be able to take trusted action. And so deterministic AI is a key component of observability and a foundation I think for you know, leveraging then probabilistic models like LLMs and so on. >> Put it from your perspective. I mean, best practices are overused word, but you know, what should be the right approach to kind of break that cycle of trust and can you even trust it? >> Yeah, well, I think it's it's we share a very similar approach between Dynatrace and Gremlin here. A lot of the work that we do in the Gremlin platform is deterministic. We have we have a hypothesis. We're going to go run a specific set of tests. We're going to measure the outcome of those tests. We're going to understand what happened and we're going to leverage LLMs around how we interface with the user so that they have a simplified explanation so we can put it into plain text form, but we're not going to leverage the LLM to process the data or to do the in-depth analysis. And I think this is one of the disservices of the current trend of AI is everything has to be everything AI has to be an LLM. But actually, there's a lot of AI that is not LLMs. And there's this great thing called machine learning and all sorts of techniques that we've been honing for the last 20, 30, 40 years that can can and should still be applied. And so that's one of the approaches that we're also taking is hey, we're going to look at all of this data we've collected across the lifetime but time of Gremlin. Millions of experiments, tens of thousands of different systems that we've analyzed and we understand the results and we're going to distill that down into credible, actionable uh, recommendations for customers. And then we're going to use the LLMs for helping smooth out the interface and and finding, you know, and and helping to find the right APIs and find the right actions to take, but we're going to do it in a way where we prompt and we confirm. We don't just blindly take action and believe what's said. >> And not just LLM, it has to be the biggest LLM out there, you know, the the better should be high. So, >> [laughter] >> but that is not the case. Sometimes a tiny one can do much better job than the big ones. And now, if you look at these two companies, Dynatrace, observability, Gremlin, the whole reliability, >> [sighs] >> how are these, you know, solutions, you know, that you are bringing together? Uh, how should we look at it, this partnership? Uh, what impact it will have on not I'm not looking at a smaller picture of teams in general, but because AI, as I talked to Colton before that, is redefining every interview I'm doing is like is about AI 99.9.9% Every discussion is about AI. Is it transforming the way we write code, we deploy code, we find vulnerabilities, we fix vulnerabilities. So, if there was a time, you know, when everybody think about software, now everything is AI. So, also talk about not the smaller picture, but the bigger picture when it comes to observability, reliability, AI, and these two companies. >> I think I think Colton and I are probably, uh, going to agree on this point, but I think we, you know, we're we're in the business of helping our customers be proactive and preventative. And so, in other words, uh, yes, failures will happen in production and we have to deal with those and from an observability standpoint, we're watching all of that and we're making sure that we solve problems, but we need to solve problems way earlier than that and we need reliability testing to be able to to help us do that. And that's where like resilience and observability really belong together and I think that from my perspective, the customers that we've helped together as a partnership uh, have really been able to achieve some of those goals of preventing P1s from happening in the first place. It's a lot more cost-effective to be able to solve those problems early than to deal with them when they're uh impacting large uh production environments. And if you add AI to the mix, that that that only compounds the problem. And so, from my perspective, being proactive, starting early, doing the testing early, and using observability to uh augment the the data that we get from Gremlin is, I think, critical for any large enterprise. >> Yeah, Philip knows me. Yeah, 100% agree. Yeah, that's that's I've been out here preaching that for the last decade, trying to get folks to get in front of the problem instead of waiting for the problem to occur. I think the other thing I'd bring up is it's an opportunity to provide more data. And back to, you know, better data helps us make better decisions and helps us to build better systems and understand those systems well. A lot of classic observability is only what has happened, not what could happen. And Gremlin gives us the opportunity to go out and to create these scenarios and see how the system actually responds to those, instead of looking at week-over-week or month-over-month analysis, instead of waiting for an incident to occur and deciding, "Hey, would that last incident be like the next incident we saw?" We can go out and proactively create these experiments and these scenarios, and we can understand how that impacts the system, and then concretely measure that, and understand what the knock-on effect is. How did that impact our metrics? How did that impact our dependency? How did that impact our response time? Ultimately, how would it impact our customers? And we can leverage that to be able to go find and fix things that we might not have other otherwise seen or had to wait until they'd already caused customer pain to uncover and to fix. >> Yeah, and if you just look at, you know, this partnership, as you mentioned, what are the key benefits of this integration? What will become possible or what has become possible now which was not possible earlier. >> Yeah, so one of the cool things that the Gremlin team did was Dynatrace is a platform and it allows customers, it allows partners to build applications in the Dynatrace platform. Uh and so it provides all of the role access, permissions, security, the actual UI framework that's required to look and feel exactly like the rest of the platform. And so the Gremlin team basically built an app to bring reliability scores into the Dynatrace platform in a way that's native to the platform, but also in a way that our existing customers already understand. And so they can be in a dashboard and they can have reliability scores from Gremlin right from within their flow and continue to use the platform in the way that they're used to. And so it's really about kind of bringing all of the great data that that is coming from Gremlin into the Dynatrace platform in a way that users can continue to leverage in in their day-to-day job. >> Yeah, one of my favorite sayings is if you want people to do the right thing, you need to make it easy. And so how do we make it easy to do the right thing here? Well, we know people want to do this, they've got many tools, they've got many panes of glass, they've got many places they want to manage things. One of the things we've learned over the last 5 years is it's it's easier to convince the engineers, the people carrying the pagers that this is good work they need to be doing so they don't get woken up, so they don't feel the pain. But ultimately, you need leadership buy-in. Especially when you're talking about an organizational-wide problem like reliability. You can't have one team fix reliability. It takes many hands, it takes many it takes coordination across the entire enterprise. And so in order to do that, you need leadership involved, you need leadership to have visibility into what's occurring, and you need to have them be able to hold people accountable and reward and incentivize the right behavior. In order to accomplish that, if they already have operating procedures, they already have places where they're looking at metrics, they built dashboards, they're already tracking this problem, we enrich that data so that it's easier for them to go, you know, to be able to accomplish that. >> And can you talk about, you know, for those teams who are already uh using Dynatrace, what does day one look like after they install Gremlin app? What kind of onboarding support kind of, you know, hand-holding they expect or that is not needed? >> The good news is that it's quite simple. Uh they've done a great job of making it super easy for for our customers. Uh if they're already existing Gremlin customers, uh it's as simple as going to our hub, downloading the the app, um and setting up the integration is very easy. Uh from there, uh they've got they've got a way to communicate with other teams in the similar language because again, they're using the same UI, they're talking the same language across both platforms, and can view the results together uh in in a unified way. >> You know, we've we again, we look at ways we can make it easy for people to get done what they're trying to get done, and so we've done a lot of work on the Gremlin side to make it really easy to set up Dynatrace to be able to pull in those metrics and alerts, to tie those into experiments. So similarly, when we built this app, we want to make it very seamless for people that are already customers of both to be able to quickly get valued, quickly set it up, quickly integrate, help help them identify which monitors and alerts are the right ones to use, be able to pull those into their experiments so that they've got that safety net, they've got that feedback mechanism. And we've created ways to trigger and take action on the Gremlin uh items to be able to kick off the tests or scenarios from within Dynatrace. So, again, just focused on making it as easy as possible for customers to get value as quickly as they can. >> Is it possible for you to kind of uh walk us through a real world example of this integration where they actually caught a reliability risk before it turned into an outage. >> We've had multiple shared customers where we've gone through this and of course we just launched it live but an example of what would happen, you know, a customer comes in, they set up Gremlin, they set up the Dynatrace integration so that they're able to collect the data that they have. We're able to go and identify a set of risks or we're able to give them the feedback about an experiment that's run. So then you swap over, they're in the Dynatrace app, they're building a dashboard and they're looking at, "Hey, what are the scores of my services and what are the risks that exist within my services?" And so in that world, that's where it's very straightforward for a customer to say, "Hey, I came in on a Monday morning, why do I have this new risk on my production service? What's going on there?" And then they can drill in and get details about that risk, about why it occurred and about how to go fix it. So we've had we've had quite a few examples of customers already who've been able to go out and either through the passive risk that we create or through the tests that are run. Um one of the other things that we expose in the Dynatrace app is the output of our reliability intelligence. And so that's when a test fails, we have a set of analysis that we run, again back to the start of the conversation, more on the machine learning side and less on the LLM side that tells the customer, "Hey, here's why it failed and here's what to do to go fix it." So again, back to the example, on Monday morning they come in, they saw a test failed, why did that test fail? Well, now they can drill in and get actual feedback about why the test failed and how to go fix it. And then in the ideal scenario, they fix it, they run it again, they see that it's passing, they're using Dynatrace during that running experiment to understand how it's behaving, to understand what's going on in the system, to make sure they really grokked what's gone wrong, so they can go fix it appropriately, close that loop, and end up back in a steady state where they fix that vulnerability right after it's appeared. >> We had a a large insurance company speak at our annual conference, a user conference, where we had a lot of people come to Vegas to come and listen to some of the innovations of Dynatrace, and the speaker uh was talking about a a cloud migration that they had to do, and uh they wanted to be able to test for things like, for example, what would an outage do, or uh what would high latency uh bring to to this particular migration. And so, they were able to uncover, when partnership again with Gremlin, uh some some risks and some P1s that they could avoid uh by doing this testing early. And so, this migration ended up with zero outage. It was one of their big success stories, and uh the lady that spoke actually was uh promoted as as a result of this. And so, uh it's stories like that, I think, that, you know, we want to continue to bring to the rest of our customer base. And let's be proactive when you're doing these big migrations, or when you're doing these big projects, like mis- make sure that we do the testing early, and that we catch these problems uh before they even make it to production. >> I love the getting promoted part of that story, too. That's back to the how do we incentivize the right behavior? And it's one of my favorite questions when I'm talking to leadership. Hey, you know, we often and we often incentivize, we often reward the folks that are great at firefighting and fixing the problems. And they're important. But what about the the team that does this work proactively and never ends up in firefighting? You know, they've essentially prevented a whole set of failures from ever occurring. How do we recognize them? And what we need is we need proof, we need evidence. And so, this gives us the opportunity to go measure those impacts, provide that evidence for folks, and allow them to go share the good news. Hey, this is this is the work we did. We didn't get paged, there wasn't an outage, our huge migration went smoothly, our app has run at five nines for the last year, and the teams can go, "Wait, I want to be like that team." And that's really the behavior we want to recognize and reward. >> Yeah. No, that was a great story as well. Now, then if something goes wrong, do they call Dynatrace or Gremlin for this? >> Yeah, if if they're Dynatrace customers, they can definitely talk to the Dynatrace team, and we'll help them through, you know, any of the issues. Of course, you know, we work with Gremlin very closely, and have partnerships at at all the levels of the organization, including R&D, where we have these discussions about how we can best serve our customers and help them be successful. >> Yeah, we got the secret, you know, bad signal line set up in the shared Slack channel. So, if something goes wrong, if something isn't working right, we're talking about it, and we're fixing it, and we're figuring out which side or where we need to go make changes in order to accommodate things, cuz again, what we care about is making sure our customers have a smooth, great experience. And it's nice that both teams have been dedicated there, and it's been great working with all the Dynatrace folks over the last couple of months as we've been getting this ready and rolling it out. Anytime we have a question, anytime we're not quite sure the right way to do something, maybe we have a little bit of feedback about ways that we could build apps better or, you know, better integrate with the platform, there's been a great partnership, and we've been able to to smooth those out, get those things figured out, and get them shipped to production. >> Yeah, and that's kind of also I have another question that and you turned it perfectly segue is that, you know, as AI adoption kind of accelerate, where do you see this partnership heading next? >> Well, I think as we as we've discussed, there's all sorts of opportunity to help us get in front of the wave that we're currently riding. And as we move to different types of systems being built, we start moving from just developers augmenting the way they're writing code to having agentic systems that are writing massive amounts of code. So, we shift from CI/CD pipelines into agentic pipelines. As we see more and more operations start to be picked up and automated, how are we going to vet that all of those are working correctly? How are we going to measure them? How are we going to provide the right safety net, the right guardrails in place? And so, to me, I think that's the the biggest potential is we have an opportunity to have a a great visibility from the Gremlin perspective into our customers' systems. Dynatrace is doing an excellent job measuring all the things and building their own set of great capabilities and truthfully they'll talk about that, but how do we how do we take those early signals? How do we take that analysis and really provide people that feedback loop? Because we want to we want to be able to enable that innovation. There's a say phrase I've found myself saying a lot lately, we want to have our cake and eat it, too. We want to have people moving quickly and taking advantage of AI and we'd want them to do that without having a lot of outages, without having a lot of defects. But, that's going to require creating these safety nets, creating these feedback loops where we're able to do that reliability analysis and recommendation and feed it back into those agentic systems to ensure that we're really finding and fixing those issues and getting them getting them fixed and getting them iterated on so that when you wake up and your code's deployed in production, it's reliable code. It has the right alerts, the right monitors. We know it's working correctly and we've already vetted it through a a series of tests to make sure that it's going to be able to withstand the types of issues that occur in production. >> Sometimes we get carried away when it comes to AI we don't want all the engines as much as what kind of caveats or warning you folks have. Yes, it's good, go embrace it, but please either have these guardrails in place, have proper culture, proper practice in place. Don't just, you know, get, you know, that everything starts looking like a nail because you have a hammer. >> Yeah, I think it kind of goes back a little bit to the earlier discussion where both Colton and I were talking about not all AI needs to be LLMs or genAI. I think relying on machine learning, relying on AI that's deterministic can be hugely helpful and make sure that we're providing, you know, accurate answers, helping engineers as they're trying to debug code and things like that to to get the visibility they need in these much more opaque systems now that that we see with these coding agents, you know, developing on their own with agents being autonomous and so on. Having that visibility and having that explainability and that auditability is really sort of the next evolution I think of of observability. >> I love that, explainability. That, you know, we need to hear that more. We need to talk about that more because sometimes you you, you know, step one, hey, did it do the right thing? Step two, okay, I got some hopes and dreams, but step three, can I prove that it did the right thing? And I think that's that's really tantamount to the longevity of the systems we're going to build. >> Once again, thank you both, Colton and Philip, for joining and sharing your insights. This partnership, and as you rightly mentioned, you know, there's so much happening, so I would love love to have you folks back on the show and to talk more about because this is a problem not going away anytime soon. We are getting more and more AI, so this field will continue to evolve, but I really appreciate your time today and look forward to chatting again. Thank you. >> Thank you.