Enterprise AI Success Depends on Orchestration | AppDevANGLE

[music] 51% of organizations rely on public AI tools such as chat GPT and co-pilot while only 20% report enterprisewide AI deployments built on governed frameworks. My name is Paul Nash and the practice lead and principal analyst and this is the appdev angle. Today I'm joined by Molly Moore from Live Ops, a company that sits right at the intersection of AI, customer experience, and very real challenges that makes uh intelligent technology work in production at scale. Molly, how you doing today? >> Great. Uh, thank you so much for having me, Paul. >> Yeah, thanks for being on. Why don't you introduce yourself and introduce your company? >> Awesome. So, I'm Molly Moore, president and COO of Live Ops. We help enterprises solve kind of one of the biggest challenges in AI today, turning um what we would say promising technology into real world customer outcomes. So we sit at the intersection of AI, customer experience and operations. Our role is to help companies determine where does AI belong, where do the humans belong, and how do those experience work together. So we're combining tech, operational to design, governance, and a global workforce to help clients move from experimentation to execution. >> Very very relevant these days. I mean, everyone's trying to figure out what to do with their AI budgets. Um, you know, organizations are trying to, you know, take that 2025 experimentation and put into 26 reality of implementation. So very very relevant real time. So here's what the data is telling us, Molly, that 92% of organizations report that AI capabilities are now integrated into at least one stage of the software development life cycle. This is a sharp increase from 71%. And the adoption story on the surface uh sounds like progress, but the adoption and execution are two very different things. The gap between where enterprises are are are struggling is is a is a is an issue. The governance uh picture is concrete. more than half of the organizations still run in public AI tools like chat GPT or co-pilot you with no govern framework underneath them that's not a technology problem that's an operation problem right so we see this as a as an issue that's that means decisions are being made workflows are being shaped and customer interactions are happening really inside systems that nobody formally are accounted for and when something goes wrong nobody owns it and that's a problem accountability is definitely really what we want to get uh make sure our organization understands so Molly Lots going on here. Um, you know, when we look at from pilot to production, execution is where AI investments win or lose, right? This is really where it's happening. And most enterprises can stand up a proof of concept. We see VI coding all the time. That's not the hard part anymore, right? That's the that's the thing that people are getting getting done. The hard part is actually AI integrating into the real workflows within the real governance structures and the real customer journeys at scale. This is where I think you were talking about, this is where you come in. Let's talk about that. What are your thoughts on that? >> Yeah, I mean, we're seeing a lot of unsuccessful AI um projects, right? And you hear about them in the news. You hear about it every time you talk to uh somebody, you know, in an organization. So, the the companies that are successfully operationalizing AI understand that deploying technology and operationalizing technology are two very different things. So most enterprises can get a pilot running that's easy but pilots happen in a very controlled environment. Production environments are where the complexity shows up. You end up you know you're dealing with your customers uh regulations, security requirements, compliance obligations and likely multiple departments. So that gets really complex really fast. So the companies making the most progress start with operating model first. They define ownership. They establish governance. They redesign their workflows to be more effective. So putting tech just on top of existing workflows is not going to help support. So those companies that are really determining how decisions will be made and who is accountable is really critical and then they apply the AI. So the the companies that get stuck in this pilot mode um are doing the opposite. They're starting with the tech and hoping hoping the organization can catch up later. What we're seeing is that execution has become the real competitive advantage. AI adoption is becoming common. Operationalizing AI successfully successfully is still very rare. >> Yeah, Molly, I I like where you're going with that. That that that definitely tracks and resonates to the market issues that we're seeing. AI exposes broken processes, right? It it it doesn't fix them. it actually just exposes what's happening and there's a belief from a lot of organizations that AI will smooth over the operational dysfunction and the reality it doesn't right the data suggests just the opposite actually we're seeing that when when you automate fragmented workflows you're accelerating fragmentation and when you automate unclear escal escalation paths you make uh the confusion faster right it just really is it's you're kind of garbage in garbage out model here right so the top pain points that we're seeing with AI and operation models is the reliability at 33% operational complexity at 31% and compliance at 15% almost 16%. And you know those aren't technology failures the symptoms of process debt that's hitting new surface areas. So the question I have for you Molly is when we when you work with organizations and you see that there's limited returns from the AI investments, what process issues tend to be the root of the cause like root cause of it all? Where's the where's the challenges here? >> Yeah, it's um kind of as we discussed a second ago, you know, organizations frequently have fragmented workflows. They don't have clear ownership. They have disconnected systems. I mean, I think that's what we see the most of is these disconnected systems that are not integrated together and then inconsistent escalation paths. They've been creating their CX journeys for years and they've shifted and evolved and they haven't really stopped to say h how do I build this from the ground up? So, you know, if ownership isn't clear, AI is going to expose it. If the escalation paths are broken, AI is exposing that, too. And if the workflows were inefficient before, boy are they even um accelerated after uh implementing AI. So I believe that AI doesn't fix the broken processes. It really exposes them. So you know the companies that we work with that are seeing the strongest results. They're not asking where can we deploy AI. Uh they're asking what outcome are we trying to achieve? Then they redesign the workflow around that outcome and determine where AI and human expertise fit within it. You know, I feel bad for companies because they're getting so much pressure by their boards, their leadership, their investors to do something with AI. Uh, but really what they need to be looking at is operationalizing. >> Oh, well, you're 100% correct. I mean, we're seeing in our research a 25% of 2025's budget, IT budget was allocated to AI projects with no real understanding of what that was a what those AI projects were doing. There was no ROI. Tokconomics are a big problem. We see that the uh you know the the measurement or metrics that people are using to understand what's going on with the AI adoption. Is their workforce increasing the uses of token? All they're doing is burning money and that's not really the right use of money. So, but let's let's look at this from a customer experience perspective. The future of customer experience is really about orchestration, right? Not automation. So, the question that most organizations are asking themselves whether AI should replace human agents and and to me, I think that might be the wrong question. The better question is where does AI belong and where do humans belong and how does that experience move cleanly between the two of them, right? And when we see in our research, we see that enterprise leaders, AI leaders are envisioning a future where, you know, these agents, human, non human, AI agents, whatever, are actively collaborating on complex task and sharing goals, not replacing one another, right? That's not really what we're seeing. It's really about collaboration. So these top agentic AI priorities and to your point about budgets uh these top agentic AI priorities that need to be kind of accounted for are around automating repetitive tasks we see that at 73% in our research decision automation is at 71% and AI assistance are at 71% all pointing towards argument augmentation not replacement and you know the question I have for you Molly is you know how do you look at the highest performing organizations that you work with and how what is the handoff between AI and human agents and non-human agents in this customer journey? What what are your thoughts there? >> Yeah, so the highest performing companies are not thinking about AI versus humans. As you said earlier, they're thinking about orchestration, but you know, I want to pause a moment here and because I think there's a lot of confusion around what orchestration means and what the definition is. So we believe orchestration is simply determining what AI should do, what people should do, and how the work moves between them to create the best outcomes. So the question isn't whether AI should replace people, it's where where does AI create the most efficiency and where does human judgment create value. As you said, AI is very effective at repetitive structured rules-based interactions. But when complexity increases, when emotions are involved, when compliance matters, or when a customer needs that reassurance, human expertise becomes incredibly important. So I think maybe an example would be relevant here. So let's say in healthcare, you have a member uh trying to understand a denied claim. AI may be able to gather the information about that patient, verify their eligibility and explain the process generally. But if the member is super frustrated or confused or upset, that's where human judgment and empathy become critical. So organizations doing this well create a seamless transition between AI and the people uh the humans that can help. So context follows the customers and information isn't lost between the handoffs. The customer never has to start over and the data tells us this is one of the biggest frustrations. We don't want to have to reexplain oursel. So the handoff itself between AI and humans that is often the most important thing over the the automation itself. >> Yeah, Molly, I I agree. I I I'm looking forward to a world where that omni channel experience is seamless and there's, you know, you pass the conversation from your text messages to your voice messages to a phone call and you don't have to repeat yourself each time. I mean, you know, you know, we all experienced it. You go into a an appointment and they give you a clipboard and you fill out the same information three, four, five times on the same piece of paper or the same thing. It's really frustrating. But, you know, I also think that there's interesting uh ways that organizations are looking at things. Most organizations are measuring AI success really with the wrong scoreboards in my opinion with that. They're looking at speed, containment, and cost reduction. And and like when we look at this, those metrics are most AI deployments are what what are getting evaluated on. But customers don't care about any of those when we look at it, right? They care about resolution, effort, trust, outcomes, that personal experience you were talking about, Molly, and the significant perception gap of inside organizations that reinforce the problem. You know, we see 45% of AI practitioners site operational complexity as their primary challenge compared to 31% of managers. They they they're looking at it going, okay, well, the practitioners have a different view than the managers, right? Um, and these managers are commonly prioritizing reliable outcomes. you know, those are the ones that we're looking at going, "Okay, is that is that right?" Um, when people are closest to the work or sitting setting the strategy, they're not measuring the same thing. So, you end up with optimizing for the wrong results, right? And and you know, I mean, I was talking to one company recently. We're talking I mentioned tokconomics earlier. Measuring success by the number of tokens that are being used is not the right measure, right? And I can't emphasize that anymore. But you know, Molly, a question for you is what does it look like when an organization realigns its AI metrics around customer outcomes and what does it take to get there? Because I mean that I think is really what's more important. >> Absolutely. Can't agree more. I think you know operational metrics tell you how efficiently the company operated which is important but uh they don't necessarily tell you whether the customer succeeded. These those metrics matter, but customers don't often experience met uh metrics. They experience outcomes. So customers measure something very different. They care whether their issue was resolved. They care whether they had to repeat themselves, how much effort was required. Do they trust the interaction? That's the key. That's the most important thing they ask. Did my problem get solved? How hard was it to get help? Did I trust the experience? Did I get the outcome I needed? So this boils down to if efficiency improves while customer outcomes decline, you have optimized the wrong thing. So one of the things that we found in our AI maturity benchmark is that mature organizations evaluate AI very differently. They're asking whether AI improves resolution quality, customer effort, operational consistency, and business outcomes. So the most successful companies don't optimize for speed alone. They optimize for resolution and trust and that is a fundamentally different scoreboard. >> Yeah. Yeah. Molly, I agree. I want to add to that. I think that a AI maturity is a leadership problem, right? And before it's a technology problem, right? I think the skill gap is and the data that we see is is is is uh evident, right? We see this that 82% of AI teams report skill gap in their AI operations, right? And 31% describe these gaps as extremely prevalent. So there's a lot of skill gap issue here and that's a big issue that we all have to overcome because you know the the the tech the deeper issue isn't technology skills it's governance operation models readiness and organizational alignment that our leadership responsibilities right accountability that really is a leadership issue and many leaders don't yet have a clear framework on where their organizations actually stand right and that goes back to some of the metrics and such but you can't build the road map without having an honest map of where you stand today, right? So, understanding that Amali, what framework do you do you use to help leaders kind of assess their organizations in real AI maturity? You were talking about your study and such, but like what framework are in place and what do leaders tend to underestimate when they're getting started in these conversations? >> Yeah, great question. And I can't agree with you more that this this all is a leadership challenge. Technology is just one component. So we encourage leaders that we work with to evaluate their maturity across four key areas. First, governance. Do you have clear ownership, accountability, risk management, and decision-making structures? I think people, this is the one that people underestimate the most. you know, you should have an AI governance committee across your organization working with different um groups and departments across the company really driving and providing direction and guidance for the business. Uh second is workflow readiness. We've talked a lot about that this but are the processes you're trying to automate actually optimized? You can't optimize something that is broken. Um and then third, organizational alignment. our technology, operations, customer experience, the compliance team and leadership working towards the same outcomes. Do you have written outcomes uh that you're trying to achieve as a business? And then fourth is execution. Can you consistently operationalize AI at scale and measure business impact? So when we work with companies, we take them through this maturity model. We assess them. We help them create a road map. We define are they in a crawl, walk, run stage and with that um position where they are h what what what roadmap should they optimize for first? Um and so different types of AI technology are more in the crawl stage versus the run stage and so making sure that you're thinking through those uh you know issues before you deploy is really important. So I I think leaders tend to underestimate the first three the organizations really creating that long-term value focus on creating the right operational model. >> Yeah, Molly, I agree. You know, and I also think that organizations do they do buy vision. They don't just buy what's on the truck today because they need to know where it's going and we're all really trying to figure this out together. But before we wrap, Molly, you know, the the listeners here, there's a lot we talked about, a lot to unpack. if they're in that crawl, walk, run stage, where should they go to learn more? >> Ah, thanks for asking. So, you can learn more about us at liveops.com. And I think, you know, if you're trying to understand where your organization stands in its AI journey, as I mentioned, we launched that AI maturity benchmark that's available online and we have an AI readiness assessment um interactive online. It's free um on our website and you can get your AI maturity score. Um but thank you for having me today. I'd just say if there's one takeaway I'd leave listeners with um regarding our discussion, the winners won't be those with the most AI. They'll be the ones that orchestrate AI people and workflows most effectively. >> Well said, Molly. Well said. Thank you for being on the podcast today. I really appreciate you being on. >> Thank you so much. >> And thank you for watching for all of you who've tuned in. We do appreciate you being part of the App Dev Angle community. But for now, that wraps up this episode. But we'll be back next Wednesday with another conversation diving into tools, threads, and talent ship in the future of the application development. Whether you're deploying at the edge, building with AI, or modernizing your cloud stack, we've got you covered. Be sure to follow us on social if you have any thoughts, questions, or just want to connect. Until next time, stay curious and stay building.