AI Agents Now Debug Mobile Apps in Real Time | Peter Morelli, bitdrift

You would think that mobile apps would be as easy to debug as back-end service, but they're not. When something breaks on a phone, teams often wait for the next release just to find out what went wrong. Now add AI engines into the mix and most observe tools were not built for them at all. But that is not the case. Bit drift says that they have cracked both problems launching the world's agentic mobile observative platform. And today we have with us Peter Morelli, CEO and co-founder of BitDrift to talk about that. First of all, Peter, it's great to have you on the show. >> Thank you for having me. >> It's my pleasure to host you here. Let's just talk about the company. So, talk a bit about how the company and what led to the creation of the company because you're a co-founder. So, let's talk about uh why you created. >> I'm happy to do so. Yeah. So, my previous job was the VP of engineering at Lyft. Um, and Lyft obviously 99% of its business comes over the phones. And so we ran into a lot of issues trying to understand a great customer experience on their phones. And so probably we worked on this problem for almost 6 years trying to understand and fix the problems that you can get. Um, and then about 3 years ago we spun this out as a company. So this is the technology we've been working on for almost 10 years trying to understand and deal with the scale of both mobile and its very unique architecture. I can go into a little bit of that but I I think that's sort of the origin of the company. >> Before this uh interview we were talking about the problem areas and it's like when it comes to mobile observer it has been more or less like treated as an afterthought for years because most tools were built for servers and backends they were not built for mobile they're kind of retrofitted for mobile. So talk a bit about what makes mobile observative fundamentally different challenging and why does the same tool that works on the server or back end doesn't work here why that approach fails >> that's a great question so generally observability tools have been built this way where you have a an agent or a collector or an SDK on the actual server you collect the data through some sync and you push it to a central aggregation point that's sort of how all most obser that's where you do the graphing and the aggregation and um sort of the analysis that observability systems are built for and they're fantastic if you're in a data center. Um the problem happens when you try to put that on a device and there are a couple of things that happen is one they're all built with those assumptions that you're in a data center. You assume that you have a lot of bandwidth. You assume that that bandwidth is reliable. You assume you have a lot of CPU and storage and that it's always running. None of those are actually true for mobile. In fact, the normal case for mobile is that you don't have a lot of bandwidth. It's unreliable, especially as you move around. certainly if you're in a car um or if you go in a tunnel, those types of things. And so what happens is that a lot of times when our customers had the worst experiences, we were missing them. They like, you know, most of the things they would try to collect some data, push it up there, they didn't have a network connection, they drop it on the floor. Um and so we were only seeing the good conditions and the good experiences our customers were having. We were fixing the wrong problems. The other problem uh was happened was when we scaled uh most you know server solutions like couple hundred thousand servers you know maybe there are more extreme cases but generally you're in the 50 to a couple hundred thousand range we had tens of millions of mobile devices every day sending data I think at its peak was pulling in 45 to 50 terabytes a day just from the phones and so when you're dealing with that sort of scale that's an immense amount of cardality it's an immense amount of data that's coming in if you're if you're trying to do a lot of logging and [clears throat] so what we found was that it either was a scale problem or it was a cost problem since everything is cost based on ingestion. Um, and so most people solve this even to this day by sampling pretty heavily like usually 1 or 2%. So you're getting 1% of only the good conditions and that's really a very very small slice of the problems your customers are having. So that's sort of why we we built something a little bit different. And if we as you're talking about your u tenor at uh Lyft as well you know and that's where you know based on their experience that you mentioned that's what uh led to the company. Can you talk about uh what were some breaking points the moments uh where existing observity tools simply could not give you what you actually needed for uh not not from your end to solve the problem but from also the customer's end you know that what kind of experience they get when they hit that debug button and then nothing happens. So talk about not only that but also the scale at which lift or modern companies work today. Yeah, most modern companies like I think the sampling that people do starts at even as 200,000 devices. It's actually a very small deployment. Um when you're looking at modern companies and the customers like that that we actually work with a lot. So the breaking points tend to be there's there's another sort of thing like in observability a lot of times you know your failures tend to be clustered a lot of times especially in a data center that switch that goes out that one port is burned the canaries have the one build but no one else does. So your your errors tend to be clustered in areas. Whereas in mobile world, I I call it like actually dealing with reality, the physical world, you have all these different variables. And so you don't have this very nice like, you know, fat like tail of the curve. You have a very fat like like uh set of errors that are happening. So you don't have like all your errors clustered. They're spread out of there. And I and then what I mean by that is like let's say you're using a phone inside like you're you're getting a ordering something you're you're getting a car whatever it is you walk outside you've switched from Wi-Fi to 5G even though those are both great like connections you've swapped network connectivity you may drop something you might walk behind a tree and that happens you were driving down the street at 50 m hour and your one cell tower that was working is now saturated even though you have five bars and that's before you add in the like different hardware devices devices that you're you're all you're all um deployed on, the different OS versions, the different app versions, the different feature flags you have. So the world in which your customers are having experience is incredibly varied and all the different combinations are very difficult to debug. And so I view it as like a world of edge cases, right? If you're trying to manage the tail and the P99 experiences, that P98 P99 is very very long and very very fat. And so if you're sampling and only getting the good conditions, you're missing most of your customers problems. And that that was the breaking point for us is that happened too many times over and over again. We would try to figure out a customer's problem, couldn't replicate it, took us forever because the mobile release cycle was 10 to 15 days, right? And if you're waiting 10 days for each set of logs to come back, each new set of things, it's a very long debugging cycle. So that was one of the other problems that really broke. Um, and after years of asking the vendors to try to solve this with us, um, and then still not doing it to this day. Um I think that's what was sort of the breaking point for us and where we said we need to invest a lot of engineering time in this. >> I would talk about company also a bit uh but I also want to talk about the AI side. You folks launched Bit Drift AI which gives AI agents kind of programmatic access to real-time telemetry from actual user devices. Can you walk us through what that means in practice? What can an agent do today that a human engineer could not have done before? >> Yeah. uh to do that let me describe a little bit about how bet drift works because I think it's pretty integral to the the system. So you know I mentioned before that uh most observability systems they collect data um aggregate it into a central point. We do it very different. We process at the edge uh almost exclusively and so we create a series of ring buffers on the phone where we collect as much telemetry as we can get uh stuff that we collect but also what whatever as a developer you put into that that sort of collects locally. We don't persist it by default. The other part of the system though connects to every device in real time and says, "Hey, here are the conditions I'm interested in." From the server side, you're able to say, "I'm interested in this cohort or this conditions or this customer's had this error four times or more, but only in San Francisco." You can have a lot of flexibility of saying, "Here are the conditions I care about." And you can either pull the data server side to actually uh do it from matching phones and analyze it or you can process it at the edge. Say I just want to count how often that is happens. You can metricize all the logs that are happening. So now you have a system that is sort of freed from the mobile release cycle. It's also freed from the ability of should I log or not, right? We're not not charging you based on ingestion because we're we're storing and processing it locally. And so all of a sudden you have this sort of distributed network and and a live view into it and saying I now want to pull data when I have conditions and problems um or things that I want to investigate. So that's great for a human when they're debugging a problem. It's even better for agents, right? Agents work in very iterative loops, right? you sort of say, "Hey, why why is my onboarding funnel broken?" Uh, you can go and say, "Hey, well, I'm going to start investigating." You know, the agent does the same very similar thing. If you were stuck with other, you know, observability providers, you would have to wait 10 or 15 days between each loop as they ask and get for more information if you don't have that up front. And so, for the first time, we're actually able to give people access to this data that they never had before. Remember, this was sampled at 1% and only the good conditions. Now you have unsampled views of all this data and you can give that to your agents and they're able to loop iteratively, ask for more information, deploy more observability, get more details, do more analysis all without having to wait for that mobile release cycle. And so agents are actually quite good at doing this. Humans generally like you know focused on the most you know the the highest priority problems. We have a customer that says like we typically only address the top 20 or 30% of problems. Once we you know added bit drift AI and started using the agents we were able to get to almost 80 to 90% of the problems. Remember I said it's that very long tale of problems right and so being able to address all of that uh proactively u is something that I think is a really unique you know sort of thing that is a combination of both our product but really what you know AI has been enabled a lot of lot of engineers to do. [snorts] So I I love that productivity gain I love the actual we're making the customer experience better for a lot of customers. It's great. >> Excellent. Thank you. And when it comes to uh observability uh building it for humans means of course dashboards charts the whole drilled on workflows but building it for agents is a totally different design problem totally different governance problem as well. So talk a bit about um uh this fundamental difference the challenges that you folks faced at the same time how your potential customers can use them because they are so used to but now today everybody's getting used to agent based you know agent workflow so that's not a challenge but I do want to understand this you know this difference technically philosophically culturally >> yeah so I think we we designed the architecture in a way that I think was built for built-in layers. We're all, you know, my co-founders and I have built a lot of platforms at many different companies. And so we approach this the same way, which is to build it in terms of layers. And so, you know, our our UX is driven off our API. And so, we have at the bottom layer our sort of API. It's now a public API. On top of that, we have different interfaces. One is the user experience that everyone sees. We've also created a CLI that makes using that API easier to use. Both of those are great for humans. All three of those are great for humans, depending on where you want to hook in. On top of that, we've added agent skills. And so now we can teach uh AI agents how to actually use these tools that are provided. So you now have a system that has nice layers. You can actually interact with it at every sort of whatever choice of layer that you want to. And so the way we built it, I think was built with, you know, programmatic access in mind, which made it very nice to actually port over and have a AI agent start to take advantage of it. So from our perspective, porting over to it, I think took a little bit of thought. We redesigned some APIs. I think we redesigned the output so it was much more efficient. The CLI was built really with token efficiency in mind so that the common things would not take you know a thousand API calls to do. So we think of we we think of you like the agents as an actual user of our system. It's a separate persona. It's a separate user and as we design them but the whole architecture is sort of stacked together in a way that allows both humans and agents to really use it. Can you talk about this whole uh ring buffer concept? Uh what it is and uh what does it also mean for capturing everything locally without killing performance and battery? Uh how do you supposed to do that? Yeah. >> Yeah. So the core the core system is like you you to to use our system you take an SDK that we provide you um and you install it into your application that applica like that application then sort of can collect telemetry and and use that that SDK. The SDK what it does on initialization it creates a ring buffer. It's a it's a sort of a a fixed size. We don't we don't want to like you know grow with unbounded things. You can you know a devices has limited sort of capacity. Um and so we we give a a limited amount of memory. Um and then as you write and collect telemetry as it gets to the end it sort of starts at the beginning. That's the ring part of it. And so you know you have a fixed size of of memory that you're allocating to collecting telemetry. um that that actually can grow and and shrink. We have customers who you know are used on kiosks or point of sale they they make that 200 megabytes right or 800 megabytes. We have also customers on mobile devices that do it at five. And we did a lot of engineering like a lot of work to to make it performance on mobile devices even low-end devices. So you know you have 16 milliseconds before a human will notice you know a freeze or a lag or something that is going on the UX. So everything is done in the background. um everything is done as async as we can. Uh the core of it is written in rust. Um so we get a lot of performance and memory benefits from that. So a lot of the work is really focus on making sure we can collect a lot of data without affecting the user experience. The ring buffers are nice. Um we memory map them to disk so that they persist between crashes or you know someone force quitting something or the OS reaping you. So you have this data beyond crashes. It's one of the nice features of our thing. So let's say you don't have a network, maybe you don't use the app for a week, you still have the access to the data and we collect it and and hydrate it afterwards. So the ring buffer allows us to say to engineers, you can collect as much data as you want. You won't blow out the the device. You won't cause performance issues in your app, but you can collect a lot of data. And that 5 megabyte buffer, even for a chatty app, is usually good for days of data. I mean, obviously, if you're putting like 20 megabytes a second of data in there, it will run out. But generally we're giving people a buffer that allows you to play with data and and add it. So most most people now you don't have to choose should I log this or not. You just log everything and we sort of manage um that the sort of the persistence we manage making sure you don't blow out the device with that ring buffer. >> Since you know as you mentioned that uh you gave beta access to customers. Can you talk about what kind of improvements they saw when they were testing it and um what that feedback led to what was it like you were expecting u what changed about uh like during that beta phase because we're launching it today so now we'll start seeing in production but what kind of feedback you receive from them where they're like hey you know this is what changed today after going to this beta and start using it versus what I was doing yesterday. Yeah, that was good question. So, um, our customers almost universally loved it. I think it really improved the efficiency in a a couple of different ways. One was in the integration and then the addition of telemetry, right? Instead of saying, hey, let's manually add log lines or manually adding things, you could tell the agents, hey, I want to I want to use Bitrift and then add the logging that we want to. So, stuff that used to take, you know, a couple of weeks or a month like was compressed down to days. The other one was you know the the system we have of saying hey go out and find matching devices and matching data conditions or cohorts and pull it back that is its own sort of like set of information and it's a cognitive load that you know people and developers had to learn and it's it's pretty intuitive from a UX perspective but being able to say and ask higher level questions like why is my onboarding funnel broken right where are my performance issues today those highle questions were able to be mapped down and like broken out into investigations without humans having to break them down and then deploy them and learn them. And so stuff that used to take, you know, even during PC's we would see people like write like 10 of these you know really high level questions and be able to sort of you know you know spin them up and get them running within a day. So I think from a speed of execution a speed of iteration lightning like like huge changes and so what that allowed one was fixing the problems a lot faster. So MTD and MTDR like really dropped, but it also allowed them to address a lot more of the problems. I mentioned that example of someone who, you know, probably best case was hitting 20 or 30% of the customer issues that were popping up, the highest priority ones. They were able to address up to 80 and 90%. And that's a huge shift in both, you know, the customer support that they have to deal with, but also in like the experience that customers are having. >> Perfect. Awesome. Thank you. Now uh if agents can query uh reality on mobile devices today u where does this go? I mean it's very hard to actually predict where things will head but but uh if you look at also these days I feel that more and more people are using mobile devices. I feel that uh even the AI workloads I don't know it's very hard to predict actually these days. But if if I say because of bit drift work what you folks are doing how it will redefine the whole debugging the whole mobile platform and things are becoming more and more prof so I'm not asking how things will look like in 10 years from now or 2 years from now but just how do you see the evolution of this space we've already seen in the last year just a huge like increase in the the volume of code that is being generated right I think the agents have especially made generating code and generating applications um really really much easier and cheaper than before. Um I think the number of app submissions has more than doubled. I think in the last year we have app generators that are doing this. I see more and more applications being built both internally and ex for external you know applications. And so I think the volume of this sort of deployment will only increase. You're only going to see more and more compute at the edge, right? Like now your TV is actually running Android under the covers, right? you see your kiosks at any airport that you go to are running Linux or Android or some variant of those. So you're only going to see more and more cases where there are devices that have compute relatively powerful compute compared to early days of like sort of IoT. Um I think the problem that you know we've talked about like being able to access all of the data right even in adverse conditions even with you know timing like challenges is only going to get more and more and so I think this for the first time allows you access to that stuff that before that data was really only done server side you're really only trying to interpolate it and trying to figure out what was happening from a very small sample so I think that is the big difference the other one is the sort of real-time nature of it right like if you think mobile phones take a long time to update like that 10 or 15 day cycle to get a new you know app version out how often do you update your TV like every 6 months never a lot of people don't do it because they don't want anything to break so the problem that you face where you're you're run your your your observability has to be more runtime versus more compile time that's the big difference and that's the big shift that this architecture allows you not you're not having to wait for pushing code out to be able to understand problems and debug them and and try to fix them and maybe mitigate them and I think this you the there's only more and more code that the the process of supporting and maintaining software where a lot of people are focusing is not nearly as developed. I think PE will continue to develop on that but you'll need architectures like this to actually feed them and that's where we see ourselves as fitting. Of course we are going through all this geopolitical crisis a lot of you know regulations are happening in Europe in US a lot of things are happening Chinese AI US AI uh what kind of impact do you see will be there because the mobile market is a global market it's not a regional market iPhone user globally Samsung globally so first of all how do you see the impact of this changing geopolitical crisis on this space and you feel that you know actually this is the right think the right time by giving users developers more control more power to solve the problems. What do you think? >> Yeah, it's a good question. I think it's complex. Um I think for Biter specifically, we built the system to be very friendly to data and privacy um sort of considerations. By default, it collects no PII. Um we don't, you know, prevent customers from doing there's no new permissions that you have to ask us. We generally defer to the application developer about what data they want to collect. Even when they want to do matching on user ids, we hash them. So we don't have access to that data. So you know our session replay is a wireframe. It's not actual, you know, screenshots. We've done a lot of things to make sure that, you know, we are we are a good good citizen and in fact a great citizen in terms of like privacy and data. Uh on the server side, we're able to, you know, data home wherever, you know, customers need us to be. And so we can put, you know, the customer data they collect in in their environment. We can run our whole stack in their environment. We can put it in in the EU or other places. Um I will say we have not done any business in China. That would probably be more problematic. But generally, um we built a system that's very flexible in terms of what data is collected and where it can be put. The nice thing about our system is that most data stays on the device, the customer's device, unless you have a clear business intent that you need to fix a problem or something. And so, a lot of our financial customers love that it stays on their their customer device unless they're specifically trying to find a problem uh and fix it and that you can turn it off then when you don't need it anymore. So, I think from from our perspective, um I'd say the biggest impact will probably be the costs, increases in chips and memory that you know is being seen across the industry. I think that will just sort of drive and sort of hinder the adoption of more and more compute at the edge. I think that's just raising costs across the board. That's the short-term uh cost that I see. >> Thank you. Now, let's talk about the market. First of all, how crowded is this market? Uh and u how are you positioning yourselves as the pricing availability and if possible if you can also talk about u some of your clients? You can name them not name them. So there are two or three questions bundled together. So generally, you know, I I've been a buyer of software and observability software for quite a long time um at many different companies and I I pretty much hated every single vendor I've worked with. And I think it's really due to the pricing model that they have. It's an ingestionbased model. So if you log more or you add more metrics or you collect more sessions, they charge you more for the privilege of doing that. Um and that's true of all observability, not just mobile. Um the the problem with that is that then you're sort of in this weird scenario where you're like, should I log this or not? I had whole observability teams that would go to other engineering teams and be like, "You're logging too much." Which is a sort of an odd incentive to sort of put into these systems, right? You're trying to fix your customers issues. Why are you not, you know, doing it? Everything's a trade-off. I get that. I think what what I really found was that it was a very antagonistic relationship with most vendors. So, we designed our pricing. Our architecture again, we process a lot of things at the edge, allows us this flexibility, but we price not based on ingestion. So, we're not you can log as much as you want. You can create as many metrics as you want. Um we're not putting any limits on that side. Um what we do prices is per unit. So think of active device or MAUs. So if our SDK is in an application, it phones connects to our our SAS or or your hosted environment. Um that sort of counts as one. It can connect one time, connect a thousand times, we sort of, you know, charge the same price for it. And so what that gives to to buyers especially is a very predictable um sort of bill. There's no session overages, there's no quota overages. you're not like surprised at the end of the month that something's been blown out. So, that's a very nice like uh sort of feature of how we price. The other one is that our incentives are very much aligned. The only time I'm going to grow is when you're going to grow and I think you have a better sense of how you're going to grow and can predict that very well. So, we become a very nice predictable player in your sort of it's a line item that's like very nice to have as as a budget owner. So, that's one of the the nice things. I think generally though it also then you know we can go to engineers and say don't worry about logging log everything because if you log it then it's available right at runtime and so again the whole point of trying to make as much of observability runtime versus you know compile time this is part and parcel of that particular um that equation. Peter, thank you so much uh for walking us through this um what you're building is I mean fundamentally rethinks how mobile operator should be done and how it should work to make it not only agent accessible but I to to make it kind of first class citizen versus a second third or fourth class citizen. So thank thank you so much and of course all the folks who are watching it please go check out bitdrift.ai AI to learn more about it. And once again, Peter, thank you so much and I look forward to chat with you again. >> Yes, thank you so much very much. Appreciate this.