Escaping demo-land: (actually!). Shipping voice AI to a million users

reviewers. Very good. But Google question for you. Pitch report. So solid high scoring matches prediction. I think 181 depends on the batting line. Blue guy and Mumbai accent. Aussie accent right mate. So you're a Mumbai Indians fan, are you? The blue hoodie guy, no less. Probably seen him choke more times than a kangaroo in the desert. All that cash and still can't buy a decent performance. Can I? >> Hey sh. So nothing like a talking monkey to wake you up after lunch, right? Um you just met googly googly one of this was one of the live streams that Googleby did during the IPL uh this year. And googlyhai is the world's first AI sports companion. Uh he had cumulatively over 1.5 million minutes of watch time this IPL and there was one particular stream where 1.1 million viewers were looking at googly at the same time. Um he just doesn't do live streams. You can actually call him personally. So users on Dream 11 can actually have one-on-one sessions with googly and he can talk to 10,000 users at the same time. you know that's the number of concurrent uh conversations that he can have and this is a story of how we built googly um just like anything new that is done this is not a linear process this took us a lot of time and a lot of prototypes three times it became so bad that we have to chuck our entire approach and the code base and start from scratch and we had a lot of bad advice which is why I'm here uh to sort of mitigate that before we go on uh this is who I am. My name is Ru. My recent avatar is that I am a staff family at Dream 11 where I founded the foundations model group where we tried to unify intelligence into a single model instead of having like a bunch of models doing different things. We were able to do that and we had 3% increased DAU overnight by replacing all of these different models and powering personalization through one model. So if you're doing B2C personalization, we've already published a technical report. You can look at that. I'm actively looking to work on post- training inference and agentic systems. So if any of you are building that, I would love to talk back to Google. Uh how many of you are thinking of building planning to build a voice agent into your app uh or in the early stages of building? Okay, very few. Um and then my next question is obviously how many of you have shipped it to production? Uh okay, there's one. But in general, it's uh there are a few. In general, it's the gap between a demo and a production when it comes to and uh I'm sure a lot of you would have already experienced this is a lot. It's very easy to get a really good demo. It's very hard to make something work in production, right? And um before we get into the nitty-gritty, the fun stuff, [snorts] uh I want to look at the problem that we were trying to solve. Why did we build this? Right? And how many of you remember Virat Kohli 6 over the top of Harris Rough at the MCG? Right. Yeah, a few of you. That was a pretty good moment. I remember that moment because when that was happening, I was with my brother and my father. We were at a Taco Bell in a mall in Mumbai and I remember like we barely touched the food. We were looking at the screen the entire time. And moments like these are special. Obviously I remember that moment not because only what Virat Kohli did but because I was able to share it with my dad and my brother right and so sports is it's a lot about these moments and who you share it with and we wanted our users to always have someone to share these moments with and that's why we built googly uh and before we began building there were some core tenets like that came out of like what we were trying to solve and first we'll cover those core tenants And the first was we are building for conversation. This is not a CX bot. A conversation is when you feel that the person across from you is really hearing what you're saying, right? And this is actually much harder than it sounds. It's very easy to build something where uh you give off the impression that you're already a bot. And that just did not work for us. Like we were not going for that. We had to solve for actual conversation, right? And because conversa real conversations happen over voice, we did not want to build a chatbot. It had to be voice first. Uh it had to be functional. So a conversation without context is nothing, right? And obviously we wanted googly to have like all the statistics, all the performance numbers, everything. But we also wanted to him to have strong opinions and also wanted to him to know what's happening off the field. All the gossip, all the controversy, all of that stuff should be at his fingertips. Right? So functionality is also important. And this goes without saying at but at Dream 11 when you build something it better scale because during the IPL for example we have close to like 12 to 13 million users and the peculiar pattern is that all of those users just because of the nature of sport all of those users want to come in at the same time and so when I'm saying scale what I really mean is concurrency and that's it those are the tenets And now we'll start the journey. We can sail on this journey. And the first stop is obviously how do you solve for conversation. And the main thing being making users feel heard like I want to start with why voice is different from text right because a lot of us come at this from building chat bots and thinking okay uh voice it's a voice bot. It's not a lot different than a chatbot, but there are fundamental differences between voice and text. And the first, this is probably the hardest one, is turn taking. When you're chatting to chat GPT, your cloud or Gemini, um, you don't think about it, but when you're chatting, there is a very structured interaction. When you're done typing, you press enter. Your turn is over. It's uh, it's the AI's turn. When the AI is thinking, it's showing you it's thinking, its output is streaming. When it's streamed, you know it's done, right? So there are explicit turn signals and the interaction itself is very structured. In voice when I'm talking to you, I don't press an enter button to let you know that I'm done and you don't press an interrupt button as much as you'd like. I don't have an interrupt button to make me stop. The model is sort of expected to just get it when you're done speaking and just get it when you're interrupting. And that's actually very hard. Um the other challenge is voice carries emotion beyond just the words. When users share their voice with you, they expect that you just not listening to the words, but you're also listening to the emotion and the tone beyond these words. So when you ask users to share their voice, you better respect the emotion that is underneath. And the third thing is latency. And this will actually be a recurring pattern that latency is death for voice conversation. I'm sure all of you have had a call from a voice board for loans, for whatever. And the first giveaway is that when you stop speaking, the latency between your stopping to speak and the bot's answer is the first giveaway that you're talking to a bot, right? In text is actually the other way around. If you get an instant reply, you think it's a bot, right? And you actually expect that there's a human on the other side. You'll see those three dots and then a reply will come after like 2 or 3 seconds. So for voice, it's actually the reverse. And with that we have our yard stake of what we want from a solution is like we want natural turn taking we want emotional fidelity and we want sub2cond latency right and that's what we're looking for. Uh the first attempt is what the conventional wisdom is in this space at least was up until 6 months ago is that you look at a stacked approach and a stacked approach is something like you already have a chatbot right so it can already chat why don't we just sandwich the LLM which powers the chatbot between a speechtoext model which converts user speech to text feed into the LLM the LM will give you a text response you feed it to a texttospech model right and this is like you know if I'm if I'm really good making cycles is me thinking, okay, I'm already good at this cycle thing. Why don't I get an engine and a silencer and put it on the cycle and I have a motorbike, right? Uh it does work. It's not that it doesn't work. And um you can actually do this and you can use a VAD for turn taking, which is a voice activity detection. Uh but let's look at how that actually sounds. Hey, Duvari googly here. How are you doing today? >> Nothing much man. How about you? How are you doing? >> I'm doing absolute damalbas. Just sitting here watching cricket and waiting to tear your IPL team to shreds today. Which team are you supporting Droo? Tell me. >> Bro, I'm not going to watch the IPL this season. I have too much work. Work is just a cheap excuse for people whose teams lose every single season. Come on, man. Don't play safe with me. Which team's fan are you really? Reveal the target, cheek. And you know, yeah, a lot of you might be cringing at that. We were really happy when we actually got there because the one thing that it got right was able it was able to pronounce English words and Hindi words in the same sentence. That's actually very difficult. But obviously that was terrible. And I I I don't have to tell you why it was terrible. Turn taking was not natural. The latency was abhorent, right? And it did try to have some emotion but was really really awkward, right? And this is almost by design because when I'm speaking something my emotion is not my emotion is stripped away and the speechtoext model just looks at the text right and not the emotion and so all of it is lost uh during transcription. So this was not surprising uh why this is like like all these models are pretty good and all these models are actually bestin-class right so there were four models the VA the uh speech text the LLM and the TTS all of these are bestin-class but the problem is that we're trying to make them pretend that this is just one model right and that's the problem and these were never designed to work with each other at a given time the best ETSs the best ST the best VA the best LLM is all going to come from different providers they were not designed to work with each other and when you try to make them work it's it's just hell right and n attempts later n is fairly large here right we got to a solution which was full duplex speech to speech full duplex means there is no turn taking a model is actually actively listening all the time and apparently it can actively respond to you anytime right there's no explicit turn taking that my turn is done now the model which is very very much like how it happens when we're speaking like we you're continuously listening to me when I'm speaking you can interrupt anytime right and let's have a listen to this conversation >> hey Duv what's up my brother how are you doing today >> hey googly I'm doing all right yeah how are you >> I'm good bro just chilling. So, which team are you supporting this season? Yeah. Ready for me to tear them apart? >> Yeah, this season I'm not going to watch the IPL. I'm pretty busy with work. What's your absolute favorite moment from the something that really stuck with you? And that was much better. And the reason is like we don't have four models pretending to be one. We actually have just one model which takes audio in and spits audio out. Right? And it's a full duplex model. So it's continuously listening. There's no turn taking problem here. So we have natural turn taking, we have emotional fidelity as I hope you could have heard. And the latency is much low because you don't have these four models. And yeah, so we solved the model problem, but we had the model, but we were having problems connecting with it. So we started off with a REST API approach. We moved to serverside events, SSE, we moved to websockets, but it still sounded pretty terrible when we tried to make it work outside the demos. And the reason we figured out later was it was a fundamental problem with the network protocol. Web soockets, SSSE, REST API, all are based on the TCP protocol which prioritizes correctness over latency. And it was never meant uh for voice or video for streaming. It was always meant to get your mail, right? It's a pretty good protocol when you want to uh text someone or send an email. UB is the protocol that is actually prioritizes latency over correctness, which is what you would naturally use if you were doing voice. And so we switched from websockets to WebRTC. Uh websockets uh TCPs to websockets were what roughly UDP is it to WebRTC. And WebRTC in general is harder to host and harder to scale. So we used live kit and they were pretty good with it. Uh we still um the model server and the live kit servers still had a TCP based connection but that was server to server so it's fine. But what you really need is a web RTC connection between client to server which is what uh solve the problem. And so that's it. We have actually solved conversation um uh with the model and the network. But let's come to functionality like again a conversation without context is empty, right? And the conventional wisdom is to add functionality using tools and MCP servers. And the problem with that is tools are essentially the models actually can't call tools. they can just generate text or tokens. So what a tool call basically is it's two model inferences. The first is the model figuring out it wants to call a tool, giving you back how it wants you to call the tool, you running the tool, returning it to the model, and then finally the model putting an output and you can see how the latency would add up there. Uh the second thing is tools and instructions on how to use tools. All of this takes up space in the context window, right? And in general you would any for anything non-trivial you would not have one tool you'll have a bunch of tools right and so a lot of context window is gone the model actually now has to think a lot whether to use the tool which tool to use how to use the tool and so you would need to increase your thinking budget if you want the right tools to get invoked and all of this adds latency right and latency as I've already said is death to conversation so when somebody asks you what the life score uh if you have 6 seconds of silence, nobody's going to care. Nobody's going to care if you um after 6 seconds you have the cure for cancer. In a voice conversation, nobody's going to wait that long. So the main insight here is that people will not they will actually wait 10 seconds for an answer, but they won't wait even 2 seconds for a response. And the difference between an answer and a response is how we solve this problem. Uh if you ever talk to a customer support agent um it's one agent can say okay I'll put you on hold and then gets back to you after 20 seconds and that's bad experience. Another agent what they can do is they'll say okay I'll check that for you but meanwhile X Y and Z and meanwhile they might have delegated that work to somebody else. So she keeps the conversation going while the heavy cognitive work is happening in the background. Right? And so we split responsibilities. We had an interaction model whose only job is to keep the conversation going. And when it figures out there's something cognitively heavy to do, it delegates to a background model, right? And this delegation happens in an async way. So the background model never blocks the conversation. The conversation keeps on going. But when it's done, it'll silently hand it back to the interaction model and say, "Okay, what do you ask for? It's ready." Whenever you want to let the user know, you can do that. It's your call, right? And so that's how you solve for functionality by delegating to a thinking model in the background in an async way, right? And so we're at scale. And this is like a graph we did for another project where we provisioned um a GPU to host a llama model and this was an A180GB GPU. And what we figured out is like you can have after a certain concurrency on a fixed um GPU, you start to see more and more latencies because your requests you start getting cued. Um the GPU saturated either in memory or in compute and then latency spikes up. For that project, this was okay because our latency budgets were uh pretty high and we actually provisioned just one GPU to make it work. But in general, the framework that we developed was if you fix the number of GPUs, you can have either an SLA on latency or very high and flexible concurrency. You can't have both. If you want to have both low latency and high and flexible concurrency, you really need to have like a high number of GPUs, which nobody can afford right now. And so the solution there is not to self-host and not to even go with smaller labs offering these models because they also have very finite res of GPUs and we learned it the hard way when we had we worked with a small lab and beyond a certain concurrency which was fairly low like it was not even 100 they said let we don't have the GPUs you need to now commit to a fixed number of GPUs which we'll provision so that was not good for us and so this is what we came up with if we really care about latency and concurrency. Um the boring the the boring answer is the right one. You go with big labs because they can give you token based pricing up until a very high number of concurrency and you don't really have to worry about where the GPUs are coming from. Right. And yeah, that's uh we've almost landed uh we solved all the three problems. We've scaled um to all of our users and we published all this. So um we deployed this in April. um in June, mid June, we got the time to write about it. Everything that I've said is actually uh just a summary of what we wrote about. It has all the nerdy details about how we manage personalization, memory, observability, all that stuff, all that boring stuff like if you're stats for nerds basically. So you can go check it out. Uh this presentation, I submitted a proposal for this presentation based on that blog uh just 2 days after this. Uh so this was 6 weeks ago and 3 weeks ago um GPD live came out the newest version of GPD live and if you go through the blog you'll see some interesting stuff there and what they've actually said is the previous version of GPT live uh was three models chained together and that did not work as well because complexity [snorts] that came at a cost high latency stated responses so essentially the same thing that we were talking And how that solved the pro how they solved the problem was they now have GPD live as a continuous interaction using a full duplex architecture right and what they've also said in the blog is we've decoupled GPD live which handles continuous interaction from deeper work and so when GPD live wants to do something cognitive heavy it goes to GPD 5.5 and this is just 3 weeks ago the presentation that you're seeing was made 6 weeks ago right and so what you have is a playbook that I've shared with view has been validated by OpenAI and so that's a pretty good place to be right but I give it 2 months uh based on what I've seen in the past year um the dust has hardly settled on this space and in 2 months most of this playbook might be outdated right and so why did I waste your time right why did I share a playbook that I feel is not going to last 2 months uh it's all because I do believe there are some lessons in the journey that we can all take Okay. And the first lesson is actually it's a cliche uh thinking from first principles but I think it's worth repeating because in this space there is actually no other option. If you feel you can get away uh without thinking from first principles and building on top of what somebody else is saying uh I can assure you what they are saying they don't know what they're talking about. Nobody's knowbody knows what they're talking about. Even I don't. So you really have to think from first principles if you want to build something right now. The foundations are not there. we are actually building them as I'm speaking and the second is it's isite quite interesting is we have this notion of a product vision emerging and then once that vision crystallizes engineering actually going there and building it but with this technology right there has to be a continuous feedback loop between what engineering can do what the models can do and has to feed back to the vision right um and this is very non-trivial real because you need to have engineers and builders who can go back and say you know your vision it doesn't make sense given the capabilities of that model and a lot of times actually in our case the vision was actually made stronger because we realize the models can do something very very easily that we thought was hard right so this is actually fairly important and very subtle and the last thing is uh you should trust no one there's a lot of money um riding right now to convince you that XYZ is your problem and ABC is the solution, right? There's a lot of blogs, uh, a lot of hackathons, right? And you should be very careful who you listen to and especially if they're out there selling something in this space. And actually, most of the hype is actually from these people. Um, and that's why I think forums like these are important. I I'm not I'm not out here to sell you something. At the worst case, I might be an idiot, but I'm not actually trying to sell you something and convince you of something uh that I feel is not right. So uh forums like these uh fifth elephant are pretty important for us to come and share our learnings and yeah that's about it. You can actually I'll try to set up uh my laptop outside if I can find space and you can actually talk to googly and have him roast your IP your favorite IPL team. Um yeah thanks any questions I can take them now. Can you tell a bit about the model itself? I mean was it uh you built something on top of llama itself or I mean what was the actual how was the actual model built? Yeah. So, um, in the open-source domain, there are only a couple of models that actually do full duplex speech to speech. One is called Moshi, that's by a French lab. And actually, the other one is, uh, from Nvidia. Um, uh, we did not use that one. So, we tried to self-host Mochi, but as I said like the self-hosting self-hosting problem showed up where we were not able to support the concurrency. Uh so we are actually now using Google's native audio model which is a speech to speech model. >> Hey hi um thanks for the presentation. So uh we also try to build um a voice bot uh which is more of a therapeutic voice. So we hit uh like using any model uh where we require a more intelligence to be [snorts] responded. So we have to rely on frontier model because that that's what they were actually responding to. So any advice on like because voicetooice models are not that um intelligent enough to respond to bigger queries or more uh intricate queries. So uh that will fall back on voice to voice models. Any um any comments on that? Mhm. >> Yeah. Uh that's actually true. Voice to voice to voice models the strength is interaction and they don't have the intelligence or the depth that LLMs have. Uh that is very true and I think the solution to that what we found was uh being intelligent before the session even starts. And so uh what we did was for each user we had a lot of history of what the user really cared about and we condense all of that. So all the information that we predict that the interaction model or the voice model would need we load it into the context window right so we give it all the intellig so the intelligence is sort of frontloaded and even before the session starts we make sure that you know at 90% accuracy whatever information would be needed in the session is already there in the context window and the other thing is obviously delegation so you like if you need some sort of functionality and depth it will only be achieved by delegating I don't see voice models getting as smart as LLMs. So the delegation async delegation pattern is actually what's going to help you I think. Hey uh great talk uh thanks for the talk. uh here uh one quick question I had was like when you moved from the multimodel acting as one model paradigm to just a full duplex voice model did you have did you actually end up with a regression on the multilingual correct pronunciation capabilities or uh no it was actually the other way uh and this is because if you're doing a stacked approach you're using a TTS TTS models are actually dumb models. They need to know the language in which they want they have to uh generate speech. There are very few TTS models that can actually do two. So English is a big problem like the biggest problem that we'll face in India is English because people just effortlessly effortlessly switch between speaking English and Hindi. And so it was the other way around. When we were dealing with TTS models, we had to first intelligently figure out what the language was. The TTS model doesn't do that. And we had to make sure that the LM outputs text in a way that is very TTS friendly. Right? So there's a bunch of prompts in the LM on only how to teach it to interact with the TTS model. And if you change the TTS model, that's all gone. If you change the LLM, that's all gone. With audio to audio models, actually you do not need to specify what language output you want. Whatever language they're trained on. and u um if you come um uh if you meet me outside I'll demo how you can actually just ask googly to switch to Gujarati or Bengali and he'll do that. So the intelligence actually uh permeates to all levels and not just the LLM. So it's it was actually much easier for us and we actually uh the code base for speech to speech was like just 20% of what it was just because of all this glue code all this code switching stuff that we had to write. Uh >> very nice talk. Uh what about the video aspect? Are you also looking at that because I mean the picture of googly is this um monkey face and you know you want to animate uh and that's a very hard problem I'm sure. >> Yeah. So I did not want to make this talk about that problem but there's actually that's actually >> equally interesting is how we made the avatar happen. Uh so that's a diffusion based model which we have um which is not as scalable. So that's why we have it during live streams. We might have it for a few paid users. And what how it actually works is it takes in voice the model and it doesn't actually generate the entire video. It just has a few coordinates on the face that it tries to predict uh that will look natural. And so it's not full video generation. It's only animating the face in a way that it lip syncs to the audio. Uh but that will be like completely another talk. And that's that's fairly new technology. Um there are a few people doing it but even we haven't figured out how to scale it to you know um 10,000 concurrent users. Hey R, great talk first of all. Um, so I would like to know like how did you handle dialects like um I worked on voice box a year ago and there were like dialects from tier three cities uh say um somewhere in Gujarat people would just mix Gujarati and and that would not be standard Gujarati. could be like uh very like specific to a region like um I remember like we uh we didn't get a good response that time like uh with the Google models speech to speech models I don't know what's the scene now but uh yeah did you >> I'm sure I'm sure that hasn't changed uh we actually did not go for solving uh dialects so I wouldn't know I I'm I'm pretty sure that model is still not to a place where it can figure out dialects of Gujarati it can speak Gujati well but no dialect. So we um we never thought about solving that problem and I'm sure that there's not enough data for even Google to solve that problem or open AI. >> Okay. And one more question like uh how do you handle interruptions? Like sometimes when uh a natural like human interacts with somebody they would just affirm something like they'll say okay but it's not an interruption. It's just like they are affirming something and u usually voice bots would just stop there. they'll send the request back to the model and then come back and there is a gap there. >> Yeah, you're right. That's a that's a a big problem. And by that is a problem is because in a stacked approach you have the VA the voice activity detector which actually does the uh interruption handling uh or the baron as it's called. And the voice activity detector is a dumb model. It just looks for human voice. So anytime it feels there's some human voice, it'll stop uh the model. Um, it'll also wait for silence to decide that the human's turn is over. When I'm talking to you, you can tell from my tone that I'm done speaking. You don't have to wait for silence. And you can tell from what I'm saying that uh I'm not interrupting you. So I'm just maybe huh that so the solution to that is not having a voice activity detector and having the full duplex architecture where audio frames are streamed uh from you to the model and from the model to you continuously uh and the model the intelligence lies on the model of when it wants to interrupt and when you have interrupted. So I don't think the VA uh although there are some very smart VADs now I don't think using a VA is going to get there and that's why I think the full duplex thing is actually what solves that >> but is it solved like uh it like when I like I'm not in this domain as of now like have left it >> no it's not it's it's not it's not solved um what is solved is when I stop speaking the models have gotten really good at figuring out that I've stopped speaking but the interruption part is not at the place that where you said like for the model to figure out uh if I want to interrupt or this I think I think that's pretty subtle. A lot of humans also might get that wrong. So the models are not there yet but they're good at figuring out when I'm done speaking. Um [clears throat] so uh uh so thank you for the talk. I I have a question before we wrap up. Right. So you presented a bunch of statistics. uh one thing that I'm wondering about is what is the average like what was the conversation duration and I'll tell you why what I'm leading to which is what happens when in a like you know a full diplex voice model uh what happens when the context window starts filling up because in a you know in a in a chatbot well it can do compaction and then something is lost now what would that look like in uh in a voice model did you run into that >> yeah so That's a that's a good point. And in general, these models, as I said, these models are not as good as the LLMs. They don't have a million uh tok in context window. In general, they'll have like maybe 100k, 128k. And so that's a big problem. But and we thought that would be a big problem. But as it turned out, uh, to keep a healthy rich conversation going, you can pretty much keep the last five to seven turns there and you can evict everything else and the and you would actually not know that's happened, right? So we actually had >> that's what you found out. >> Yeah, that's what we found out. Like we found that you you need to have the complete history of the interaction. That's how the richness will come out. But we found out even if we kept the last you know maybe 10 turns and you need to understand that each turn is actually very short right when you're talking to someone it's actually very few tokens right so the context the system prompt actually ends up being the biggest culprit here but when you're talking uh short snappy sentences that's actually very less when you're texting and typing and you have tool calls and everything else that's when the context window bloats right and we don't we are not letting this uh model the interaction model do any tool calls Right. >> I see. Okay. But I I that's interesting because I think you uh on your slides you had uh you know where you would delegate to a tool call. Yeah. In all generality I would expect there to be tool calls right and so when those tool calls written because you know you want to you want to respond or answer uh and so those tool calls would also but you're right. Okay. So speech in general, I think what you're saying is that speech as a modality um uh turns are short and the context window doesn't fill up as fast as one might as as probably you expected. >> Yeah. Yeah. It doesn't fill up and like uh we would like to think that you know when we talk to each other we're actually tracking like the entire thing but we found out through experiments like if you keep the last 10 turns that's more than enough. You don't even need to summarize what was before. We tried that as well but it did not add to any quality. >> Right. Right. Okay. That's interesting because I thought that you know the the impact of what what uh you know what uh desktop tools call compaction would be sort of disastrous because that's always lossy right. Yeah. >> Uh so okay very interesting. Thank you D. Thank you for that talk and uh thank you for um I think great talk and great work. Like before that slide came up, I was like, "Oh, Chad G open AI up announced this like just 3 weeks back and they talked about the same problem." So great work. Kudos to the team that did this. Thank you. [snorts]