A peek behind the curtain on how we run an open source, global, and privacy-first AI Infrastructure

All right, hello everybody. It's almost done. You almost made it. A couple of more hours of DrupalCon. So yeah, in this session I want to do a little bit of like a more technical under the hood. How does this stuff actually work? Right? I'm an engineer myself. It's crazy to see all the magic of these AIs, but like how does it really work? So I I want to open the or show you behind the imaginary curtain of a tool or of these tools that we all use every single day. Like what actually happens if you go to Drupal AI and you press enter or if you go to Cloudways and press enter. Like what is it actually happening in the background? But first about me, my name is Michael or people call me in the Drupal community under the name Schnitzel. I've been This is my 37th DrupalCon. Which feels crazy. So yeah, I've been in Drupal community for a long time. I'm originally from Switzerland. I live in Virginia now. Yeah, and people talk about me that I'm a tech nerd that is now a manager. So I'm happy if I write one line of code today. I do a shitload of open source. So I call myself an open source maximalist. I used to be way more involved in photographing. I used to do all the DrupalCon group pictures, but I now find other people that want to do this. Yeah, and I'm a dad. So what I'm going to show you today is everything that I know how we run this. And people always ask me like why are you so open? Right? Why are you showing everything? Why are you doing and it's In the end, it's very simple. Like as a company, as an Amazee, everything you do is open source. I've never understood why Drupal should be open source and then everything else below like MySQL, Linux, and all this stuff is open source, but somehow this little slice that we call like a hosting platform should not be. And so yeah, we're we open source everything we do. You can download it. You can start your own IO tomorrow if you wanted to. Because I believe, right, our edge is not the code. Our edge is the people, the operations, and how we execute. I can tell you running a 24/7 hosting company is freaking hard. Um so, if you want to try, go ahead for free, um but it's not easy. And in the end though, I do believe if we do open source, we're going to build better software, which in the end is going to help all of us and make us all better. So, All right. So, who Why did we start an AI? Like, why do we offer AI hosting? So, we started as amaze IO, and the idea of amaze IO was to basically make Kubernetes easy. Making it in a way that you don't even have to learn Kubernetes to use Kubernetes. If you ask the Kubernetes community, they're saying, "No, no, no, everybody should learn Kubernetes." Right, that's fine, everybody is But if you ask a car company, they're not telling you to first learn how a car works. Like, who here actually knows every single cable in a car, what they're what it's what it does. Nobody anymore. But we can use cars every single day. So, why should we learn Kubernetes? And that was the same idea of amaze IO or Lagoon. So, we made Kubernetes accessible for application engineers, software developers to leverage Kubernetes, because I can tell you it's awesome, but we shouldn't learn it. And so now the same we're doing with amaze AI. It's the same philosophy. Basically, we make the abstract, the complex, um and we make it easy. But we're going to give you control. If you want to know how it works, if you want to know read every single line of the code that we're executing for you on our servers, you can. That's the open source. And in both of them, and you're going to see this a lot today, is data protection and privacy is extremely important. Myself, like I mentioned, I'm from Switzerland. The Swiss banking system is known notorious for um not telling the government who their customers are, and that's ingrained in all of us. So, I by I do not trust anybody. I do not trust the governments. I do not trust the companies. So, I'm building stuff that also you don't have to trust it. You can actually verify. You can read the code and that's for everything we're doing. And for the people that have been at the AI summit, they have also heard me talking about right now it's horrible. The data protections and the privacies that are given by the regular Drupal company uh sorry, by regular AI companies are horrendous. They are really really really bad and we're trying to build better tools so we can use all these tools but in a safe way. All right, today's plan. We're going to do a technical deep dive. Um I'm going to show you why this is different. We're going to do a road map and then hopefully we have enough time for Q&A. All right, so this is what we built. We're going to go into this step by step that you understand how this works and if you wanted to, you can start it tomorrow by yourself. All right. So, let's start in the middle. The first thing we need to understand is even though we talk about AI, it's mostly actually two things. One, it's an LLM or a large language model. This is the part that can reason. This is the part that can generate text. In the end, it's a text predictor but just a very intelligent and very fast one. However, we're going to go this into this a bit deeper. If you want to actually um ingest a lot of data, you can't send it all to the LLM. Like if you want to do a Drupal search for example, you cannot take your 10,000 Drupal notes and send it to the LLM and say go find me the best note that is based on this user. So, we need some intermediate step and we're going to go into this. That's done in the vector DB. So, that's the first complex thing. You don't only have one service, it's actually two services. An LLM and a vector DB. So, as I mentioned, the LLMs are the ones that understand the language. Um they support the they support context windows and they by the way get bigger and bigger but we cannot still import a Drupal site. And the vector DB is where stuff is indexed and the LLM can ask the vector DB for information. And what is really cool about this is that you do not or you will not have the problem of um hallucinations. Right? The problem is the model is hallucinating if it doesn't have enough context. If it doesn't actually know what you're asking it about. But if you provide it enough data, for example, in a rack in a vector DB uh through rack, I'm going to explain what that means, you can reduce hallucinations almost completely. Okay, so how does such a vector DB actually work? So, the first thing you need to do, you have all your documents. These can be PDFs, these can be Drupal nodes, these can be whatever. Like it can be all kind of document. And in a first step we're running them through a so-called embedding model. Now, an embedding model basically converts text into numbers. Why? Because computers are very good with numbers. So, all the LLM stuff that you're seeing is actual math. Because what happens, it converts the num the text first into numbers and then the the LLM is doing stuff with numbers or aka math. And we store these, they are now called vectors. Right? The text goes into embedding model, it becomes vectors, and these vectors are stored in a vector database. Think about it just a regular MySQL database, but instead of tables, it's a multi-dimensional database. They're called vectors. You don't need to know how exactly it works, but this is what it does. Now, if I'm as the user at the left here, if I have a question, let's say, I want to know when is Michael's session at DrupalCon. And what we did already, we took all the sessions of DrupalCon and we put them into a vector database. My AI app, let's say, Drupal, that has access to all that vector database, it will actually first ask the vector database, "Hey, tell me anything about Michael session in DrupalCon." And then the vector database will return with only that, right? The vector database maybe has 50,000 sessions of all the different Drupal events that ever happened, but the vector database will return that to me. And then the AI app in the step three actually sends it to the large language model. So, people always think the large language model talks to the vector DB. That's wrong. It's actually the AI app, let's say our Drupal AI, that talks to the vector database, gets the data, and sends it to the large language model. Because the large language model all it can do is receive context. So, you if you ask it, "Hey, when is the session of Michael at Drupal con?" If you look actually in this request here, what happens is this app here, let's say Drupal, sends the question of mine and the context that it got from the vector database. And then you instruct the large language model to only search for an answer of the provided data. And then the large language model will not start to hallucinate. Like if you can tell it, "Hey, large language model, if you do not find the answer in what I provided you, do not respond. Do not make up anything." Anyway, the large language model will then give you back to the AI app and the back and the user gets it back. So, that's how what we call rag. Um it's the part that we have a vector database in between. And most tools, as soon as you want to ingest a lot of data that would expand, would make this context window too big, that's currently the way and the only way we can do this. All right. So, we've learned now what the vector DB does or why do we need a vector database and what the LLM does. We thought, all right, there is all these different AI apps that we need. So, what are they? Now, there's a lot of different ways, a lot of different tools that use AI today. Obviously, we all know that one, Drupal AI, right? Drupal AI can talk to LLMs, it can talk to vector databases, and all that stuff. There are though a lot of other tools out there that can do this. You maybe heard of like LangChain or LlamaIndex or OpenWebUI or things like that. We at the Mewsic, we have our own private AI systems that we allow people like talking like ChatGPT style, like it's a chat interface, you can upload documents and things like that. Now, all of them need some kind of APIs, right? They need to talk to the LLM, and they need to talk to the vector database. Now, we standardized on two APIs. One of One is the OpenAI API. Now, this is kind of funny. I tell you don't use OpenAI, right? But, I tell you use the OpenAI APIs. Because they are the standard, right? OpenAI was the first one that actually made all of this these APIs, these LLM APIs, um really useful, and so everybody started to use these APIs. Um there are ideas to come up with new APIs that are actually like proper APIs that we as a community agree upon and how they should work, like we do it with a lot of other stuff, like let's say with HTTP or with other protocols, but right now everybody uses the OpenAI API. Vector DBs, we use Postgres, and so the Postgres already has an API. We don't have to teach anybody like it. It's a bit funny. Outside of Drupal world, everybody uses Postgres. We internally in Drupal still use a lot of MariaDB and MySQL, but I can tell you outside of Drupal, Postgres won. Like they everywhere, everybody uses Postgres, everything supports it. And so, yeah. But, what is really cool, we can use this with all these different tools, like the fact that our infrastructure runs OpenAI APIs and Postgres, and all these tools above here talk to them, you can use it out of the box, no problems asked, etc. We do see though there's a third thingy here, and that's the UI and the API. So, we still need a way where people can actually create API keys, right? They want an OpenAI API key, they want a Postgres API key. So, there's one last thing that we had to build that's also open source, but it's a tool that actually can manage all of this. We'll see this in the demo how this runs later. But, this is basically everything that happens above, right? You have the tools and they need to talk to different places. Now, let's talk to the models. There are at least today over 50,000 different models you can choose from. We maybe hear of only like the Clouds and the GPTs and and the and all the different the deep seeks and the mistrals, but I can tell you there's a lot more out there. So, one of the hardest thing is you have to choose which one you want because you can't run all of them. It's just not possible. Um what is really cool, right? The open source ones, I can just download them and I can run them on our own vector architecture. I don't have to be worried at all that the creator of the models can access them because they run on my own system, right? That's like open source. If I download something and run it on my computer, that's um where it runs. However, there's also these proprietary models. And what a lot of people don't know is I can go to Cloud and I can go to Open AI and I can tell them, "Hey, I want to run your model in my own infrastructure." And they will give you, if you're big enough, they will give you the model. And they will give it in a way that you cannot extract it like and I can tell you the NDAs that you need to sign are super heavy. Um because they don't want you to obviously reverse engineer and steal it, but they will give you the model so you can run it in your own infrastructure. And you can run it in a way that like say for example in in in Cloud or Anthropic's case, Anthropic, the company that created Cloud, does not have access to what you're asking the models anymore. And that's what we're doing. Um this is mostly done together with the infrastructure providers. So, AWS, Google, and Azure, they all have these agreements with these model providers and I can go to AWS and saying, "Hey, I want to run Claude 2.1 at 4.6 in my own AWS account where Anthropic, the company that created Claude, has no access to it anymore." So, that's the way that we have to run these models privately because we can guarantee that the company Anthropic does not have access to it anymore. If you go directly to Anthropic and you run it through their APIs, yes, they will have access to the data and they will do all whatever they want and that's where the privacy policies are bad. So, that's what I'm saying. It's not that we're not saying, "Don't use these proprietary models." Understand how they run. Understand the infrastructure provider that runs them and make sure that their privacy policies are really good. The fun thing is though that um all these models, they have different APIs. They have different ways how you talk to them. The structure is different because everybody comes up with their own little magic sauce and how this works. So, if we go back here, you maybe saw there's an LLM proxy. So, this LLM proxy is actually understanding all these different providers. So, all of these, like Claude, Deep Seek, and them, they're all behind that LLM proxy. And they also understand, right? Some of them I can run as LLMs as a service. Some of them I have to buy GPUs and all that stuff. So, that's where the magic actually happens. The other interesting thing is some of these providers at the bottom, their actually their uptime is horrible. Um right? Because everybody wants to use these LLMs and there's massive spikes and stuff like that. So, what our LLM proxy does, it also automatically load balances. So, if we see, hey, AWS has like slow response times, we can automatically switch over to Azure or Google. And that's really the only way right now. Like you cannot just choose one provider and hope that it will be up. Um it's not there right now and so you have to basically have multiple accounts with all these different infrastructure providers and to make sure that you can provide a good service or otherwise, yeah, you're just going to have outages and you're not responses and stuff like that. All right. Now, I talked about um the you not only need to make sure that like the the privacy is guaranteed or the privacy policies of these companies. The other thing is, specifically outside of the US, it also matters where these models are running. Because in the world of how data protection works today in the world, even though it's in the cloud and online, the data protection is depending on where the data is stored. Right? So, if I store something in Switzerland, the data protection policy of Switzerland applies to that data. If I store something in Germany, it's there. If I store something in the US, it's there. That's how it works. It doesn't make a lot of sense, but it comes from the fact where we stored data in safes or like in physical paper. The laws have just not up they have not been updated to the world that there's like a global privacy policy. That doesn't exist. And so, it's extremely important that where these models also are running. And so, that means we as a company that says we a privacy is extremely important to us, we need to run different endpoints per country. Right? Because if I have, let's say I want to run something in Germany, but the endpoint that you're talking to is in in the US, the data even though that the data is forwarded to Germany, the fact that it passed the server through the US, the US privacy law already applies to that. So, it's extremely complex. So, if we look at this, right? We talked about we have our applications on top, we have our LLM proxy, and then we have the infrastructure. The problem is I actually need this for every country another time. So, this is how the infrastructure actually looks like. Right? So, if I go to Germany, I have, for example, I buy GPUs and LLMs from Germany. I have my own LLM proxy, my Vector DB proxy. And when my Drupal wants to talk to, let's say, a German model because it wants it Let's say the Drupal itself runs in Germany, it will talk to that LLM proxy directly. I need to run the whole thing again in Australia, and I need to run the whole thing also in Australia. So, we're basically duplicating the infrastructure for every single country or every single region, every single time because that's the only way that I can guarantee that if your data lives in Switzerland or in Australia or in Germany, never leaves that country. The only piece that is once is this UI and this API. Right. So, there's one single UI or an API that you can create API keys for these different places, but after that, I have everything duplicated. So, all of this that you see, so this whole thing with vector databases and different infrastructure providers and different LLMs and all of that because like not in every country you find the same, we just run multiple times. And by the way, we also have a staging environment, and we also have a dev environment. So, this all actually runs So, we have around 50 of these running um in every different country to make sure that the data protection of privacy is actually properly. And so, yeah. Um that's how we do it. Now, obviously, yes. Sorry, did you say you have 50 running in each country No, in total. In total. Right now, at least um right, every country that comes Yeah, that would be crazy. Um No, and like that's the only way to do this is like you need to fully automate this. Right. I can go now and say like hey, I want the whole thing running also, let's say, in um in South Africa. Like AWS has a region there, and I can start it. Um AWS is currently launching in a region in New Zealand. Um so, as soon as it's up, I can point my infrastructure there and it builds it up automatically. If you want to do this manually, you're never going to finish. Question, yes. Why does Switzerland have a different model? Why Switzerland and not the other countries? Is there a particular reason there? Yeah, so not the infrastructure provider don't offer you every model in every country. Yes. So right, the problem is they need to reserve GPUs for these different models. Because you can if you try to start a GPU when the request comes in, it's too late. So they they always have to keep specific amount of resources available. And these infrastructure providers, they are looking at the current usage and demand and and decide which models are available where. Depending also right on the privacy policies and all this different stuff. So yes, in every region that we are, we have to individually check with Google, AWS, Azure or other providers which models do they have available. And which models do they give me access for how long and all this stuff? It's a mess, right? The cloud promised us that we can just run the same everywhere. And that's true for CPU and RAM. Right, I can go to any country and just start CPU and RAM and I don't have to worry about it. In the GPU world, this is not the case at all. And so yeah, we have in every different Now we try to find common um like that it doesn't matter in which country you have you go. For Drupal AI, what we actually do, we do a so-called chat model. So that we automatically decide which of the models is the is works with Drupal. So you don't have to worry about like if you choose Drupal in Switzerland and Drupal in Australia, um you can though still select individual models if you want to try something out. But we give you by default the model that we have tested that we know with Drupal it works. But again, it's all about control so you can do this. Okay, so how do you get access to this, right? So, we have a self-service portal where people can basically create all the usage, the team access, the token life cycle. I will show you a demo of how this works. But, you can basically create multiple of these keys, right? So, in every region, you can create keys, um and every key has its own region-specific endpoint. That's extremely important. That's the only way that you can guarantee the privacy and and the security. Then, the other cool thing is like we have customers that's saying, "Okay, this is all great, but I wanted it in my own account." So, we have more and more customers that have already agreements with AWS or with Azure or Google. And because I have 50 already of these running, right? The 51st is not that complex anymore. So, what we can actually do, and we're doing this now with multiple companies, where we actually run inside their AWS account. So, they give me access to their AWS and I deploy one of these things into their account, and that's their their own. And they have full control over it. So, they that's an even better because, right, it's not even running in my infrastructure anymore. It's running in theirs. And because it's completely open source, they could also kick me out and run it themselves. And that, at least I can tell you this is extremely helpful in in the world today because companies do not want to get locked in anymore. They for too long bought some kind of SaaS product or something, and then they wanted to leave. The company was like, "Yeah, feel free to leave, but by the way, if you want to now migrate, it's going to cost you hundreds of thousands of dollars." And we're fixing this with deploying something into your infrastructure. So, it's continuously running and it's open source, so they can actually maintain it. And does that actually meet an on-prem requirement in your experience if somebody has an on-prem requirement? Do Is it sometimes on-prem? Yes. Okay. Um if you want the highest data protection, you can do this on premise. Um He didn't pay me to ask that. Say again? He didn't pay me to ask that. Uh yes, so you didn't pay me to make this transition, correct. Um no, we can deploy all of this also the infrastructure. It It happened a couple of times in the hosting. The problem is now I need GPUs. And these GPUs, we're talking like $20 or like Um so, if you have them, sure, I can help you. But And by the way, if you buy them now, you get them in 2 years because a lot of people want to buy them from Nvidia. So, unless you have like secret um connections to Nvidia, you're probably not If you think about it now, it's probably going to not happen in 2 years. But there are companies that are there. Um so, we can We can make this completely in on premise. Um and we have seen We have seen like universities that somehow got huge GPUs donated, and we can run there as well because in the end all we need is a Kubernetes cluster. Again, obviously it runs all on the existing infrastructure, so you can deploy all of this. And then you have no worry at all because then it literally does not even leave your data center anymore. And the really cool thing is like it's the same developer experience. So, you can the developer will not even actually realize that it's now running locally because the endpoints, the usage, and all of that is exactly the same. And so, yeah. Um what is actually cool because we do this privately, like we do it in their own region, it's actually faster. Um And so, you don't have any cross region and stuff like that. And of course, we can still scale. Like all of this didn't talk about this yet, but all of this is auto scaling. So, each of these 50 things, they automatically monitor the usage and they scale up and down and stuff like that because right we specifically in the AI, you can clearly see when people are at lunch because that's when they don't ask their models for stuff. So, you can see pattern. And you don't want to run like specifically if you run GPUs, they cost like $5 a minute. So, if you can scale them down if you don't use them, it's very useful. And we see this like people go for lunch, they scale down, and after lunch they come back up and stuff like that. And yeah, and then obviously the last thing is you need to run this 24/7. Um and that means right, we as a team we have um I'm easy right now, we're on 60 employees, around 20 to 30 are are somehow involved in support and on call, and this all needs to run 24/7. We solve it with having people in all different time zones. So, we have teams in in Americas, a team in in EMEA, so Europe and um Eastern Middle and um South Africa and Asian Pacific, and that's the only way, right? Because these things will break. This technology is so new that things are just going wrong all the time. And um yeah, and so all the other things. The good thing is like we figured this obviously. We had to do this already. This is not nothing new. That's our legacy coming over from all the hosting. We obviously had to do all of this. Yeah, and then the last question is like what do we actually track? Um and so what is really important to us, again, like I said, um privacy is extremely important. We do not log any prompt or responses. So, whatever you're asking the models, we do not log it. Um it goes through our servers, but I don't need to know what you ask the model to charge you. I just need to know how many tokens you used for for the services where we have token-based pricing, um but nothing else. So, the only thing we do is we store the metadata, meaning which key, at which time, and how many input and output tokens. And that's it. I don't want to know what you what you ask these models because it's not my job. Um right, and I'm not interested because I'm not training models either. Um and so there's not many providers out there that do this because people always ask me, it's like what are people using your infrastructure for? I'm like, I have no idea because I don't track it, right? It would be great to know what people are using our infrastructure for, and we maybe would be able to provide a better service with that. But for me this already is a problem. I do not because what is really like what we have to be aware of like right now Open AI has like 300 million users. They're becoming a massive honeypot not only for hackers but also for governments or other companies. And we saw it with 23andMe and that suddenly got bought by a Chinese company. Right now all your DNA is over there. Who knew this 10 years ago when everybody was like oh yeah your DNA, right? So I'm not just doing it for my own sake. I'm just I do not want to become this honeypot. And so I think everything that we're doing we're very deliberately discussing is it does it now make sense to store this? Because we maybe are able to find out more data or better service or are we opening ourselves up for a problem? So if the government ever comes and says give me what this user created I can tell them the key, the time, the input, the output tokens. But I cannot tell them what they actually what this user actually did because I'm not storing it. That's any government. That's any government, correct. Yeah. I mean some governments have better data protection laws but I can tell you most governments right now they're cracking down on this more and more. Like we've seen it in Europe which used to be like the the the the the good private privacy. Like France just passed a law that forces you to implement a backdoor in every SaaS product. And so companies they're leaving France now. And because they're saying this is this like I'm not going to implement a backdoor that allows the government to read everything. So yeah it's unfortunately it's bad and we're trying obviously doing this for as long as we can and we're not going to be luckily we're in countries that we don't are not forced this but if this ever happens yeah we might need to change headquarters and things like that. All right what's next? Um obviously our customers they want a lot of analytics and insights like I mentioned. They want to know and so while we cannot tell them exactly what they're doing, you can still figure out like how much is it used and stuff like that. One other really interesting thing is I don't want to go too deep in this, but there's so-called secure enclaves or confidential compute where the LLM request is actually encrypted on the client side, so on the browser side the the request or prompt is encrypted and it's only decrypted inside the GPU and along the path it's completely encrypted. Which is probably the the best and the safest way to run all of this. It's a bit more complex though, right? Because you actually need to give a public key of the GPU that is going to run to the browser. So it's a bit fun, but it it exists. And then the last thing is that people ask us about fine-tuned models. So it is possible with open-source models to automatically to to take these models and fine-tune them in a model. Nvidia is pushing this quite hard with their newest model the Nemotron where you can actually buy a non-finished not buy, it's an open-source model. You can download a non- a non-fine-tuned model yet that you can train on your specific customer data. Or your internal data or like we could make a Drupal 7 coding module because the problem is, right? Drupal has changed so much in the APIs that the models are sometimes confused. Is it now Drupal 10, Drupal 11, Drupal 8, Drupal 7? The problem is we're talking about $100,000 per training run. So this is right now at least not not easy to do, but it's possible. And and we also believe that this is going to become cheaper, right? So it was definitely possible in the near future. We actually or some companies will have or some communities will have their own models that they run and fine-tune, but that's where the world is going. Oh yeah, um I said already it's all open source. Um so yeah, so you can download this if you want to read um Kubernetes YAML YAML helm code. Um again, it's not easy to deploy. It's broke, but maybe with some help of the AIs you can download all of this. It's all open source. You can use it. Um but we mostly do this for people to validate that the stuff that I'm telling you is actually true. All right. So, I want to show you a little bit how this works. Now, to give you some insights, um this is not the full-fledged finished version of this. This is like an uh an a kind of half-mocked page um how this works. But I'm going to shortly show you um how how this works. The easiest though, if you want to try it, so we actually have we are the main we run the triple AI demos on Drupal.org. So, if you go to the AI page, there's demos. You can fill this out. You can actually create different experiences, right? You can tell us and we can put it a bit higher up. Come on. A bit higher up. Um so, you can choose from the existing experiences that you can play with. And you will use the infrastructure that I just showed you, right? So, that's all of that uses that in the background. You don't see it because it happens fully automatically. But now you know, if you start an and demo here, we automatically generate the key. You will see an endpoint for the LLM. If you choose the the AI search, you will also see an endpoint for the vector DB. And it will go through this whole system that we just talked about. However, this is just in Drupal. What we are seeing more and more customers are asking us is that they want to just have AI keys. They want to use the AI key in other tools, in WordPress, in Symphony, in Open Web UI, in their open clause, wherever. And that's what this dashboard is for. So, you can basically say, "Hey, I want an API key." Um we're going to hit and explain you how it all works, right? So, we have um then the first thing is, right, you need to select all the different regions. Um so, here on the left side, we see Switzerland, Germany, etc. Let's say I choose Switzerland. And I can see already my keys. So, I can see the production app and the staging server. It tells me how much I used already of it. In this case, the customer is um or this is new installs, and I can see the usage and the history. And I can see um I bought like credits of $50, and then the API app um the production app used it. I can create the key. So, this is the demo key. I can select the region again. I can create it, and now it comes back, and it gives me an API key, and it gives me the endpoint. Right, again, and this is specifically Switzerland. So, this is an endpoint that runs in Switzerland, so the data never ever leaves Switzerland. Um I click done, and then I have it back here. So, that's really how simple this is. And you can take these keys, and you can put them in any application that supports um Open AI. Um And then the models you will select actually in the application itself, right? So, the models are always the client um so, uh chooses which model. And yeah, that's um that's basically what we're what we're working on. That's just the UI. That's the part that we saw there. And yeah, that's it. So, any questions? Anybody want to start a company like this? Yes. Thank you. Also, what is the activation Do you have a different touch for that? Oh. Interesting. Let me fix this for you. That's nice. All right, other questions? Yes. Back to the NBA, so with the other project what is it? Yeah. With that you're taking a model from them and running it on your service state. You never really interact with them after the initial agreement, right? Or are you still hitting their servers? No. I do never hit servers from the company that built the models. So, they give me the model. I run it in my own infrastructure. They never have access to my servers. I never visit their servers. So, right? So, like sometimes if Anthropic is down and like Claude is down and everyone on X freaks out, ours is still running. Um so, it's a bit of confusing, right? Because it's not you like people are just not aware of these model sharing agreements and stuff. Um but yeah, no, there is no after the model has been delivered, there's no more data transfer or data uh happening in any way. Yes. Is there any kind of lag between when, for example, Anthropic releases Opus 4.7 before you can get it? No, because they tell us before. Okay. So, we know or in in in some cases AWS knows and tells us already. So, whenever they release release a new model, we need to do some more testing, right? We don't immediately release it because we had problems where we also immediately released it, but um most of the time within a couple of hours we can use it as well. Yes. What's the term of the NDA? Like in other words like every year you're re-signing and they can decide blah blah blah you're out of the program here anymore or is that like for the next 5 years I can tell you it's so new, I can't I can't answer that question, right? These they are um every company does it differently. Um but basically what their what their biggest concern is is that we will take the models and create our own models from the train, right? So you need to somehow prove to them that you don't take the models and train with them. And then the other one is can you make sure that the data or like because you what is funny, you pay them still in tokens, but the model runs on your side. So there is some way that you need to prove that you're actually counting the model tokens correctly. And there might be some ways that they want to verify this, something like that. But it's not it's not easy to get any of that stuff and we also don't have it with everybody, right? You can And but you can also just go to AWS and they do the same. So depending on which models you use, um yeah, but they basically realized that if they want their models to be used, they need to provide them in an in a more private way, aka that they don't have access to the prompts anymore. Because right, these big companies, the big enterprises, if they wouldn't offer this they would just use another model. So you So the the creator of the models would just lose out of um of of revenue. Yes. If you're paying them like tokens Yeah. And then you're also paying for the infrastructure that's running the the model and using the tokens. >> Yes. Seems like you're paying paying twice for that usage. I obviously don't pay them the same price that they if I would run the model running on their side. But in the end, yes, it is per token. Um but what we can do is like if you go to Anthropic and you go to their price list in the end we can roughly offer the same price as they do. Um because though, right, we need to run it in specific regions I can't just like let's say if if if the if the if the amount of traffic or like the amount of GPU resources are exhausted in Germany I can't just go to to Australia and say, "Oh, I'm just going to run it there." So I we need to over provision a bit more in these countries and so that makes it a little bit more expensive. So you will see that the price per token is higher because we offer it in specific regions. But AWS, Google, and Azure do the exact same, right? So they for example, they allow you to run clouds on it globally and don't tell you where it's running. Or you're saying I want to run only run in Germany in Europe, it's a bit more expensive, or you want to run it only in Sweden, then it's like another 10%. So it like goes up like in 10% usually in 10% increments because that you just it's a it's an economy of scale. Yes. I'm not really curious about the pricing given how much you talk about security and making sure that probably not going to ditch Um I mean basically first just curious how much of your current client base is overlapping with what you do with Drupal um and possibly if you can divulge like what industries you're getting business from. I'm curious. Yeah. So right now it's all Drupal because what we first launched is a Drupal AI provider. Right? So for us the Mazy IO comes from a Drupal world. Like we we're mostly active in the Drupal community. However, we have many customers that use Drupal but also other stuff, right? Our infrastructure is not biased to Drupal in any way. It's It's optimized or we have base images and deployment scripts and stuff for Drupal, but you can run JavaScript, Node.js, Python, Ruby, Go, whatever. Um and so we are um So, we decided deliberately before we launch all the other stuff, right? The underlining LLM and all that stuff, we actually launched the Drupal AI first. Because that was a lot easier to then but we had to build it, right? But you couldn't access it, right? The APIs were not up publicly accessible and stuff like that. So, right now most customers are doing Drupal things. Okay. However, while we did this, uh more and more customers that came to us and said like, "This is all great, but I also want to use it outside of Drupal." Yeah. So, that's where things like, yeah, they want to use Llama Index and all this other stuff. Um What we also see happening is that within companies, like they're now using tools like Open Web UI or um N8N, like that's going like through the roof right now, right? They want to They have existing N8N workflows and now they want to connect their AIs and they they talk to the security team and then the security team says, "Nope, you're not going to use Open AI, right?" And so they then ask on LinkedIn or go to Reddit or uh just Google and or ask their AIs to find an a private AI provider and then they end up with us. Um so I would say, for your question of like what type of customers they are, it's really interesting. It's not There's something weird happening with the very big companies, like so the Fortune 1000s. What at least right now what we're seeing is that a lot of them actually did an investment stop into AI. Because so many teams and people started to use AI and they cannot really show an ROI. So, what's happening right now is that we need to start very small. So, like we even like with some of the companies we started like $600 a month type of deals, right? Where they just try and they need to show some kind of AI approval board inside the company that their their spend is actually providing revenue, right? Because it's easy to say, "Oh, I'll just buy all these AI stuff and I'm faster." But can you prove it? And so, that's that's happening and so, with the really big companies, that's what we're doing. Like we're in these programs where they test us and they validate and things like that. Interestingly more it's like in the SME space, which they don't have these AI approval boards, their people have a lot more flexibility of trying out stuff. The problem there is a lot of shadow IT or shadow AI, right? That people just randomly use a tool and we don't know. Um so, it's it's not I can't really say it's one. It's just it really depends on how far they are in their journey of using AI. How good is the security teams to tell them they're not allowed to use open AI and Anthropic? Um and in the end really individual people of the companies that understand, "Hey, we should use these tools, right?" But I can also tell you like so many people still think that AI equals ChatGPT. Right? The fact that you can use this in other tools and others just is people think it's the same. AI means a chat interface, but it's it's obviously not. I mean, we all know in Drupal AI. There was a question. Yeah. So, as we have these models, so how frequently do you need to refresh these models? How do you explain that you buy and host it and then it's done? So, how frequently do we need to refresh these models? Yes. It's also a big question that we don't know. So, right now new models come out pretty much every month and we update them. Usually they are just new generations, but what is interesting like for example, there's still people that love GPT-4. Um so, we still run GPT-4 mini in our infrastructure because some people just feel that GPT-4 it gives them a better answer. Um the problem is though that in some cases you actually are not allowed to run them anymore. So, there's not much that I'm defining. With the open-source models I can run them for how long I want. But right for us it's also a problem like like I said, we need to keep capacity existing to run these models and so if there's not enough data anymore, we might also need to shut them down. Um so, it's a hard question, but it's definitely it's a lot faster than like Google versions for example. Like it's yeah. Question, yes. Do you measure how much faster your system is compared to like running on the web asking ChatGPT the same questions you're doing? Um I don't and it's very hard. Right because they do a load of caching everywhere and stuff like that. Um but what I can tell you is that yes, if you just compare API response calls, you can just see that sometimes like let's say you are in Germany and you asked Anthropic AI, um they're the same speed and then suddenly you have like a couple of requests that add a 200 millisecond delay and that's just like it clearly went to another continent. Right? Because you can just for for the speed of light you can estimate now how much further the request went. So, I would say it's maybe not so much about faster. It is faster, but it's also more like um it's it's it it's less spiky. Cuz I'm I'm curious like holding a web page amazon.com does a test like if if it takes two more than two seconds, people are just leaving the site. You know? So, if you have a site Yes. >> and you want the user to stay on the University of Waterloo wants to use semantic search, you know? Yeah. They want them to stay on the site. They don't want them to go to wherever to just do ask the same questions. They want them to stay there. So, if if your solution is quicker, right? Then they'll probably stay they'll come to Yeah. >> versus doing all the other things. >> I probably wouldn't say that you can measure it on the site, right? Because the on on a on a semantic search, there's so much more like there's requests to the rack to the database involved and stuff like that. What I mean with quicker is like if you just measure the pure API call. However, if you if you look at the whole stack that is involved in a request from Drupal or like in a say a semantic search, there's so many other possibilities to delay. Interestingly, at least right now, it's perfectly fine. People are suddenly okay with waiting a bit longer. That's crazy, actually. It's a crazy stat, yes. I I do think this is going to change. Right? I mean, this whole the web needs to be fast wasn't on the first day of the internet either, the fact. So, I think this is going to be more. What I do believe though is that the fix there is actually is more like a small language models. I always feel like today we're using an 18-wheeler truck to transport a pen. It works, right? The pen will arrive safe and sound at your destination, but you just spend more gas than the value of the pen. I mean, maybe if the pen is out of gold, then we can maybe talk about this, but and so that's just in general with every technology that at the beginning, we're just throwing hardware or infrastructure or whatever at it. And if you look, there are first examples where people build SMLs, small language models. Um and and these can be a lot quicker. You can actually run them on the edge, right? So, you don't need massive GPU forms anymore because some of these models, right? Um you need four GPUs chained together to run one model and that does one request at the time. So, that's not something you run at the edge, like I mean you like in a data center or even maybe on a CDN level. But, these uh SLMs, you can suddenly run them much closer to the user. Um some people are even saying we can run it on the computer itself, right? Like that or like in the browser. And so, I think that's where we're going to go is that instead of us taking these massive language models that learned everything, right? They from a PhD in neuroscience, they can also solve um run math equations and code, right? Why do I need a model that can code if I just ask a question about a school admission, right? That doesn't make any sense, but it's a we haven't had had enough time to optimize them. So, I think in the future we're going to have these smaller models that are hyper optimized and maybe instead of learning one, you maybe have 50, but there's maybe a small tool in the front that figures out, okay, which of them do I need to use now? And then you can run them and then we're going to be fast again. Then we should all be talking about sub-second like Amazon did the um but at least right now as soon as it says AI, people somehow are expected to wait. It's it but I don't think that's going to be for much much longer. Yes. Is there a gotcha that you would be willing to share that uh you wish your clients were more aware of people trying to get going on your infrastructure? Oof, I have so many. Pick a favorite. I I think right now that's what I talked about in in the AI summit session, right? It's instead of RTFM, read the manual, read the privacy policy. Right? And that's with your setup as well, is what you're saying? Uh yeah, read mine, please, as well, right? Like, read the In general, be be aware what these what these companies are doing. Um because they are right now they're fully banking on you not understanding how it works. Plus, they also hide behind this shield, "Oh, I'm not using your data for training." That's maybe fine, but you're still sending it to 20 sub-processors in 50 different countries, and you don't know what they are doing, right? Because you don't own them. And so, that's really the power of this. This is like Yes, let's use this. Like, I'm not saying do not use this AI stuff, but please be aware what you're doing. Because I think some of the stuff we're doing is worse than if you would go to Twitter or X or Facebook and just ask these questions to the public. And because again, these companies they become massive honeypots. And not for hackers only, but for for any type of thing. And I Yeah, so and I think there is going to be people that are really in a couple of years are saying, "Fuck, I would laugh it's if I in 2025 didn't know the decision." So, that's And I just And it's not about And I'm not I'm not trying to say come out to me, but it's more like just be aware, right? I'm using ChatGPT myself, but I'm only asking for stuff that I would be perfectly fine that it goes into the internet like that it's public, right? And then And for the stuff that I don't feel comfortable, then I'm using like a service like this. So, yeah. Is amazing AI provided a service where you can run a full CLI and get amazing results? Yes. Yes, you can. Yeah, so it is possible that most of the question was, "Can I use cloud code or open code or so?" Most tools today that come from a provider, you can actually switch the LLM provider with like environment variables, and you can run them um through our infrastructure. So, Cloud Code, we tried Open Code, Visual Studio Code, Cursor, like all these tools, you can connect them to our LLMs. Because, right, they have these OpenAI they have these endpoints. Um I have to say though, it's going to be a lot more expensive. Right, because you have to understand these $20 or these $200 um um provider or like per month subscriptions, they're heavily subsidized. So, we asked some of our teams that are very heavy users of let's say Open Code, we ran them through our own LLM providers and they used $300 per day. So, and there was other people that did the same estimations, so these these $200 accounts are sometimes up to 30 times subsidized. And the question is for how much longer is that going to go? Like as long as there are VCs pouring money in on top, it's going to go and work. But maybe at one point these companies will say, "Nope, sorry. We're now uh creating limits. We're going to make it more expensive or stuff like that." All right, we're going to have to stop. One more thing, um the uh the team the people behind all of this that did the session submission uh selection, they're doing a new thing that you can give um session feedback. So, please give some feedback on this. It's not only for me, but it's also for the for the people behind. Some of them are my friends that are doing crazy hard work to select these sessions and they will be extremely grateful to provide some feedback because it's really hard if you don't if you don't know if you selected the right sessions. So, thank you.