A retrieval-augmented-generation pipeline to help users query system-provided documentation

Good morning everyone. I hope you all had a good breakfast. Let's jump into the next session. It will be retrieval augmented generation pipeline to help users query system provided documentation. And it will be given by Zoltan and Gunnar. A warm welcome applause for them. >> [applause] >> Okay, so one of us has to start speaking. That that should be me, I guess. Well, hello. We're we want to share with you a proof of concept an an example implementation that we made to help users well, as it says, query system documentation. >> Yeah, and well, this whole idea came up one day we were we met together. We were talking about what would be the possible next step uses between that would fuse AI and Debian. So we came up with this idea. It's a rack pipe rack pipeline. Something that has been going on for quite some time. So we are using these methods that are that well, they arose these methods in 2018 some more or less. So is it too low? Okay, yeah. So we use these methods that were first polished in 2018. So these are very novel methods. >> Okay, so what is the problem that we're facing that we're trying to help solve? We tell developers to write the F man page to we tell users to read the F man page, but but we know that it's culturally difficult for a newcomer to understand to to query the system provided documentation. We have a lot of documentation that's very good, but it's written in a form that's not very easy for a newcomer to explore and to understand. Besides, we have over 50,000 packages, and people that don't have a given man page installed, well, they will not know about it. So, well, there are many ways to query for information, but we try to make I I mean my our target our objective is to propose a pipeline that enables natural language interaction with system documentation encompassing all of all of Debian. We're not giving much the I mean I I just set up this proof of concept, but you can see it online. It's working now, and you can query what we're about to to to explain now here on on on my personal network space, but well, never mind. It will be somewhere else at some point. Please. Oh, okay, this is mine as well. Okay, so well, we have a lot of text. We have very little time, but the thing is uh if a user knows which tool they will be using, it's easier to query, but we want to suggest what packages can help them and how to how to get to the right documentation. We know the unique style leads to having telegraphic man pages that say the things once, that says the things in the right place, but they're not very discoverable. This style has been criticized. We have written guides that encompass multiple tools, but they're not very discoverable anyway. Um uh It's hard to query a system as to how can I achieve a given goal. Uh as a related development, less than 2 months ago, we heard we we were we read about Red Hat's new tool that came out with their version 10 release called Light Speed. Uh we haven't tested it. We we asked some people about it and it goes along the same line. Of course, ours is just a first step into this, but and we were also told to something about the Microsoft's well, they have a similar facility in in their PowerShell, but the thing is one thing that we wanted to achieve and they we we managed to is to have everything queryable on a system on a on any given system installed locally without going to the network, without revealing user information with a reasonably small package installed without requiring powerful GPUs. >> I think this is yours, Frank. >> Yes. Yes. >> Um um oh, yeah. Uh so, one of the main tools that we're going to use is the attention mechanism or the transformer architecture. Uh these type of models, they they were published in 2018 and uh uh with the paper very famous paper attention is all you need. And what these does is it really changed the paradigm of how how we think about semantic in the in I mean, we've been talking about semantics here and there. Uh but the the main advantage that give us that gives the attention mechanism value is that we're starting to talk about semantics in a contextual way. And now we can choose like the span of the attention or the context that is going to be taken into consideration in order to extract the actual semantics of a given word or of a given sentence. So, well, I invite you to to read the paper attention is all you need. It was published by Google and well, next. I think this is here. >> Okay. Okay, so particularly what we're doing is retrieval augmented generation. We were discussing quite a bit because all the rage in language models is the the generative part. How a chatbot answers to you in natural language. Well, we wanted to understand the the natural language and in the end we decided not to do the generation because it well, we we are still discussing whether it adds value or not. But what what we do here is to take a simple large language model as a as a Gemini or or Lama or GPT and add to its training the the set of knowledge that that we have. So, we have the retriever that learns all of this information and the generator that converts it back into into human language. Well, I I I I don't think we we have time to explain the the three types and all all of these details. But the thing is we are teaching language model all of the all of the manual pages that that we're using to train this. I think I'm getting ahead of myself in terms of the of the presentation itself. Okay. So, uh we can choose to use an encoder-decoder. This gives it more more information and more learning, but it has a high computational cost. So, the the training was uh quite intensive. Um or we can use a B encoder that has a an efficient uh an efficient interface that that's a bit easier to uh to work on. So, we ended up using a modular way. The where we did some fine-tuning so we can use B encoders for finding the not not just the answer, but the relevant man page that that a user would be looking for to achieve their task. >> So, um I think this is >> That's mine as well, I think. Okay. So, what what we did is to to keep things within the reach of the hardware we got we we got a somewhat old Nvidia card to to do our training. We decided just to use section one of the man pages. So, we took all of the user share man man one from all of the packages available in Debian and trained our model with it. Uh and well, of course, that gives us some blindness. We're not covering a lot of use cases that something like production tool like this one should have been covering, but it's just as a proof of concept. Uh we're we're training Yeah, we're training with 34,000 man pages out of four times as as much that we have. And the man pages are between very small like less than 200 bytes to over 2 megabytes 2 and 1/2 megabytes. So, that's that's what the our model had to learn. >> Okay, yeah. So, um the way in which we chopped the problem was first we use an encoder model and then uh decoder model. The what the encoder model does is it's improve say the understanding of the of your training data or of the data that you're going to use. And the purpose of using this is to find a numer- a numerical representation for the text that you have. So uh using different how do you say using layers and neural network layers and attention layers. So these attention layers are um what what I what I explained before. But what we're going to do with this is we're going to start mixing the queries. The queries being the questions that you're going to make to the system say what is the best tool that I can use for sorting my files in natural language. And you also have like all this universe of chunks of the of the man pages little pieces of of the man pages that are that are going to respond to that question. So what we did is we took we took a B encoder model. What does that mean? We're going to encode the query and we're going to encode the respective chunk where the where that where the answer of that query is found. And then we're going to put like in the attention mechanism like attention mask when we encode the query we we put the the chunk the corresponding chunk and then we train we embed or encode and then we take separately we take the chunk and we embed the query inside and that is what B encoder means in this context. So um so in order to be able to do this training what we did was well we started having these problems. Where do we find the correct the necessary data for training our models? There's no such a thing. There's no data sets that will tell us okay this is the ground truth. So we had to generate our own uh data set. And for doing this, we used GPT. Uh we of course are assuming that GPT is would be our uh gold standard or ground zero. Uh and we uh we sampled uh 1% uniformly uh all the chunks that we found in uh in the man one uh documents. And uh uh we extracted uh posi positive pairs, meaning uh chunk query are our uh well, the query uh the the response to the query is found in that chunk. And the output were these type of pairs that we're going to use for training our model next. Um oh, yeah. So, uh as we as I told you, there we chose an encoder-decoder model. So, we're we wanted to search for pre-trained pre-trained models that that is models that have already been trained using a uh general purpose data sets, and we we want to adjust the weights and the biases of those models to be able to uh to answer questions on this specific context of man pages. So, we went through searching for different encoder models, and we found uh we chose these: T5 small, mini LM, uh BGE um small in English, and multilingual E5 small. Uh some of the reasons why we chose these is, of course, they have these architectures. Uh their their architectures are not exactly the same, uh but they have they meet some other of our requirements that are um they're very small. They're really small models. They can fit They can be used in CPUs, and uh they were also um open source. You can download and play with the architecture and such. So, that meets that met our our requirements. Um Uh so, here in this table we show you the sizes of of these models. So the the highest being T5 small, it's 60 million parameters. Uh but this is this is not the size for the encoder. The encoder is smaller because T small is encoder decoder. And and the uses for them. So here we can see the the uses for each one of the models. The T5 of course it's a full rag. So we will see that this turned out to be a obvious afterwards. And um and the the rest is we have embeddings local embeddings for Mini LM. We have BGE that is good for query alignment meaning you're able to mhm in your to make the query and the chunk come closer together in your embedding. That that means query alignment. Um multilingual 5 that is for multilingual rag. And um Oh yeah. So once we pre-train on Well, the next stage is to fine-tune our models. That means for that we use the Adam W optimization algorithm meaning that we are able to not always when we're doing our our um our adjustment of the loss we can always adapt the learning rate meaning we can every time work on smaller steps. That is why we chose Adam W. And the contrast and we for the loss function we use the contrastive loss. The reason why we chose contrastive loss is that we only use positive pairs. So we didn't use any negative. So we had to find a uh function for which the the positives would have would be meaningful as opposed to the rest of the queries and chunks that do not correspond. Uh so uh that's uh that's what we used for our our fine fine-tuning. And um some of the um uh characteristics of our of our training was we only used 50 training epochs and the size of our batches was really really small. The learning rate uh that we uh used was also like very very tiny. We didn't do any regularization. Uh and even with this we we get to we got to some um interesting um outcomes. So um so we um pre-trained well so we encoded all the rest of the of the um directory of the of the man pages. We encoded it all. Uh we got the our our embedding space and then we started making questions uh to this so that we were able to evaluate what we what we uh obtained uh or or the quality of our encoder. And uh oh yeah, something I I had not uh hadn't uh told you is that we had to write some wrappers for some of the models in in particular. We had to write a wrapper for T5 so that we so we mean had to calculate our mean pooling for for the T5 model. And uh well, what we got was uh here I think we have a well this is the difference between what we the well or the of the weights between the trained the pre the fine-tuned models and the vector store what we got uh what the vector store meaning uh that once you have the vectors or the the encoding or the embedded space, you have all your representations, but all these representations are mainly geometrical. Meaning that uh well, you don't have lists of uh or tables of your data, but you have um um uh similarity or semantic similarity between them. So, we needed a vector store so that we were able to search in our space uh for uh for neighbors of these uh for the neighbors of the embedded queries. Uh so, oh, yeah. >> Okay. Uh I I think this one we should skip. We're running a bit late, so so let's get to the next one. Okay. Uh it's your turn again. >> Yeah. Okay. Um Oh, yeah. So, for generating the queries for which we're going to evaluate, we we also generated like a completely random queries, and we started uh seeing where these queries fell. Once we embedded a query, we did a uh K-nearest neighbor inside our embedded inside our embedded space to retrieve all the chunks that were uh significant or that were the nearest the nearest in the vector uh in the vector space. And uh and then to evaluate, we used uh precision and recall, and we used uh the ROC curve. Uh for this, we used the distance the the distance in the uh vector uh well, in the embedded uh embedding space. And what we got was this uh being the area under under the curve, the greatest that we got was T5 using T5, which got 0.67. Well, there's a lot of uh things that could be said about um the the results that we got, but um it's important to note that uh in these using these uh oh, sorry. Uh it's important to note that we did a very very tiny training, so there is a lot of opportunity here to refine the results that we have. >> Okay, this is just a an example of the interface we had until 2 days ago. Uh as I showed you that at the beginning and I'll show you later, we also put this on online so you can query this via the network. The important thing I wanted to show you here is that you can see how how we can query how can I submit a bug in a package? And this will give us something that that's on a given distance, uh so it's a confidence measure. It's a very very bad confidence measure, but that's what what what we got. And uh well, uh our time is I I I'm sorry, just let me Yeah, please go back to to where we're sharing the URL, so >> Oh, yeah. >> It was the second page, I guess. So, uh yeah, here. Here. So, it's published there. And well, I would love to have questions now, but yeah, it's too much information. We could not convey it more quickly. >> Okay. Right. >> Oh, well, I will Yeah, I will Here is our GitHub and the source of our checkpoints of the models. >> Mhm. Thank you very much. >> Yes. >> Thanks. Thanks, everyone. Does someone have a quick question on the room? >> Yes. >> Do you have plans to integrate this in Debian proper? >> Sorry? >> Do you have plans to get this into the Debian archive? >> Uh no. Uh that's the short answer. First, we as many of you know, we have been discussing in the project whether pre-trained models such as those distributed by Hugging Face are acceptable for our free software guidelines. We haven't arrived to a to a decision on that. So, at least until then, we cannot do it. We would like to at least offer this as a service. Right now, I lack the hardware to do so. Uh we were told of some ongoing developments. We may get the hardware to to put something like this online. As it is right now, I think it still leads to too many false answers, which is the usual case for for language models, but we want to refine this. But yeah, the the the answer is do you can I make it into a dev package that you can install? No, we cannot right now. >> Okay. So, last one. >> Uh uh yeah. So, so um So, I noticed that you used a very generic language models for embedding for using Uh have you considered using a more specific small model which is adapted to the domain? This is the first question. The second question, if you've uh fine-tuned your own model, are you planning to publish it on Hugging Face? >> Okay. So, for we we didn't find any model that was trained on this specific for this specific use case. So, that is why we used a pre-trained model and then we fine-tuned it. And if we would be planning to publish it on hugging face, yes, of course we we would like to do so. >> Would be nice. >> So we are running out of time so next paper please.