Flock 2025 We Need A Detective To Inspect These Logs

one of those people. So, let's deep dive into how to inspect the logs automatically with LLMs. Please warmly welcome Tomasz and Jiri on stage. >> [applause] >> Thank you very much for the intro, Carolina. You were spot on because we have like 50 minutes of content and 20 minutes of time. So, let's dive into it. Uh So, slide number one, uh logs log detective, that's what we are here to talk about. Uh it's the project we are working on for almost 2 years. We had this idea and started doing some research and it's snowball into now actually many, many things. So, right now we have two deployments actually. One is hosted in Fedora Infra and you can go to logdetective.com. That one is running on Fedora community ship. So, you can use it right away. Uh one of the functionality is that you can feed in your log and get it explained by AI and it will tell you like what it thinks and give you some explanation and suggestions. Uh which means that I didn't even say what the project is about. So, log detective is an AI tool or service uh that can explain failed RPM build logs to you. Uh it can probably work even with test logs or flatpak builds or image uh builder logs, but we are right now focused mainly on RPM builds uh so to focus only one thing. Uh yeah, so Yeah. Yeah. Yeah. Uh yeah, and one more thing I wanted to add for this slide was that it's already integrated directly in the Fedora Copr interface. So, if you are using in building in Fedora Copr and you have a build there and you will list on the look at the list of the truths, you'll see two buttons there. One is called ask AI the address teach AI. Uh ask AI is that you'll be forwarded to logdetective.com and it will explain that log to you. It takes probably 2 minutes to do because I mean, it's slow. I mean, even though we are running on a GPU, it can take some time. And the other one is teach AI and that will actually give gets you to the interface as well, but that gives you opportunity to annotate the log for us so that we can include it in the data set and we can use your contribution to provide better results. So, Eureka, can you tell us more about how it works inside? >> So, um So, it's actually very simple. Um Log Detective, as it is right now, is hooked up to our GitLab instance. Anytime somebody tries to make a merge request and that merge request fails in CI due to, well, build failure, effectively. Uh that activates a webhook and we get a submission for the Log Detective to take a look at. Log Detective server does some very basic processing. Uh it effectively uses just the standard template mining approach to extract the interesting parts of logs. Oh, sorry. Um Does it >> Does it work? Oh, okay. So, it uses template mining to extract the interesting parts of logs or to be more precise, to extract the most significant samples from the logs. Uh those are then fed to the LLM, which tries to explain them all. As you can see in the Yeah, you can see it, I think. Um you can see how we're making multiple requests for the chat completions endpoint at the LLM inference server. So, for each one of those small parts of the original log file that was deemed significant by the template mining algorithm, we make one request so that the LLM can kind of explain what was happening at that specific spot. Those are then compiled together and used to prompt the LLM one more time to give a final summary analysis, find, you know, figure out what actually happened. Um right now what we're using to do that, uh we're using what some of the models that are deployed on our internal internal infrastructure, but it's agnostic as to what kind of models you're using. You could hook it up to Open AI offerings, um Gemini, anything that's using the standard Open AI API. Um it's pretty much uh yeah, it's very simple. It's just a question of adding two lines in our config if you wanted to. And finally, when it gets that analysis done, it displays it to you either as a, you know, um message on your merge request or alternatively on that web interface that Tomasz was talking about previously. And uh yeah, uh since we are Open AI uh API compatible, we can hook up to anything else except including VLM. Uh VLM is actually offering almost entire Open AI API as part of its uh interface. So, any VLM instance running is enough for us. Um since we're using uh just um you know, 7 billion models uh the 7 billion parameter models, it's kind of limited in what it can do. The bigger ones would probably work better, but it's still working decently enough. We're getting like reasonable actionable results most of the time. Um even if you quantize the model down to like just 4 bits, like uh the models are usually trained on 16 bits. So, you're basically uh cracking down to 1/4 of the original size. You can still get useful results. So, if you like want to, for example, deploy um entire Log Detective, all the infrastructure, you can do it on your laptop. And that's actually how we're doing it for development. So, we make a new change, we can just deploy everything on our laptop with Podman Compose, and run the inference and get answers. Uh Granted, if you do it on your laptop, it's going to sound like an attack helicopter, but um just for a bit. Uh after that, it um gives you the response and you can see if you messed up and you have to do it again. Um we have two modes there. Uh the simple one, that's the original we have used, which basically just works by taking all the individual pieces of the original log, putting them into the prompt, and then asking the model, "Hey, what can you make of it?" Um that's the original mode. Then we got a staged mode, which is what we're using now in deployment, which basically, as I've said, prompts the model multiple times to get the response, once for each snippet, and then once again for all the snippets at the same time, which are already annotated. Um and um that integration um we have with uh GitLab right now. It's working pretty well. Uh we have a nice little account there uh for Log Detective. We can actually watch it making uh comments on all the merge requests. Um we are trying to make it better all the time. Um right now uh we have uh this feature there where you can respond to Log Detective. Uh you can respond, obviously, as you would to any other commenter on your merge request. But what we're specifically looking for are emojis, thumbs up, thumbs down. So, we can kind of record how people are reacting to it in the long run. The idea is that we will eventually use it to kind of fine-tune the model on the responses, so we could give it better, you know, feedback and eventually produce something even better than we have right now. Um but in the short run, we're using it to judge whether or not any changes we have made to our deployment are impacting users. So, please, if you have uh you know, your your uh merge request commented on by the Log Detective and you like it or you don't like it, don't hesitate to respond there. That's what it's for. We're also adding a little disclaimer about this onto every comment we make. So, it's you know, very transparent and you don't have to worry about, you know, posting a wrong emoji or something. Now, about a template mining I mentioned, we're using something called rain tree algorithm. It's something that's totally unrelated to LLMs. We have slightly adjusted the heuristics that it's using to determine where individual parts of the logs end and begin. That's because the original system was made to be used on continuous streams of logs, for example, from firewalls, you know, trying to find the significant events. And it's not necessarily well suited out of the box for build logs, which are, you know, basically they are long, but they're not continuous streams. They are in the files that are already existing. And also, they sometimes contain messages that are split on multiple lines. So, we need to account for that. But that adjustment itself is very simple. It's just like 10 lines of Python code. It works pretty well. Um By splitting the logs into those snippets and then asking LLM one snippet at a time what it thinks about them, we can achieve sort of quasi chain of thought behavior, even though you we're not using a reasoning model. Because effectively, reasoning models, those that are popular right now, all they're doing is that the LLM is first generating a bunch of text describing a fictional thought process. So, here we're kind of doing that on the side by ourselves. And um yep. >> How much? >> Okay. So, now you know who the mastermind behind the project is. Uh so, there was lots of text in in slides, so let's look at some pictures. Sorry, we don't have demo because uh we don't have time for that, but we have screenshots of how it looks what Yurika just described. Uh So, here first we are looking at the merge request command uh that's on a CentOS Stream merge request. I don't think I need this because I I'm getting echo from >> Yeah, sorry. >> Uh So, yeah, so when you contribute to the microphone, please >> Uh you can just go go over here and use the >> Ah, okay. So, if you contribute to CentOS Stream and you do a merge request uh and your build will fail, this is what you will uh see in your merge request. This is the command from Log Detective. Uh as you can see, it's pretty long. Uh that's definitely one of the bigger issues we are having right now that it contains just way too much text, even though we tell our line like, "Please be concise." It's not. Uh so, we are trying to trim it down, but in the end usually like the gist of the issue is there. Uh then you can see the three arrows uh at the bottom that can be expanded. So, the first one details is here. That's what Yurika talked about the snippets. That's how we extract the important lines from the log, and here they are. So, it's usually the line in the log file, then the actual line, and then the explanation. So, this is like I would say pretty useful if you are new to packaging or new to Linux, uh you can easily get explanations of every line here. And typically, one of the lines is is the like the big reason why the build failed uh like and that has the explanation, and then we are doing the uh big query uh as Yurika explained uh to get the like proper explanation. So, you can expand the details to look at this. And the Yeah, as and and finally, there is also that that if you if you think it's right, please give it thumbs up. If if it's bad, let's let's give it thumbs down. And we are aggregating the this emoji feedbacks so like we know how we are doing. We that we know if we push a new update out like it's better or worse because since LLMs produce always something different like it's very hard to test the system. I mean, we usually do an update so we run some uh like we open some MRs and see how the feedback looks and it's like, "Yeah, it's better. Ah, it's worse." So, it's really hard to tell. uh And if you are going to the log-detective.com website, uh that's one that's hosting at Fedora, uh the interface looks like this. uh So, it it it is split into two panes. This is the right uh sorry, the left pane that has the overall explanation again like very similar as to what you saw previously from the uh merge request command. It's still like under the hoods, it's the same. It's just the interface is different, but this is on demand. You can go to that link and use it right now. But if all of you do it right now, I'm pretty sure that it will explode or like so- something like that like we didn't scale this service up like at all. And on the right side you see the snippets and ex- explanations of them. And thanks to Zbyszek actually, he suggested that like we improve it and we actually show the whole lines. Uh I mean, we are not UX designers so like that's why it it looks how it looks. I mean, we we did try hard uh but uh we would definitely need an overhaul at this point like to improve the interface. Uh but again, it's it's very similar what you saw in the uh MR command phase. Uh yeah, and we are at the final summary slide. >> Okay. So, um all the things we pretty much already There's only one thing remaining. We have an annotation tool for your logs. So, if you have your build failure and you want to, you know, resolve it, but not just for yourself, but for everybody, you can submit it to our annotation tool at www.logdetective.com. Uh there you will basically perform manually the operations that Log Detective should perform automatically. You will describe what happened, how you can fix it, and just highlight the portions of the original log, you know, as you would want Log Detective to do. That way you will contribute to our public data set. It's published under, you know, very open and non-restrictive license. So, yeah, everything you put in there is going to be public. So, please no submissions that are containing other stuff than build logs. We are going to sanitize it, but it's going to be available for download eventually. So, um we welcome all contributions, not just to the data set, but also to the code. Uh we host everything on GitHub. Um we're appreciating any sort of feedback on the output formatting, everything practically that we're doing there. Um and we're going to have a workshop on Saturday morning. So, if you want, you can join us for that, and we can, you know, take a deeper look. Oh, and if you want to watch Log Detective work at one last URL at the bottom, that's the that's the link to the user in GitLab. So, you can check him out now how he's contributing. That's that's everything. >> Yeah, so Yeah, thank you. Time for questions. >> [applause] >> So, um thank you. First, thank you for that. This looks quite promising. I'm wondering whether how deep could you configure the model itself or LLM from within the log detective config? So, for instance, would you be able to I don't know, configure the temperature of the model just from the log detective configs or uh how do you tune that out? >> I I I can I can manage. I can manage. They can't hear me. Okay. So, um yeah, uh we have the temperature parameter available exposed in the config. Um so, you can just edit a a YAML file and change that. Um you can just swap the model out just by pointing to a different URL as well. So, that part's easy. You can also edit the prompt through a YAML file. So, all of those things can be done. I would have to advise, however, that changing the temperature of the model doesn't have to be necessarily that effective. Like, it used to be that it was one of the parameters that determined the model behavior, but these days it's not having that much effect as it used to. So, yes, you can do that. And also, of course, because we're using the OpenAI API to do the job, uh we can expose additional parameters up to and including all of those that are exposed in the OpenAI API. It's just a question of seeing a need to do it. So, if you see something that's not exposed and you want to, just submit a PR and it's going to get there. >> Another question? >> Um I I know you're using a 7B model, but uh it seems like you're not passing in other elements of the build other than the log. Um beyond the fact that it's called log detective, is there a reason like I would I didn't notice in the snippets that you showed of it saying like, you know, and on line seven, I can see that you passed in this and that's probably the actual root of your cause and it should have been that. As an example. Like, are you not Is there a reason there's no more context being added? >> Does this mic work actually? Oh, yeah, it works perfect. Uh yeah, that's actually a very good question, Brian. Uh I I mean, yeah, we are only using the log file the log file itself. Like, we are not passing any additional info. And yeah, it I think it could be useful. I also think that we could also go to the sources and basically check out the sources and see latest commit or something like that and pass it in there. I I mean, that could be helpful. >> Yeah, um Just the um like, yeah, you can do that. The problem is that if you're using a small model as we are right now, the model has fairly limited um Okay, um any Everybody here is familiar with the word context in terms of like, you know, well, at least somebody is. The point is like um the more the more context is like the memory of the model in terms of computing. If you overflow that, it forgets what was at the beginning. What's worse, even if you don't overflow that, if you approach that limit, uh the model starts to put less emphasis on what was at the beginning. Uh and unfortunately, with smaller models, you get to that limit much much faster. If you're talking about sources of like um the package, um you could maybe devise some sort of heuristic where you would pick out only the changed files, and then you would pipe them into the model context, but you would need to be very careful about not to overflow that. You would probably need to employ additional heuristics just to chop off the not interesting pieces. And if by a chance you would hit something that was important, you know, it's often is that the packages contain more than just one file with source that's executed. So, if that actual problem was somewhere else than in the file patched, yeah, it would increase the complexity a whole lot and uh yeah, I'm not sure how quickly we could implement that or if it that's even like reasonably possible to do on what we have available. Um yeah, so but yes, it's it's technically possible to include other files as well. It's just that the models wouldn't handle it, not those we have right now. >> Yeah, to me sounds like like we would start turning it into an AI agent like to actually be smart about what's happening and try to aggregate it and do something extra which I mean yeah, we could do it but it's it's much more work than just this. >> Hey, I I might have multiple questions but I'm going to try not to ask all of them. So from the description of the log detective, I'm thinking like is it efficient to use like one general model instead of you know, all the like suitable embeddings and re-ranking and vector databases and all of that. And yeah, the other question would be like uh isn't aren't there existing products are you not reinventing the wheel? And I guess the third one would be how do you do evaluations like you know, how do you know that you're improving? >> Okay, so that's three questions really. I'll start with the first one, reinventing the wheel. Um So the That's the second one. Okay, the second one. Well, it doesn't really matter. I'll start with that one. The point is no, nothing like this quite this exists not with the GitLab integration and the closest analog would be copy and pasting the log into a prompt window of your LLM agent or chatbot. Or you could try Copilot, which didn't exist 2 years ago in the form it exists now. I guess that may work, but none of those alternatives are open source in the same way, and none of those can be controlled as well as Log Detective can. So, no. I don't think there's anything that quite fits the niche. Um The third question was how we're seeing if we're improving. Well, that's what the emojis are for. Um there's no way, literally, like it's very difficult Well, cannot no way, okay? That's maybe too broad, but it's very difficult to measure performance of LLMs on any actual tasks. All of those metrics that are commonly employed, starting from perplexity and ending with various chatbot arenas, they are very easily subverted and toyed with. I've seen papers on it, and it has been done before, and it's done now. I mean, there are some things you can check for relatively easily. For example, if you give it yes or no questions and restrict it to that one kind of response. If it's binary, you can easily measure if it's going wrong or right. If you're going for a long-form text and trying to measure that, things get tricky very quickly. And automatically measuring it it becomes more and more problematic. Also, if you want to do that and you want to turn it into a CI, you need at least one additional deployment of the system somewhere running that will not be ever queried by your users, which is something we just don't have resources for. We would need to spin up additional Log Detective instance somewhere else for everything. Then we would need to launch a barrage of tests at it of tests at it tests that, as I've just noted, are not always extra accurate. And then we would need to get some sort of, you know, measurement of the accuracy from that. So, that is making the whole proposition kind of yeah, not so easy to do. Um it's easier to do if you have like a deterministic computer program. There you can just launch unit tests, integration tests and such, but this this just doesn't behave the same way. Um also it's much much more expensive to run. So, uh what was the first question again? Okay, so it wasn't important. Okay. Um well, as we have it described and I mentioned Oh, there was You have a microphone. He's He has micro- microphone. Okay, so um Oh, okay. Okay, so I don't remember the question anymore, so So, what was the >> Okay, it's working. >> Yes. >> Is there enough data to train a machine learning model rather than using an LLM which may have data origin concerns? >> Okay, so no to that first question, and that's why we've said several times that we're gathering data. And because we're gathering the data under license that everybody who contributes to the data set agrees on, we don't have concerns about the origin of the data necessarily because we know the origin. We are the origin. Um so, that's I suppose an answer to that. I If I understood the question correctly, um >> Hope so. It's It's from chat. >> Any any questions? >> another one from chat, so >> Okay. >> Um apology if this has been covered. Uh I missed the beginning, but what is the accuracy of this from your experiences? >> Okay, I would say it's generally accurate when it comes to simple issues that pop up during during the build. For example, anything from missing packages, uh bad um bad options for the compiler, stuff like that. It can pinpoint those very easily. If it's anything more esoteric, then it can be tricky. Uh it gets worse at that point. There's also the point that if by any chance whatever the line the Okay, if the information about the failure isn't in the build logs or if the information gets doesn't get extracted by the template miner we're using, then the LLM has absolutely no hope of figuring it out because there's no information available to it. It's not magic. It's just an LLM. So, that's the limit. So, I guess those would be the failure modes. If it's not in the build log or if the template miner can't pick it out. >> Thank you Yuji and Tomasz. You can ask more questions in the foyer during the break. Thank you for your attention and uh >> [applause]