EuroPython 2026 - Security and Ethics in the Age of Generative AI Panel

I am extremely excited about this topic and this panel and this all around for these amazing and talented people. I will not going to invest a lot of time to introduce them because I think that they are going to do it much better than me. Then I would love to start with everyone talking about who you are, what do you work on and why do you think that security and ethics is important please? Um Julian can you start >> from France where I'm software engineer soon in my career I realized that testers automated testers are very important for software quality and that software quality is very important to have good products and so I iterated on that and started to learn about operations and product and now cyber security. So, I'm working for a cyber security uh company focused on fishing, fishing detection, fishing prevention, fishing technon. >> Wow, sounds amazing. Please, Mike. >> Hi. Uh I'm Mike Feedler. Uh I live in New York City in the United States. Uh I work for the Python Software Foundation on pipi.org. Uh doing security and safety uh for the past three years. uh I think that in today's landscape and historical landscape but more so today uh having an ethical responsibility towards each other and towards humanity uh is taking uh a backseat to some other concerns. So we have to kind of bring it to the forefront whenever we can uh to talk about how we take care of each other in this landscape. >> My name is Sylvia. I'm a security researcher at GitHub security lab. I work with finding vulnerabilities in open source software. So the topic of security and uh open especially open source security is uh very close to what I work with. Um currently I'm also doing a rotation with another software development team uh working on the private vulnerability reporting feature. Uh and this is the feature on the github.com platform uh that allows security researchers to report vulnerabilities privately uh to the maintainers. And what we've uh experienced recently is uh this that is being uh used a lot more in the last few months. Uh and with and so uh this is uh also yeah very relevant to the panel. So my name is Visha and I'm from Kov from here. Uh now I'm working as a Python developer but my background is in philosophy and social sciences. uh during my academic years I was more mostly interested in the human agency and also cognitive science. Uh there was also a notion of cyborgs this very interesting way of uh approaching certain uh questions about humanity and about the boundaries of it. Uh so I feel that may you know LM somehow are allowing us to extend our mental capabilities. uh so we are like there are similar uh let's say uh aspects to the cyborg notion uh and about ethics I think that uh for me ethics is not so much about obeying the rules and finding the best uh sets of you uh rules to to to comply with but it is about like practical application of what you're thinking about what the self is what what we are uh so in this way uh also uh like the geni like make me focus on on us. What does it mean for us as a humans? >> Hello everyone. I'm Seth Larson. I also work at the Python Software Foundation along with Mike. I focus more on the security of the Python programming language, but then also packaging tools. So everything that's outside of Pippi that you end up installing Python packages with. I focus there. And I feel like security to me is about like infrastructure making it so that it's easy for people to do the things that they want to do with Python without having to be an expert in every single thing. Right? Everyone in this room has a different set of expertise. And so like one of my goals and one of the goals that I try to inspire others to when it comes to security is thinking about doing it for other people so they don't have to learn everything about security and they're able to focus more on the domain that they're interested in or that they're an expert in. Amazing. Thank you so much. And as you will see why I'm so excited uh with all this diversity of backgrounds and let's start with the same the the first part of the panel that I want to have a more general overview and I would love to start with Sylvia maybe asking could you tell us a bit what is your research about in GitHub? >> Yes. So um when I do my research at u GitHub in open source security usually what I do is uh first of all I read the code manually. I use static code analysis for finding vulnerabilities um and maybe some dynamic analysis run the code with the debugger and uh the way that AI has changed the picture a bit um is that it fills some gaps that uh we have had for many years in automated vulnerability detection. Uh so if we take web vulnerabilities as an example um uh we have a lot of tools for example for static code analysis that will find vulnerabilities like injection type vulnerabilities like SQL injection, command injection, cross-ite scripting uh but the way these tools work is that they have these rigid rules and as long and uh they would miss a lot of types of vulnerabilities. uh so for example authorization types vulnerabilities or uh authentication bypasses business logic vulnerabilities and um and here where AI is well it's more flexible in a way that it can read understand the code uh and help us find vulnerabilities in that way and so AI in that way became another tool in my toolbox in finding some also some other types of vulnerabilities um yeah so that's That's how I have been using AI in my security research work. >> Yeah. And yeah, there was a small thing in here. Oh, good. Okay. Yeah. As um as Silia was sping, generative AI is having an impact. Uh and Julian, what do how do you think in which areas generative AI have given defenders uh some kind of advantage? For example, something that the platform does today that two years ago or six years ago, maybe even better the comparison you can do down. It's very difficult to detect what is fishing and what is not fishing because most fishing websites are made to look like the actual website and some companies publish websites that look like them the original one but not very the same logo or there are differences. So we have really good actors that are acting like like fishing ones and fishing ones that are acting like good actors. So it's very difficult to tell them apart and we try to use AI and generative AI to do that but that's not that much efficient. We don't have much much gain on that. Mhm. >> But for example, we are using some NLP techniques to do analysis of content and for example on that I know is very much helping us because it's very much NLP on steroids you can very easily analyze what is the intent what what is the urgency of the messages that you receive >> and so to determine if it if it may be a fishing attempt so it's one more sing but not a really good answer I is not solving that for us. Oh, super very interesting topic. Uh I have been working a lot on NLP and I didn't thought that this could have been a big advantage in this in this area in this way. Oh, very interesting. Uh someone else has some other uh point of view or thoughts on how your life changed in the last uh years with generative AI in general. Maybe >> I I'll I'll try this. Uh so what what what what Sylvia mentioned and what Julia mentioned are very much uh taking these tools and practices that we have historically done manually and now we are kind of accelerating our our capabilities to use them, right? Uh lots of people have uh described generative AI models as a very eager uh unrelenting intern, right? Somebody who has infinite coffee and energy and is not going to stop. they are probably going to get a lot of things wrong. But if you kind of teach your intern the same way you would teach an AI LLM, you can get actionable and good results. But you as the human discerning have to have to apply that critical eye to their output so that way you are not kind of led down the path by your intern into a dark tunnel that ends in sadness. So for me, I'd say like recognizing that reality has just helped me kind of apply that critical eye. Okay, the output is the output is do I trust the output? Would I trust it from a human? Maybe. Would I trust it from an intern? Maybe less. So continuing to kind of apply the like this is helpful. This will get me further than I was before. It's an accelerator, but it has drawbacks. So being candid and talking about those and and and kind of recognizing those has helped me kind of isolate what practices in my day-to-day I am willing to delegate to an intern versus which ones I would definitely not. >> Yeah, definitely. Super good point. I will also now talk a bit about um code generation and I would love to talk from another perspective. In this case, I'm going to ask to Maria, what is your take on how generative AI is doing changing the human side of coding? >> So, I think that uh at this point we are also in an early stage of the option. So, we all use it but uh the things haven't settled down yet and we need to like find our ways of uh uh like uh inter uh interacting with this tool. uh I think uh well we there was a talk like human envelope is tired so uh yeah this there is this this thing that uh genetic AI produce so much output uh that sometimes it's hard for our brain to process and it's not only because it's so sophisticated but also there is so much of it and we are not used to it uh and uh it can be intimidated intimidating and it also can like uh We may feel reduced to somehow just accepting it and just letting go and we are not fully uh we like cannot be fully focused on what's going on there. So we are not able to uh uh for example really engage in well there is this this uh we could but there's this danger that we are not engaging in what is there. We don't know anymore what is there. we just clicking and accept accept >> and um so this taking like the sense of agency of what we are doing uh so there is a danger like this uh I think also that there is a lot of anxieties because uh you know we may uh like make a joke so of that or anything like uh like it but you know there's a spread of losing jobs uh because of AI uh there is this thinking that do we still need to uh learn like manual programming. Is it something that is really needed any uh or it's just some kind of uh ancient art? Uh it makes me think about uh like uh carpestry and power tools and flatp pack furniture like everyone now can create their own furniture. So do we still need carpenters? Uh and on the other hand, yes, we do need them, but they became some kind of um luxury products. Uh so they are affordable for a few maybe. So are we going to what what will we have a similar situation with with programs? Uh yep. But on the other hand, I just want to uh like um be more balanced. Uh we have also other opportunities. For example, I am dyslexic and like for me I struggle every day. I struggle with these little things of spelling of like putting my thoughts into coherent uh uh words uh etc. And LLMs are so brilliant in it. So in this understanding uh it really helps me to uh overcome my shortcoming and to concentrate fully on on what I want to do. So I don't need to struggle with this all little bits and yeah I'm selecting but there are a lot of a lot of conditions like this others and yeah so u sounds super interesting thank you for sharing all of you was super insightful um I would love uh to also start another topic and move uh to some situation that I had uh maybe one year ago when I was uh using one LLM or some LLM attempts to brainstorming and in very specific sub field and apparently provide me a solution with a peep install library name and sounds so good. It was so so so good that it was like this feels so weird. I'm going to go to my PIA website. I'm going to check it if it's right. Surprise for no one. It was an hallucination. This library was not real. There was some names that were close but was not. And this may bring me to the next topic and the next question to Seth. Tell us a bit of a coin concept that you you did that is probably I'm gonna pronounce it very bad quing and how do you see the um the scene about this and if you expected the growth of that >> yeah so slop squatting it's like a port manto on typo squatting I will say so I I coined the term slop squatting but there was a researcher that was doing research on this for a few years before I even was interested in this. Uh Joseph Sprackcklin, so look up his work. Um but yeah so this phenomenon is basically an LLM will suggest you a package to use or maybe I mean packages is the big one but it will be a hallucination and if it is a hallucination and you run pip install whatever the package that they suggested and an attacker who also has access to the same LLM as you decides okay well when I ask these specific questions I get the LLM to suggest me to install this incorrect name I'm going to go and register this name and then when people install it thinking that it's a real package, they will get malware on their system. Right? So, this is the the attack pattern. And I feel like it is still a thing that exists. It's LLM are are not perfect. They will still hallucinate. They have gotten better. I do think that LLM companies that are doing these sorts of trainings are aware of this issue and probably have tried to train it out. I do not know how successful they are. Uh, but it is definitely something to do just in general when you're using the output of LLMs is to if they're instructing you to take an action, especially one that crosses a security boundary like that where you're choosing to install something, you should do your due diligence. Make sure that the thing that is there is actually used. It's not malware. We try our best to take malware down within a matter of days or less, but obviously there's always an opportunity for there to be overlap. So, please be diligent when you're using LLM outputs. >> Yeah. Yeah. And related to that, I wanted to follow continue with you and Mike because I would love to know has Pipi considered proactly reserving packages names that uh the LLMs commonly uh hallucinate or could hallucinate before attacker resist them. So uh I I I find this topic so fascinating because Python packaging is very unique uh compared to many other ecosystems. Uh and and some of these are strengths. Some they're all trade-offs, but some of you might consider them bad or behaviors, but there's reasons for all of them. We might not know them today, but they're there's useful there. So one of the reasons that that I have found that hallucinations like these happen is because a Python distribution file does not have to match the import name that the file is using. So if you import pil you've probably installed pillow. Uh and those two names are different and LLMs don't necessarily have that context of what that difference is and what the import name is versus the installation name is. They'll also use uh console scripts or entry points to again see that that is a name that people have published in documentation to say these are commands you should run but that's also not the package name. So LLMs are just as fallible as humans when it comes to context but they are far more confident in their representation of you should do this. >> Yeah. So there's reasons for all of these divergences uh and and they exist and trying to kind of go after every single possible one is a is a I'll say it it's a fool's errand right because there's no way we're going to catch them all. So there's the the the point of like well if we try to put effort in that direction are we giving end users a misplaced sense of security that we have done that for you and you don't again to Seth's point you don't have to do your own due diligence. >> Something everyone needs to remember is you are downloading random software off the internet from people you don't know. >> That's terrifying. We should all be scared of that all of the time and we should do our due diligence and do whatever processes and practices we we should to vet what it is we are downloading and consuming and publishing out and betting our our careers on. It's dangerous out there because it's free and anyone can do anything. So trying to proactively find every single potential hallucination. We also have the same problem with typo squatting. So we we have decided actively to not try to ban that because package names or project names are sequences of characters and and that's all they are. They are a semantic symbol. Just because you have a semantic symbol that is an acronym of the PhD research you are doing this year and someone else had one that is one character off doesn't guarantee you ownership of that name or that symbol. It's just a sequence of characters. So, uh, the the guidance I'd say is use copy paste very carefully. Check your characters that somebody hasn't hidden some Unicode in between those characters because that's another way to get at people with non-printable spaces, but packages won't accept them anymore. Yay. Um, >> but uh but but it's like you you you have to be very very uh careful and there are tools out there and methods out there that will help you be careful such as pip audit, such as copying names off of Pippi as opposed to uh what someone posted in a discord in response to a help because that is another kind of security vector or what your LLM is out is producing. uh so many people are basically not adopting the like I want to be interested in the ingredients where my my food has come from and this is the same problem like you should be you should be curious where your software comes from >> completely agree and thank you for all the advices I think that all of us we should follow them uh as you are already with with the work can we discuss about what happened with the light LLM uh topic uh uh and also provide some context to the audience and some overview and how was solved and what could have been done better or any insights that you can provide us. >> Sure. So for for context, light lm is a very popular uh Python package on pipi.org. Lots of people use it in order to interact with their different LLM products. And as you may have heard, AI is kind of popular. Uh, and lots of people write lots of software and there's a lot of interest and lightm gets millions of downloads every single minute or day and it's it like like lots of software, right? It it kind of hits a popularity and hockey stick growth and everybody loves it. Attackers love going after big popular things because they use that as a multiplication effort. If I can get into something big, now everyone has it. If I have to go after Julian's company or Maria's uh company, then it's harder. But if I go after your supply chain, I can get you easier, right? So, they're looking for weaknesses. And with Light LLM, they found a weakness in their in not in their build or or they were using a tool, a security scanning tool that was intended to be download and scan your your project for security problems. And this security tool had a security vulnerability and they had a malicious project published. So anyone who was using the latest version of that security tool is now subject to all of the supply chain attacks. Like they weren't doing anything wrong per se other than letting the latest come into their universe without any vetting, auditing or or a human in the loop making a decision. Uh some of us may use GitHub actions or use dependabot or renovate bot to suggest updates and some of us are like oh that's so noisy why don't I just take latest is why you don't take latest uh latest will give you something that you did not necessarily know about approve and it it happened in minutes lightm was then exposed by this other tool their credentials were were kind of exfiltrated to the attackers and a new version of lightlm was published the uh to Pippi that contains malicious behavior that now millions of people or hundreds of thousands of people were were kind of systems not necessarily people systems were consuming this in a very rapid rate. Uh, and it's it's scary how fast these things happen now, right? We talk about acceleration in the age of AI. And it's true like uh Seth and I did a a blog post analyzing the traffic pattern specifically around light LLM and we can guesstimate about 120,000 installs in the time window where these uh malicious versions were live on Pippi. Whereas uh modern practices that we are advising most people to use which is called either pinning right pin pin your dependencies don't use latest uh and then secondarily use dependency cooldowns so that way you aren't the first right let's let let a security team who you know their company their they love downloading and analyzing things let them be first don't be the first right uh and like maybe you're waiting for a specific patch or behavior So, keep an eye on that tracker, but don't use latest. Uh, the the incident here got reported to Pippi while I was asleep, right? Like it it happens 24 hours a day, and I'm not awake 24 hours a day. Uh, uh, all the coffee in the world won't do it. Red Bull doesn't do it. Uh, and I'm too old for that. Uh but the uh the the the part that I kind of have to remind folks is that like the the remediation of this happened when I got up and saw the notices. >> We I've been working on different systems to try and automate and and minimize that time. But there was three hours of window where this is just running rampant across the the the world by anyone and any system that are doing this and it's propagating that attack. it's stealing more data and that that time window is very fast. It happens very quickly and then companies or or individuals who have been subject to this take days to either find out that they that this happened to them and then even further to analyze the impact that it happened to them and their systems and their companies to try and figure out how bad has this been. So I I encourage everybody to like take a page out of that book and just don't use latest. Good. I will take u this uh as a recommendation even for myself because sometime we think the latest is the best right is the better is it's the last thing is this human feeling of w this going to be the >> yeah okay >> I mean latest is good but it's it's it's sharp right so if you follow like the Debian project and their release cycles they have stable unstable testing right uh I may have gotten those out of order but one of those is the sharpest edge and it's like if you want to opt into that that's great that's not the default >> you can still opt into it and you can in Python >> and that is currently many tools default but again maybe that shouldn't be the case maybe it shouldn't be the default >> yeah yeah well as we are talking about trust uh and supply chain maybe Maria can also give us uh uh her thoughts about uh people have the possibility to share code that they don't fully understand that they don't they don't fully even write and that is the still the same kind of trust that is that the same kind of trust that we have been building in our open space communities. What is your thought on that? >> Uh so uh well first of all this is not the new phen phenomenon. We have stack overflow for years >> so people were still able to do it. Of course, now it's faster and more and more and it's multiplied. Uh but uh yeah, it's still there. Uh and um we also have to have this this this notion that we can do this, but should we? So there's also the sense of it's us that makes the decision and uh you cannot really take away this trust uh because uh even if the code is uh generated I'm still uh can take the accountability for it and I just want to describe free notion of responsibility which is like uh um you can talk about so the first one is like obligation and or duty so it's good for uh some kind of security or complying to safety. But uh if you hear about responsibility in that context then you probably oh it's heavy a bit dull I somehow prefer to isade it you know you you don't feel maybe very good about this kind of heavy responsibility but there are also two others uh so the second one will be something like that I want intentionally to do right job and that means that um I'm thinking about the outcomes both positive and negative of my code, I'm thinking about the risk I uh like put into this code and I try to mitigate it and of course uh it's only my effort. So I might not be succeeded but at least I try and I also am willing to be accountable for it. So I'm not like evading from no this is my code this is my decision that I wanted to contribute that I wanted to share. So uh I'm ready to take to take you know to to then to then yeah be responsible and but there is also the third uh notion of responsible responsibility and this is the ability to response uh so in a way uh the word the yeah around us asking us a question this is like kind of method meth uh okay like meth methodic ah no I won't say it today. So, so yeah, you can say like it's asking us a question but it's like also like uh um pushing challenges in front of us and this is about ability to respond whether we will respond to this challenges will we do something. So this is like the call of action really like are we ready to to take the action because maybe we maybe this is something that is really needed right now. Yeah, definitely agree with you. Someone wants to add something to this more humanistic point of view or should we jump into other topics more into attacker defenders discussions? Should we jump? Okay. Uh then let's continue with uh thank you so much. It was amazing this part. Let's also continue with Julian. And as I spoil already to you, I add this as spoiler based on your experience and okay, which is your um what do you think that has the biggest advantage right now? The defender, the attacker as everyone can have access to this generative AI and all the knowledge previously. What is your point of view? So today many people nearly everyone can access very good LLMs and produce a lot of code. But the question remains is it good code and so that matters both for attackants or defenders. So as attackers you want to create fake website and that's very very easy to do. Now you can ask an LLM to copy any website and he will do it apart from the new ones that may not be very happy about it but you can force them and any unalign LLM you can ask whatever you want and they will provide they are very eager to serve. So it's very easy to replicate. If you want to add a backend server that will receive credentials that are get excfiltrated, it's very easy. You can ask them to do that for you. You can ask them to deploy that for you on cloud served servers. It's very easy to automate that. So the the work of attackers has been made much easier to start with LLMs because now like you can do a website over the weekend for your association. Now you can do a fishing website over the weekend for your criminal activities, but LLMs are not very good at creating fishing. Like they're very like most of the time they're not very good at the task we ask them. And so when you are doing fishing or criminal activities, what you actually want is not just to fish victims, but to not get caught. And the problem is there. LM are not very good at being perfect. And so they will do mistakes and if you are not good at illegal activities, you won't spot the mistakes. Like if you are not good at Python coding, you won't spot coding mistakes or anything like that. The rules are the same for them as for us. And so we are seeing a lot of people starting to create fishing websites, but it's very easy to to catch them because they are not very good at web security and they make rookie mistakes. So on on one part it's easier for them to do it but it's easier for us to do it because um before more experienced people were doing criminal activities. Now regular people try to do it and they are not very good at it. So they have kind of an an advantage but for experienced groups that do criminal activities as a regular basis that is their day-to-day work they get a pretty good advantage from LLM because they are they can more quickly evolve against our detection methods. So like you and I, we can more easily solve the Jira ticket and move it to to the next column and they do the same. So for more experienced groups their job is a bit easier but like ours us too we want to detect their website. We want to catch them. We want to find proof that they are doing that they are wrongdoing and so we are as any other developer we are helped by LM to go faster on that to be more efficient to try new things. So we we are not seeing that much of an advantage >> over the previous actors, but the new actors are not very good and so easier to catch. >> Okay, good to know. I am still curious to know the opinion of Sylvia that probably she's more into the defender side, but let's uh hear her opinion. >> Yeah. So um recently the situation I feel is so dynamic and every new week there comes a new AI tool for uh aiding for example vulnerability detection as well as uh attacker tools. Uh so in some way I feel we are uh on somewhat even ground in the sense that uh there are both tools for attackers as well as for defenders. And so for example at GitHub security lab uh we've developed an AI agent framework tool for open source maintainers to find vulnerabilities in their projects uh which we've been uh using uh on a lot of uh open source projects. Um it's basically an AI agent with some MCP tools and some special prompting and um what it does it runs in some loops to verify the vulnerabilities that is fine. So there are fewer false positives and so uh the quality of reports it produces are uh better. Um but what another topic I wanted to touch upon here was how much uh of these vulnerability reports we are producing um also thanks to those uh due to those AI tools and uh we can definitely see this uh trend in the data for example from the GitHub advisory database. Um so even if we compare for example to last year the number of CVE assignments by uh GitHub CNA uh in the last year GitHub CNA assigned uh over 2,9 around 2,900 CVE and in this year today when I checked in the morning so July 15th uh we've already assigned over 7,400 so it's over two and a half times more and we are halfway through the year. So we are on our way to have five times as many uh assigned as five times as many CVs for the whole year of 2026. Uh as well as uh for the private vulnerability reporting. So um basically there are both uh users as well as AI agents who use these tools to uh report vulner uh to report vulnerabilities to the maintainers. uh in January there would be so for the whole month of January there would be around a thousand repository uh thousand reports and in the month of May I believe there were 13,000 per month so this is 13,000 13 times more >> wow wow it's imp it's impressive I want life to move a bit of topic But I'm very curious Mike about uh who is working when a package get quarantined is this automated is a human there can how is this working? Um >> yeah so uh Pippi for for many years uh was an amazing tool for the community for people to say I would like to share software with others. Here it is Pippi and everyone can use it which is a wonderful thing. you are giving a gift to the world. Sometimes that gift has little bits that you didn't expect in it. Uh and that's a problem, right? And the the ability to continue and sustain this this machine that like is a is a really big engine. Nobody thought Pippi would be used for malware. Nobody nobody thought about that because to to Seth's earlier point, not everybody is a security expert, right? And we don't expect everybody to be a security expert, but we now need to be some degree of security expert in some part of our lives because it's too dangerous to not be. >> So, Pippi had to grow capabilities over the years to start being a little more discerning and kind of being able to put uh uh projects in different states or life cycle states. Uh and this this grew out of a need where me as a Pippi admin and responding to reports the only lever I had to pull was destroy >> right and that is a very aggressive move and you don't want to get it wrong so you defer that decision until your last possible moment so you don't make the wrong choice because destroy is irrevocable that's it it's gone uh and so I I introduced this notion of pro quarantining a project which is basically saying we don't know right now. Put it in timeout. Put it in a place where it is not destructive. It can come back, but it is not installable by users. It is still visible on the website. If you want to, you can find the content addressable URL to download it. But pip install or UV install or whatever tool you use should not respect that and should not work uh for a new install. If you've pinned it, that's a whole other story. But this notion of quarantine grew out of well, again, I have to sleep. uh what do we do when things are happening while we sleep? How do we allow security researchers uh responding to an incident to continue to respond in a non-destructive manner? >> So creating this life cycle status gives the the Pippi engine the ability to say timeout not install until a human can review the quarantine status and either verdict and and destroy or clear it from quarantine. And uh that started out as very much a manual operation of just like I'm I'm out. I get a pig on my phone. I can click quarantine and then check it in the morning, right? Like I'm I'm not going to deal with that right now. And very quickly it was like well the utility of this is limited by whoever is an admin receiving a ping making that distinction and decision >> and why should we only let humans do this? So uh I built out the capability to have a set of huristics and other uh signals that can place a project in an automated quarantine session. >> So uh I' I've built out a program with a variety of security research teams, individuals, companies around the world who they are either interested, curious people report malware to Pippi from all corners of the earth. I don't know why half of them do it. I don't care. They're doing the job. It's great. Thank you. Keep doing it. uh but then there are other security companies who have products to sell and they they are invested in this. So they are partnering and that's also very very cool. Thank you for your support doing that. So now that we have this variety of of of individuals corporations performing these initial rounds of triage I added a heristic to say okay if two or three of of this group or this cohort agree then let's put it in automated quarantine for now and then when I wake up I can make that verdict. I can do the decisioning for for the final verdict but in the meantime we are protecting the community from this proliferating and it expanding. Uh so it's like it's already out but can we min minimize and mitigate the time in which this is a problem. So there is still a a question of the people who are reporting are they human are they AI and the answer is maybe as long as they have a high degree of true positives >> right if they if they start submitting false positives they I have my own personal brain reputation meter and they start falling and then I don't trust them as much uh and there uh one of my most notable researchers uh is a gentleman Kamill Monkovski from Cert Austria. >> He does this as a hobby. This is something he does for fun. And he has self-reported by himself well over 2,000 incidents. And he's been right all along. >> So people who love doing this, I support you. Please keep doing it. Uh but if you do it for work, please get your work to pay for it and and help. uh but having these these trusted community members who truly they are passionate they care about the security story for the Python ecosystem like those are the people who are the hidden figures of our world who they're not up on stage right here I'm trying to represent their work uh and that's that's a hard task too because that puts some focus on them that they might not want so I try to be cautious about who I name but the uh the the ability there is how do I take these people's work it IP AI augmented or accelerated, but as long as they're getting 100% accuracy or 99% accuracy, I will continue to take that work and we will all benefit from it. >> Sounds amazing. And I love that you mentioned community because this link with the next correction that I would love to be more focused and kind of start closing the panel with the community and I would love to uh to invite set to answer the next question. Of course this going to be share question because I want to have different um perspective. uh but the question is is the volunteer maintainer model still sustainable when generative AI increase both contributions and attack that is what you were very well mentioning. >> Yeah. So right now what I see with this new age of LLMs is that there's a lot of opportunity and the opportunity is maybe changing how we're thinking about how we're doing work these days. the maybe the contributions you're getting are looking different. The vulnerability reports that you're getting are look different or there's different volumes. We're going to have to figure out what this means for like on a projectto project basis and on an individual to individual basis. It's going to be a kind of a a transitionary time is what I would describe what we're in right now. So there's lots of opportunity in a transitionary time. We're allowed to experiment. We're allowed to try new things. Just because we were doing things in the past doesn't mean that we have to do them in the future or vice versa. We get to choose our new path forward. Um, and so I would make sure that everyone has an experimental mind when they're coming to open source and how things are done. Uh, keep keep curiosity at the forefront. Uh, and especially in security. I think that one of the things with security that I'm most excited about is this idea that uh in the past there was no contributor ladder uh for security teams, people that are remediating vulnerabilities because there's this trust boundary, right? There's this hierarchy that existed because we were trying to keep people like users safe and so only security researchers are doing this work. Now, in this new age of LLMs where anyone with an LLM can find a vulnerability, sometimes it's only a low severity or whatever, but because now that is more public, it makes it so that there's potentially at least one rung on the contributor ladder for security researchers or people that want to work on security teams for open source projects. They can contribute to these vulnerabilities because they're they're basically public. Anyone with an LLM can find them. And so in that way it it provides an onboarding ramp that did not exist before for security teams or people that were interested in that type of contribution to open source projects. So that's just an example of of something new that it unlocks, right? And we would not have expected it until we started to live it. >> Yeah. Julian, what do you think? What is your opinion? So we we get asked a lot to go faster using II and still today I'm not very confident about the agents that I use because we have to trust them a lot both on doing the right job that the the brit branch is working fine at the end but also that the process to get there is safe that they don't install un unwanted dependencies and that they don't want they don't do action on your system that you wouldn't do. For example, reading your own file and sending it off in China, it happens to me. Or installing uh untrusted software on running commands to remove files. These are all actions that we are not very confident about. And today I I don't think we have solved it and we are still in the transition. We want to go faster but we are increasing the risk that we take and so we need to build trust in II and right now I'm I don't know exactly how to do that. >> Yeah. Sylvia, what do you think about that? Yeah, I also believe there is there is needed some change to the maintainer u to to the model and one of the ways that uh we can support open sources of course for companies to support uh as well as as well as I wanted to talk a little bit about some of this new initiatives that uh came up uh recently uh to support open source software and offload some of the security work from the maintainers. So for example there is GitHub secure open source fund which provides funding and security training for uh maintainers as well as potential u possibility for uh collaboration. Um there is uh OpenAI's patch the planet. There is project AC uh which also came out recently uh which is uh meant to uh provide a platform for triaging uh new vulnerability reports and they duplicate them because what happens often is that uh maintainers get a lot of similar vulnerability reports because they use the same models and so they produce similar vulnerabilities. And so I think it's important to um engage with the community and hear learn about these initiatives and hopefully and also take part in them. So uh which this should hopefully support this um community and open source model. Mike or Maria, do you have any thoughts on that or >> I think that everything was said like already. So, >> well, as >> well, I I have one just tag on that is we we talk about community a lot, right? And one of the biggest kind of security signals that that trigger corporations concerns is a single maintainer project, right? Uh and there are lots of them out there. I I have a few of my own, right? Uh and when we talk about community, it's how do we foster a community where we are okay sharing our projects with each other so that way we can not only represent my perspective. Maybe Julian and I would work on one, maybe Sylvia and I would work on one and we can represent different perspectives and build a broader community base, but also that limits the security boundary. It's no longer, well, someone's going to attack Mike's laptop. they have to get both Mike and Sylvia or both Mike and Julian or all three of us to agree that this code should get pushed out. So by increasing our own personal project communities, we are then kind of leveraging the wider community and kind of bolstering each each other that way as well. >> Yeah, I would love to close this section with a last question for Maria that's going to be a bit more deep. Um is generative um this generative AI moment a real turning point in humans uh progress or is more a bubble and will be in the books in 100 300 years as it was just a bomb and it a bubble and it's not something >> uh well I think that at the moment is really hard to tell because we had even this discussion yesterday uh because there's yeah we don't know there may be some threshold that will uh hold us back or maybe not uh it's hard to tell but uh I think it's very important to like handle the problems that we have now what we know about and there is so much to do uh so I don't think of course like speculating about I don't know AGI or something it's fantastic and uh exciting But let's we should focus on what we can do what we see we see a lot of problems already. So I think that we need to like uh yeah do something with this uh and of and yeah and then later on prepare for what may come uh if AGI like hits uh the market like we don't know this is like we honestly don't know what will happen so let's prepare how we can and yeah we there is a lot of tooling to be done education uh leg legislation uh we should also be like because um right now there is this uh kind of way that uh the progress is pushed by mega corporations from the just few countries and come on do you want this like significant event in the human history to be drived just by corporate profit or maybe by some so-called dark LLMs that just are focused on malicious exploits. So I think we should be there uh also to be involved in this in this process. So it's also shaped by the needs of community >> and yeah open source is just wonderful like uh community to do it. Yeah. >> Yeah. I would love to kind of start to wrapping up uh the panel before if we have a couple of minutes maybe we can also give the chance to the audience to give uh to ask a couple of questions. But I would love to ask um or to all of you what do you recommend to all of us? I know that we you share already some tips but I would love also to hear from all of you what should we do better and what should we do to be prepared because this room is full of te more technical less technical different um section of tech and I truly believe that all of us we need to be prepared and have some kind of minimum amount of knowledge in security and I would love to know from you what do you recommend Please Seth will you start? >> Yeah. So I I can start by maybe people these days will like ask for open source contributions go to a project that you're interested in but I wanted to specifically highlight uh something that would be a differentiator which is to try to make a human connection uh with that community before contributing because what we are seeing these days is very different than what we were seeing before. before people would come to a project and signal interest on a human level and now it is mostly just submissions with code. And so if you can go back to the basics and like connect with the community before making contributions or talk about what you're interested in or try to figure out like how you can contribute to a project, it sets you apart so far compared to what other like other people are doing what we're seeing. So I would I would recommend try to keep that human element involved in open source contribution because it is really really important to have and it's the type of network and tool that we'll need to be able to have that like future ingenuity and be able to take advantage of opportunities in the future when they arise. We need that human network to be there because if we're all kind of in our own corner doing our own thing we kind of lose that ability to act collectively. So >> uh so for me I will go back to this uh human agency because like everything now is happening so fast and can be overwhelming. So we might like be in this kind of passive attitude or there's nothing much we can do. So but we we shouldn't really go that direction. Uh at least we can do is like change our way of using LLMs. It's not like that we are overflooded by it by it but it's this tool that is supposed to help us. So if we start just uh looking from this perspective that this is the tool and we decide what we do with this tool it can maybe it will just be for our better own like good but we also make have ideas how to like uh contribute participate in other projects and what can be done. Uh I also wanted to touch upon the uh human connection and maintaining this human contact um in open source in vulnerability detection and reporting. A lot of us uh experience similar problems and uh so I and so what I wanted to say is whether you're overwhelmed by a number of um vulnerability reports your open source project is receiving, you're also not alone. And um what I would recommend is to engage with uh other uh community members uh also here at Europyon and uh to exchange uh what worked, what didn't uh what could be better uh as well as uh share that with uh the companies that might be that could uh help and um as and these initiatives that I uh that I mentioned before. So uh something that I I've tried to think about is how do we take what we are sharing here because this is very much a a a like a selection bias right you all selected to be here which is awesome thank you for coming >> there are so many people who aren't selecting to come to either Europyon or specifically to a cyber security panel uh who need to kind of hear these messages of how do we go forward so I would ask you all to kind of take these messages and fan them out even further to the folks who either did not want to come and didn't feel the need to come or just didn't have the resources to attend to continue to share that it's it's not simple, it's not easy. We're all working to try and make it easier, but being aware awareness is half the battle. So, just knowing that downloading software off the internet can be dangerous is is a good message. So continue to share these ideas. Share share a link to the talk. Share share whatever you can with the people who have either explicitly opted out or could not come. >> I like Seth's idea of trust boundaries and to recognize them and to be careful when you are crossing one. And for example, your credentials, your passwords, your tokens, they are very sensitive and most of the time we deal with them manually. And for example this morning there was a talk about should you trust trusted publishing and it was very good because it was it made clear that the name is is misleading and that it's a special part of the supply chain that it is about and not the wall supply chain and so understanding better how the system work makes you more robust to attacks. And for example, a very simple and practical step is to use password managers so that you don't have to remember them and every time checking that you are entering them on the good website. It works for tokens too. You can use vaults. There are many practical steps that makes your life easier and more secure and so to make to not make cyber security always painful but sometimes very useful. Thank you so much for all these uh advice and tips and yeah definitely I completely agree with all of you. Uh we have two minutes left. I'm not sure if someone has a very quick question. If not we are going to Oh yeah I I see people that is going to the mic. >> Um >> yeah there's a microphone in the center. >> Microphone. Yeah. In the middle of the room. >> Yeah. Hello. Um, you spoke a lot about security and not so much about ethics. So, some easy questions will be about ethics. So, using AI has a questionable ethics. There are people who believe it's unethical. There are people who believe it they rather not know about that being unethical and stuff like that. And um my question is is it okay for me to submit LLM generated stuff that I trust to people I don't know without their prior consent. So to make it more realistic if I go to an open source project and provide my contribution that I trust I I verified it. I I put my name on it. I believe it's doing what it should be, but I used something that the open source maintainer on the other side might consider unethical. Am I uh bordering them with with my contribution? Should I uh not do that without an AI policy on their side or something like that? I don't expect a yes or no else answer. Just that would be too easy, I guess. >> Yeah. So like you already mentioned kind of uh part of the answer which is that if a project has or does not have an AI policy some projects have adopted these and have either made it explicit that they're against it or like you said they're willing to accept contributions from LLM substantially or unsubstantially or whatever right to varying degrees as long as you're willing to put your name on it as long as you're willing to represent that what you're submitting is at the quality standard that you're okay with right so there's there's different degrees so like Like you said, there's no yes or no. I would definitely if you want to be careful about this, you can always engage in a humanto human conversation with the project and ask like how do you feel about this? Um because in the end that humanto human connection is going to be even more valuable than that individual contribution will be. >> We're going to have the only time for the last question. I'm so sorry. >> Yeah, no problem. Hi. Um good afternoon. Thanks so much for the panel. Um so I'm an AI engineer and tomorrow I'll be speaking about um prompt injection and things about AI engineering. From the panel discussion I see there are two layers. So there is the AI security part of things where you're trying to secure the AI and there's also using AI to search for vulnerabilities. Now um as an AI engineer I'm moving towards this AI security thing and I'm not so much into this um into the community of cyber security engineers. How do I get into that community so that we can secure this AI because as things improve AI the responsibility we give AI is increasing. So now we have tools, we have APIs, we have MCP and all these kind of things. The responsibility improves and we need to secure it in a way just securing securing the end part of it is not enough. We need to also secure the um AI itself. So how do we collaborate to get to this point where we can secure the AI? That's one thing. The second part of it is I think CIA you spoke yesterday and you spoke about having like 80 CVE that's what you guys call it. So in AI we now have a lot of CVS. How can we how can we try to get more of this so that we avoid the general public getting exposed to this um security um security issues. Thanks so much. Um could you repeat the question? >> Okay. Um let me let me explain from the beginning again. So we have two layers to the security issues. We have the AI security where you need to secure the AI and we have using AI to search for vulnerabilities. So we have two different problems or two different layers. Now, how do we I'm an AI engineer and I'm trying to see how we can secure AI systems. You are cyber security engineers. You know how to secure web and other systems. How can I move from where I am to collaborate with you so I can use some of the tools you use to secure AI systems? So, that's one part of it. The other part of it is I know you guys have a way of doing this old CVE and checking for vulnerability things and now from what I do in AI security I can see there are a lot of them that we are like the zero prompts from Microsoft and from copilot and the rest of them what we are seeing these days how can we get more of these so that we can also protect the opensource systems for AI systems So I think answering your first question I think it's uh securing whether we are securing uh applications or AI systems but it's still a threat modeling exercise. Um so the similar concepts apply and uh we're just looking at the different kind of systems. So I would recommend u diving still into how uh well more into threat modeling as well as um how uh as well as well there are some uh community projects uh especially from uh OASP and some companies uh that provide some more information about uh about these uh securing these kinds of systems. Um >> I I guess maybe for the second part of that how do we find more CVES like like Sylvia shared stats right like we we are finding them uh I think it becomes a matter of uh after you have threat modeled and figured out the kind of scope of your system is now go attack it right just just keep poking it and seeing what this is what human researchers have done for a very very long time they have different techniques and some were easier than others so they followed those paths but use use your LL M you you know develop your prompt try try again and just keep poking at the system and you'll find new things that you didn't find before. >> Thank you so much. We need to close here. The speaker is going to be around the conference and thank you so much for joining the panel and have a nice afternoon.