PyPI Security: Past, Present & Future

okay welcome everyone it's uh uh it's late in capital Andi is going to talk about pip thank you round of applause please thank you um hello um actually I just found out now how it is pronounced I've been saying uh pipie for the all of my life um confusing with the other pie of course uh but I'm talking about the repository uh so I will probably keep saying Pi pie because that's uh how I've been saying until now if I remember I will say pipy as uh um I'm very opinionated uh I think myself as a troll so if you disagree with my opinions feel free there's some facts here and there so you might want to agree with those at least but uh feel free to disagree with anything else um I made that slide about myself because I feel very important so I have um done some great achievements in my life I found a security bug in Internet Explorer and um I managed to steal cookies from a website uh Micosoft said no it's fine like this uh it should be like this so I could um you you got a popup like uh oh is this safe this is not safe do you are you sure you want to load it yes or no but it loaded before asking uh but um yeah I got banned from an Allin it place by for eating too much not for nothing strange and I can play and sing a whole song or a couple while on one foot so uh if anything can top me I will uh I don't know feel diminished um to be more serious um my first package in Debian got in in 2009 um I'm interested in Linux of mobile I use the phone as a main phone for a short while uh but my interest is more in application side so I've been packaging um applications and U writing ones um I also I'm involved with python um partly as a personal interest and partly as a work thing and um I would like to be to have some more Sicilian in Debian U right now there there's only one package is the Sicilian fortunes uh and there is a pending the language files for a little game that I've been in new for I don't months uh yeah anyways let's move on to the uh real topic of uh of my talk um basically when I started to to use Python um my my project was a relational algebra compile um I didn't know pii existed at all so my experience was you get stuff from Debian that's uh that's all like there's Debian and then there's the internet so like um some guy's homepage or or Source Forge back then or GitHub or whatever um I I had I didn't know p p sorry existed um yeah and um for example once I needed a python package to to make the colors um like RGB colors on the terminal and I P did it because like okay I need this thing in Debian so I should make it go there uh and for me that that was the way like um I didn't even question how things were um one uh day I got a job uh I finished University I went to work and uh and there I found out that um there there's other things about uh that are not the Linux distributions um it was terrible actually because they were even doing like autoscaling and and it took like about 30 minutes to boot up a machine and download everything from pii because they were downloading everything every time um anyways um at my job I forced them to stop using pipie and use the Debian packages um and this has been uh good for licensing oddly because uh we publish our um server image on Google uh cloud and they say that uh if you use whatever from a distribution then it's fine but if you include anything else then you need to provide licensing information so my team didn't have to provide anything because we just use de packages and uh yeah then it was no extra job for me and those guys using go they had to work a lot for their sins um yeah so pip uh pip sorry as a user um before 2018 it uh allowed uh signing things and uh it allowed users to publish a public key which um has some utility but limited in the sense that if you hack the user and replace the the key then it's useless but you can know if the the key got got replaced and then uh it was just checking the short ID from the keys and the clients were not checking anything so it was um but I so it is claimed that no one was using it but I actually have no idea if someone had some tools to to check it or not anyways that was the previous Pipi implementation but then they reimplemented it completely like with a new software uh where uh signatures got hidden and the public stopped existing there so then you you needed um project by project to know a way to to find the key so if there was someone using some tool to check the signatures then the tool became useless because then you needed to know where to find the the public key um at some point I released my first python Library uh and I released it on it is in Dean and also on pip um because I thought okay it's it's a library uh it should be there so others can use it um then nothing happened for a while and then python said that the critical projects they must use second Factor authentication and I hate it like I I mean it's more secure but I don't care about security I mean I work with security it is my job and I but I only want second Factor authentication for my bank that's it you know like LinkedIn Facebook email I don't care you want to ask me okay I will get a new account no problem uh I mean for me it would be easier anyways I thought okay po those guys who have critical projects that other people use and they will have to do this extra job with uh you know doing the second factor to publish and uh and so on but um so um my project is the I don't know if you can see that there is a very thin line there this is a download counter uh these two projects they do the same exact thing but uh one is more popular than mine so I felt very safe about the second Factor indication okay it's going to fly very high over my head not doesn't concern me at all um uh I download a good metrics of popularity as we know here with dealing with popcorn I would say they are not uh I made a little side project from this talk so I became in the top 20 of most popular python projects and then I got an email saying quit C it out or we ban you because you're abusing our terms of use um so I Ian this project was very fun to make because my first iteration was like oh just uh use Peep And A bash Loop and download the same thing over and over and it doesn't count because it Cates locally so okay I made the ram file system I delete and redownload delete it's very slow doesn't work uh don't do it like that it's terrible but um then okay I use curly in a loop H slightly better but still slow then I made it in in Python then uh then I made one packet in Python where I sent just uh tens of thousands of requests in one packet then ah then it's fast uh unfortunately the replies are really slow so so you need to you need time to to get all this data that you don't want you're just going to throw it away but uh HTTP has a nice header which is the range header so you don't need to get like your whole pocket you don't care about the pocket you just want to think that you got the pocket so he only asked for one bite and that is how I got I mean it takes me like um 20 minutes to do like that to do like 14 what is 15 million downloads in Home Connection nothing special I'm not the c or anything uh yeah so I I actually think this is very gamed because uh the the previous uh popular project from before they it's now a startup so it got uh with companies and so on and they hired developers so I think this popularity thing is uh there's some conflict of interest there I mean uh and and I'm sure they know because I'm I'm still using this for my library they haven't said anything or they keep counting they they know it all comes from one IP every day at the same time it's like 14 million requests that the for 20 minutes and then it stops like but uh yeah they keep counting and okay so I'm pretty sure I'm not the only one who thought about this but maybe the first who thought to make a project the project still EX but it's not on piie anymore but on codb or codg uh you can get it from there if you are interested in becoming popular um but what are the metrics to decide what is uh an important project and my answer is I have no idea uh because all of these things can be gamed um number comets yeah you can just do delete and out the same line a million times and uh you can just create fake identities is in the commit uh header and um stars GitHub stars are also a a thing and you can buy them so there's fake accounts um I don't know if you use GitHub I I stopped but I still have to use it for work um but uh I get um stars from like accounts which have no projects nothing no activity and they are fake accounts that also put stars in other places so so they confuse the recognition algorithms um another way to possibly find out which project is important would be reverse dependencies but um in my experience I mean there's some websites that should do this but in my experience they don't work like I published projects that use that project but uh that project is marked as having no reverse dependencies so I don't maybe I didn't use the correct syntax that they expect I should be using anyways um I I mean it could make sense like especially for a library for a client project or like a leaf bucket then it it wouldn't make any sense to but uh yeah uh I mean if it gets improved maybe it's a good way at least um anyways to to make sure that my project remain very unpopular I I put this in my my contributing um I never actually received any patch as an email I tried my best but uh yeah never happened but I didn't stop there I put this as a license on a library and I got many bug reports telling me like you're stupid what are you doing this is you use an MIT license why are you using this I want to use this so what's your problem uh um then I put a code of conduct uh which forbids to call me stupid for using GPL so I can just bun you if you if you complain because uh people were doing that and the USA thing is unrelated to another project but uh Americans tend to not like it even though I think it's very fair like it doesn't say like Americans are less special just says they are equal they don't like it I don't know uh um anyways my project became critical despite all all of my effort to make it like no one should use ever my project it became critical and this was before I invented the download automatic downloader thing so it uh someone else must have downloaded it a million times on on their own but it wasn't me um so I had to personally deal with all of this uh authentication and security things and um so the to sweeten the deal they made a token giveaway so they were giving this uh USB tokens that you have to press um the thing is that I live in Sweden and uh they've delivered in many countries in Europe except uh yeah Sweden so but uh of course I'm Italian so I got it delivered in Italy so at least this that's a photo of the tokens that I received I I think uh Google paid for them um personally I think they are very annoying to use H and of course you know if it's for my bank I am very willing to go through uh you know difficulties but uh but otherwise I uh I find it very annoying um the thing is that um I use fire so fire jail by default doesn't allow USB access so in the beginning I thought this doesn't work on Linux maybe it's only Windows supported I don't know but actually they do work it's just that I needed to change some rules there um and then you're supposed because they give you two so it's safe so but you're supposed to keep them separate so if your house burns then you only lose one but the other one you still have it but you also need them both together because every time you enroll a new website you also need both so you cannot like just put it somewhere you know in a I give it to my friend who lives in Spain or something no I need both of them at hand um so it's a bit uh complicated I don't know but um I still have both of them I haven't and they still work um of or you can use OTP which I think most people use from the phone um so phones break or at least mine do a lot uh and until recently you know you know I mean of course there is a million ways to do this and uh but the let's say normal person way would be to use a Google authenticator until recently you couldn't back it up now you can back it up but then Google has it also so I don't know which one is uh well before without backup was more secure but uh then you break your phone and it's gone and you need to rescue your account somehow um and you're supposed to print this uh this codes like these rescue codes print it and keep it in a safe place um so I don't know how organized you are but I am not so first of all I don't own a printer so if I to print them I have to go to the library and um it's a public network I don't know is someone reading my whatever I'm sending to the printer uh I don't know maybe it's not that secure anyways probably not but um uh then I I would lose the piece of paper or the text file wherever I save them I I I think um I have no chance of of of finding it whereever whenever I need it like maybe 10 years later um and then you can ask uh PP stuff to recover your account but it takes months for them to do so and uh if you have printed the rescue codes then they I know this because they they have a public issues about this so you can see the conversation happening so normally between like opening and resolution it's one to four months and then but if if they see that you have printed the codes or you have downloaded the codes uh then they will not rescue the account because they say like oh no maybe you're trying to hijack someone else's account so we don't touch it uh and and once this happen your project is gone like it's forever there you no one else can get it it will not be deleted it is just there stuck forever so you can just Fork your own project with a different name uh of course um I put everything in my password manager on my machine so the the second Factor authentication is in my uh same first Factor authentication um anyways I mean my fear about this was that I actually had to do the second Factor authentication to do that the upload so that I needed to carry physically this USB thing it and put it in and press it whenever I wanted to do an upload but that's not how it works at all basically you generate a token so you go on the pp website generate a token save it in a text file and that's it for forget about your Tok you can throw it away you don't need it anymore just um don't don't lose your token um so in theory this uh the to because you can get a fully scoped token but in theory you should not use them because it's insecure so what you're supposed to do is that get a fully scoped token because it's the only way to upload something then upload something and destroy the token and create a new token that is only valid for that project uh but I suspect no one does it and they just and people just generate one token and keep using it forever um then in uh since May 23 uh signatures are discarded so if you upload a signature you get an angry email saying stop doing that don't uh stop signing your stuff um um in the more or less same time they made trusted Publishers which means that um that you can um uh compile your project on GitHub mostly and upload it from there and then uh you don't have to generate the token and you don't need a global token if you upload it from GitHub then you can just create the project from GitHub you don't need the the dance with the global token for some reason um some uh com nice compromises have happened recently or in this same time span soly torch got compromised twice once because uh they were using their own Runners so if you open the pr to them you could uh leave a process running Forever on the runner and then uh it would uh intercept the secrets when it was used again the same Runner to to publish for example uh and then uh because someone published a package on on the on the real Pip with with the same name as a packet that they were using in their own pip and and but the real pip has a priority so and the pet was the same but it was it contained the stuff to steal all the credentials from the home of the user um also the administrator of pip uh published this on uh Global Tok uh by mistake on dockerhub so for for a while it was possible to just go and um and write whatever you want on whichever packet and uh in PIP but uh this got found out very soon apparently and uh anyways I mean to me it looks like we're doing a lot of effort into securing things and then we allow Global tokens to keep existing especially administrator Global tokens to keep existing and I mean if we count the amount of hours spent in the security I think could have been better spent into doing other things um recently they introduced attestation which uh so this was last week oh this week um so um it means that instead of signing now GitHub signs for you so it's signed in the way that yes it's uh coming from GitHub uh and it sign um to to go towards the end of my talk so pgp is not popular so they removed it and second Factor authentication is now very popular because it is compulsory but uh I would like to know how popular it was and if it was more popular than pgp before it was compulsory because I mean if we make it compulsory then everyone is using it okay but uh is that that really a surprise and uh and anyways uh so we still are relying on a file on the dis to authenticate because first I mean with with second Factor authentication you rely on having the token on your dis which you need but uh if you say no we do trusted authentication then okay we don't need that token but you still need your SSH key or GitHub token to upload to GitHub so in in a way it's like similar I mean if if my machine gets stolen my token is there like you need you have the token whichever kind of sort of token and how many jumps it takes um but I I so I think they're calling it security but what they want to do is actually more towards the authentication um so I have a conspiracy theory here but uh you are very free to disagree with this um this is is all my sick mind thinking of that um so I think that basically they want to to slowly towards the ears push people to just push uh code using GitHub or other companies like that and uh and then Microsoft can take care of Bunning um Russian guys for example as they've done already uh and then uh when you download code from pipe then you say okay I know there is no Russians because they are banned from GitHub and we trust Microsoft to do this job and so me American company can uh freely use whatever I download from pipe because there's no Russian guys H and and there is all this s bomb thing and so on so so they can uh uh you know what is s bomb does everyone know okay H yeah yeah so so I think it's it's geared uh in that direction of uh of securing uh government uh like not government contract but like that if you have a government contract you can keep using pipe and uh and and keep doing whatever you want and and know that Microsoft with all the resources is on it to to ban whomever the government is saying that must be banned um that United to the fact that he really hate pgp so I think also this is a very important goal for some people to get rid of pgp forever and uh and be done with it uh so I have a quote here from a fellow devian developer and troll you might read it on your own I mean I agree with this once in a while I agree with him uh that uh yeah I think distributions still have a place in in the world even though because all of this it's like okay we we are doing all this effort to authenticate people but no one is checking what the code is I mean I am still very free to go there and and publish a virus on pip like no one is is checking but uh but it's very like yes all the steps are authenticated you have second Factor authentication and the file went from there to there also they have no way to do Repro reproducible builds um so you can have like okay I uploaded from this comit and then in my build there was a car pipe shell and then you have no way like okay what happened here in the middle from this source and this binary because you can use the network while doing the builds so yeah I don't know to me it feels very um performative security rather than actual security um but um yeah I don't know feel free to have your own opinions uh I'm done with the talk if you have uh comments or questions or um feel free thank you [Applause] one minut for one question about by by peace security to factorization you're the wrong crowd I thought everyone would disagree with me and oh you're an idiot why looks like we have terrible lots of people seems very happy thank you yeah man pgp is all very well good but every time I try to use gpg on the command line uh I hate it I have to look at the manual page it's it's completely unusable okay maybe you should use a graphical [Laughter] interface thank you thank you very much thank you very much I think we are out of time um we have still one last talk from the DPL and after before closing ceremony see you soon in five or something 10 minus oh today's fine thank you [Applause]