The Adventures of pip and pacman: A Tale of Package Managers - László Kiss Kollár - PyLondinium19

right [Music] I hope you enjoyed and snacks upstairs objects were very nice to get us started again we have another senior engineer from our Python infrastructure team Laszlo Laszlo also contributes to various pi PA projects including pit and audit will I think you said yes he will be talking to us a little bit about PIP and pac-man today thank you thank you they're working yeah it's working so good morning everyone yes I'm alas I work in the infrastructure team in Bloomberg and I'm here today to talk to you about or to tell you a tale of package managers let's start by saying making the statement that managing Python packages is actually simple many people will say I guess mostly seasoned by the users that this is not this is actually - and why even give a talk about it my experience is that especially novice users new new people to Python struggle to work with Python packages they can find issues when they encounter permission denied issues or incompatible dependencies or they just struggle to figure out what is the right tool to install packages in Python so in this talk today I'm going to try to go through a few examples and get to a few best practices to see how can we make managing Python actually simple because I do think it is actually simple so let's start off by actually looking at what our package managers they are tools to install search remove and just you know do general operations with Python packages when we are talking about Python packaging we definitely probably all know PIP which is the main by far most popular package manager in Python and does anyone know what PIP stands for no it tends for pip install packages right it's pretty cool so pip is an open source project it's a very robust and used by a lot of people it does all of those things I mentioned early plus a lot of other things it's a very advanced tool by default it will install packages from pyah pyah at the cheese shop so this makes it possible for us to get access to the all those amazing open-source projects on PI API but you can also use it to install local packages or use private indices and a lot more on the other side we have package managers from the UNIX like or like the operating system package managers which are like aptiom pac-man and and many others these are much more general-purpose package managers they usually take the upstream source packages potentially apply patches to them run a test suite if there are extensions which need to be compiled they will compile them and package them up for the appropriate distribution this means that these are generally these work better on any given distribution because they are well tested for the particle distribution but it also means that it takes more work to get them to work which means that the selection is generally smaller much smaller than what you will be able to get on pi PI and the updates are generally slower so there is a trade-off there so the question is and actually before I if I get to the next one I just want to quickly mention that this is a this is a short talk and I'm completely going to just avoid Windows and klonda and I'm very sorry about that it's a very very important area but I just don't have time to talk about that today if you do have questions I we can talk maybe after the after the presentation so yes I'm sorry but I'm going to focus on unix-like operating systems and most Linux so which one should we use the big question and the question is of or the answer is of course it depends and it depends on many factors but I think it makes sense to look at things like how or where do you want to install these packages and it's it's useful to separate installations into categories like system-wide package installations what these are or basically when we talk about the system-wide package installation we consider one global package installation location which is shared across that machine by all users for the particular packet Python interpreter let's see how this works so we Google we're brand new user to Python and we we just want to try some fancy package we found on the internet and we we want to try that we Google how do I install Python packages and then we get the answer on Stack Overflow somewhere oh you just pick install package name alright so let's do that people install PI ml let's see we in stuff imo and we got to come in at hound pip is actually now installed ok that's interesting so what do we do with that so pip is actually part of the C Python distribution so if you download it from python.org you should get it to the Installer but on many Linux distributions it's actually split into a separate package so you do have to install it separately in this case we actually use pac-man the Arch Linux package manager to install Python - pip to get installed depending on your distribution you'll have to see how exactly do this but you can already see that you already need the operating system package manager to install the Python package to get this to work ok so we have that next thing before we move on I want to give you a tip which is that don't do that do this instead if you run pip install that's going to invoke pip with whatever version that pip was installed for whatever version of python was installed for and unfortunately still today there are many people using distributions where pip or python is on to seven which means that now you're installing pip calling people to python to seven and you're installing packages to Python to seventh sign packages you might not actually want that and it can be very confusing when you are not able to import packages with I don't or seven so yeah just explicitly specify the interpreter and use - and pip it's going to save you headaches okay so now we all know all that and we actually start to install package rather than three seven - install IMO and what we get is a permission denied error right here why is this happening so what happens here is that pip is trying to install into user live Python three seven side packages which is the global site on this machine this is like I said shared by all users of the machine so you won't be able to install as any regular user into that only root channel or administrative users can do that okay so what do we do in this case we run the same thing with sudo right what else let's do that and yeah no errors it worked excellent right everything works what can go wrong we have a problem though so now let's say a few days later we find this cool project called beads which we don't even realize is written in Python and we install it on our favorite Linux distribution with pac-man and in terms that it is indeed written in Python and it depends on this yamo Python package which we've just installed with pip using sudo and now the package manager says it says error filter commit transaction conflicting files when it's trying to install files into this location which we've just installed files to wit pip so yes there is an inconsistent between the package managers internal view of what should be installed and what is actually installed because we have basically messed up that global site so this is a bad situation to be in and it's actually it can be quite tricky to clean it up so Swee do pip is not a good idea it's going to cause problems now there's another thing here which is important to mention executing setup the py as root is a better idea seller dot VI is by far the most popular way of implementing the build and installation steps in Python projects this is changing today or nowadays but still this is by far the most fuller way to install Python packages so essentially when you do pip install with sudo you're executing a Python script with root privileges and anything can be ended setup WI script so for example I wrote a nice example here let's just import Shu tail and rm3 everything on route and just for good measure ignore the errors to make sure it works and you might think this is this is silly no one's going to do this right but what if I name this URL and then I upload it to pi PI and all the people who think that the popular your live library which is actually sped you early 3 is the one they are installing and they miss that 3 which you do they run this which would which is really bad right and actually this is called typosquatting and in 2017 there was a similar incident on the pipe ei PI pi where some unknown person impersonated several packages and the downloaded set of the VI scripts were just collecting information from the Machine so nothing malicious happen but an Hakeem anything could have been done in those scripts so actually the PSF is actually funding projects on the PI P I to make this more difficult or or just to improve the security but it is still a really bad idea to run with ok so what have we learned about system-wide package installations well I think the biggest takeaway what I can say here is that just use the u.s. package manager just just stick to that because that's going to work best if the package is available obviously because the package selection is much smaller you should just do this you'll be fine so apt back man or young or whatever your favorite exists which uses what happens though if the package is not available whatever you want to install is just not available to package repository oops it's falling apart yeah so in that case we can use known system-wide installations these are ways of installations where we restrict the installation into maybe a user's home directory or even to just a separate directory and we don't interfere with any other package installations other users the machine or anything else so the the simplest way of doing this is with the user user site installation so this is something python has supported since 2.6 so it's really widely available and we can use it with piping store management user if you do this when I try it and we let's install this not so well-known but very cool project called black and what you see is that everything works fine you just installed it there is a warning here which we're going to ignore here but I will come back to that the installation just works fine and then let's say let's see what is in our system path just print that and you see that there is this directory on sister path which is in dot local under my home this is the user side and if anything is in the dark that directory exists then cpython will put it on the path and it will look for files there when you're trying to import something so this is what the - my user argument doesn't rip so now if we try to import black and print where that file is from you can see that it is indeed installed into that local my home into the Python 3 7 site over there so it's great everything works and it's really simple you don't have to use sudo or anything now if we want to if you want to run the black script though f not going to work unless we put this dot local bin directory on the path and this is what the warning was referring to from pip because it was smart enough to detect that we don't have that director on the path and it can't really do much other than telling us that you need to do this it's a one-time setup that's very easy to do and after that everything just works as expected so this is cool there is a bot and that is that basically all applications and libraries install - bit - - user will share the same location and this means that if you have application a which depends on some library with one particular version and you have another application which depends on the same library but with a different version that's not going to work because you can only have one version of that be available in the user side to overcome this we can use virtual environment so virtual environments take this a step further basically they they contain an entire virtual Python installation in a single directory they do some it does some clever tricks to make it seem like it's an actual full-blown Python installation it is not really it's it's kind of it's really lightweight but it works really well it is actually built into Python since the release three point three so you can just use - mvf it will work you have to install anything if you if you happen to use some older version a legacy Python for whatever reason you can use the awesome we're children package just in so it from IP I using either your OS package manager or - - user as we've just learned and then you can do by tomorrow some - hmm okay so how do we actually use this thing it takes two steps to get to a virtual environment first of all we just run 1 3 7 - 3 M - n BN and enemy the name of the environment and this is the name of the directory which will be created where the environment will live it will be created in the current directory and that's all done and then we have to activate it the activation is basically just sourcing this activate script from the environment itself and this activates scripts does a few things for example it updates or prompt to show us the name of the environment that we are inside a virtual environment it will also update a few environments we're able to make sure that everything is contained within that environment and everything works as intended and it will save the previous values of those environments environment values ok now let's try to use this if now we try to install a package we can just go ahead and pip install and in this case this is perfectly fine you don't need to write the explicitly the interpreter version because there's only one interpreter in this case self-contained in the environment and yeah everything just works as expected no warnings it's an insult and inside this virtual lamp if we can try to import black and then look at the file we see that it is indeed in the my vm directory and it leaves there self-contained if we try to run it it also works nothing we didn't have to change anything because the activation script updated or path so it's already available to use if we want to exit the virtual environment we have to call this special deactivate function which is also something the activation script made available and then everything goes back to normal it restores all the environment variables and total amnesia nothing works anymore black is not there like nothing happened the director base is still there so you can just go back again and activate it and use it we don't actually have to activate it we keep what we can do is if we if we just write out the full path and then do a script in the bin directory and try to execute it it will work perfectly fine and that is because virtual memory writes the feedbacks so the first line of the script and it will put the Python interpreter inside the VM onto the onto the shebang line so this will actually work perfectly fine you will be able to potentially put this directory on your path and just use the tools from over there you only need to activate it if you want to make modifications you want install scripts or you want to do development and you really want to import things for example from the virtual end ok so now with all that what have we learned let's recap doing at install packages BIP BIP system-wide besides the fact that is probably not even possible it's going to cause issues especially if we do it as root and that is also going to expose us to potential security issues so just avoid this instead you zapped back mania or your favorite package manager to install Python packages system-wide if you can but it's better to prefer non system-wide installations user mode installations if you want to keep it simple and you just want to make up tool available to your user or if you have a more complicated situation or you just want to try out something quickly just use virtual environments and that is all I wanted to say okay we have some time for questions if anybody has a question thank you what do you experience with maintaining packaged versions in your environment like think package dot lock and also what's your experience miss Conder sorry maintaining packaged versions in my yeah so let's say you created your environments by people installing black one months ago and then you recreated this environment on a different machine you get slightly different versions you have a difference in minor versions but you have different behavior between well I guess it depends so me III personally for development purposes something like black I would just installed it in user mode on whatever machine I'm working on a new soda version available there if I actually need a specific version for a specific project for development I will probably make that make that known in the requirements in the project itself and then make it possible to install it into the virtual environment or through talks or some other tool to actually make it part of the development process and require an explicit version of that so I would say it really depends on on the situation I do often like I have black or PI tests and all of these tools installed in on the user side because I use them all the time and I might just have a simple project where I just want to run my tests - and it's available but you can put it in your virtual lab and require an explicit version my experience in Kannada is very little I don't use I don't use it I know mostly what it is and the anaconda distribution I think for Windows it's it's amazing it's about the best way to get started with Python and Python packages I think there's actually a lot of collaboration happening between Python packaging like python packaging Authority and the cold or the anaconda corporation to get ideas from each other and just collaborate it seems to me that pip is actually getting more and more features from Conda and the other way around I think it's really good the one big drawback with Conda is that you have a smaller selection of packages than what's available API which I think it's probably fine if you are doing machine learning or data science because that's really widely covered on anaconda but it might not be fine for more niche Thanks any other questions hello I had more of a question of package managers in general it feels like we have a package manager for every day of the week and there's always a new one on the horizon other any plans in the open-source community to try and kind of merged package managers into one universal tool foreign languages what do you mean my package manager is because I think pip is still the de-facto package manager for Python if you mean more like additional tools like I don't know you might be thinking of depends and things like that I meant more package managers made for programmer and I could just say NPM kind of yarn oh I see like just we have so many of them are there any plans just to try and kind of put them all into one I think that that would be very difficult because all of these languages have their own specific quirks and implementation details which the package managers need to keep in mind so creating one I think would be a very challenging task what what that intersection today is I think is the operating systems package managers which are general-purpose package managers and you can always use them to install like NPM Python packages if they're available in the package selection but I think you're encouraged to just stick to the like like you saw in this presentation if you can't just avoid that and use more self-contained localized installations and just use your own programming languages package manager okay thank you I think we've got time for one more question is there any command in which V install all the dependencies let's say some package but it's dependent on SQLite so as people stall that package one SQLite is you start with this right you mean native the native dependency of that package yeah so that's actually a problem the PI P I PI PA is looking into right now the fact that you cannot track or even Express the native dependencies in a Python packages it is a problem you basically just have to know that you have to install it and what ends up happening is that you get an error while you're installing the package that X is not available and you have to google it and then someone on Stack Overflow says that all you have to install sequel Lite or something which is not not a great experience I have to say that there is no solution to this at the moment it is being work looked at and the idea is that and this actual example where I think the pipe is looking for collaboration with the Condor or the anaconda group because they do have something similar and and to make this work in a way that that the people can somehow or - packages can express external dependencies which people can somehow make available to external package managers like the operating systems package manager but this is very very preliminary and I think there are other people who are make sure I see one in the audience was probably much better suited to talk about this yeah Paul yeah maybe maybe yeah you can catch him after the talk and yeah not today cool we have to draw it so close though thank you very much Larry [Applause] you