How to best package your library? - Bernát Gábor - PyLondinium19

welcome back down to the ground floor hope you're enjoying your day so far and need to keep the time so quick introduction of burner number another senior engineer here at Bloomberg this time from our data technologies department helping all of our global data analysts get data onto the system as quickly and efficiently as possible he's big-time contributor to tox and various other packaging tools so he's gonna take an opportunity here today to talk to us about how best to package our libraries thank you thank you very much so yeah earlier about how you install some packages this talk will gonna be more about how do you get to that phase where you can actually install stuff and hopefully also understand what actually happens under the hood when you actually say to people whatever other tool you're saying okay install this package okay so the sake the has a lot of let's say not the nicest things so I brought a few nice puppies to cheer you up a wee bit so yeah just who am i yes I said I work at Bloomberg our senior software engineer I'm also the maintaining of the virtual I'm stool which makes me a member of the Python packaging authority and the maintainer of the talk stool so hopefully I should have a good picture of what I'm actually talking about thank you so yeah this is gonna be again as similar to lasso about the point of view of the Python package of authority meaning that will not talk about these lovely technologies such as Conda or your favorite operating system package manager which is the package or yama or whatever as Packman you're actually using but in order for me to tell my story I actually need some example project so I brought this lovely little project this is how let's say a basic Python library kind of like works nowadays it basically has some kind of logic whatever you want to implement it after the logic you write some tests cuz you don't know it actually does what you want before you write some tests and then you have some packaging information these are just does okay this is kind of how I want you to package stuff and then some additional files which are basic just maintain yours therefore your CI information or your git repository and all that kind of stuff okay so this little library this in the question is the the only thing it does it tells you lovely quote from the point of view of the elaborate parks like in this question it tells you that the lighted Park knows how to make the best of whatever he has to work with so let's try to make the best of whatever we have to work it in the Python packaging so the question is how do we make if I have a right project how do I make it available for someone else running on a totally different machine on the other side of dirt now the way we do is basically when I say make it available I want them to type in this import box I think discuss the library and I run them to be able to do the same thing I'm able to do on my machine and in this case it's basically just calling that code generator okay so whenever someone I actually type in this import box if I actually look at the representation of these bugs module you see that it actually is a representation of a file on the file system somewhere now this file system is something that's called by the side packages Python you sure doesn't really know if a package is available or not it the way it actually finds out it actually tries to do it it's in kind of like a monitor of the Python try first as forgiveness later it just tries to import it and if it manages to import it it will give it back to you that package now the way it does this it actually has a few package loaders registered and this package loaders have some configurations on it and one of these configuration is the suspect and the suspect basically just tells your location where the file system loader will actually try to look for existing packages or not okay so going from this the way we actually can what it means for us to make it available to rather developer another machine it'll basically we have to put it to other that other machine side packages somehow okay and this is basically the goal of the Python packaging how do we take something from my developer source tree packaged it up ship it up to some cloud from where some other people can actually pull down the package and install it into their side packages folder okay so you can see a few tools which like you probably used it if you doing this by the set up to assist have usually packaged stuff then we use the twine to actually go to the PI API which is live it is shop inside Python and then we use peat mostly both to download and install into the assigned packages and discover this kind of additional packages that the user might want to use now so if you actually look into the side packages folder you will see that inside the side packages first we have two things one of the things that we have the actual package files this is like my business logic with some additional compiled file generated there but more importantly we also have some additional metadata okay so when we talk about we want to install a package we want these two things we want to pull in the business logic and we want to pull in also the additional metadata into that folder in their side packages folder so the question is how do we generate this ok now the way we have two options here one of the options is we make available our library as a storage distribution or we make it available as a video distributions and there are some differences between these so what is the source or distribution so so distribution basically is whatever you have in your working directory - the project management file and the maintain your files these are like the CI files your actual git repository that kind of thing okay and still has all the business logic the packaging and the test okay now basically if you take in this project we receive just remove this special information but kept everything else which allows the user to both still actually do our packaging and the reason why we should always also package tests is that whenever the user of the packaging he needs something to validate that the packaging succeeded that's why you should always include your testing your distribution okay it's a validation for the people on the other side actually then try to do the packet the more installation packaging so a will is basically you no longer care about the test the only thing you actually care is about Billy CLE your binary files your business logic okay we're basically it one on one taking just this information like the actual business logic and in the fastings actually whatever actually gets put in its present their insider system side package okay now if you look for the read how do we actually ship or generate a storage solution the way we actually do this is basically like this we first start with developer store three we collect some files from the file system we package it up inside for example like zip it in a file we package it up passive to the pipe I and then the other side how the installation actually now has to work is that the total solution gets pulled down you extract it to a folder you generate those metadata but our metadata is also like for example we saw in the real case you had this additional information about the package plus pulled basically the business logic and put it copy it over to the user machine side package folder okay so this is what you want to end up in the end everything the metadata and the other file now how do you how do you actually ship a will compare to this in case of the wheel the only difference is that now all these building operations generating this metadata and selecting the business logic happens on the user machine ok the user machine meaning that here on the developers or three I directly generate exactly that information that what I want so at this point whenever I get the user machine I literally has to wear it basically I just have to extract whatever I get in that wheel and just copy it over to the site package of the user now you can already see that this means that the operation that we perform in this case on the user machine is very simple it's a lot less that can go over drunk and that's why this is kind like the preferred way how you should do it the body are cases when you this is not not going to be possible I'm going to see later so how do we generate the wheel as I say just generated is generated by C files and this is the reason this for blood point is the reason why it might not be always possible if you actually have C extensions in this case you actually have to know the target platform to be able to generate the binary files and in this case you might not have access to all the platforms that you want to ship to your users so if you are having the C extensions you always most likely will try to shape a few wheels which is the most important main platforms and leave the soul discipline bare as anyone else who doesn't have your main platform to actually fall back and generate their own installation packages okay now this is where the Python packaging Authority and the Condor differs condo sales peeps a that we are not responsible for actually providing you the C++ tool change at least up-to-date maybe this will change in the future what kinda actually takes up and himself to actually also if this build operation is like a compiler actually provide that compiler or actually provide even more dependencies library's header files orders of cooling basically the approach of that the moment of the Python package culture it is that it's up to the user to provide it okay so now there's an initial constraint right for example you actually have to generate wheels in this format widget based specified that from the file name an installer should already know if that is compatible with my platform or not this is just a technicality detail for required using the life for the packages so let's see what you actually need to make this actually this operation happen correctly on the user machine you see that to in order to select a copy files you require if you use setup tools or your packaging you care the correct version of the setup tools if you to generate the metadata you need the wheels package which is on the pipe I this is what actually knows what format to generate this metadata files that you have to put into side packages it also knows what kind of fighting you're targeting because for example it might need to know how to generate any kind of paisa files and also you have the other information like you need to know the art actually actual F C C++ first that you actually compile again tit but more importantly you require that in any case in order for this operation to succeed you need to provide correct setup tools and build version now the way historically worked what actually and the reason why did sees me that is for example that if you're using the feature of setup tools that was added in a new version if you try to run it with an older version basically it's undefined if what's going to happen because you using new features which are not available there so it might work it might not work if it doesn't work if actually if they actually work it actually worked and it doesn't work then you're not sure if actually succeeded or just have them to succeed and you have late and box into your software so it's important that whenever you actually perform this installation you generate your dis build exactly with the right versions of both your packaging ecosystem meaning in this case setup tools or ways that we're using in this example project okay so short history packages have the actual Python packaging ecosystem developed so it all starts basically back in 2000 when it was enough project started to get shared rounded people were like in the core system that maybe we should have some kind of way of defining how we actually package stuff and when you're just starting out and you don't know how you actually and what you what you actually need what's the best way to wing you just basically it seems a great idea that for now we're not standardized what we actually can do we'll just give you basically a script file where you can specify what you want to do and this is how it this could you started out the interview the setup I hear you they provided a core framework of what packaging gives but you could write anything at the setup dot file this mean that you could easily iterate extended base iteration and adapt it to what exactly you needed now set up tools the disabuse was kind of like baking the sea Python which means that it had a long turnaround like if you needed a new feature it took a while to get it out but in 2004 set up to squared around and add a lot of sugar syntax on top of discourteous and made it just a lot nicer to use it be quickly been adopted and it became so much de-facto standard that people I started assuming that it was already there and that's why if you go to the secular Florence you have some packaging issues you can see so the first step is to OB your setup tools if something doesn't work but in 2008 we had people to suppress it installed then we had in 2014 wheels this is very worked the first approach was they saw this problem of not having the correct set up with versus maybe let's not require the set of tools and the user machine we just build exactly what was needed there but even as I said this doesn't necessarily cause maybe you have C extension that kind of stuff is required still to me build in that gives you these are still kind of like assume that you know what order your bill dependencies or the dependent versions of your blender tools okay now one thing is that set up tools and we will still use the set up that pipe which basically allows you to run arbitrary code which is great for flexibility it's horrible to actually use the ability because whatever people who are less experienced coming to the ecosystem and they start writing random Python code into the setup file is basically the great recipe to actually mess up things and people start complaining it's not good so fleet in 2015 while the initially introduced it's kind of like the first attempt to have a decorative or dynamic build system and it uses the PI project on text should define what are your dependencies and this kind of like seemed to work so this is where we actually because it was easier to understand and was order harder to get wrong we try to see how we can move more into this direction having how we can allow the ecosystem to actually have builder tools which are nicer interface than what setup tools usually has okay so how does a build work well a build in the previous aim where I see the actual typical details is that you basically call this set up by eldest and then you are plotted by using the set up by upload which with the bracket deprecated concerned out there is no safe way to actually make this work and HTTP so instead we created a separate tool called twine which is kind of like guaranteed to do it over a safe network protocol and whenever you basically do a pip install what actually happens in the background that peep goes to the pipe I discovers that package pulls it down and after it develop these tracks it it basically calls the setup I install command which basically then use the setup tools to actually generate and copy over the forest into that target that side packages folder okay so now the thing is in this case what happens if the user doesn't have the correct versions of setup to resolve Revere's what actually happens is if we basically end up with this cryptic errors that the Installer starts complaining that I don't have this no module name sleep item build you can see this is kind of like error message targeted at the programmer we actually knows what is an important how what is using this definitely is not a message for the front end user to what is wrong so how we can solve this one so the way we can actually solve this problem and the way we desert it okay maybe let's have a decorative way to define our built environment dependent heat instead of just assuming that the user machine knows what kind of set of tools or wheels we actually need or other or other kind of dual dependently we should allow them ship them tell them exactly this is what this package needs to be able to build okay so what happens in this case what would happen this can virtually we can create a virtual environment if the user machine knows our dependencies we can actually exactly in store that will depend s internet environment and then we basically call this generation of the wheel by using this is a little built environment and this allows us that we always can guarantee that on the user side machine we exactly run with the same versions as we run as we are required in the production now this is what per five on eight is about it's about allowing you to specify exactly what kind of packages you want for the built to perform correctly and if uses the PI project that Tom and Tom mainly cuz we did not have any other better alternatives and yes yam was not considered a better alternative neither Innes so moving on from this basically at this point you see that now we have actually have two jobs we have to do one of the jobs like in this side you can see that now we instead of just actually running the build operation and copying we have two one these we actually first have to generate this easily to build environment and then we have to generate the actual wheel and then install it now the question is who will actually do this kind of is related build generation and the decision has been made that this should not be the actual build I can't and the bill back NT in this case is like set up to fill through poetry instead people will be the one responsible who actually do this operation and people actually consider the build front-end meaning it will not do any build operations but it will provision the environment for the backend to do its job and it will guarantee that that the dependency declaration specified for that back-end or met when its current okay so yay we have one we no longer have this awkward errors that the user will say that hey the installation is Nestor should sit on my machine but there's another problem here if you see in our operation we're still calling basically set up that pi okay so this meant like in case of fleet do have a fleet actually managed to work is that they had to basically derive everything back to the setup PI and the set of tools ecosystem cuz that's the only good thing that people actually understands to call and we have now another standard which basically is pair five on seventeen and this defines okay let's throw away the setup by legacy which is kind of like release there from all the voltages which allows the user to run arbitrary code but instead we should be able to just generate the wheels by having a proper API and this is the API we came up with basically you can actually see the same PI project on file you can actually specify this is my back-end and this is its builder API endpoint which you can actually use so you can see in this case now we actually have some programmatic API and the other advantages of this is that now we can actually have really nice way of calling it from applications which hopefully means that the front-end has a better communication protocol to talk with the backend rather than using basically textual now we actually have more pythonic way of speaking with things so this various back and start adopting this kind of new way of interfaces and set up to provide it from 40.8 flip provides it basically where the early ages and poetry is another builder that provides it for a long time already now and we have also have front-end supports like people 19 already comes with a five on seven so if you have people 19 on your system you literally can use any of these backends you no longer need to set up tools and your bills will or guarantee that never will be having a bad build dependencies unless you specify this wrongly so and toxic another tool or like a front-end which actually does this there are a few caveat though so one thing is our pair five or seven eight of five one eight so the two perhaps did not really address editable instance this was kind of like well initially the powerful design was like this draws a lot of country a lot of non agreement let's just drop the subject and let's not focus on this this is something that hopefully will manage to address in the next year meaning that you can actually use editable in stores cue in the future it requires a fairly new peep which may be a problem array now do you P problem for you it requires happily it only requires a new peep only on developer machine if you actually be sued as a will but if you actually have a dis restores it would acquire the new people only on the user side machine ok so there is a big build comment press called the question is because ok so how do I now because my real stress contributions to upload to pi PI there is a ppl comment hoped slash planned if anyone willing to implement it but in the meantime you actually can use the peph five on several library directly this is basically something you can just install and this actually has an interface to generate binaries or generate source solutions with this single comment and this is what you should prefer using going ahead okay now the benefit that we have reproducible decorative builds no more need for setup by well at least that once we have editable instance ironed out and it should be a lot less simpler now there's one caveat for talks talks at the moment if using it in your CI it doesn't actually enable it by default this is mostly for backward compatibility reasons so you should actually have to specify by using this lovely flag - you want this is related build environment and more importantly going ahead we have this to pepper on 8 + 5 & 7 which ensure that you no longer have all setup we know are going to use this at all legacy set up that PI and we also ensure that we have declare a represents the environment and this should make us a happier hopefully every one of us and in future if you want to build google source solutions please check out this lovely backends you don't have to set up to set up which is very powerful but you it's not the only option you have poetry which may be more simple and much harder to get it wrong and consider setup tools especially in these setup that by format more like something that only ended when you have are something complicated something advantage but not for simple use cases and use the lovely per file 170 library to actually build your source solutions and we this should mean that you have a lot less errors even if you change machines it will automatically pull in all the correct dependencies for the build to succeed correctly so if I talk with anyway you look got lost I have this article blog article which I've wrote it's basically explain this in much more long hoarded and much more in detail and yeah I'll take some questions now Thank You panel will make sure that gets onto the attendees chat on slack if anyone needs a invite slack let me know have we got any questions for Bona before we break for lunch oh right in the corners you just want to see me run yeah thank you for the talk I had one question is there any layer of security or do you all pip install packages in our own risk as in between us uploading packages to PI and then pulling them back down there is a security letter on the pi PI itself like pi PI does some inspections you don't should expect like actual checking what get imported what gets run now the Python 3.8 does have a security API which means that you can actually inspect whatever is running on your application so kind of like you can do some evaluation I mean so that's something that you should look on if you actually want to check what running that library would do to you but otherwise there's only or is like pi PI does have like if they detect anything is done in something horrible they'll remove the package but that's more like if you have to report it and flag it as being a bad package okay I think we've got time for one more question yeah and thanks we talked just in terms of do you have any insights to share any tips kind of running this system let's say in production or in like a practical point of view would you have an internal PIP server or get sub-module well kind of best ways to get it from the history to let's say let's rather than distributing it to the world it may be a controlled manner for organizations or corporates that kind of thing I think we're running it from like like pulling directly the get models I would not say it's horrible it kind of like works the problem is you don't have a way to like easily change it and you like pulling it into version control and you don't need it I generally definitely recommend having some kind of piper server run it locally on your company's machine something that already take care of lot more features for example it automatically allows you to cache whatever you're downloading that kind of stuff because if you're pulling from get every time it's going to be even the pulling down is going to take a lot of operations while if using the pi PI server in its protocol it allows for various improvements over each communication with people cool thank you very much better [Applause] you