Publishing a Python package on pip [PythonLiveLessons #6]

[Music] hi everyone welcome to episode sixth the seventh episode of python live lessons my name is sebastian matto and this is a weekly show about python code and the general workflow around developing a python project and what we're going to cover today is continue basically where we left off during the last episodes in which we developed a python application called markdown code exec hi and our goal for today's episode is actually to publish this project that we previously developed on pip so that other people can install it so what i mean by that for those of you who are not super familiar i think most people know what pip is so there is a so-called a website a repository called pipe by the python package index hi welcome welcome to the stream um and uh this is where most of the this is sort of the standard way the app store of python libraries so to say so then if you want to install you write you want to install some kind of python package say our python package markdown code exec then you would type in the terminal pip install and then the name of the package that you want to install and it would be downloaded and installed in your python environment this is the main way to install python packages not the only way there are also for example other ways like anaconda is a python environment that has its own package package management tool if you work under ubuntu ubuntu has its own package management tool so there are various ways to install python packages but pip pip install is by by by a landslide most famous one the best known one and our goal is to basically allow users to install our package markdown code exec in this easy way by typing pip install and as you will see that's quite easy to do so how does that work um so i'm going to just to get this out of the way there has been a shift in the python ecosystem where for a very long time the normal way to actually in package your software was to create a file called setup.pi and setup the pi do would basically do everything related to packaging to it would would be the entry point the script that would create a wheel package which is this the for the package format that python uses um it would be it would allow you to ins to download the source code and run setup the pi to install your package from source there it did a lot of different slightly different jobs and then there was a tool called twine that would upload your package to pipe by and this is the old way and actually quite recently ish the python ecosystem has shifted to a emerging new standard this probably will change somewhat i think over the next releases but the idea is that there is a single project file byte by tom's markup language and a tool called flit and together these two this this file that describes your project and flit that actually reads the file and builds a package and uploads the package to pipeline together they actually perform the packaging operations um so that's what we're going to do we're not going to create a setup.buy we're going to create a pi project and use flit now as always the main goal of doing this live is to to engage with questions in the chat so if there are some questions or comments or suggestions please don't hesitate to post them in the chat if you are watching this live of course so um let's actually get to work um so switch to main scene all right so here we have our python project that we created so far um so what did we do well we made it we created a nice command line app that we can execute and and a python module called markdown code exec if you're interested in that just watch back to previous episodes because that's where basically we walked through the whole pro process of creating this library this module python watch now and for today the first step that we need to do as i already said what we need is a file called by project. so let's just create an empty file called by project up new file my project up and i will open it and here in this file all the stuff that we need to describe our project that's where it will go so it will contain the metadata things like where the documentation lives who the author is things like that so let's put it aside up and then in the browser i will go and this what you're looking at now is the documentation of flit so um it's a bit messy i'm not gonna lie the ecosystem around packaging python packages is a bit messy but this if you google for flit and pipeproject.tomol you will find this documentation and it to a very large extent it just describes the the stuff that you need to put here in the in this by project or tomo file so at the top it just says okay for any new project using flit it will look like this so every and we are building a new project using flit and they are saying if that's the case just copy paste this into your pi project what does this do i don't know exactly but basically they try to make flit also backwards compatible with other older packages and then you could indicate other things in here but for our purpose we just want to sort of look towards the future and copy their recommendation this is this is the metadata that we need to have in any case without really understanding what it means this is at the top this is a bit of black magic but for the rest the stuff that we're going to put in our pi project automo is pretty easy um it is kind of sort of simple metadata that describes basic properties of our python project so let's also just copy this and and put it in there so yep this of course is related is is metadata for sort of a an example example project file um for an example python project but we need to just adjust it accordingly right so up my name the name is not ast check but it is uh markdown code exec so this the name refers to the name of the package as it will be called on pi by if we upload it the authors well that's me of course so my name is sebastian matto um email address the one that i use for this stuff in any case the cooksidle to now there is a readme file we are actually not using a rich text we are using a readme.md right so here you see we have our readme.md that we actually created in the previous episode and that is our readme file we have a license that indicates what the licenses under which our module should be should be distributed here they're saying an mit license i prefer the new general public license not because i'm that's a very well informed choice but just because that's kind of what i'm used to i'm not super familiar with the differences between all the licenses but the mit license is a little bit more permissive and the gpl is sort of the standard the standard open source license i saw i just copied this this string i had that prepared in a little notebook and this string that you have here this very long thing wrap is the standard way the standard machine readable description of that license that's what it is so you need to make sure that it actually really matches this standard machine readable format which you can just find if you if you google then it requires python version larger than 3.5 that's to me fine and but of course you can adjust it right if you have if you have other requirements and then there is a dynamic attribute and it says version and description are dynamic what does that mean well it means that the version will be read from the from the init of the module the initial pi of the module and description will similarly be read from the init dot pi of the module so let's actually open init pop by and then you see this is the version so flit will be smart enough to basically read this and put it in use that as the version what is the description well the description is the dog string of the init so if i hear say i'll just copy it up execute right so during the last episode we looked at dog strings and we saw that the dog string is something between these triple quotes that is generally under a function definition but you can also have dog strings for dog strings for modules and classes and other things and this right here is a dog stream for the module and flit will actually read that automatically and use that as the description um all right so and then i prefer should say executes not execute well um alright so and let's also bump actually the version because now we're going to do something new okay so that's what will be used that's how flit will know the version in the description pretty easy right and pretty uh well uh described also now then um we can say say that our module had dependencies when we could also create a list of dependencies here we could say for example all right numpy actually markdown code exec doesn't have any dependencies so we can leave this empty but you could specify that here and that would mean that if people pip install your package all those dependencies would be installed along with the package what other things can you specify well one thing that is important you can specify also links to documentation etc but one thing that is important for our package and for many other packages is an entry point so i will say project.script and then i will say markdown code exec is and then i'll say markdown i will type this and then explain what it means up so we have a command line application um and what and we want our flit when we build the package and when we pip install it we want actually our application to be available as a command line app so we don't only want to be able to import markdown code exec as a python library but we also want to be able to execute it as a command line app and we want the name of the command line app to be markdown hyphen code hyphen exec and what we want to happen is that when we call markdown code exec that from the markdown code exact module and the submodule underscore app the app function is called that is what this means right here so if i open that app the app module here up make it full screen you see that we have an app function and this app function is what you would call the entry point of our program so basically calling the app function is the same as executing our command line app now and what will happen now if we define that here flit will automatically build a script then that script automatically executes this function that serves as our command as our entry point that's basically the logic behind it so that is very important there are a few other things that are somewhat important such as for example we want to have some project urls i will copy copy it from my notebook so what do these do well basically they describe where where the users can find more information about uh about the project right so the documentation lives on github and the source code actually also lives on github right so we don't have a separate documentation site but say that you would have a a big project that has its own documentation site on read the docs.io for example then you could put that here okay so now we have um now we have a pro by project.file that in principle has all the information that we need to build our package there are a few things are going to go wrong that we few things we still need to do before we can actually execute it so let's actually switch to into an editor here or a terminal here we have it and if i would type now flitz build that's basically the command that builds it up it would say um all right config error description file readme.md is not exist does not exist so um i'm missing a few things let's put this inside oh yeah here it is read it's capitalized so it should be lowercase readme.nd like this up if i if i try to execute it now oh wait clear try to be up flit build up it is doing some things now let me make it full screen but it is still generating an error so let's try to understand where that error comes from so flit by the way is a python package that you can pip install so at some point in the past i said pip install flit and then i had flipped and that's the tool that i'm using now um what it is saying here is let's run it again flip build multiple files or folders could be markdown code exactly and then we are running into the issue actually that if we go switch back to the editor i should put them i'll try to keep it as structured as possible right you see that i told i told the pipe project that we want to have markdown code exec but that could actually refer either to this python file markdown code exactly by order to this folder markdown code exec with the init inside it and that conflict is not acceptable to flit so we need to be a little bit more explicit how am i going to solve it well what i actually wanted to point down point towards is this folder so i will just rename this file to actually use hyphens instead of underscores up alright and if i execute up try to make it structured flit build up all right now it is actually doing something but it is still giving an error message it's saying that there are interact or deleted files in the source directory and now we're running into another another sort of property of flit namely that flit if you have a git repository it absolutely wants all the stuff in the git repository to be clean otherwise it will not work i feel that's a little bit finicky to be honest but okay so if we say git status now then we see that our git repository is actually not clean because we have deleted or actually renamed this markdown code exactly pi to the version with underscores we edited some things in the init.pi and we added the pi project.domo file so all of those things are not clean so we need to actually sort of make commit those changes so as i as as i usually do you remember from previous episode episodes where we started working with git two episodes ago i think um i tend to use git gui for this i will open it so i'm not going to go into a big explanation of how git works oof and we have the ugly font again i will up simply all the changes that there are i will stage them so here in the red field that's where all the things live that git doesn't know about in the green field those are all the changes that we want to give a name by committing them you can go back to the git episode and learn about that and i will say prepare for packaging commit and if i now go back to the terminal so the the the slash simply means carriage return so we start at the beginning of the line so it's a bit clearer for you and if i type flip build up now you see that we actually get a whole bunch of green messages and it says a bunch of things right but the most important things that it says is that we built sdist markdown code exec.turtle gz and built a wheel markdown code exec dot wheel now the dot wheel package that's the standard python package that you actually use when you pip install stuff and we now have it it's automatically built for us let's take a look if i go switch back to rapunzel now under this well let's say not use git ignore under this in that folder there are now two packages here one called dot wheel and one called dot tar gz and that is basically the end product of our packaging process that is our python bracket python package of course users can at this point not yet pip install it right because it only lives on my computer so no one will be able to say pip install markdown code exec but we can now very simply actually upload it to pip to pipe by i should say so let's take a look at how that works so the next step will be to say here i will say flit upload that's everything that will actually work for me but let's take but of course there is no black magic involved you need to actually undertake a few steps before you can upload something to pipe i the first step is to actually upload something to actually create an account on bye-bye so i have done so and it's very easy it's free there's and you just create an account on pi by and then you have a username and you have a password that's how easy it is then um what you need to do let me actually switch back to the browser i feel like i'm switching a lot back and forth between different things i hope that's not too confusing for you but it's just because we're managing a lot of different things right the terminal the editor the documentation and i'm going back and forth between all of those so let's go back up to the browser and here on the documentation of flit you see they have a section called uploading oh flitz yeah controlling packager plots i click on that and then they say well you know you should have a file called pi dot pi rc dot pi pi rc and in dot pi by rc you basically indicate your credentials so you say the only thing that you really need is this the other stuff indicates okay if you want to make use of the test server you can do that but if you just want to upload to pi pi directly then you have this you have a repository that is a fixed link to pipeline you have a username that is your username on pipeline and you have a password that they don't show here but you should have it that contains your password i'm not going to show you my rc because it contains my password but i have a file like this on my computer in my home folder and when i say flit upload it will work and it will be able to upload to pi pi because i have this i have an account on pi pi and i have this dot pi pi rc file in my home folder yeah so let's actually see how that works so if i now here up clear top if i say flit upload flit will first build the package as we did before and then it will try to upload it to pipeline and i'm doing this live as you know so let's hope that it actually works and as soon as it has worked we will be able to pip install it we will be basically done let's see up user oh sorry it's not flit to upload it is flit publish up there we go okay and you see there it goes it actually really first built everything and then uploaded it by default to the normal pi pi repository so not a test pipe repository for beginners i would say first play around with the test pipe otherwise you end up uploading a lot of junk to the actual pie pie you can actually no one will stop you from doing that but it's a bit impolite maybe but here i actually uploaded it immediately to the regular pie pie and if i now switch to the browser and i go to up pipe.org up i will log in log in and you will see here are all my projects i have lots of projects on pi by for your installation pleasure so to say um and the markdown code exec is the latest one so if i click on view you see that actually it will have a very nice readme that is copied from the readme on my computer it will have a link to the source and to the documentation etc and basically it will have all the stuff that a nice python project needs can do with a little bit of polish maybe i think the readme can be a little bit nicer but you get the general idea right all of this has been automated bomb just flip publish and and up and and off you go all right now and then to actually show you how easy it is to uh to install it i can say well let's go let me switch actually to my home folder because otherwise otherwise you might get confused between the version of markdown code exec that i have actually coded on my computer and the version that i'm not now going to install so i'm going to install pip install markdown code exec bomb there we go easy peasy now remember if i now say python and i say import markdown code exec that will work did now we're importing the module the version of the module that i just installed through pip and if i want to call it as a command line i can say markdown code exec hyphen hyphen help bomb and we get actually the command line script so we see you see that we're now actually basically running the app function inside the module nicely sweetly encapsulated in the script markdown code exec that i can simply execute as a regular command line application now that's it actually we have uh created a pipe pipe let's actually switch back so we can see it again we've created a pi project optimal file we added some metadata in the pi project automo you see it's pretty easy it's not that complex all the stuff that you need to put in there the main thing to know if you want to have a command line app is that you need to indicate this pro you have to have to have this project.project.script section that has an entry that points to the correct function in your code and then it will automatically be a be sort of exposed as a as a command line app um you can specify dependencies that will be automatically installed there are no dependencies here because markdown code exec doesn't have dependencies but if it did we could it would have specified them here and it would have been automatically installed and that's it so i think to close this off let's um let's actually tag a nice new release how am i going to do that right there we are so git status everything is clean i'm going to say git tag release 0.3.0 refer back to the git tutorial if you if you're unsure what this means and i'm going to say git push origin release 0.3.0 and now in addition to uploading the package from pi pi we've created a release stack on github for the exact same code right so that we keep everything nicely synchronized all right um well that is about it actually in less than 30 minutes we were able to go from just the python module to actually uploading it to pi pi and installing it i hope you see that it's not that complex what i've shown is also the up-to-date way to do it so a lot of information that you will find online involves the setup.buy which by now has been deprecated um for those of you who are uh watching this back or offline thank you for watching this actually also concludes the python life lessons episode we've now done in seven episodes seen a basic application from the very first conception the code sketch in episode zero the first episode to now actually having a pip installable package in the seventh episode episode six um and that's basically a it's a nice moment i think to conclude these uh this series of python life lessons or at least this let's call it the season maybe i will make another season at some point but let's say that this is the end of season 0. for those of you who are actually watching watching live i'm happy to take take some questions or to answer to some comments that you might have thank you very much for watching [Music]