Documenting Python code (docstrings) [PythonLiveLessons #5]

[Music] hi everyone welcome to python live lessons episode 5 the fifth episode sixth zero based indexing sixth episode of python live lessons um today we're going to pick up as as every episode where we left off during the last episode with our project our python project markdown code exec and today we're actually going to focus on documenting the code so we're going to add user documentation to the code so that other people can actually use a markdown code exec hi morpheus good to see you again um as always i'm doing my best to add a little bit of interactivity in this this live stream so if you have questions don't hesitate to post them on the chat and i will try to engage with that as as much as possible alright good to see actually some returning some returning small but but loyal crowd so to say so what do we um what do we have here when we talk about uh yeah sorry got distracted there so what what are we talking about when we're talking about documenting python code so um one way a very cheap way to document your python code and we've already done that to some extent is to add a readme file to the to your source code right so a little text document in our case in markdown format to to explain how you should use uh the the project in our case markdown code exec um and that's fine and it's also very convenient because tools like github and pi by often automatically take this readme and make it very visible to to the users or people who visit the project site so that they can read that so that's a low-hanging fruit and we're actually going to use that also today then we have so-called dog strings and dog strings are a little bit of an intermediate you could say between regular code commands in the source code and documentation so i would say comments in the source code are mostly for either yourself or other developers to understand how the source code works they are not really intended mostly for users outside users to understand how to use your code a docstring is strictly speaking a comment in the source code but you attach it to a function in a way that i will show you in a bit and that helps you and other developers to understand how the function should be used but you can also automatically extract the information from a docstring and include it in the documentation for the project such as a readme file so that's what we're going to focus on today sort of automatically generating documentation from a dock stream um then there are some let's call them frameworks or utilities that automatically take a whole software project and generate sort of documentation from it a pi doc is included in the python standard library it's a utility that generates documentation based on source code it's not as far as i know used that much it's kind of outdated and not so popular but it's included in the python standard library another utility that does the same is called sphinx and sphynx is used a lot and i would say maybe other people disagree but i would say that sphinx is actually the most used documentation generator for python projects if you go to if you sometimes use the site readthedocs.io on which a lot of documentation for python projects is hosted that is usually and perhaps even always generated with swings so what are strings and pi docs they're basically just utilities that automatically generate documentation what we are going to do though is take do it one step simpler we're not going to generate the whole documentation site that's overkill for our small project instead what we're going to do is generate a readme that contains information that is automatically extracted from the docstrings from our functions and the cool thing is that we're actually going to use our own project markdown code exec to do so right so that's a little bit of inception right here for you because we're going to actually use our own project to document our own project so before i actually go switch to the editor what does numpy dock look like here you have an example it's a bit small for you to see maybe but so what how it works issue here you have a function definition summarize and it's from a project time series test that i developed and then you see that just below the def statement so just below the function definition there is a bit of a comment a string indicated by these triple quotes and inside the string there's a description that says well summarize generates a string with human readable summary blah blah but it also contains some semi-structured documentation so you see there's a parameters which sort of underlines with these hyphens and then it says that one parameter is results colon it's a dict and then there's a description of it and then it says returns again underlined with these hyphens that should be a string and and you see that this is kind of semi-structured documentation and particular utilities such as numpy dock actually can read this these kinds of strings and automatically interpret them and turn them into pretty looking documentation that's basically the idea so let's switch to the to the code editor and again if you have questions or suggestions about other ways to document code don't be shy to to suggest them up to presentation mode and no there we go so what i have here um i'm not going to repeat again what our project does in a lot of detail you'll see it in a bit in action um but basically this is the project that we developed in previous episodes and we have a folder markdown code exact in it we have a submodule underscore markdown code exec and in that sub module we have two functions parse text and parse file and those are the two function that functions that we're actually going to document so the first step that i'm going to do is add documentation to these 2d string to these functions in numpy doc format um and i'm being a bit late here actually by doing this so normally i would this is now with the project is almost finished and now i'm adding these dog strings normally i would add dog strings earlier on during the coding process just so that i don't lose track of what i'm doing essentially even though i'm maybe not using them immediately to generate documentation so i'm pretty big on using doc strings myself so how do you use docstrings well you already saw it in the example you basically start by having a triple line string here and everything in between will be a dog string so it's that simple so anything i type here would be a dog string but i'm going to use a specific convention the numpy.convention to actually make this dog string in such a way that it is readable by machine readable by by utilities other utilities so what does parstax do takes a str off containing markdown text now text so that's what it does right so here's also the explanation for those of you who don't know what the what the code does it takes a string call line markdown text finds code blocks in the markdown up in the markdown sorry oh and markdown executes these code blocks and captures the output and then embeds the captured output in the markdown um maybe we can give an example of what that would well actually let's say i go i switch to demo.md just to make it a bit more concrete for you what does this do if we have a code block here in the markdown the code block is indicated by these triple triple blocks triple backticks there are two print statements that generate output of course what you print then that output would be captured and it would actually be inserted as text below right and the idea would be that that way you can illustrate basically what code does when it is executed that's the idea so here the top level the the dog string is basically a description and then we're going to be say okay there are parameters there's actually only one parameters parameter but you would call it parameters um and it's called md and it is a string that's what it is the input markdown string and it returns something up and i have to underline that as well why well that's part part of the markdown convention it's also a string and then i would say the let's call it the compiled markdown string that's it this is the markdown numpy doc docstring right and it's quite nice and it is short and it describes what our function does now below we have another function called parse file it is a little bit more of a higher level function in the sense that it actually takes an input file and an output file but does more or less the same thing as parse text so it takes a markdown input file processes you can you can be a little bit better than i am right now in my cr crafting a nice explanation of what the function does right but i don't want to spend too much time on that parses executes code blocks while capturing outputs puts and writes x and writes the result to a new output file and then we have a source which is a and what is it exactly exactly it is the full path to the input file and we have a destination source our string and it is the full path to the output file let's not do that all right so now we have documented our r uh our um we have documented our functions now what are we going to do just documenting the functions obviously doesn't automatically generate generate doesn't automatically generate documentation but we can use this as a start to generate documentation maybe i should actually let's go you know up just to kind of give you an idea of how this works so now i have my my terminal here at the bottom if i would say import markdown code exec and then i would say markdown code exec dot parse file sorry parse text this thing here above then python will automatically use the doc give a underscore doc attribute to it and that will contain the dog string and you see how this basically allows other utilities such as numpy dock to actually take the dog string and turn it into something nice uh nice right so basically let me print it out and it would look nicer so up so basically that's the idea the doc strings are automatically exposed through this underscore underscore doc underscore underscore attribute this by the way works for functions but it will also work for classes and other things okay now um what is the next step the next step is that i'm i installed a python library called numpy.mp.2md you can install that just with pip install pip install mp.2md and what mp.2md does is uh it parses these these dock strings and generates nice documentation so how what what is the next step so here we haven't read me the readme file that we created before so but this readme file has been hard coded right everything that's inside it i typed manually myself that's what i don't want to do anymore i want to create a template and i will call it readme template.md and in this readme template i want to include information that is extracted automatically from the source code and then compile that into the actual readme i hope that that is a little bit clear right so i want to sort of instead of manually creating a readme i want to automatically generate a readme based on the based on the documentation in the from the docs strings so and i'm going to do this in the following way i'm going to say okay here i'm going to have a function reference so underneath there's going to be a function reference or a description of the what what it looks like you know what let me actually show you what it's going to look like approximately that will give you some better idea for what i'm doing yep so let me switch to a time series test this is a um if you this is another project that i've been working on and if i then scroll all the way to the bottom you see this is a nice project with lots of documentation you see that here we have a function reference and it has some kind of nice description it's not that pretty i think but it has some kind of nice description of how how you should use the functions from this module and all of this has been automatically extracted from these dog strings from numpy dog formatted dog strings so we want to do the same thing basically for our markdown code exact project so that's what we're working towards so how we're going to do that well we're actually going to use markdown code exec so i'm going to say well we have three backticks and this contains python code and this python code should be executed silently what does this mean again well if you watch the previous episodes you may you you may remember that basically everything that is executed in a silent code block results in the output being captured and then this the code block is replaced by that output so what i'm going to do here is type some code that will basically print out the dog strings and then this this part right here will be replaced by those dog strings in a nicely formatted way that's the idea kind of cool right and pretty fancy also i think then i have to kind of have a cheat sheet here because i need to actually know a little bit what i'm doing um so the first function that we need to know need to have is the functions that we actually want to document from markdown code exec import parse up text and parse file those are the two functions that we want to have and we want to um use numpy dock to md to actually parse them so from numpy mpdoc yeah mpdoc to md import and then there's a render md from object docs string i've i've seen prettier function names i have to say but basically it does it is quite clear so this is a function that renders markdown from the dock string of an object and then things are suddenly very simple because we can say i want to print out render md from object docstring and i parse pause parse text and i call it markdown code exact.parse text so this means i want to parse the docstring from this function parse text and then you have to specify for a reason that's not entirely clear to me what the function should be called maybe in case you want to give it an alias i don't so i just call it markdown code exec parse text this will print out that nicely formatted dock string then i will say up to new lines and i will do the same thing for parse file parse file oh now that's pretty cool right um so let's see how we can actually execute this i hope the logic is clear if not interrupt me with questions at any time right so the logic here is that everything that is printed out in this code block should actually be captured and the code block itself should be replaced by that so let's see how we can do that i will switch to an editor here's or sorry a terminal and i will call python markdown code exactly.pi and i will say the input is the readme template and the output is the actual readme dot md let's see what happens i hope this works you never know this is life after well doesn't crash it's already a start so let's switch back and let's put them side by side up okay so here on the left we have the template of the readme and here on the right we have indeed the function reference it doesn't look very nice now right because it is kind of like marked down mixed with a little bit of html etc etc but that's okay it doesn't need to look very nice like this because it will look very nice once we actually upload this to github and then you will see how it works and how nice it looks and you know what we're just going to do that let's actually post upload this to github so you can see i start get gui right so two lectures ago i think we saw how git works and i'm just doing the same if you're not familiar with how git works watch back that lecture so what did i change here we have to unstage changes i added some dog strings that's what you see here up add dog strings that's one commit i created a readme template and a readme and readme and machine generated readme build md i commit this as well right so what i've done now is i've basically frozen these changes the unstaged changes were the things that were changed but git didn't know about it then i made a commit that sort of locked these changes and i gave them a name with this commit message and then if i say push it will actually upload all of these changes to github where our project lives and that we talked about two episodes ago so that's done close this switch back to the github and now if we go not to time series test but to mark code exec we will see that here we have our nice function reference right for parse text and for parse file pretty simple pretty straightforward pretty nice looking so this is part of what i wanted to do i guess the main main thing main nice thing is already that now we explained to our users how they can actually use our library programmatically by calling these two functions however um you can we the last app previous episode actually we used to turn our python module into a command line application so that you don't actually need to say from you know markdown code exec import blah blah but you can just execute it on the command line which by the way is what i just did right when i converted to readme and this is not documented here yet and i also want to document that and i also because i hate doing things manually and copy pasting i also want to do that programmatically so how can i do that so let's go back so let's say uh command line command line usage and now we want to do more or less the same we want to do something here dot python silent that will actually result in the explanation of the command line usage being printed out and this is a bit finicky i have to say but i will show you how it works anyway so the general idea is if i call python up if i call python markdown python markdown code exec with a help right like this it will actually print out a simple help for how you can use this on the command line and i want to actually embed this this information in our in the in the in the markdown that's what i want to do but as i said that's a little bit finicky so let me show you how that works because what it involves is calling my python markdown code exec as a command as a command line app from python and that's a bit finicky but we can do it so i'll walk you through it okay let's close this and i have my cheat sheet because i wouldn't be able to do this from the top of my head so the first thing that we need is subprocess subprocess is a python module that executes stuff that basically allows you to execute things then we can say sub process dot run and subprocess dot run takes a list of things that should be executed so that not on the terminal you would type python space markdown code etc here you wouldn't use spaces but you would pass a list of the separate words so it would go like this python markdown code exec dot pi up help all right that's the first thing then we need to do a few other things we need to say we want to capture the output otherwise it will go lost and we want it this is text so we want this is a bit of a technical detail but we don't want this to be bytes but we want this to be actual unicode text objects that we're capturing and that's it essentially although if i were to execute that i know because i tried it out this would crash because python doesn't know how to fight python which is a bit strange so just calling python like this wouldn't work but we can say instead of just python we can say systoled executable then you ask what is systoled executable well i will show you import sys system executable this is the full path to the python interpreter so if you are writing a python script and you're wondering where is the python interpreter that was actually used to buy execute this script this is sys.executable so what i'm basically saying here i know at least one python interpreter where it lives namely systoled executable and i'm going to use that to execute markdown code exec pass the argument help capture all the output i want this should be text which is a bit superfluous to me and then that will return a process object which i assign to p and i know p dot std out will be contain all the output that was generated during the execution of that code let's see whether this actually works if i run like this up check uh invalid syntax why is this oh yeah comma run trip you see yeah this actually works right so if in rapunzel i click quick run it will detect actually that you want to run everything between the back ticks and it will be capturing it all right so this looks pretty good to me so what i can do now is again switch back to the terminal and do more or less what i did before up take the readme template as input the readme is output up and if i now switch back to rapunzel let's take a look at what the output or what the actual readme log looks like all right and now you see in the readme we also have for the command line usage a bit of an explanation and our function reference of course still oof well that's about it right so there's lots of room for improvement here but at this point it's mostly aesthetics i think right making it look a little bit nicer adding a bit of a nicer descriptions to the dog strings adding a little bit more information to the readme file but in principle now we have a very sweet way to automatically generate an up-to-date readme file avoiding us from having to manually change the readme all the time right that is kind of the advantage that we have here um yeah so at this point what i want to do i think this is a nice moment also to say okay up in the unit i think we are pretty fancy now i think we're going to bump to version 0.2.0 um i'm going to commit the code tag it as version 0.2.0 and um then we have a very nice new release of markdown code exec so uh yeah right let's start get gooey again we have a few changes right but now they're mostly limited to the to the readme update read me up oh wait actually this should also be and i i tend to say usually if i change the version i just change the version and the thing and i say bump to 0.2.0 uh i push this up to the master so that we have an up-to-date thing on github i'm doing this a bit quick because we've covered all these steps in previous episodes and now i'm also going to tag a release how does that work again git so i just use the git from the command line now i say i want to tag release 0 2.0 all right this this slash the only reason i'm typing it is because otherwise the line is a bit long right it's just the carriage return and now we say git push origin release 0.2.0 so i push the tag to github as well resulting this from appearing on github there we go switch back reload the page and we have a fresh uh up to updated markdown code exact right here right you see here this is the command line usage isn't that nice the way it is printed out here so you can use a little bit of aesthetic tweaking but other than that all the crucial information is here and i think the nice thing also is that in this kind of inception kind of way we've used markdown code exact to generate its own documentation it's bootstrapping of documentation of sorts and i will actually use markdown code exec myself quite heavily in other projects that's the main reason that i started this to begin with because i thought it would be nice to share with you all during python life lessons but also because i actually want to use it um and that's it what i wanted to cover today next week i hope you will be back and then we're going to i think right now we have a nice package it is time to upload it to pi pi to actually make a nice release that other people can pip install so i hope you will tune in then to see how you can actually publish things on pipeline um if you're watching this back the recording then this is where python live lessons ends thank you for tuning in if you're watching live then if there are any few questions or comments then i'm still more than happy to take them [Music]