One Binary, Every Package Manager: Shipping a Rust CLI To PyPI, Npm, Homebrew, Winget… - A. Kumar

Good afternoon, everyone. So, yeah. So, I will be uh presenting uh one binary every package manager uh shipping a Rust CLI to uh PIP, NPM, and all. So, let's get it started. So, who am I? Uh I am uh Ajit Kumar, and I have done my PhD uh from Pondicherry University. Basically, I come from India. And I have been in Korea for last 6 year. I did my post doc from Sungkyunkwan University, and then uh from last 3 years, I have been working as a software developer. And my current role is uh in Belmatics, where I work as a senior software architect. So, my take a stakes are like I have been a Python developer, and then uh from last 2 years, I have been doing Rust web technology and all. So, I have interest on running, hiking, and of course, open source. And in uh Seoul, I am uh actively participating in uh Dev Korea, which is a English-based IT meetup. And then uh part of Seoul Rust. So, these two are run by Florin and Jan Wagner. So, these are the sources where I have been uh actively involved. If you want, you can connect uh with me there. So, what is my agenda? So, I have uh as you uh come to know about my brief introduction that I have been a IT developer not too long, just 3 years, but before that, I have been doing software for research and publications. So, then uh when I started my job, I got this .env file. Everyone who have been doing development, they will be knowing it exist uh still. And then uh how Rust development and distribution system work, and then what is problem with uh one binary, and then how different package manager work. So, like in previous talk, we come to know about uh uh supply chain of software. So, these packages many time uh you will have that they have malicious package and all. So, maybe that will solve the problem. And then what are the lesson learned and result. So, .env and .env's story. So, if you have been developer, you know that there is basically a traditionally we use a text file .env for storing our key and value pair. It can be anything for API key to database setup to configuration. But nowadays, with different software, now these things are evolving and we are going different way also. But uh still many of the package they use. So, it is used by Python like uh in Django or fast API, we use this. We also uh Next.js, most of uh front-end developer, Next.js, React, they also use it. And in this we keep all the API keys, environment configuration and all. But when we nowadays when we go for deployment, we have started using secret manager like in AWS and all other uh places also. So, when I started, there was information uh spread here and there. So, first I started collecting all the uh configuration and we have put into this .env space. So, it's tell about what it .env. There is no standard specification. Suppose in Docker, you can have uh spaces uh equal to after equal to. In some, you don't have like in in uh AWS, they don't allow to have a spaces. So, there is no specification. So, it's very difficult when you migrate from one uh package or any uh like if you are deploying your project to AWS and want to convert to secret manager, your .env start uh breaking. So, there was many problem. So, I started uh first learning about it, and then I put a place where anyone can contribute and add their information about what they know about .env file. >> [snorts] >> Then, over time, I started facing more problem. So, I thought to okay, let's build a tool uh that can help me to manage my .env file. So, .env .env is only one part. Then, we have started having .env.example, where you put all the dummy values, so other developer or if you are putting open source software, then other developer can know what are the keys and their value will be needed. So, then it become uh .env.example, and then if you have different uh environment, a staging, a release, and all, then you are started having .env.release.test, and all. So, then now from one .env, they are started uh many .env. And then at the end, when you go for suppose deployment to any cloud or anywhere, then maybe you decided to choose AWS secret manager. Then now you have to convert your .env to secret manager. So, again conversion. So, likewise, this evolved. So, this all was like being new to software development, this all was creating trouble for me. So, I thought okay, let's uh do something about it and I start a tool that can help me to manage this uh eventually. So, over time, that tool become EVNX. I wanted to have EVNX, but that name was taken with some other project. So, I started it as a EVNX, and now this is a tool that help you to manage .env. It help to have a setup with default variables. It help to manage different version. It also help to convert to different like if you want your .env to Docker complete uh to work with Docker, then you can convert. If you want to work with a secret manager, it helped to convert your ENB into those format also. So likewise, there is many feature, but this talk is not about ENBX work. It is about how we can distribute a simple CLI. If you want to distribute to different set of developer or you can say different set of community, then this talk is about how you can distribute your tool. >> [clears throat] >> So before I move why I started EVNX in Rust because I was trying to write a CLI. My options were Python and then if I have to do some low level, then I was thinking about C++. But I have been introduced to Rust recently. So I thought to give a go with the Rust and then it was very nice to work with Rust. I have very good experience using Cargo because it is integrated everything like your test, your publish, everything is all the tool chain. The as a developer you use Mac, you use for testing different tool in different Java or any other programming environment. In Rust, Cargo make all in one. So you don't have to jump from your a system to run one tool to another tool. With Cargo, you can do all the software life cycle for Rust project in one one place. So this was one of attracting point for me to work with Rust and Cargo. So I just It is my personal choice and then Rust language is also powerful for security and for memory usage and all. And I was building a CLI. So I thought okay, Rust will be a good choice. So that's that's how we are started in Rust. So as I mentioned, Rust development and distribution, either CLI or any big project is very simple. So let's have a brief outline how it's work. So sub- uh package manager is called crates. So, c r a t e s crates.io, that is Rust package manager. Like in Python, you have PIP. NPM, you have NPM. So, likewise, Rust have their own package manager. So, with Cargo's CLI, you can just do all the development, and you can also publish to the crate. So, if you are working on a binary project, maybe tool or something, you can publish. If you are working on a library, that also you can publish to crates. So, one software, one environment, and you can develop, publish everything all together. So, that's how uh it's simple. Everything is in Cargo. So, for uh starting the project, cargo new. If you are checking implementation and testing, cargo test. If you want to build and release, cargo build and release. So, >> [clears throat] >> this is very simple. And then, there are many other command that you come across in software development life cycle. You can have everything uh in this. So, that's how we have started this. And then, once you want to publish, just before publish, you can have a dry run, and you can publish. cargo.toml is your um you can say metadata, where you put everything about your software and everything. And this is used in indexing. Then, uh when you have to publish, you have done with your uh software, and you can just cargo login. You have to register on the site, and then you can do cargo login and cargo publish. And your softwares are published on the crate. So, it's very simple. And one thing just to remember that whatever you publish on crates, there is no recall. You cannot recall your software. So, whatever you publish, it is permanent with the version and everything. You can ank They use the term ank that can be just removed from the When people search, it will not show, but uh still it will be there. So, whatever you have to to you can just publish carefully. So, that's how uh I have built my uh CLI. So, any user who want to use my tool, they can just have cargo install and use the name and they can use. Or if I share that with GitHub or any other places, then they can use curl and they can use. So, now I would have a stopped here, but this having problem because I wrote a tool, I wrote a CLI that is very general in nature. Every developer can use and when I want to distribute it, the first solution ask user to install Rust and cargo ecosystem. Maybe my end user is Python developer or React developer or Next developer. Asking them to install Rust tool chain and everything is not a right way of doing it. Second part, maintaining second uh solution is also difficult and then user may not familiar with curl installation and everything. That also uh difficult and it can also uh raise security risk because install. sh what I am keeping, maybe end user don't know and they reluctant to use it. So, either of these solutions were not suitable for my use case. So, now I was looking for some other way like being a Python developer how I can use this tool with Python ecosystem also. So, from where I started uh trying this. So, distribution problem, that's what I was mentioning. So, Rust give us a single aesthetic dependency-free binary. So, when you write code in Rust, it you can get a single aesthetic uh binary. So, and using the multiple platform setup, you can compile your source code in multiple platforms. So, write once and according to the platform, you can have multiple version of same binary. So, that's that's like good and then every packaging system ecosystem have their own packaging format like in Python, you can use pip. Uh if you are from React, you can use npm. If you are using Mac Macintosh, you can use brew even on Linux also. And window also nowadays have repository base installation using winget. So, all these are different ecosystem to download tools and software. So, now creating a simple a CLI general-purpose, I want to explore all this. So, user, whatever platform, whatever ecosystem they are using, they can install our software. So, they have their own a schema and everything. So, I this talk is about how we can do it and what are the benefit of doing it. So, Python data, you have repository PyPI. For front-end node teams, they use npm. For Mac and Linux, they use Homebrew mostly. For Windows developer, you can directly download binary and install, but nowadays they have a repository base. Before 10 or Windows 10, they don't have. Always we download and then we install. So, this all have the repository. So, we don't want to have uh Rust in everywhere. So, we want to have follow this and then user can uh download our software. So, that is what title says, build once and ship everywhere. So, if you are building a binary or a software tool, you can build only one one source code and you can have multiple binary and you can ship everywhere. So, [snorts] then the the system will become like this. You are writing your code and then cargo build will create all the single native binary and then with uh GitHub or any automation CI/CD CI/CD, you can use to publish to all PyPI, npm, Homebrew, Scoop, Docker, or whatever uh releasing thing you want, you can uh follow that. So, in Rust, we have cross-compilation. So, what we can do when we are doing the build, we can pass targets, which OS or which system, which CPU we are targeting. According to that, our source code, they will create a single binary. So, for all the available target, we can have binary files. So, now it solves a problem that multiple platform, we don't have to make changes with one source code. We can have multiple binary, and all are individual binary, single by file. So, it will run in all the system. So, it will give something output like this. Uh unknown Linux, Arc, and all this like there are many other OS and platform, so you can get all the binary. So, this solve uh problem of having multiple target. Then, there are some target, especially targeting for low-end devices, where you have lots of issue with the security and installation of the package. That was not our target, because we are targeting developer. So, our OS are mostly high-end computational OS and devices. So, uh later I have removed that from build, and we target only the PC and all other target. So, now it's very simple CI. You build your your CI can build the target, define whatever you have targeted, and then attestation, you can verify. Currently, all uh it's actually a little bit threatening also that what I have observed, none of this uh package, PIP or NPM, they don't ask any signature or any certificate, except Bing get ask digital certificate or other uh scanning method. Otherwise, all this repository, they just allow you to update. So, that is supply chain problem. I think they all enforce certification and verification process, but that's very simple for other package manager. So, then you have your artifact, and then you just put all the artifact into multiple channel that you have decided, and then according to that you create your change change log. >> [clears throat] >> So, the same thing uh in a steps. So, now how we achieve this? So, we got binary for all the platform. Now, what we have to do according to the ecosystem or channel, we just have to apply wrapper. So, we wrap for PYPI the binaries, we wrap for NPM, and then all these package mana package repository, they accept our submission, and then user can download using their system. So, uh different package having wrapper like PIP use S distribution, NPM you have post install, Homebrew have some concept called formula where you can put into GitHub and directly you can download. Winget have uh manifest. They have three files that you have to create. Cargo already we have discussed, and a scope also work like that. So, now go through the PYP. So, we can package all the pre-compiled Rust binary according to the different uh target, and that will create different Python wheels, and then for this PYP I have used Maturion. It's very uh good where you can convert Rust to PIP packaging. It's very good package and software that you can use if you are working cross Python and Rust. That's very suitable. And then uh it's don't have any uh dependency on other problem because the binary is in one uh one file, and you can just have uh use Python modern CLI workflow also. So, this is Rust part, and then for having a Python uh PYP you have to create uh PY project TOML that is used by Maturion, and then they create the wheel for multiple target. Like this build, they will create target, various target, and various wheels they will create and then uh when the wheels are created, you can just push to the repository and your package is live on uh PyPI. So, in this what I was saying, they were not asking anything like even the MD5 or any any hashing or any just login with your account and you are good to upload. So, if my account get compromised, then anyone can upload any kind of binary. So, these are the risk uh in this platform, but that's how normally PyPI work. So, then once you have uh done, then you can your Python user can use pipx because pipx is now if you want to install any library globally in your system, then you have to use pipx. You cannot install with pip uh due to uh new changes in the system. So, now my tool was written using Rust. Now, a Python developer, if they also want to use .env file, they can also download, install, and then they can use the tool. So, now I can target Rust developer and also I can target Python developer, whoever uh I want to use the uh .env file. Similarly, moving forward, uh .env is also used mainly in all the React and JavaScript ecosystem. They have so many library that load .env file. So, basically, they have this this uh uh .env.local, .env.test, all this is there. So, that is also a big very big user group. So, now we want to distribute this via npm also because if you are a front-end developer, most of time you are installing JavaScript uh library and our tool is not a JavaScript, it is a Rust-based binary. So, then we uh follow this approach where we have to just wrap our software into their ecosystem and then it will work. So we have to package various according to various platform like for Linux, for Windows and all and then based upon this packaging whenever we publish this to NPM and then user want to install, they match with their platform underlying platform. It's Now it's become the NPM duty to match and find is a matching binary is there or not. Depending upon the matching, the it will be installed. So we don't have to do now anything. It's It's the repository. It's the distribution system that have to think whatever user is using, which platform they are using and then they will download and install. So my binary we just create once and the system will install it. So they The mostly it is sidecar kind of thing. This JS launcher they do all this permutation combination and which platform they have to download. Accordingly they just wrap and download resolve the binary and install. So now after Python, now if a JavaScript developer or TypeScript developer they want to install, they can just install using NPM or NPX if they want to install software. So now my software which was also useful for front end, they can also use my binary and they can also install from their their environment that what they know. They know NPM, they know NPX. They don't have to think about Rust, they don't have to think about binary and our tool is distributed to that community also. So now just I was keep going and then I tried to see how Homebrew work. So Homebrew have uh two-tier system. If your tools is very popular, then you can it will be part of Homebrew core. But, if you are new, then they don't accept direct submission. Then, they say you create a tap or you create a formula. That that is very simple. That is just a GitHub where you can have your uh formula and then user can So, this is just two-tier architecture, much safer than other two, and very simple also because it is just uh a small file that you have to create and a GitHub repository, and they will take care from there. So, this is how you create a formula in your GitHub repository. It is a Ruby file that you have to create. And then, you just give your source uh repository file name, and then the binary file and everything. And then, when user install, this will be run, and then the your binary will be uh resolve when installed. So, this is very simple. Uh we call it formula, Homebrew formula. So, it is >> [clears throat] >> simple in your CI also. It is very simple that you can just trigger and bump your version if you are releasing the new version. And the formula will just keep updating because when user install, they install from the formula. So, this is how uh Homebrew it is available on Mac as well as Linux also. So, they can just use either two a step brew and add your tap. You are with ENBX. This is the code. And then, they can just normally install how they install other software. Brew install EVNX. Or if they want, they can just merge both command and just install directly. So, now with this, ours binary is available for Linux as well as Mac Mac user also. And uh remember, I have just written one source code in Rust. And using this uh distribution system we can call we are distributing now to uh multiple uh platform. So, as I explained in the Homebrew, they have two tiers. One is core where the well-known software you will find, then you don't have to add the tab. And if it is new like my project, then you have to just use tab. So, the criteria like a star fork and all, based upon that later you can submit to the core. So, the the last one is uh winget. I think uh this one was, as I said, they have a very uh automated process. They use their because they have antivirus also, and they have put everything together as a scanning automated scanning. So, when you submit and uh my tool is reading file and all those, so they create a signature that match with some Trojan or malware. So, every time I submit a new version, they raise a uh Trojan warning, and then I have to go through their manual process and pass that process, then again resubmit our request, then only they accept. So, this manifest file we have to create, and then when we submit they create this Trojan uh warning. It is false alarm, but it is matching the signature, so they were raising uh the turnaround is I need to have a digital certificate, and then once I start signing my binary with those, then this will go away. But now it is a new project, and I don't have a digital certificate, then I was submitting all the version, and I have to go through the manual verification where I have to submit uh for this one, MMPHC, where I submit the binary. I I say the category and everything, and then after some time, hour sometime, 5 hour 6 hour, they clear it, and then again I have to resubmit submission to the their repository that PR then get clear and then my binary get released to their repository. So, this is little bit harder, but over time when the project grow, it will also solve. But, this is a good way that Window have moved from direct downloading EXE and installing 2 OS to a repository model and now they have a automated pipeline to verify also. So, this brings some security to the platform. So, this is the long-term solution that I see I need a certificate to sign. So, I I think I am out of time, so I will just a little bit rush. So, now similarly now it is a binary tool, so we can also have GitHub action because whenever you do a pre-commit or something, you can have as a part of your pre-commit code where this tool can run and do all the we have a scanning security scanning and all in the future in the tool, so you can use as a GitHub action also. So, I have created a separate GitHub action and published to the marketplace also, so it is available via GitHub marketplace and it can be consumed in any of the pre-commit like if you are using pre-commit hook, then you can also use this one. So, also nowadays everyone using AI for coding, so I have also tried to have a skill for ENV and then based upon that a skill your AI tool can use this tool because like if you have submitted your password or API keys or anything, you're having this a skill they can a scan your repository before making a push to the repository. So, those agenda skill also I'm trying to build and it is nothing but just a GitHub repository and then you have a skills. assets where if your assets are being used then it will show in rank, otherwise you can just download and install the agenda skill directly to your uh system. So, like NPX, it is by NPX a skill add. You can add the skill. Now your uh agent have these a skill and they can try to use this tool internally also. So, that is uh a skill. >> [clears throat] >> So, that was the how we have uh approached different channel. And then I also try with a scoop and also have a playground where you can test it. And normally, if you want, you can also distribute as a docker. And later I will look into the uh data. So, the aim of this talk was one binary, if it is general purpose or if you are targeting a large community developer community that come from different background, then how you can distribute your software via exploring this channel. And remember you are writing code only once and using the CI, you are publishing to all these platform. And this is only one time job. Once you have set up your CI/CD, you don't have to think about this channel and how they distribute because you just push and CI/CD, all these channels are automated. So, you can have uh your release automatically. Uh having a good uh software supply chain that will also help when you have set up this one. So, >> [clears throat] >> this all are the findings. So, once you binary artifacts finish, there there is a time taking because we are uh compiling for multiple version and then we have to retry, cache and everything. So, uh our signing key and token are also. So, all this we have to manage, but all this, as I said, it is one time. So, once you have set up and fix all thing, it will be like keep going. So, release time, it can be very fast. You are targeting lots of registry, and then your installation will go up. So, funnier story is the the Rust installation is about 300, where Python installation is in thousands, like 9,000 or something. In NPM, also, it is in thousand. So, if I have I would have a stopped at Rust, the distribution would have been very low. But, when I touch Python and NPM, the download of the binary tool have gone uh very high. So, that's all from mind. Thank you. Uh we can have some question if Yeah, lunch time I think is 12:10. We are 4-minute delay. Yeah. If you have any question [applause] Yeah. So, if you want to contribute, all project are open source. Yeah. Both project env-x.dev and .env.space, one is .NET specification, and one is about the project. So, if you want to contribute, you can also contribute either way. Yeah. Yeah, any question? I think all are hungry. Okay, then. Yeah. Okay, thank you. Have a nice lunch. Yeah.