FAIme web service hands-on workshop

Thank you. Good morning everyone. Thank you for joining us. Today we have Thomas Langanger and I'll let you introduce yourself. >> Let's all get started. >> Thank you Thomas for being here. >> Yeah. >> So good morning everybody. This is a workshop, a hands-on workshop. I will explain some things and then you have to do it on your own and then finger crossed because you want to download your own customized ISO and we have this internet connection over there. So hopefully it's not taking too long. So who knows five one two three four yeah maybe half of it. So um this is an an old project of mine means fully automatic installation. Uh it started as an network installation for bare metal. Later we could also install virtual machines. Uh then I've added support for creating an installation media. Uh but you can also create a change route or build environment. uh the cloud team of Debian is using FAI for creating the official cloud images. You can also create live images with it. And what was very important when I invented FAI that a system administrator can easily edit some config files. So should be plain text or a certain format and uh yeah make adjustment because you do not want to have the standard configuration the standard installation but do this as you like but then I was thinking about so in the past the target audience was more the advanced CIS admin but I thought wouldn't it be great that also end users could use fi the fi technique for installing calling and maybe they do not care about all the technical details. So that was the idea. Let's create a web service or fi as a service. Uh this is the web page how it looks like. So what you can do there you can create your own customize your personal installation or live or cloud image. Um, you can uh choose the language. For example, it's not that detailed as in the Debian installer where you can say I want to have the French language, but the time zone should be maybe South America or the North Pole. And uh the keyboard layout should be a German keyboard layout because I thought most users if they say I want to have a French ital installation they want to have a French locale the French time zone and the French keyboard. So on the web interface you have less possibilities to choice but then it's easier to use. You can also choose which distribution you want to install. Do I want to install stable or testing? I provided the old stable in the past, but I think like two months ago, I disabled this possibility. You can select a desktop. Which desktop do you want to install? And what's very important that you can uh add a list of package names that will be installed later on. And a custom post script uh post incript is also possible. you can upload it. Um, today we start with an installation image and if you do this on your own, please only try the installation images because that will take like one to three minutes to build on the server. And if you say I want to have a cloud or a live image, this will take like five to seven minutes. And the server can only process one job at a time. So you may see uh if you say build my custom image that you are in a waiting queue. So do not yeah start with the installation image and maybe without the desktop. So for your first try just say I want to have a plain text installation. So no desktop environment just to give it a go. And that's all for now. Uh I think did you all got the URL? So it's it's not fi.mme it's fi minus project. H there's the URL on top. So if you go on this web page there is also you can choose it's it looks a little bit different nowadays. I will switch. So it looks this is the web page and then here you say the find me build your own then choose the Debian install image and then you can fill out some things if you're think oh I'm advanced user you can toggle then you would see more options for example you can this is very handy you can directly upload or install the public key for the root account. So you have passwordless access to the machine that will be installed. You can select a partitioning scheme with LVM with a separate home partition. So just give it a try and look. And if you have questions, this is a hands-on workshop. Just raise your hand and then we will do this. What I want to say what's currently not possible with the official Debian installer to install the stable release with the backports kernel. This is because the Debian installer environment is very complicated. So FYI is the one that provides you an installation image for the stable release with the backpods kernel advertisement end. So um I think I do not click here because you can everyone can do this on their own. Any questions? >> Just speak up very loud or wait for the microphone. >> But it's more important that we can hear you. >> Yeah. So what does FIA uh F AIM me use in the background? This is um de uh debootstrap. Is this some other solution? >> Uh this is the fi. So so in in the background yeah we are using deb bootstrap and we have our own pearl wrapper around aptget install. So for the installation ISO there are two steps. First um create a partitionial package mirror of all packages that are needed for your installation. So we just do the usual package dependencies resolving and then say aptget minus s or minus d which means download only. So then I I have a partitionial mirror of all packages and the second step is to build the installation ISO and that means during the installation we need a running Debian system in fi we call this NFS route. This is the first step you do if you create a or if you set up a fi server. You create the NFS route which is used by all machines during the installation. We call this NFS route or change route. And there we have all packages and tools that we need during the installation and this part is put onto the ISO image. Then your partition mirror is put onto the ISO. And then we have the config space, the configuration space which is just a subdirectory maybe with like 20 to 40 files, readable files. And this is also put onto the ISO. You can also if you created your ISO, you will see a link. You can also download the tab of only this config file. And this is all put together onto a bootable ISO. And if you boot a virtual machine or bare metal machine with this ISO, this FI environment, this Debian environment is booted up. And then FI has some mostly shell and pearl scripts which control the the different tasks to do the installation. We do not use parts of the Debian installer. So the Debian install is a very special environment. We do not use the petition thing the preceding we can do preceding but for the partitioning part we have a very nice config file which is very similar to an FS tab. So if you see like two examples uh you know oh I want to have another partition and you can say the size should be from 10 GBTE to maybe 20% of the overall disk size. So things like this are possible. You can create butterfs, LVM, software rate, crypt things. That's also very easy to describe in the file configuration. And then we have some files uh where you just list the packages you want to install and in file there's a class system included. So you can say I create a class department A and department B and maybe this classes these files only describe different list of packages or we have a class XFCE. Sure this will install the XFCE packages you need. So anyone already pushed the submit button on the page and >> position one >> pool position one. So, who is the f who's the first? So, you were the first and you already have the download link? No. Anyone already has a download link? Ah, you have it here. Nice. Yeah. So, the overall size is less than 800 megabyte. And it says it took 40 uh no 54 seconds to create this ISO. So let's see how long it will take to download it. [laughter] Please do not uh stream anything currently. More questions. Yeah. Oh, there. Yeah. Yeah. >> Yeah. >> Well, so and for for so for building the ISO, do you use live build or you custom scripts? >> No, I have a custom script. We do not use live build. >> Yeah. I was just wondering if you if you plan to integrate live build uh with or become the new live build. >> No, no. >> So, uh how do I get my ARM 64-bit server installed with that? So, is this a limitation of FI or of just of the web interface because I couldn't select any architecture there? >> Yeah, this is a limitation of the web interface. Uh what's what's the video? So um I have a script that builds the ISO and I think like half a year ago or less I've added ARM 64 support. So if you want to do more things with FI I always suggest set up your own FI server but ARM 64 is possible. I still have no hard risk. [laughter] I I didn't do this. So I still need some hardware to test it but or if there's something missing I will always ask the CD image team. They know how to call saw EO. So we are using the same tools. The only thing that find us a little bit different is that we do not use the init RAM FS tools but DR cut for creating the init RD for booting up and also for the live system. Hello. Uh I have a little question. Um is um is it easy to add your own uh Debian derivative in FA? Sure. So your own derivative means you have maybe an additional line in sources.list list and then if you have your own repository it's possible there to provide a line for the sources.list list. So for example, if you have some non-free or uh packages which are not in Debian. So if I toggle to advanced, I think here where is it somewhere? It's there. Yeah, here you can add one public repository. So uh what's I sometimes see in the logs that people um add the prox mox package repository because they want to have the installation with the packages and if you say I have a repository which has then the packages for my distribution sure it's possible we can also this is the debin installation you saw there's also a available and it's not yet published but uh there will be also the possibility to create a custom Linux Mint ISO. >> Okay. So, if you can add another repository, is there a way to set up the package pinning to? >> Yeah, package pinning is just copying an etc apt file to the correct location. It will be not possible with the um with the web interface but start with the web interface then you get the file configuration and then set up your own file server. >> Okay, thank you. >> Oh, very good. Is the password generation I noticed it allows an option to do that for you. Is that static at build time of the ISO or dynamic at installation time? >> So if you do not provide a password here um if you push the submit button you will see that your job will have an ID here. This is a random ID and the password will be set to the ID and uh sure you should change this and in the back uh at the bottom I say please uh uh for the keyboard you you should change the user and root password. Uh you can also provide the hash. You do not have to type in the clear uh clear text password but the hash is also possible. And sure if you say, "Oh, I want to uh have uh more secrecy," then set up your own or change the password afterwards. So some success or how fast is the network currently? The download >> around one minute. >> One minute. Okay. So So you already downloaded your ISO. So do you have a virtual machine where you can boot the machine? Yeah, just do it. Everyone who already downloaded uh it's the ISO should start and tell me if there's an error message or yeah microphone um V is fully automated on the install side but how is it on the generation side? Could I fully automate this because it still has a web interface where I have to type some things. Yeah. Can I script that? >> Yeah, you can script it [clears throat] there. Ah, show RP URL. So, if you switch to the advanced mode, you have this button, show RP URL. I think I can do this. And then you see a URL like this. This is not a perfect URL. So, I'm using some CGI. And then you just set some variables. I have a page. Oh, yeah. There's a documentation of this RP. So, this is really a poor man's RP, but it works. So you could use a command line uh for yeah creating ah it says curl and then this command line. >> So how about the reproducibility of these images that we get out? >> So it's all unique so to say. If we call the same thing twice I get different images. >> Yeah. So for me reproducibility is not that important for me. >> Is it hard to do or >> the the the question was is it hard to do? Help me give it a try and we will see if it's hard. >> Well so so one reason I'm asking so I I notic that Squash FS uh so make squash fs is not reproducible. >> Yeah. So, >> so I wonder if for you it's easier because you don't >> so it we are using squash fs and also I'm not sure if the init rd generated by dracut is really reproducible. I'm pretty sure it's a lot of work and there will be some minor things you have to tweak to make it and for me it's not worth so [laughter] >> the it supports UEFI. So >> yeah, >> how did you do that? And and uh what does it look like then in my BIOS list of installed operating systems? Does it represent itself as Debian or FAI me? So um the ISO can be booted in legacy or UF UEFI uh environment. Uh in the end we boot a the grub and then you will see the grub which says uh we are starting the fi installation and um this is a usual Debian installation and then the Debian um kernel package will um add to the UF gr uh UAF entries. So in the end you will see a boot manager minus L or something and you will see an entry for Debian. So any success or still? Yeah. So tell us what worked or >> Well, it definitely boots. Huh? >> Well, I downloaded the image and it definitely boots. So, so the installation boots. >> Yeah. So, if anyone wants to show what's going on in the virtual machine, just come up and connect your laptop. So, we may see something more experience. Who is still downloading? Still downloading. [clears throat] Yeah. >> Yeah. >> So, HDMI. >> No. >> Yeah. >> No. Display port. >> Display port. Okay. Video team, do we have an adapter display port to HDMI? I think they have they have everything. [clears throat] So maybe what is the idea or where do you like to use this customized ISO? Do you have any idea? Oh, this would be good in my environment. It's because I need uh installations. Any ideas that you already have where you like to use it [clears throat] for other installer with a new kernel to support bar metal installation. For example, we need the backport kernel. Then it's easy to build a >> install with. >> So back ports is important for you. >> This would have made my life easier like a couple of months before we had to build something out of a mint dro. I know the mint is still coming. So this is pretty easy. And the API even though it's so like uh basic, it's still useful for me. >> Yeah. Okay. just write me an email. I'm not sure, but I think I will publish the Linux Mint after Dcon, but you can also trigger me and I may give you the link. >> What? >> My use case is over for now. Maybe later. >> Yeah. [clears throat] Yeah. Over there. No, I just want to share that it's working here and I'm booting starting the installation now. >> Yeah. >> Mhm. >> So, and and you do not do not have to touch anything because in the BR menu there's a timeout I think about >> some seconds. So, if you just boot it up >> uh without touching the computer, it will start the installation and remove everything from your disk. Oh, there I see an installation running this also. >> No, no, this something different. [clears throat] >> Yep. >> Uh, so I assume this also works with PXA booting. So I can use this pro to provision hosts. >> Yeah. Yeah. >> Perfect. >> So the the nice thing is that it's always the same control script that run, it's always the same installation. Either you do pixie network installation, you do the installation from the ISO or even if you create a raw file for booting up, the file process is always the same with the same configuration. Sure, for the pixie boot network, you need some additional things in the DHCP, but uh if you see this red text, the red logo, text logo, then everything is the same. >> Yes over there. >> So, so it's a different ISO between Pixie or it's the same ISO that is generated. >> So, for Pixie, you do not need an ISO at all. if you do pixie boot from your thing. So, so uh you would boot a machine from network card then you do not need an ISO at all because then everything comes from the fi server and an NFS route. The configuration will be mounted by NFS. Um there's a possibility. So you could boot from an ISO, but the ISO only only contains the kernel and the inert RD or maybe the FI NFS route, but the configuration then comes from a GitH repository or from an NF server. So we can choose whatever part from the file installation comes from local from the ISO or from the network and you can uh grab everything from the network with P starting Pixie boot kernel init RD NFS routt everything via network without using an ISO at all. >> Very good. Thank you. Now that we're talking about u pixie boot um and you you're talking about NFS um that's the one thing I want to get rid of. So is there another way to supply this HTTP or or SMB or whatever? >> Yes. So so D cut is very nice. So you can boot an ISO with the kernel and the init RD and there you can have a URL which fetches the squash FS via HTTP or HTTPS or CIFS or what whatever Dracut provides. So doesn't work with the video. I should have prepared an ISO on my own. And [clears throat] yeah, but but then let me try. I I have an ISO. I remember. >> Okay. >> Yeah, no problem. So 5 CD 5 KVM minus U3 CD. Let's see. doesn't work. If you have more question just ask. Okay, here. So, there's a pre-generated ISO on the web page, but you want to create your custom things. So, let's see if Okay. So, I also have a wrapper script called five minus KVM which boots up. >> Text bigger. Okay. the cooler. Yeah, thank you. 10 minutes left. No. Okay. So, this is just a simple wrapper that I use for testing. It boots up a virtual uh CUMO KVM machine and the disk will be on /tmp because I throw it away after trying the installation. So it's very fast and well that's not good. So boot from first partition. Now we want to do the client installation. So this is not the fi IO but the fi IO. So and this is not fully automatically because this is like a demo ISO where I boot up the system and then some manual selection can be done. So you can create with the F me web service a complete installation without touching anything or you can say I want to create an ISO and the user can select what to install and that's what we will see later file like this uh it's because of the network that may we may need some time out but then we see an curses menu where where we can select what to install. Fi can also install RPMbased distributions. So it's not only for Debian derivatives. Can I have the micro? Another question over here. It's it's not really question but maybe you can show our simple config git repository so that people have some ideas how to configure it. Yeah. Okay. This is now booting up. Yeah. I didn't configure the local network correctly but meanwhile [snorts] I'll show background. So let's go into simple X. So these are the subdirectories we have. Um so the most important there's a class subdirectory in the class subdirectory we have some scripts you see the file names so we have an order and these scripts are executed thank you and then everything what those scripts write to standard out will become a fi class and we the fi classes are written in uppercase so for example the 50 minus host classes There I have a shell script which depending on the IP address or on the host name that I get from DHCP I define the class Debian for example or German and this class is then later used for doing some configuration stuff or for selecting packages. I think for German I will later select the task German. Uh you see there's also rocky and mint and thevar uh files are just shell files where I can define shell variables and use them later. So defining classes is the very early stage of infi installation. Then we have no the disk config because we want to configure some uh local disk. And here you see this is the example very similar to an FS tab. So create a primary partition some log logical partition which file systems uh file system types and some options to MKFS or tune FS. Um I think yeah then the EV uh UFI looks different. The LVM looks like this. And I hope this is really easy to understand. Uh how does this this solution compare to uh iPixie with with preede? I'm currently using Philip Hans uh SIS solution with the hands off. >> So, so uh preceding uh especially for the petitioning part is very very ugly. You need to add some dots at certain places and if you forget it, you are lost. It's hard to debug. Um this is much easier. So use just this. So we we in fi we can use the preceding for the packages like answering the question do you want to start the open SSH server at boot time or not but for p partitioning we do not use the preceding. So for some other things you can use preceding but not for partitioning. >> So basically what you say is the main differences in how you do the partitioning. >> Yes. and and and the overall control and the overall installation environment is not the Debian installer. So we do not use any part of the Debian installer. >> Okay. >> Uh so you you still use a pretty old style config with shell variables. Uh do you have a plan to add an option to use like unified files say toml similar? >> No. Fair. >> Keep it simple. Stupid shell is the thing that most people can use and know. So I think for me there's no gain in using some domain specific language which is not touring complete and so on. During the installation you have root account you can do everything. So uh for security reason um there's no need >> fair no blaming you. >> Yeah. No no. >> So this was the example for the disconfiguration. So do we have time till 45? No. 40 >> till 40. Okay. 3 minutes. >> 3 minutes. >> 39 now. >> 37. >> 39. [laughter] >> So just to show how to select packages. You see here we have a keyword packages install without the recommen. This is now new. Uh you can add classes and log logical operation on classes. So this list of packages will not be installed if the class mint is defined. These packages here will be installed if the class non-free is defined. Uh for an ISO for the class ISO of or for the architecture we add some packages. So it's very easy. And here you also see that you can use a variable name for uh a class name. So thanks a lot. Uh I hope your download is still working and if you have any questions later I will be here the whole conference. So just come to me give me feedback if it works what what is missing and thank you very much. >> [applause] >> Oh, this is the menu where you can select my first installation