GitLab CI - A Template for Reusability

so welcome to galaxy on a template for usability I'm Francis and I'm a project specialist at cold front labs I work on a lot of mating tests and we're closed right and I'm Nick I do a lot of like front-end JavaScript stuff that cold front labs as well as Drupal theming and a little bit of module development so today we're going to talk about get lab CI and before we get into get lab and what you can do all the amazing things you can do with it we're gonna quickly go and talk about continuous integration and what it is just a note on terminology I'm going to use the words merge and merge quest quite a bit today these are similar to the terms pull and pull requests in other systems but even get lab you say merge so just so I don't lose you on the terminology it has to do with the practice of integrating code into a repository or yeah the act of integrating code into a repository so at its most basic continuous integration is the practice of merging changes into a central repository on a regular basis and running automated processes builds tests etc to verify that the build is functional instead of keeping code on individual computers developers merge their code into a shared repository frequently generally on a daily basis or more often so that everyone has access to the latest version of the code depending on who you ask there are anywhere between like 33 and 30 mils of CI today we're going to keep things simple and highlight just a few that everyone pretty much agrees on the main tenets I guess would be that code is stored maintained and updated in a single central repository this doesn't mean on your production web to it this means by changes to the code should be small and modular and merged in on a regular basis and every commit should be tested automated self testing builds should be run to ensure that the build is working so some of the benefits of CI have to do in reducing time spent tracking down bugs speeding up code integration and increasing visibility and transparency because you're constantly running tests on your code bugs are found quickly it's also generally easy to isolate about be cut or the positive about because code changes are kept small and in small manageable chunks another benefit of making incremental changes is that there's less likelihood of integration conflicts when coding honk let's do our eyes they're generally pretty easy to deal with because you're only dealing with a small number of changes that displeases the chance of making mistakes when dealing with conflicts and also avoids compounding issues memorizing food since the code is stored in a single place everyone always has access to the latest version and could tell the status of the build at any time without having to track down individuals and ask him like how features doing or where bottom is at or all of that stuff so the major takeaway about CI is that it saves you time having everything in well-tested manual manageable chunks speeds up the development process and here automated CI processes give you confidence in your code because it is being built and tested with each commit [Music] so the workflow for the CI Pasha's is simple you have your central repository where your code is saved and this might be your master branch if you're using git and Jesus to this code are made by making copies of the code and merging in small changes back to the main repository on a regular basis in our project we follow a feature branch workflow for CI we create branches of code for building and testing and then of these branches are merged back into our main master branch finally processes such as code code checks tests and builds are automated so that they are run on every commit every change you make to your code is tested and verified some examples of automated tasks include running a linter it to checks your coding syntax and standards running compilers to ensure your code is building and running validation tests to ensure that functionality is working and everything is we're going to check for regressions - whether using feature branches or committing directly to the master branch it's important to remember the continuous in continuous integration to realize the benefits of CI you have to commit often and keep changes to a reasonable size for example it's much easier to read something like this in which a dependency a single dependency has been added to the project then to read something like this where there are many many changes like images being added dependencies added components created and configurations changed all in a single change so sure you can read through large volumes of changes but it's much quicker to deal with things in those small chunks that we talked about previously okay so Nick told me I shouldn't spring pop quizzes and presentations but does anyone know what CDs dancer does Yellin oh okay continues delivery so actually it was a bit of a trick question but I guess it actually gave more people the opportunity to get it right can this continuous deployment or continuous delivery basically have is following the CI process but then tacking on automated deployments to your automated processes these deployments might be triggered automatically which would be continuous deployment or continuous delivery in which case you are manually trigger your deployments but they're still part of your automated processes in our examples today we're going to include automated deployments as part of our CI okay all right so now that you all are masters of the CI seedy lifestyle we're going to get into our main topic here get lab CI so before we go into the nitty gritty good lab CI stuff we're going to tell you why and how we use it so the first the most obvious reason is that gitlab we host a lot of our projects on get lab be it for internal use sort of personal projects or clients and get leb ci is built in to get lab and it's free to use the next one is it's really easy to configure we'll get into this a little bit later but essentially all you have to do is add your get lab CI file into the root of your repository and you're done it'll run every time you push next it offers really good process control you could run individual pipelines without necessarily having to make code changes so if you just needed to run a test again you could go through the gate lab you I go into pipelines and run your your stuff manually without having it to pick up on changes and go and you can also save artifacts to your build and what this is basically when we run a production deployment get lab CI saves a compiled code version of our build which is the artifact so that we can go and download it later if we need to we need to figure out what went wrong or was it broken when we built it kind of stuff and for those of you who know our boss he also told us to use it and so we had to so at copán we use get web CI for a lot of different things we run a lot of code checks and linting on our PHP in our sass and our JavaScript code we use it to build and deploy our Drupal seven and eight sites UJS projects and even popular images and we run validation like backstop Code section acts and a couple other things on our sites and we run security audits using dress composer and again so now that we told you what we use why we use it and what we use it for let's start things off with some basic terminology get the home people exit process are kind of broken down into four main things runners pipelines stages and jobs so a runner it's basically a machine typically a virtual machine that takes all of your code from your repository all destructions in your CI file and if we execute any tasks it runs things get lab provides a set of free runners for you to use if you're going to get lab comm there are some usage limits they're pretty high though I think you are allowed to run 1000 pipelines per month is it typically you would create your own runner so you don't run into any limitations and you can optimize it for your own processes runners are what makes your CI process go they run your pipelines pipeline is a pretty generic they like it's the wrapper term for your jobs that you're running it will basically just contains all your series of automated tasks that you run on your code and they contain multiple stages a stage is a category within your pipeline you may have a little stage or at the chlorine stage probably all of those things and the order in which you declare the stages or the order in which they are run they run consecutively which means basically one stage must complete before the stages after it can begin to run so like the lint stage must finish before the build stage begins and the deploy stage has to wait for the builds to finish before it can also run an important thing to know about your stages though is that while they define the general order of your pipeline they don't actually execute anything that's what jobs are for so a job is an individual task contained within the stage it's where the functionality of all your CI processes are written you can have multiple jobs contained within the same stage so for example when you're linting something you probably maybe have JavaScript and SAS so you're gonna want to run tests on your es lint or JavaScript style int for your SAS code it doesn't make a whole lot of sense to run es lint and styling in the exact same job yes they're both linking processes but they're for two different languages they should be separated into two different jobs so within your lint stage you could have a es lint job and a style lint job another important thing to remember about jobs is that unlike the stages that they're contained in they run in parallel so when your lint stage begins both your es lint and your style and jobs will start running at the exact same time once the third matters as long as there is once all your jobs in a stage have completed that stage is then completed itself and then the next stage is all of the jobs inside of it can start running so let's visualize so here you can see we have some groupings of tasks all of these tasks are what makes up your pipeline within our Python we have three categories lint build and deploy these are our stages within each stage we can see some individual tasks these are jobs we have es lint or silent in the lint stage we have a job called build in the build stage in the job called the deploy in the deploy stage so when a pipeline begins all the code and instructions are sent to your runner and all the jobs within the first stage will begin to run but now just because all the jobs started at the same time does not mean that they will finish at the same time fortunately before the build stage can begin all of the jobs in Billund stage must succeed once all the lint jobs have succeeded our build can go and this process basically just repeats throughout the whole pipeline each stage must finish before the next stage can begin all right so let's take a look at configuring so Nick mentioned the CI file which is used to control your pipelines right I forgot a slide which I always do here we go to create a pipeline so Gill at CI is controlled by file called the gitlab CI yellow file this as I mentioned is used to tell get lab what to do it defines the pipeline and get lab will automatically detect this file when you place it at the root of your repository so let's go have a look at that file so you can see at the top of this file I have defined the stages in this case we have a super simple example we have three stages lint buildin boy down here you can see the jobs I have an es link job in the lint stage a build prod in the product state or the build stage and deploying the deploy section now can any to see how well you've been listening and he wouldn't tell me why we would have three stages when we only have three jobs and you know we could just have them in a single stage why would we have these jobs in individual status so you guys exactly so you need them to run one after another so in in this case you can see that we have very simple scripts the scripts are what tells the run are exactly what to do during the job we just run a linter to check syntax run a build and then deploy that build out to a server you'll notice that in the build stage we create an artifact and that's what is used grabbed by our deploy stage to copy over to our to our web server let's go take a look at this in the gate lab UI that's a little oh and something I didn't mention here you can control by an artifact size expire so you probably don't want these you know if you're if you're building as much as we are you probably don't want those artifacts sticking around too long so you can control when how long you want get loud to hold on to those for you all right so you can see that the latest on the pipeline's page of the gate lab UI you can see that I have my demo branch here and I've actually had all the stages succeed um you can click on a stage to see the builds that were run within that stage and you can see here that my linting task actually failed earlier so let's go have a look at the terminal output for that job here you can see that it ran the npm install just fine but it did fail on the engine from linked because of syntax error on in my main J's file if we go back to the pipeline after fixing the linting error let's go have a look at that build that we that we ran you can see here that those job artifacts can be accessed downloaded or you can browse to the files that were that were saved in that artifact we specified in the file you can also a kind of a handy feature that I like is if you create merge requests you can go and see the stat the most recent status of the pipeline that was run against that branch with a merge request another handy little tip you you can also run jobs again which is something they can mention and you can also run whole pipe lines if you if you have a reason to to want to run a specific branch at a later point okay so going back to our gate lab CI file I'm going to go to version 2 there we go so some of you might have picked up on in the last version of the file I'm not sure if anyone was paying that much attention but we had some repetition going on in our job skirts because I had npm install written in every script parameter in this version I've moved that up to the before script which is a special keyword in github CI which will just run all of those commands you list in the before script array prior to every job in the file there are also a series of other keywords such as after script so that you can kind of reduce redundancy in this CI file as you can imagine as we add jobs things easily get you'd have a lot of repetition going on in this version of the file I've also added two new deploy jobs I've added a deploy dev and deploy QA because I didn't always want to be deploying to my production server once we introduced jobs like this I don't actually want to run all three of those jobs every time I run the pipeline so we've added these only and accept parameters to make sure that we deploy to dev when for a commit as made to a branch except in cases where we commit to the master branch and then the deploy to QA will happen whenever we make a change to the master branch I've decided to trigger my deploying to prod manually because I don't want to accidentally have that happen anymore so the taking a look at these scripts you can see that actually there's a lot of duplicated code here because pretty much everything is the same between my deploy jobs except for the location where I'm actually copying that build to this gets especially troublesome when you imagine adding multiple validation jobs to all of your cert or to test code on all of your servers and adding like we we get to the point where we have you know nine stages and probably 20 to 30 jobs yeah and so if you're you know it it adds a lot of bulk to your CI file if you're repeating code so what's the answer to this problem of dilute 10 please okay so templates the reason you probably have come to this presentation okay so get lab see our templates are pretty much exactly what they sound like they're pre-made jobs that you have created that you can extend to fit the needs of your of all your stages and these do you can have script stages only accepts dependencies anything that another job could have that's what a template can also have all these parameters are inherited by the job that's extending the template and you can customize and override each parameter whenever you need just in case you need something a little different in a previous example they had three different jobs for deploying to three different servers we could easily create a generic deploy template to handle all of these so let's take a look at how we would do that so here we have our production deployment job I've taken the liberty of removing some of the functional the screen went away yes yes so it is essentially yes I'll let you attach to the dog and so they're accessible by the other runners it just makes the next I'll be able to has like a storage it has some temporary storage that you can use typically you would make your own caching server not ours for the most part we also have our own good lab instance so we control the whole server yes we sure do yeah so I actually just had a build issue and I was like trying to make the demo and I wasn't gonna troubleshoot it so I just went back to recording it's because fun fact your NPM walks like the package - lock JSON file is read first and then the package.json file is read to install any updates so your log file does squat all unless you use okay carrying on so we have our deployment job here for Prague so I've replaced some of the functional code with comments because you guys get we don't need to see all of the code so we know by looking on a previous examples that this job looks pretty much identical to our deployment jobs for dev and QA except for the server so they're all part of the deploy stage that although the SSH key they all are sync like they're all basically the same so the first step in to turn is into a template is to rename our job so we've named it deploy template and so we're just telling everyone what it is so you'll note the period before the name this is a handy little tool that tells gitlab that this is not just an ordinary job it won't run it because it's hidden from the CIE process so next we'll get rid of our only and accept parameters you could have default values for them if you wanted which is totally cool to do but for our purposes right now we won't worry about that so now looking at the script parameter the only thing we have left is the big problem the server is hard-coded right into the script to solve this we will take advantage of variables so variables allow us to really harness the power of getting leb CGI templates just like any other language variables or data storage key value pairs you're good you can declare variables in a few different methods and levels which we won't go too far into but essentially you can declare them per job per pipeline per project and even per group inside of gitlab itself so you can use variables pretty much anywhere you want within your builds except for a few small exceptions and those exceptions can be found in the gate lab see I documentation about variables so yes so depending on where you would declare them that's where its scope is when the CI process begins on your runner all of the variables in your various scopes are loaded as environment variables just into your terminal so you would access them like you would any other command line environment variable so let's see how we can use variables to finish off this template so Clare a variable in a template we use the variables parameter which is a basic Hamel array so now that we have the target server variable available to us we can simply remove the hard-coded server and plug in our variable so now as you can clearly see our slides are using bash so we access our variable through the dollar sign and then the variable name obviously if you're using what does match your PowerShell or something there'll be a slightly different syntax but you get the gist so the next step is to actually use our template to allow drops to pull from this newly created template we use the extend its functionality we use the extends parameter this basically tells get lab that when it gets to our job it needs to go look at our template first I use that as the base job then look at our new job and make any overrides or customizations based on that you know like like a template so let's look how we can use this template in our deployment job the first step is to tell get lab that we are extending our new template using the extends parameter next we need to make use of our target server variable and give it a value for this job so in this case just drop 4.com finally we'll add our only except rameters back because they're specific to our job to make sure that we don't accidentally have a dev build deploy to queue at production that would be unfortunate so following these steps we could upgrade our dev and QA stages as well there were jobs as well so here you can see we've extended our template declared our variables and made our own lean except rules pretend the real only accept so while these templates help reduce duplicate code it may not seem so important in this situation we only have a few jobs to execute the same code but when you have multiple types of builds multiple types of deploys a whole bunch of tests that run before and after those deployments they the repetition can add up really quick okay so extends and variables is great they're all very nice to use but what if you have multiple projects on git lab that needs these templates so you start off with one project with a pipeline in a template then you add another and another one and more templates more projects more pipelines more bills it's just getting ridiculous so you may be like our good friend Leo here and think that that's just way too much work to support all those projects with their CI bills and all their templates and you're right it is way too much work we would know we used to do it we would have all of our clients projects and all of our projects with their own CI build files all using these sort of templates we've made in our files all extending them and then if we updated one of the projects we would then have to make sure to go through all of our other projects and update those as well to make sure that all of our builds are on the same on the same stage there and this is where the final key feature of templating comes into play the include parameter the include parameter is a global parameter just like when you're declaring your stages it goes like right at the top of your gitlab CI file it functions as a basic list of URLs all pointing to different Gambel files when you add a gamma file within the includes parameter get much the IVA basically look at the URL go to that URL and embed all of that ammo as if it were written in your CI file itself this allows you to have a single source for template files all of which can be included extended and customized throughout all of your without having to update constantly so as an example these two slides are exactly the same as far as get lassie is concerned so not only does this allow you to include templates across multiple projects but it also drastically reduces the amount of code you have in your CI files so we'll end our official slide presentation with some important notes to keep in mind when you're working with templates all the parameters like script and stage and all that they're unique this means that if you have if your template declares a stage and your job declares a different stage when the gitlab CI runs your job stage will be the only one that's respected will not look at both of them a really important example for this is your scripts you may have a lot of default scripts written in a template if your job has a scripts parameter all of your default code in your template crushed is gone you cannot have two scripts so if you need to customize the script for this one job even by like just ever so slightly you would have to rewrite all of the scripts for that job unless you can somehow work it into variables like we do the exception to this rule is when a parameter is basically just a list of key value pairs like variables in that case get lab will merge those two parameters so let's say you had your target server variable in your template and then you added a site alias variable in your job itself your target server will be respected building into a list another important note is that jobs are limited to one template you can't combine multiple templates into one which is only a temporary limitation coming and get lab 12 they're changing the extends parameter to take a list of templates so you could have multiple templates combined into one job where that could be useful is like for us we have our basic set of like only and accept rules for deploying to different environments that we just caught and throughout everything all of our projects follow the same tagging and branch naming standards it would be nice for us to have a template that is just be dead deployment categories and extend that with another template so that when we make a dev deploy we can say take the deploy take the dev deploy and then we're good so we do have I believe we have some time left so we can do questions although we had a couple however we can also show you how we deploy Drupal with git lab CI and it's going to be very technical because we're basically going to show you how we template it all out okay I actually have it oh yeah here okay so it's big enough you guys want a little bit bigger good okay so essentially when we're making a build we have our template for building Drupal which essentially is just making sure composer is run and NPM is run and we add some default artifacts just naming our artifact and how the artifact will be saved and then we kind of customize it what we do with our templates is we kind of make templates from our templates to import into our jobs because basically our dev Bill is going to be the same pretty much every project we do our Drupal dev wants to build the exact same way so we have our build template then we have just a job that is the dev build and then we include that throughout all of our projects so that if we change the dev build once anywhere all of them will update and make sure they all deploy the same way so here you can kind of see some examples of the artifacts here is a good example of where parameters that are key value pairs are merged and not crushed so in our template we declare the name of our artifact and what pads it needs to save as the artifact and then in our dev build we have artifacts again and we're adding a new key to tell it that our dev field should expire in one day scroll that up a little bit so in this case artifacts in both the template and the build are kept there just merged into one so then we have a slightly different process for our release builds where we get into a lot of crazy templating is our deployment stage so we have two different kinds of deployments I won't go into initial deployment just because it's a weird use case and we have to deal a lot with drush which i don't want to talk about dealing with rush so essentially how we deploy to Drupal is we build it all through through composer and then just used rush site aliases to arsenic everything where it needs to go so the whole site aside from the database is built on our runner and then we just copy it all over so you can see a lot of like this is where variables come into play because you see like we have site aliases options ssh options our alias group our alias their alias type and then we SSH everything and then we run a bunch of other commands to get dress up to date we even wrote our own backup script because drush 9 took out the ability to then backups on your site because why not don't work yeah so we made our own little PHP script that is essentially Drescher so we do a whole lot of things we run the Drupal fix permissions script that you can get on drupal.org with a little customization by us and essentially it's all automatic we can go from no dev site on the server at all to a completely functioning dev site with just clicking the CI build run so it depends on which one we're running we have our initial Drupal deploy this actually creates a database and gives it all that with the MySQL URL our normal deploy we just update the data then move on and then after we've deployed everything we actually run a bunch of updates so we automatically run like brush-up dB we import all of our config and we do some more permission fixing just because we've noticed that get lab CI can sometimes screw up some of the permissions I did consider showing you a full demo of it like running but sometimes it can take upwards of our entire presentation time to fully deploy like a functional Drupal site so that didn't really make a whole lot of sense unless I hit play at the beginning and then showed you the end result yeah and so then we have it split out so we have our update template which has our basic stuff and then we extend this template further so that we update our future environments specifically we update our dev environments and update our QA environments our product buyers and most of these are just taking from the template with the small exception of like we changed the dress alias type variable just so when we run rush it knows what server to go to what to do another example are we on we are we are yeah are we okay so our SSH key is loaded I can actually show you we have we've done like all out in this template essentially for all of our projects we just have a list of imported files and that's all our CI consists of for the most part so when we load SSH keys they are variable stored on a project level so within gitlab you can go into the UI and actually make variables and those are inserted into every pipeline in that project which is where we the actual keys yeah well they're as they're not committed to the repo so they're they're stored as like secure variables that only maintain errs and owners can actually access and then basically we have this before script run and all it does is it looks for our variable and just loads it in and that way all of our jobs can access our Devon and prod servers as needed and another cool thing although if you want to talk to me like after this you can come find me because we also have stuff that like when our lint fails it actually runs a fix and gives you that artifact so that you can download that apply those changes and commit them back up it is technically public I'm just going over with Matt our boss just if we are actually going to like give it out fully we will yeah when the slides become available you'll you'll get there okay thank you [Applause]