hey what's up everybody my name is moss and in this video i'm going to introduce you to gitlab ci more specifically i'll walk you through the creation of your first gitlab pipeline in order to follow along all you'll need is a gitlab account and a gitlab project to work with if you find this video valuable please consider throwing a like on the video and subscribing to the channel for more videos like this if you haven't already go ahead and grab a coffee and let's get started so the first thing i want to do before creating our pipeline is provide a high level overview of gitlab ci cd and some key concepts gitlab ci cd is essentially an automation engine for your software development process it includes a subset of gitlab features that allow you to perform devops practices like continuous integration and continuous delivery slash deployment it's comparable to a tool like jenkins which also serves as an automation engine that facilitates devops practices with gitlab ci cd you can automatically build test and deploy your software project using gitlab pipelines a gitlab pipeline is a version controlled yaml file called getlab.hyphenci.yaml that lives at the root directory of a gitlab project if you're not familiar with the concept of a ci cd pipeline it's essentially an automated set of sequential steps to build test and deliver slash deploy your code get lab pipelines are made up of two main components the first component is a job a job describes what needs to be done and could include commands to compile test or deploy code the next component is a stage stages define the order in which jobs will be completed a pipeline is basically a set of instructions for a program to execute and the program that executes jobs in a gitlab pipeline is called a gitlab runner get lab runner is a separate program from the gitlab application itself and you can run gitlab runner on your local machine a vm or even a docker container it is similar to the concept of a jenkins agent at pipeline runtime gitlab assigns pipeline jobs to available get lab runners there's more to discuss about runners but i'll cover that once we get into the pipeline creation now that we've covered gitlab ci cd and its key concepts at a high level let's start creating our first pipeline ok so i'm currently inside of a gitlab project that i created it's called the gitlab hyphen ci hyphen introduction and there isn't anything in the repository as of right now there's just a readme file uh in this gitlab project and i'm going to create a gitlab pipeline for this project so in order to create a pipeline for a gitlab project we would navigate to the ci cd section of the left hand side menu and we would either select pipelines or editor first i'm going to navigate to the pipelines page so i'll click pipelines and on this page since we don't have a pipeline created yet it shows us this list of curated uh ci cd templates that are based off of um tech stacks so you can see here we have uh an android template uh there's also a template for go projects gradle projects maven projects if i keep scrolling you can see there's a template for python projects as well for just about every tech stack that you can think of there is a gitlab cicd template pipeline that you can that you could start with if we didn't want to use one of these templates we could also just create a gitlab hyphen ci dot yaml file at the root directory of our repository but i think if you're not using one of these templates the best way to create a new pipeline is to use the pipeline editor on this page so from here i'm going to select create new ci cd pipeline and this file is going to be created on the main branch and it then takes me to this browser-based pipeline editor and if i scroll down here you can see we have this template pipeline that's just a very basic bare bones template pipeline that we could use if we wanted to it's not technology specific in any way it's just kind of assumes the most basic steps of any cicd pipeline before we review this pipeline though i'd like you to notice at the top of the page here it says this gitlab ci configuration is valid and one of the benefits of using the browser-based pipeline editor is that as you're editing your pipeline uh gitlab is evaluating it and and it will tell you whether or not your your pipeline configuration is valid and if you have any you know syntax errors it will let you know at the top of the page here that you know the the pipeline is not going to run now this validation is also available as an extension in vs code if you use vs code and you'd like to edit your pipeline locally in your ide so you can install a a get i believe it's called a gitlab workflow extension and it includes this functionality where you can validate your ci configuration from the command palette another really useful feature of the browser-based pipeline editor is automatic syntax highlighting so as you're writing your pipeline and you add certain keywords so for instance the artifacts keyword what it should be doing is automatically suggesting uh matching keywords like artifacts keyword and it will also provide like a short description of what that keyword does the only problem you notice here that it's not automatically suggesting keywords there's currently a bug that is preventing this feature from working and there's an open issue in gitlab's project on gitlab that is supposed to address this particular bug after some research on google i found a matching issue you can see here it says pipeline editor autocomplete should trigger based on cursor position this was created two months ago and uh yeah it's labeled as a bug so you can see here what is expected as the person types it should be suggesting keywords so it's unfortunate that this feature is not available at the time of recording but i did want to mention that as a benefit of using the online pipeline editor two other nice features of the pipeline editor are these two tabs at the top we have a visualize tab and a lint tab so if i navigate to the visualize tab it will give me a graphical version of what this pipeline currently looks like and then we also have the lint tab which gives us a breakdown of our ci configuration lets us know if our pipeline syntax is correct so let's navigate back to the edit tab and i'm going to delete these comments here and let's quickly walk through how this pipeline is currently structured so on line two we have the stages keyword statement so this pipeline is pretty standard it has a build stage a test stage and a deploy stage stages are executed in the order that they appear in this list below the stage's keyword so we see that the build stage is the first listed and that's also represented in the visualize tab you can see from left to right the build stage is the first stage and if i wanted to i could switch the build and deploy stage so if i made the deploy stage first then we take a look at the visualize tab you can see the deploy stage would be executed first then the build stage and then the test stage stages are typically defined somewhere near the top of the pipeline as you can see in this example now after the stages definition we then have a job definition so starting at line seven we have the first job defined and it's called the build job as its name suggests the build job is associated with the build stage so it should be compiling code or building our application and to associate a job with a particular stage we would use the stage keyword under the job definition so under the job name we have the stage keyword and then the value is the name of the stage one thing to note is that a job doesn't have to be associated with a stage it can execute as a standalone job however the one keyword that is required for a job definition is the script keyword the script keyword allows us to specify commands that the gitlab runner assigned to this job should execute similar to the stages definition these commands are executed in the order that they are listed here and as you can see it's basically invoking shell commands like echo so anything that you could do at the command line you could pass in to this script keyword so if you needed to build a new docker image or compile c code using make or compile java code you can invoke those tools and utilities from here if i continue to scroll we can see that there are uh two test jobs there is a unit test job and a lint test job and what you'll notice is that both of these jobs are associated with the test stage so they are going to be executed inside of the test stage and this is represented visually as well so under the test stage you can see both the unit tests and the lin test jobs are listed under the test stage and what this means is that by default both of these jobs are going to execute in parallel they're not going to wait on each other to finish in order to execute they'll both start executing as soon as the test stage is has begun and then the final job in this template is the deploy job there's nothing really special about this job it's just associated with the deploy stage and then it has these kind of you know fake these echo statements saying that it successfully deployed the application so what i'm going to do is modify this template a little bit the first thing that i want to do is remove these sleep statements so the first one on line 24 sleep for 10 seconds and the one under the unit test job sleep 60 seconds i'm going to remove those sleep statements the next thing i want to do is modify the build job to produce a quote unquote uh executable uh binary file so the output of the build job should be some sort of you know binary application that you can run and will be used by the tess the test job and the deploy job so under the script keyword in the build job i'm going to add another command and i'm first going to make a new directory so make directory and then we will call it the build directory and after we create the build directory i'm going to add another command and we will echo my application binary file and we'll write that to a new file within the build directory called executable binary file v1 okay so this is just a text file but we'll pretend that it is a binary executable that is being produced from the built job the next thing that i want to do is modify the unit test job to utilize and consume that executable binary file that was generated in the build job so below the script keyword in the unit test job i'm going to add a new command under the script keyword and we're going to cat the the binary executable file so that we can see what's written to it which should be this string here after the invocation to cat i'm going to reference a predefined environment variable called ci underscore project underscore directory and this is an environment variable that i did not create myself it is provided by git lab and it's made available to a gitlab runner when it's executing a job after specifying the project directory we will then specify the build directory and then finally the executable file so executable binary file v1 so now that we're referencing that file in the unit test job i want to run the pipeline so that we can see the output and review the output of the pipeline so i'm going to scroll down here and we're going gonna leave the commit message uh as the default commit message and as soon as i select uh commit changes the pipeline will start running so i'm gonna select commit changes and once i select commit changes it checks the status of the pipeline and then it will show up here once the pipeline has started running and we can open up the pipeline okay so i'm going to select a view pipeline in a new tab and from here we can see the graphical view of of the pipeline and the status of each of the jobs within the pipeline and what you'll notice is that the build job succeeds the lint job succeeds but the unit test job the job that was referencing the quote unquote binary file that we created from the build job uh failed so let's take a look at the console output of the unit test job to see what failed within the job so i'm going to select it and it takes us to this console output of the job and let's walk through uh this output so it first starts out by allocating and running a specific gitlab runner once it allocates the runner and it creates a new environment for the job to be executed in it clones down the repository to the local workspace and then you can see starting at line 21 the commands that were specified under the script keyword begin executing so first we see the echo statement for running unit tests then we see the echo statement for code coverage and then on line 25 we see the cat statement where we try to cat our you know our executable file and it says that there is no such file or directory and this appears to be why the job failed now the reason that it couldn't find the file that we created from the build job is because each job runs in a fresh environment and jobs are not being executed in a shared workspace the same way that a jenkins pipeline is so a jenkins stage has access to the artifacts that may have been generated by a previous stage such as the build stage but in the case of git lab each of these jobs are being executed in a separate workspace and they don't automatically share the artifacts that are produced from one stage to another one job to another and real quickly i'll show you the documentation for this the gitlab documentation it says one important difference is that jobs indepe run independently of each other and have a fresh environment in each job so if we have any artifacts that need to be passed between jobs that is controlled using the artifacts keyword and the dependencies keyword so in our case if we want that binary file to be available to the unit test job we need to utilize the artifacts keyword in our pipeline so let's navigate back to the pipeline editor and then i'm going to scroll down to the build job and i'm going to add a new keyword under the build job definition and this is going to be the artifacts keyword the artifacts keyword allows us to specify files that should be shared with downstream jobs in the pipeline and this keyword also allows us to download those specified artifacts to our local machine after the pipeline has completed so for instance if the job generated a test report we may want to download that and review it after the pipeline is completed and to do so we can utilize the artifacts keyword in our case we want to specify the path of a file and to do that we would just specify paths under the artifacts keyword and then under the paths keyword we can specify one or more file paths that should be artifacts produced from this job so i'm going to specify the build directory and the executable binary file now after completing the script section the get lab runner that's executing this build job will upload this file to git lab and then any downstream jobs like the unit test job will first download any available artifacts from gitlab before executing their commands in the script keyword so let's go ahead and commit our changes and see if this fixes our pipeline so i'm going to select view pipeline and i'm going to open up the unit test job now in the unit test job we can see that the job succeeded and starting at line 16 the unit test job downloaded artifacts and says downloading artifacts for build job and then it specifies the token of the artifact and we know that um the file was downloaded successfully because the cat command works and we can see what's inside of the of our executable file on line 26. a couple of other things that are worth noting from this console output page is on the right hand side it shows the duration of this job it took 12 seconds and we also get time stamps or not time stamps but we get a duration within the job as well so here we can see it took one second to download artifacts and for each step within the job we get this duration it also tells us how long ago the job finished and then we have this timeout period which is currently set to one hour and that means that if this job is still running at one hour it will be timed out automatically and it will no longer run and this timeout period is very useful if you're using gitlab runners that are hosted by gitlab because you're paying for the time uh that those that you're using uh those gitlab runners and then we can also see which runner uh executed this job so let's navigate back to our editor actually before navigating back to the editor let's navigate to the pipelines tab and from here if we take a look at our latest pipeline and i navigate to the right here this is where artifacts can be downloaded so if i select this button here i can then download the artifacts that were produced from the build job which should just be that single file and uh and download them to my local machine okay so now let's go back to the pipeline editor with the artifacts keyword we now know how to pass the output from one job to a downstream job but let's say that we needed the jdk installed so that we could compile our java project how would we go about doing that within a pipeline well in jenkins we have the tools section which allows us to install tools like the jdk onto the host that is running the jenkins pipeline so the jenkins agents will be running on a particular host and if a pipeline has a dependency like the jdk the jdk will be installed on that host and then on future runs of the pipeline on that host it won't need to reinstall the jdk gitlab doesn't have uh an equivalent to the tool section in jenkins instead gitlab suggests using docker images that contain all of the dependencies you would need in order to you know build test uh and deploy your code now how would we utilize a docker image inside of a gitlab pipeline well we can actually specify a target docker image using the image keyword value of the image keyword should be the name of the target docker image that you would like the gitlab runners to use so for instance if python was a dependency we needed python installed in the runtime environment of this pipeline we could specify the python docker image and if we needed to we could also specify a particular tag along with our image like the latest tag or an earlier an earlier tag now what will happen after specifying this image when we run the pipeline and a job is executed by a gitlab runner the gitlab runner will pull the docker image that we specify here run a container from that image and then it will execute all of the script statements that we specify within the job definition within that container so if that docker image has python installed on it then we will be able to invoke python from a command under the script keyword and that's exactly what i'm going to do so i'm going to add a space here and then scroll down to the deploy job and let's add a command under the script keyword in the deploy job and we're going to invoke 3 and then the version command so this should print out the currently installed version of python and let's go ahead and commit those changes and see what happens when the pipeline runs okay and i'm going to select view pipeline and i invoked python in the deploy job so i'm going to select the deploy job here and the deploy job should begin pretty quickly okay and the deploy job has begun and uh starting at line four you can see that it says using docker executor with image python and then it is now pulling the docker image for python okay and it looks like the job has completed so if i scroll down after it's cloned the repository it's downloaded artifacts uh on line 21 it says deploying application and then on line 23 we see the invocation to python 3 and on line 24 you can see the version of python is printed out to the console so if we have any dependencies like python or the jdk that need to be installed in the environment that the job is being executed in then we can utilize the image keyword and pull a specific docker image and if we have a custom docker image we can also use a custom docker image as well from you know a registry container registry either within gitlab or outside of gitlab okay so let's navigate back to our pipeline editor now it might be the case that you need to authenticate with an external service during pipeline execution or maybe there is some sort of environment variable that that needs to be accessible to one of the jobs and you might need to define those environment variables so how would you do that for a gitlab pipeline in order to define variables that can be used by gitlab runners when executing jobs we can use the variables keyword under the variables keyword we can specify key value pairs of environment variables so for instance let's say i needed to authenticate with a particular service so i needed a username environment variable what i can do here is specify the key of the environment variable and we'll call it username and then i'll set the value of the environment variable to tech with moss now if there is a password or a token associated with this username environment variable should i define that password under the variables keyword in the pipeline the answer is probably not because this pipeline gets version controlled with the repository and so that password would be also version controlled in plain text and that's not uh that's not ideal so if we have any environment variables that might be sensitive like tokens or passwords we can actually create those variables from the settings tab so if i scroll down to settings and then i'll open the ci cd settings in a new tab and from here i'll scroll down to the variables section and under the variables section we can create new environment variables that would be available to to our pipeline so you can see here it says variable store information like passwords and secret keys that you can use in job scripts and they can either be protected or they can be masked or they can be both protected and masked protected means that they're only exposed to protected branches or tags and then mass means that they're hidden in job logs so i'm going to select add variable and the key will be called password and then the value will set to my password and it's of type variable and we'll leave it with the default environment scope we'll also leave it as a protected variable and then we're going to select mask variable and when i select add variable you'll notice that the key shows up here but under the value it is not shown so it's masked but if i have the right permissions in this gitlab project i can reveal the values and we can see that the value is my password so let's reference this password environment variable in our pipeline so if i navigate back to the pipeline editor i'm going to scroll down to the deploy job and under the command where we invoke python3 i'll add a new command and what we're going to do is simply echo the username and password environment variables so i'll say echo using credential and then username password and let's go ahead and commit those changes and take a look at our pipeline job okay and it looks like the deploy job has begun so i'm gonna open that job up so the job has succeeded and if i look at line 25 i can see the echo statement using credential username and password and then on line 26 we can see the output of that command and we see that tech with moss is printed out to the console but as we specified in the cicd settings the password value is masked in the job log output so it looks like our pipeline is running as expected and i don't want to make any additional changes to the pipeline but there are a couple of other things that i'd like to show you before closing out the first thing that i'd like to show you are the settings for get lab runner so if i navigate down to the settings tab and we navigate back to the ci cd page there is a section for get lab runners and i'm going to go ahead and expand this section and it says runners are processes that pick up and execute ci cd jobs for gitlab and if i scroll down here we have specific runners and shared runners and what i've been using in this tutorial are shared runners and these are get lab runners that are hosted by gitlab using shared gitlab runners is free but it's limited to 400 ci minutes per month per group for private projects now if i wanted to use my own compute resources instead of shared runners so i wasn't limited by this quota i could download the gitlab runner program and i could run git lab runner on my local machine a vm or a docker container on compute resources that i own and then i would register that git lab runner with this particular gitlab project and then when the pipeline is ran gitlab would utilize the runners that i specify on the compute resources that i own and assign jobs to those particular gitlab runners and under the shared runners section we can see a list of available shared runners as well the other thing that i wanted to quickly review is caching in a gitlab pipeline so i'm going to navigate to the git lab documentation on caching and by the way i'll include links to this documentation that i've been referencing in the video description so in addition to artifacts and related to the artifacts keyword you have the cache keyword and this allows you to cache files that a gitlab job downloaded so for instance if your pipeline was running on a python docker image but your project needed some set of python packages like numpy or scikit that weren't included in the python installation then you would need to download and install those dependencies at pipeline runtime but you wouldn't want to have to download those dependencies every time the pipeline ran so we can cache those kinds of dependencies using the cache keyword in our pipeline and if i navigate to the the keyword reference documentation for cache and scroll down they have an example of where we're caching a particular file path so under the cache keyword we specify a key for the cache and then we specify the paths which might be a set of binaries um or you know this config file so going back to the caching documentation um with cache subsequent pipelines can use the cache and subsequent jobs in the same pipeline can also use the cache if the dependencies are identical there are so many features that i haven't mentioned in this video that are available to you with gitlab cicd and git lab pipelines but i only wanted to cover a subset of features that i thought could get you up and running with your first pipeline and that you could apply to your own project initially this video is just your first step into get lab ci cd and gitlab pipelines as i said before i'll include links to the key documentation that i referenced in the video as well as documentation that i think is important to review if you want to continue expanding your knowledge around get lab ci cd and gitlab pipelines but i hope you enjoyed this video and you found it valuable if you did please consider throwing a like on the video and subscribing to the channel for more videos like this thanks for watching