Using Gitlab CICD Pipelines to run Bash Scripts on a schedule
On my homelab, I've got a bunch of tasks which I need to run automatically on some of the servers. Checking certs are up to date, moving files around, clearing logs down.
Historically the way to do this is as a CRON job, these work, however, don't scale well, are difficult to centrally manage and unless you write in a bunch of error checking how do you know if it worked?
Over the years I've migrated from CRON to Jenkins, this worked well for me, and despite the hate, Jenkins gets, for what I've been using it for it worked well.
Then I moved to Rundeck an excellent product, with at the time awful onboarding with a terribly documented onboarding process (search this blog if you want to know more and how to get Rundeck On-Prem setup)
The current tool of choice for running pipelines on a schedule which I'd like to get working are Pipelines using Gitlab CI/CD
All my code is in gitlab.com and using the pipeline tooling within it seems to be the obvious thing to do to have one less thing to manage or run (not Rundeck or no Jenkins)
It seems from my reading that the GitLab pipelines support bash, running things on remote boxes using GitLab-runners so seems to be the place to head to..
💡 I'm not a developer, I am at best a system administrator. (some who work with me might disagree) however, while I manage people day to do I run as much tech at home as I can so I understand the technologies when I'm having a conversation about it.
What I was finding by searching
To get to this point, yet again as I find time and time again when I try and get my head around setting up DevOps style tools.
There is lots of documentation however I found it all over the place, very focused with a lot of posts and youtube videos obviously spawned off specific tasks.
Unfortunately, they were usually based around Docker and I wasn't looking to use Docker.
What Do I want to do?
So this is what I'm looking to do with my pipeline
Setup a GitLab runner on a remote server
Have the GitLab runner use the shell (locally) to execute code
Have the code in a Gitlab.com project run on that remote server as a bash command
Run that code at 8 am every morning
Know if the code has run or not.
Connecting Remote Servers to GitLab (SaaS)
So the first question is "What is a Gitlab runner?"
It's an agent that runs on a remote server and talks back to the SaaS version of Gitlab
💡 There's a different method for the self-hosted version
and dictates what method of execution (shell, ssh, docker container etc) can be run on that remote server
Runners come in multiple flavours, shared, locked and associated with a project and it's the latter I'll be using moving forward.
Runners can run on Windows, OSX or Linux servers.
Choose your Project
I'm assuming you have a project set up in Gitlab and some code to un under that project
Within that project view installation instructions in GitLab by going to Settings > Build, expanding the Runners section, and clicking New Runner.
This will display the new runner page.
Choose your OS, add tags to indicate which pipelines should run on the new runners, add a description and click on Create Runner
You'll be presented with the command to run on the Linux server designated as a Gitlab runner
Register the runner against your project by running the command on your Linux server
sudo gitlab-runner register --url https://gitlab.com/ --registration-token GR1123456qwertDJMEKEIMDUwcLk
During the installation, you'll be prompted for which type of executor you'd like to run, in this example select shell the options available would be
Heading back to the Gitlab.com interface the runner(s) you've installed will be listed
Important things to remember when installing the runners
Tags are important, if you run multiple runners the pipelines can be directed to run on specific servers based on tags. Examples of tags could be Ubuntu, OpenSUSE or Ansible, terraform a runner can have multiple tags separated by a comma
Installing the gitlab runner creates a user gitlab-runner on the remote server which will be where (unsurprisingly) gitlab will run things.
In /home/gitlab-runner/ on the remote server remove .bash_logout if you don't you'll get an error like the following when you first run the pipeline code
The Runner of type Shell don't work: Job failed (system failure): preparing environment:
What have we done?
At this point we have
Setup a project in GitLab SaaS
Assign Gitlab Runners to the project
Setup tags and method of running.
The next question I had was how do I make the GitLab repository a pipeline to run things?
Setup a GitLab Pipeline
The .gitlab-ci.yml file
The pipeline is controlled by a file called .gitlab-ci.yml which is a YAML file that contains blocks with instructions to be run on the remote Gitlab Runner server as the user gitlab-runner
💡 there is a dot at the front of the filename .gitlab-ci.yml, which makes it a hidden file on a Linux system
In your project repo you want to run the pipeline by creating an empty file called .gitlab-ci.yml
This can be done either in a cloned copy of the project repo or directly in the GitLab web editor which if you use there are lots of example templates.
The shell template for example looks as follows
# You can copy and paste this template into a new `.gitlab-ci.yml` file.
# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword.
# To contribute improvements to CI/CD templates, please follow the Development guide at:
# This specific template is located at:
# See https://docs.gitlab.com/ee/ci/yaml/index.html for all available options
# you can delete this line if you're not using Docker
- echo "Before script section"
- echo "For example you might run an update here or install a build dependency"
- echo "Or perhaps you might print out some debugging details"
- echo "After script section"
- echo "For example you might do some cleanup here"
- echo "Do your build here"
- echo "Do a test here"
- echo "For example run a test suite"
- echo "Do another parallel test here"
- echo "For example run a lint test"
- echo "Do your deploy here"
For my automation I want the following to happen
Download the repo
Run ansible code
However to show an example the pipeline runs two blocks a test block and a deployment block.
We could run pre and post execution blocks to set things up, and build blocks.
If for example, I was running some automation to create a docker image I could have stages to build, test, upload and deploy
The code looks like this:
- echo "Do another parallel test here"
- echo "For example run a lint test"
- ansible-playbook movefiles.yaml
So what's happening here?
The first test1 block runs on the server1 gitlab-runner
💡 This is how the tags are used when setting up the gitlab runner
It runs 2 bash commands on the server1 gitlab-runner server
If the test is successful (admittedly it's not much of a test, it's just an example) deploy1 is run.
If the test1 block failed for any reason the deploy1 block won't run.
The second deploy1 block again runs on the server1 gitlab-runner server and executes the ansible-playbook code.
💡 To stop it from running each time I update the code in the GitLab repo I've added the only: section to instruct to only run the pipeline on a schedule.
Commit this code using
git add .
git commit -m "first pipeline upload"
Head to Build -> Pipeline and click on Run Pipeline (blue button top Right)
Inevitably the first few runs of these pipelines are going to have problems and you'll want to troubleshoot and see what is happening on the remote server when the code is run.
To troubleshoot head over to Build -> Jobs and you'll see what is working and what is not.
Clicking on a job block will display the output of the job
At this point, you should have a working pipeline and you'll want to run it on a schedule, this is done by heading to build-> Pipeline Schedules and clicking on New Schedule
This will display a pretty simple form which is based on the CRON syntax for times, as we are working out of master leave the branch the same and click on Save Pipeline Schedule.
This will display the pipeline
And you're done.
💡 A mail will be sent out to the email associated with the GitLab user account you are using on success or fail.
So this was very high level, designed to get up and running. It provides examples of key areas which need to be set up to run Gitlabs build pipelines
Ideally, there is more to be done here
Run this all as code
Add a better testing block
Understand the runner executors more
What other options are available in gitlab-ci.yml file
Once I got my head around this and solved a few issues it's actually pretty powerful as a solution, I do like that I don't need to host it, it runs off YAML and will run anything which can be run in bash.