This is the third part of the three-part gist for setting up a basic CI environment on a single machine (e.g., a laptop), for dev / test / proof-of-concept, etc.
This CI environment consists of:
- Gitlab (Community Edition) as a source-code repository
- Jenkins for build and continuous integration Gitlab cannot be used without a mail server (Postfix). So, we also set up Postfix.
This part of the three-part gist outlines the steps to create a pipeline in Jenkins wherein the steps of a Continuous Integration process are properly configured. Here we explain:
- Reconfiguring Gitlab after installation
- Creating users and project repos in Gitlab
- Accessing Gitlab through SSH
- Installing Jenkins and accessing the Jenkins dashboard
For reference, please follow the three links below:
- Continuous Integration with Gitlab and Jenkins: Gitlab Setup
- Continuous Integration with Gitlab and Jenkins: Gitlab Configuration and Jenkins Setup
- Continuous Integration Pipeline with Jenkins and Gitlab
Here we describe the setup steps on Ubuntu 19.10 platform (Ubuntu Desktop), allocated 5GB RAM and 4 CPUs.(We actually set this up on a Ubuntu Virtual Machine on VirtualBox, running on Windows.)
A bash-like shell will be needed to run commands for the set-up.
The setup on other Linux flavors will be very similar.
Note: In our setup, the machine name (hostname) was archlab and the OS user running the commands was techie, included in the sudoers list. You will find these two string tokens in the rest of the gist. You should be able to replace these with suitable values for your environment.
In this set-up, the following versions have been used:
Gitlab: Community Edition 12.9.3
Jenkins: 2.222.1
We want to try out Continuous Integration feature in Jenkins. For this purpose we want to:
- Create a Java project repo in Gitlab CE
- Create a webhook in Gitlab so that a notification is sent to Jenkins whenever any code changes are pushed into this repo
- Create a CI pipeline in Jenkins that builds the project whenever any changes are pushed into this repo, using Gradle
Create an OS user named devusr on Ubuntru: sudo adduser devusr.
We will create a Java project in the home directory of devusr and build it using Gradle, just to be sure there are no compilation errors. Later, this project will be used for our CI experiment.
Note that JDK is already installed on the local machine, since Jenkins depends on JDK.
Now download Gradle and make it available for all users on this machine.
The latest version of Gradle (v 6.3) can be downloaded from this link.
Extract it into the /opt folder, and create a symlink for convenience: sudo ln -s /opt/gradle-6.3 /opt/gradle.
Set appropriate ownership, e.g., sudo chown -R <your_OS_user>:<your_OS_group> /opt/gradle.
To enable all OS users to invoke gradle, create a file /etc/profile.d/gradle.sh with the following content:
export GRADLE_HOME=/opt/gradle
export PATH=$PATH:${GRADLE_HOME}/bin
Assign executable permissions to this file: sudo chmod +x /etc/profile.d/gradle.sh
Login as Administrator into Gitlab Dashboard (root user with administrator password) and create a new user named devusr.
From the local mail inbox (~/.mail/new) retrieve the URL for devusr to access Gitlab dashboard for the first time. Access this URL, and set a password for the Gitlab user named devusr.
Now, as devusr, access Gitlab dashboard, and create a new (empty) project named demoproj.
Back to a Ubuntu terminal, login as OS user named devusr: su - devusr
Enable SSH access for devusr to Gitlab repos: run ssh-keygen -t rsa -C "devusr@<your_machine_name>" and do not supply any passphrase.
Copy the SSH public key (from file ~/.ssh/id_rsa.pub) into the 'Add_keys page on Gitlab dashboard.
Back in the Ubuntu terminal, run the following commands:
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
ssh-add -l
Now as devusr we want to initialize and build a project in our home directory, by the same name (demoproj). Then we will push the files in the project into Gitlab.
In the home directory (/home/devusr), create a sub-directory demoproj and navigate to the sub-directory: mkdir ~/demoproj & cd ~/demoproj
Now initialize the gradle project: gradle init. Choose the following options:
Select type of project to generate: <application>
Select implementation language: <java>
Select build script DSL: <groovy>
Select test framework: <JUnit4>
Project name (default: demoproj):
Source package (default: demoproj)
Create two Java files: src/main/java/demoproj/App.java and src/test/java/demoproj/AppTest.java.
- Source Code:
// App.java
package demoproj;
public class App {
public String getGreeting() {
return "Hello world.";
}
public static void main(String[] args) {
System.out.println(new App().getGreeting());
}
}
- Source Code:
// AppTest.java
package demoproj;
import org.junit.Test;
import static org.junit.Assert.*;
public class AppTest {
@Test public void testAppHasAGreeting() {
App classUnderTest = new App();
assertNotNull("app should have a greeting", classUnderTest.getGreeting());
}
}
Now, a directory structure like the following is created:
demoproj/
build.gradle
gradle
wrapper
gradle-wrapper.jar
gradle-wrapper.properties
gradlew
gradlew.bat
settings.gradle
src
main
java
demoproj
App.java
resources
test
java
demoproj
AppTest.java
resources
Actually there are some more hidden files, particularly, a .gitignore file suitable for a gradle project like this one. Run cat demoproj/.gitignore, and the output will be:
# Ignore Gradle project-specific cache directory
.gradle
# Ignore Gradle build output directory
build
Ensure that the project builds without any compilation errors or test failures: run ./gradlew build.
Set up the environment for the Git client:
git config --global user.email "devusr@<your_machine_name>"
git config --global user.name "devusr"
First login into Gitlab over SSH: ssh -T git@<your_machine_name>.
Now we have a project named demoproj in the home directory of devusr, and an empty project repo by the same name in Gitlab. So we should not clone from the Gitlab repo, but add the contents of project into Gitlab, instead, as follows: \
git init
git remote add origin git@archlab:devusr/demoproj.git
git add .
git commit -m "first commit"
git push -u origin master
On this occasion, the git push will not trigger a build in Jenkins. But we will now set up the CI pipeline, so that any subsequent updates to the source code does trigger a build.
This is a multi-step process that involves configuration in both Gitlab and Jenkins.
In response to a build trigger, Jenkins needs to clone a project repo from Gitlab. Therefore, Jenkins needs to access Gitlab using suitable credentials. A separate Gitlab user may be created solely for this purpose.
Login to Gitlab dashboard as Administrator (user root), and create a new Gitlab user named buildusr.
As usual, retrieve the URL for buildusr to access Gitlab dashboard for the first time, from your Postfix mailbox, and then set a Gitlab password for this user.
Jenkins will access Gitlab (to retrieve files from project repository) as buildusr (the Gitlab user created above). But it will access Gitlab through APIs, using a token. So, a token for buildusr has to be created in Gitlab first.
While logged into Gitlab as buildusr, click on the Profile icon on the top horizontal menu (extreme right), and then, from the left-hand panel select 'Access Tokens'.
Give your access token a name (say bldtoken), and select all the scopes. Then click on 'Create Personal Access Token'.
Copy and save the generated token, since it will have to be supplied into Jenkins UI.
Before creating a CI pipeline, we will plug in the access token for buildusr and create a _Jenkins_project.
Log in to Jenkins Dashboard. From the left-hand menu, select 'Manage Jenkins' and then 'Configure System'.
Scroll down to the Gitlab section and apply the following settings:
Enable Authentication for /project endpoint: Uncheck
Connection name: gitcon (you can supply any name)
Gitlab Host URL: https://<your_machine_name>
Credentials: Click on 'Add' and then 'Jenkins'. From the pop-up:
Domain: Global Credentials (unrestricted)
Kind: Username with password
Scope: Global (Jenkins ..)
Username: buildusr
Password: Gitlab password for buildusr
ID and description can be left blank.
Click on the 'Add' button.
Click on the 'Advanced' button and check 'Ignore SSL certificate Errors'.
Click on 'Add' button at the end to save the credentials (Gitlab access token).
Now we will create a Jenkins project within which a CI pipeline can be configured.
From the Jenkins dashboard, on the left hand top corner, click on the 'Jenkins' link and then 'New Item'.
Let Item name (i.e., project name) be demo, and select 'Freestyle Project'. Then click 'OK' button.
A Jenkins project named demo is created.
For our application, we will install the Gradle plugin for Jenkins.
From Jenkins dashboard, click on Jenkins on the top left, and then on 'Manage Jenkins'. Then, click on 'Manage Plugins'.
From the list of available plugins select and install the Gradle plugin.
From the Jenkins dashboard, click on 'Jenkins' (top left corner), and from the list of projects shown, select demo. Then click on 'Configure' on the left side, in order to set up a pipeline.
The pipeline consists of several elements. Follow the settings applied for each part of the pipeline, below:
-
General: In this section, set the 'Gitlab connection' to
gitcon, i.e., the connection created in Jenkins, using the Access Token ofbuildusr. -
Source Code Management: Select the 'Git' radio-button.
For 'Repository URL' specify :https://devusr@archlab/devusr/demoproj.git. This is the correct format for accessing the repo from Jenkins.
For 'Credentials' click on 'Add' and then on 'Jenkins'. In the pop-up, specify the following:
Domain: Global Credentials (unrestricted)
Kind: Username with password
Scope: Global (Jenkins ..)
Username:buildusr
Password: Gitlab password forbuildusr
ID and description can be left blank.
Click on the 'Add' button.
Click on the 'Advanced' button and check 'Ignore SSL certificate Errors'.
Click on 'Add' button at the end to save the credentials (Gitlab access token).
For 'Branches to build' specify*/master -
Build Triggers: Check 'Build when a change is pushed to GitLab'.
Important: Note the GitLab webhook URL:http://<your_machine_name>:3030/project/demoThis is required for creating webhook in Gitlab.
For 'Enabled Gitlab triggers' check only 'Push Events'.
Click on 'Advanced', and ensure that the following settings are configured:
Check 'Enable [ci-skip]'
Check 'Ignore WIP Merge Requests'
Check 'Set build description to build cause (eg. Merge request or Git Push )'
Allowed Branches: Select radio button 'Allow all branches to trigger this job' -
Build: Select the radio button: 'Use Gradle Wrapper'.
For 'Tasks' select 'Build' from the drop-down. -
Post-build Actions: Enable the option: 'Delete workspace when build is done'.
Click on the 'Save' button to save the pipeline configuration.
Back to the Gitlab UI, log in as the user (devusr) who created the Gitlab project named demoproj.
From the top horizontal menu select 'Projects' and then 'Your Projects'. from the list of projects, select devusr/demoproj.
Now from the left-hand panel, you can select 'Settings' and then 'Webhooks'. Apply the following settings:
URL: http://<your_machine_name>:3030/project/demo (i.e, the URL mentioned while configuring the pipeline in Jenkins - see above.)
Trigger: Check 'Push events' only
Uncheck 'Enable SSL Verification'.
Now, whenever there is a git push into demoproj in Gitlab, a build should be triggered in the demo CI pipeline in Jenkins.
Back to a Ubuntu terminal, as OS user devusr, make a small change in the demoproj source code:
In the file src/main/java/demoproj/App.java, change the greeting from: 'Hello world' to 'Hello new world'. Push the changes:
git add .
git commit -m "code changes"
git push -u origin master
From _Jenkins Dashboard_, click on demo on the list of projects. You will find a recently completed build with console output like the following:
Started by GitLab push by devusr
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/demo
using credential fd5cf3f7-cbc9-4a6b-9764-fd48819e1fa6
Cloning the remote Git repository
Cloning repository https://devusr@archlab/devusr/demoproj.git
> git init /var/lib/jenkins/workspace/demo # timeout=10
Fetching upstream changes from https://devusr@archlab/devusr/demoproj.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials https access
> git fetch --tags --force --progress -- https://devusr@archlab/devusr/demoproj.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://devusr@archlab/devusr/demoproj.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://devusr@archlab/devusr/demoproj.git # timeout=10
Fetching upstream changes from https://devusr@archlab/devusr/demoproj.git
using GIT_ASKPASS to set credentials https access
> git fetch --tags --force --progress -- https://devusr@archlab/devusr/demoproj.git +refs/heads/*:refs/remotes/origin/* # timeout=10
skipping resolution of commit remotes/origin/master, since it originates from another repository
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 9d7dc009ecc3cfe3a8e64e3a5b15c59431662276 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 9d7dc009ecc3cfe3a8e64e3a5b15c59431662276 # timeout=10
Commit message: "code updates2"
> git rev-list --no-walk 5b376c00b11feb3fac5cb2ab8fd076d302bbc1a2 # timeout=10
[Gradle] - Launching build.
[demo] $ /var/lib/jenkins/workspace/demo/gradlew build
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileJava
> Task :processResources NO-SOURCE
> Task :classes
> Task :jar
> Task :startScripts
> Task :distTar
> Task :distZip
> Task :assemble
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses
> Task :test
> Task :check
> Task :build
BUILD SUCCESSFUL in 20s
7 actionable tasks: 7 executed
Build step 'Invoke Gradle script' changed build result to SUCCESS
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
Finished: SUCCESS
For completeness, we specify the appropriate firewall settings: only HTTPS port 443 (nginx reverse-proxying Gitlab CE) and port HTTP 3030 (the port we set for Jenkins) should be accessible externally, apart from SSH access.
sudo ufw allow http
sudo ufw allow https
sudo ufw allow SSH
sudo ufw allow 3030