The project provides a pipeline DSL plugin that allows users to use buildpacks while developing a pipeline script in Jenkins.
- ABSTRACT
- CONTENT
- LINKS
- GENERAL INFO
- PHASES
- WHAT DID I DO ?
- HOW IT WAS DONE ?
- FUTURE WORKS
- CONCLUSION
GSOC 2021 JENKINS PLUGINS PROPOSAL - FATIH ATES
FULL DEFINED PIPELINE DSL
Buildpacks PLUGIN's README
Jenkins Buildpacks Plugin Repository
Latest Version(v0.1.1)
Mentor
Full Name: Javier ROMERO
E-mail: root@jromero.codes
GitHub: jromero
Web: https://why.jromero.codes/
Student
Full Name: Fatih ATES
E-mail: fatiiates@gmail.com
GitHub: fatiiates
Web: https://fatiiates.github.io/
-
Phase A user is able to provide a Buildpacks specific DSL that specifies a builder, a path, and image name which results in an image created as <image name>.
-
Phase A user should be able to fully use defined DSL expressions.
-
Phase What can be added in addition to DSL ? For example: colorful outputs, connected java-buildpack-client outputs to plugin...
-
Phase This phase, includes the integration of the constantly evolving Buildpacks features into the Jenkins Buildpacks plugin and making improvements and bug fixes on the plugin according to the constantly improving Jenkins system. This step never ends.
I developed this Jenkins plugin from scratch.
The repository is a java client for buildpacks developed by the snowdrop team.
Issues:
- snowdrop/java-buildpack-client#12 (Closed)
- snowdrop/java-buildpack-client#13 (Closed)
- snowdrop/java-buildpack-client#15 (Open)
PR:
- snowdrop/java-buildpack-client#14 (Under review)
This repository is the same PR sent to the java-buildpack-client repository. It is a repository created to include in pom.xml as if from a maven repository. Development was made on this repository until the PR was accepted. How it is included from Jitpack.io can be seen in this line.
Research Cloud Native Buildpacks
First, I sought answers to questions such as what Buildpacks are and how they work. Since I'm using the Heroku platform, I had some ideas about Buildpacks. To correct my misunderstandings and learn more about the tool, I first watched the talks on their page (especially the "Pack to the Future" presentation was very good), then read the documentation from top to bottom.
Setup Jenkins development environment and test Docker daemon access
After the research process was completed, we started the implementation. Jenkins, Docker, JDK 8 and Maven had to be installed on my computer in order to get into the implementation process. I used a machine with Ubuntu installed for development. I had Docker and JDK 8 on my machine as I used it before. Additionally I installed Jenkins and Maven. After installing the environment, I read the documents prepared about the Jenkins pipeline and got information about the pipelines. Then I moved on to the development step.
Being able to develop a plugin for Buildpacks was based on accessing the Docker socket. I tested that we can access the Docker socket by restricting the file permissions of the Docker socket, and when we restricted the permissions, we were getting an error, but if we gave 777, we could easily access it. Setting the permissions of the Docker socket file to 777 can compromise your system. So we solved this by including the Jenkins user in the Docker group. Then I gained some experience about Buildpacks, Docker, Jenkins pipelines by using sample applications.
Research for Docker daemon integration options (libraries)
We searched for references to be able to use docker with java codes in a Jenkins plugin. As a result of my research, we decided that people trying to directly access the docker socket are constantly having problems, and we described it as a worst case. Then we found the docker-workflow and docker-build-step plugins. We decided to use the docker-build-step plugin and then we completed the library search.
Define a DSL for the plugin
We would either start development directly or proceed by planning. We preferred to go with a plan and first of all we designed DSL. We created a DSL based on pack-cli's features and Buildpacks docs. We now had a fully defined DSL. However, it would be pointless to try to realize all these features in one, so we divided the DSL into phases.
Implement DSL and test integration
They have prepared a nice guide for developing a plugin. However, this only allows you to create a startup application. Basically two files are needed to define a DSL expression. The first is a Groovy script named provider(Buildpacks) and a Java class as providernameDSL(BuildpacksDSL). Groovy and Java compatibility allow us to pass any data in the pipeline to the Java class. I tested that we could get the parameters by printing the parameters to the console.
After passing the values to Java, we started working with containers. I tried to implement the system using the docker-build-step plugin that we found before. But no matter how hard I tried, I couldn't do it. I tried to implement it by doing research and development for a long time, but I could not succeed. We asked questions in the forums, tried to get support from developer groups. We even tried to reach the providers of some plugins. Then, one day, we got an e-mail that the mailing list java-buildpack-client was ready. I tested the demo application directly. It totally works. But before creating a dependency in this way, I did some research on docker-build-step for the last time. As a result of my research, I found that this plugin is more suitable for a step rather than a DSL pipeline. Since we couldn't see a sample plugin around using this plugin, we decided to eliminate this method.
Work with snowdrop community to make snowdrop/java-buildpack-client compatible with Jenkins
We chose to use java-buildpack-client but this required us to act with the Snowdrop community. Jenkins plugins were working with Java 8 while Java-buildpack-client was running with Java 11. This was preventing us from compiling the project. I have created an issue about whether the system can be adapted to Java 8. But they said they were busy and I would save time if I tried it myself, so I did. I struggled to get java-buildpack-client to be implemented in java 8. I was able to apply it to java 8 with a small change. Then I sent it to the repository as a PR.
However, there was still a problem. Logs are only printed to console and Jenkins was not able to capture them. I contacted the Snowdrop community again by opening an issue. After doing some research, I saw that they defined a logger and we implemented this logger in a class and transferred the logs to Jenkins.
Another problem was that including java-buildpack-client from a local maven repository was counterproductive. I sent them a new message asking when they could review the PR. Unfortunately they won't be able to review it anytime soon. We opened a new repository and transferred the client here and started to include it from the remote server with Jitpack.io.
Integrate snowdrop/java-buildpack-client with our Jenkins plugin
The java-buildpack-client was really well designed. The comments of the codes, the readme file, everything that can happen in a code were obvious. Now what I was going to do was pass the DSL variables as a parameter to the java-buildpack-client. Afterwards, we confirmed that the system is working in the development environment and we compiled an hpi file and tested it on the local Jenkins server. In this way, we ensured that the plugin was integrated into Jenkins, and then I prepared a comprehensive Readme.md file.
Add support for environment variables.
After we got a working plugin, we started adding new features. Undoubtedly, one of the most important features is environment variables. So we added this as the first feature.
This feature can be defined with two different parameters. env and envFile. The env parameter takes a string or a string expression and contains string data in the format "env1=test". envFile file takes a relative path according to github repository. The envFile file parameter gets a file path relative to the github repository containing the env variables. Env variables are supposed to strictly conform to the format. So I wrote a function to parse those that fit a format and those that don't. A dependency could have been added instead, but it's a simple thing anyway. If there was no problem with the values, it would take all the values to the local variable and then transfer them to the /platform/env directory in the container with the provide of java-buildpack-client. This is the process required for envs to be used during build.
After this feature was added, it was made available in v0.1.1
I would like to thank my mentor @jromero(my mentor) for his endless patience and helpfulness.
I would also like to thank @cmoulliard, @BarDweller, and @aemengo for their help.
Thank you Google.
Buildpack Jenkins Plugin made with ❤️
Phase-1 already completed here.
So, the next step is to transfer the features in pack-cli to the jenkins plugin, that is, to complete Phase-2. Some progress has been made in this phase, and the latest version can be found here.
After Phase-2 is completed, the process now comes to Phase-3. Phase-3, again, includes adding some features to the DSL from pack-cli, giving colored outputs in console outputs, connecting the logs of java-buildpack-client to jenkins and discussing what can be added.
When it comes to the last phase, named by Phase-4, the plug-in is operational with all DSL features. Only the routine checks of the plugin, the newly added features to the buildpacks system are also included in the plugin, and the innovations/requirements brought to the plugins by Jenkins are applied to the plugin. This step will continue as long as the ecosystem lives...
Terminal commands were already available in Jenkins. So this allowed you to use the pack command. However, when you want to write a pipeline script, it didn't give a chance to have a system that works compatible with jenkins. In addition, it was a non-recommended use.
Jenkins pipeline DSL Buildpacks plugin allows you to use buildpacks compatible with Jenkins. The base system(Phase-1) is currently available. What needs to be done is to fix the bugs and add new features.