Skip to content

Instantly share code, notes, and snippets.

@prclibo

prclibo/ros_buildfarm_tutorial.md

Last active August 12, 2019 23:12
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save prclibo/90fc9587f8382c069a3fb525291d2a39 to your computer and use it in GitHub Desktop.
Save prclibo/90fc9587f8382c069a3fb525291d2a39 to your computer and use it in GitHub Desktop.
Download ZIP
ros_buildfarm_tutorial.md
Raw
ros_buildfarm_tutorial.md

Deploy a customized ROS buildfarm for your group

Setup environment

Setting up environment requires >= 3 machines, namely master, repo, and slave. Denote their IP is 123.123.123.1, 123.123.123.2 and 123.123.123.3.

Setup the master

  1. Install Jenkins and its dependencies Java 7. First do:
sudo apt-get install openjdk-7-jre-headless

Next install Jenkins 2.32.3 from e.g.

wget https://pkg.jenkins.io/debian-stable/binary/jenkins_2.32.3_all.deb
sudo dpkg -i jenkins_2.32.3_all.deb

Note that newer Jenkins version like 2.60+ does not work after configuration of the enviroment and results in a HTTP 503 error.

After the Jenkins and started (should automatically), visit 123.123.123.1:8080 via your browser and setup your Jenkins. Install all the plugins recommended and create your administrator account.

  1. Clone the environment setup config and modify it:
git clone https://github.com/ros-infrastructure/buildfarm_deployment_config.git

Modidy master/common.yaml. Configure the following values.

  • jenkins::slave::ui_user:
  • jenkins::slave::ui_pass:
  • jenkins::slave::masterurl: The URL of the master, http://123.123.123.1:8080 here.
  • master::ip: The IP of the master, 123.123.123.1 here.
  • repo::ip: The IP of the repo server, 123.123.123.2 here.
  • user::admin::name: The admin username of Jenkins, e.g. we use admin here.
  • user::admin::password_hash: The password hash of the admin user. Find it from the passwordHash tag in /var/lib/jenkins/users/admin/config.xml.
  • ssh_keys: The public keys with which you can access the master server.
  • jenkins::private_ssh_key: generate a private key and save it here. This key will be set to /var/lib/jenkins/.ssh/id_rsa and remember to authorized its corresponding public key in repo server.

See https://github.com/ros-infrastructure/buildfarm_deployment/README.md for reference.

  1. Install required dependencies, by
sudo ./install_prerequisites.bash
  1. Do the configuration
sudo ./reconfigure.bash master

The last step with log Running puppet might take some time. In case of debug, edit ./reconfigure.bash and change -v in the last line to -VV -d, which means dumpping more verbose log with also debug info. Logs are dumpped into /var/log/puppet.log.

  1. Install the missing swarm-client. A jenkins-slave account will be created on your system. Check sudo ls -l /home/jenkins-slave/swarm-client-1.22-jar-with-dependencies.jar. You will find that the file is of size 0. This is because its download link is broken. Re-download it by
wget -O /home/jenkins-slave/swarm-client-1.22-jar-with-dependencies.jar https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/swarm-client/1.22/swarm-client-1.22-jar-with-dependencies.jar
  1. Re-run the configuration after swarm-client is downloaded.
sudo ./reconfigure.bash master

If things go right, you will see the jenkin-slave running by sudo service jenkins-slave status.

  1. Log in admin to your Jenkins master server by browse http://123.123.123.1:8080. In "Manage Jenkins/Configure Global Security" config. Check "Enable security" and select "TCP port for JNLP agents" as "Random". See https://stackoverflow.com/questions/38969044/jenkins-kubernetes-jenkins-slave-node-is-offline for reference.

  2. If things go right, now in the Jenkins master page "Manage Jenkins/Manage Nodes", you can see the "master" computer and the "slave_on_master" computer. If it does not appear, check the following might give hint.

sudo service jenkins-slave status
sudo vim /var/log/jenkins-slave/jenkins-slave.log

Setup the repo server

  1. Clone the environment setup config and modify it:
git clone https://github.com/ros-infrastructure/buildfarm_deployment_config.git

Modidy repo/common.yaml. Configure the following values.

  • jenkins::slave::ui_user:
  • jenkins::slave::ui_pass:
  • jenkins::slave::masterurl: The URL of the master, http://123.123.123.1:8080 here.
  • master::ip: The IP of the master, 123.123.123.1 here.
  • ssh_keys: The public keys with which you can access the repo server. Recall when setting the master server we generated a private_ssh_key. Now put its corresponding public key here for the access to jenkins-slave user.
  • jenkins-slave::reprepro_updater_config: Select the targeted architecture architectures. For examples, since I only care about x64 systems, I set it to amd64. The given example config uses 'E5944336 0776D936' as the signing_keys. The later reconfigure will create E5944336 key but I do not know where 0776D936 key is from. So to signing work I change 'E5944336 0776D936' to E5944336. Note that ' is also removed since there seems a bug that it cannot be handled correctly.
  1. Install required dependencies, by
sudo ./install_prerequisites.bash
  1. Do the configuration
sudo ./reconfigure.bash master

The last step with log Running puppet might take some time. In case of debug, edit ./reconfigure.bash and change -v in the last line to -VV -d, which means dumpping more verbose log with also debug info. Logs are dumpped into /var/log/puppet.log.

  1. Install the missing swarm-client. A jenkins-slave account will be created on your system. Check sudo ls -l /home/jenkins-slave/swarm-client-1.22-jar-with-dependencies.jar. You will find that the file is of size 0. This is because its download link is broken. Re-download it by
wget -O /home/jenkins-slave/swarm-client-1.22-jar-with-dependencies.jar https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/swarm-client/1.22/swarm-client-1.22-jar-with-dependencies.jar
  1. Re-run the configuration after swarm-client is downloaded.
sudo ./reconfigure.bash master

If things go right, you will see the jenkin-slave running by sudo service jenkins-slave status.

  1. If things go right, now in the Jenkins master page "Manage Jenkins/Manage Nodes", you can see the "master" computer and the "repo" computer.

Setup the slave server

  1. ssh_host_key

Configuring jobs on your master server

Some more dependencies

See https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst for reference. In summary you should install the following:

sudo apt update && sudo apt install python3 python3-all python3-pip

sudo pip3 install empy
sudo pip3 install jenkinsapi
sudo pip3 install rosdistro
sudo pip3 install ros_buildfarm

Provide credentials for Jenkins master

Save the Jenkins username and password in ~/.buildfarm/jenkins.ini so that the configuration code can access your Jenkins. See https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst#provide-credentials-for-jenkins-master for reference.

[http://jenkins-instance-url.example.com:8080]
username=admin
password=changeme

Customize your config file

Checkout https://github.com/ros-infrastructure/ros_buildfarm_config/ and modify index.yaml. See https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/basic_configuration.rst for reference. Basically you should set:

  • jenkins_url to your Jenkins address e.g. http://123.123.123.1:8080.
  • rodep-generator
  • custom_rosdep_url

Upload your customized ros_build_config somewhere online, e.g. push it to Github http://github.com/YOUR_FORK/ros_buildfarm_config. Do not use local address e.g. file:/// or http://localhost/, because jobs ran in docker can not resolve the local address.

Generate jobs

See https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/deploy_configuration.rst#generate-the-administrative-jenkins-jobs for reference. Clone a ros_buildfarm and run the generate_all_jobs.py.

git clone https://github.com/ros-infrastructure/ros_buildfarm/
cd ros_buildfarm/scripts
generate_all_jobs.py https://raw.githubusercontent.com/YOUR_FORK/ros_buildfarm_config/master/index.yaml --commit

Import upstreams

See https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/deploy_configuration.rst#import-packages for reference. Visit to your Jenkins on browser, trigger the job import upstream.

USE BACKFILL_ROS

Generate devel / release / doc jobs

  1. These jobs require to run a system Groovy script generated. Since Groovy 2.0 disallows using classpath to import external libraries. We should manually download some Java imports. See ros-infrastructure/buildfarm_deployment#147 for reference. In my case I need to download diffutils and apache-xml-xerces.jar into /var/lib/jenkins/plugins/groovy/WEB-INF/lib. Before downloading you might want to do su - jenkins to make sure jenkins owns the libraries. Then expand /var/lib/jenkins/scriptApproval.xml with the one mentioned in issue #147.

  2. Visit to your Jenkins on browser. Install plugin Dashboard View . Then trigger the job Krel_reconfigure-jobs.

  3. If something happens like

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod hudson.model.View createViewFromXML java.lang.String java.io.InputStream

then use https://stackoverflow.com/questions/38276341/jenkins-ci-pipeline-scripts-not-permitted-to-use-method-groovy-lang-groovyobject

I have approved

staticMethod hudson.model.View createViewFromXML java.lang.String java.io.InputStream
method hudson.model.ModifiableViewGroup addView hudson.model.View
another
hudson.model.View updateByXml javax.xml.transform.Source
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment