- At MeetBSD 2014 in San Jose, Craig Rodrigues had a discussion with Kevin Bowling, Jason Wolfe, Jeremy Singletary, Chris Christensen, and Jason Wolfe from Limelight Networks (LLNW) about the Jenkins cluster. Craig expressed interest in leveraging the knowledge and skills that the LLNW team has in setting up devops frameworks, such as Saltstack, for improving the ongoing maintenance and scalability of the Jenkins cluster in FreeBSD.
- Craig wrote this document, based on ideas that were discussed at MeetBSD.
##GOALS
- use a modern devops framework to setup and deploy Jenkins cluster for FreeBSD
- be able to shut down all the existing Jenkins nodes, and redeploy Jenkins on new machines, using the devops framework
- automate more of the ongoing maintenance and monitoring of the Jenkins nodes using the devops framework
- add more nodes to Jenkins framework very quickly via devops framework, to make it easier to increase scale the Jenkins cluster
- use the Jenkins cluster as an open source example that people can follow for how to use a modern devops framework to deploy a Jenkins cluster with FreeBSD
##DESCRIPTION OF JENKINS CLUSTER
The list of machines in the Jenkins cluster is here: https://wiki.freebsd.org/Jenkins/MachineList
- havoc.ysv.freebsd.org, chaos.ysv.freebsd.org, wreck.ysv.freebsd.org are machines of different CPU/RAM/disk. They are all running FreeBSD-CURRENT/amd64. These are the hypervisor hosts, and are running bhyve VM's.
- the Hypvervisor Hosts are running FreeBSD-CURRENT/amd64, and are updated via source upgrade/buildworld/installworld. This is necessary because bhyve is new so it is necessary to track the latest bhyve changes in kernel and userland to get fixes and new features.
- the Hypervisor Hosts have rc.d scripts for starting the bhyve VM's on boot. These scripts are stored in https://github.com/freebsd/freebsd-ci
There are several bhyve VM's that are running on top of the Hypervisor Hosts
- jenkins-9.freebsd.org is running FreeBSD-9/amd64. This is also the alias for http://jenkins.freebsd.org. This is running the main jenkins web server. Builds of stable/9 and stable/8 branches occur in this VM.
- jenkins-10.freebsd.org is running FreeBSD-10/amd64. Builds of head branch occurs in this VM.
- scan.freebsd.org is running FreeBSD-10/amd64, and is used to run the clang static analysis build on the head branch. The results are published at http://scan.freebsd.org
- freebsd-update is used to upgrade the OS in the bhyve VM's
The main Jenkins config is backed up in https://github.com/freebsd/jenkins-config via the SCM Sync Configuration Plugin. There are scripts in https://github.com/freebsd/freebsd-ci for booting the bhyve VM's mentioned above, running kyua, etc.
- devops framework should be something mainstream, and run under FreeBSD
- devops scripts for Jenkins cluster should be open source, preferably checked into https://github.com/freebsd/freebsd-ci
- devops framework should be able to do source/buildworld/installworld style of upgrades on the hypervisor hosts
- devops framework should be able to do freebsd-update style upgrades in the Jenkins VM's
- If changes to scripts are made in https://github.com/freebsd/freebsd-ci , then it should be easy via the devops framework to deploy the changed scripts to all the hosts
- devops framework should be able to report the OS version on each host, and report any CVE/security issues to jenkins-admin@freebsd.org
- devops framework should be able to run "pkg audit" on each host, and report any CVE/security issues to jenkins-admin@freebsd.org
- devops framework should be able to install/upgrade required packages on each host. Example packages would be jenkins, openjdk, nginx, postfix, sudo
- devops framework should be able to do an initial install, and subsequent updates to necessary config files. This includes rc.conf, /usr/local/etc/sudoers.d/jenkins-build, nginx.conf, web server certificates for https://jenkins.freebsd.org
- devops framework should be flexible so that in future we can add nodes running in VM's (QEMU, VirtualBox), or nodes which are bare metal on non-amd64 arches (i386, mips, arm64, etc.)
- devops framework should not be dependent on any people or scripts used by FreeBSD clusteradm. :)
##NICE TO HAVE
- devops framework should be able to do basic monitoring of Jenkins cluster, and report problems to jenkins-admin@freebsd.org. If this is out of the scope of the devops framework, then being able to quickly deploy a monitoring framework like Nagios via the devops framework would be desirable.