- Introduction
- Prerequisite Tools
- Initial Cluster Setup
- Helm / Tiller
- Nginx Ingress
- Load Balancer
- Cert Manager
- Persistent Volumes
- Jenkins
- Harbor
- Kube Dashboard
- Conclusion
This guide will go over how to set up a single node Kubernetes cluster on DigitalOcean and is targeted towards those that have at least a basic understanding of Kubernetes architecture. In addition, it is assumed that the platform you are using is macOS, and that you're comfortable using the command line. We'll cover the specifics of setting up and configuring the following:
- Prerequisite Tools
- Initial Cluster Setup
- Helm/Tiller
- Nginx Ingress
- Load Balancer
- Cert Manager
- Persistent Volumes
- Jenkins
- Harbor (Private Docker / Helm Registry)
- Kubernetes Dashboard
First thing is first. In order to set up and manage our cluster, we need to start with the basics and install a few tools beforehand. Let's start with Homebrew, Doctl, and Kubectl. These will be the tools we use to interact with and manage our Kubernetes cluster.
If you're a Mac user then you're probably already familiar with this. If not however, Homebrew is a CLI package manager for Mac (and Linux). That's pretty much it. It allows you to install packages onto your machine and manage them with ease.
To set up Homebrew, simply run this command in your terminal.
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
This will pull the latest version of Homebrew and set everything up for you.
Note: Homebrew requires the system to have Xcode Command Line Tools installed. You may be prompted to install these. If you are, follow the instructions and you'll be good to go.
Once Homebrew is installed, you can invoke it with the brew command.
Doctl is a command line tool created by the DigitalOcean team. It is used to interact with your DigitalOcean account and manage your resources via the command line.
We can install this via, wait for it... Homebrew! Simply run the following command:
brew install doctl
In order to use doctl, we'll need to authenticate with DigitalOcean by providing an access token. This can be created from the Applications & API section of the DigitalOcean Control Panel.
Login to your DigitalOcean account and navigate to the Applications & API section. From here, select "Generate New Token", and then give the token a name. We'll need to grant the token read and write access to our resources.
Copy the newly generated token and run the following command in your terminal window:
doctl auth init
This will prompt for the auth token we just generated. Paste the token into the terminal session and doctl will authenticate with DigitalOcean. You should receive a confirmation message that the credentials were accepted:
Validating token: OK
Kubectl is the command line tool that we will be using to issue commands to our Kubernetes cluster. Anything we need to do to our cluster will be handled via this tool.
We can also use Homebrew to install this as well.
brew install kubernetes-cli
In order for kubectl to be able to interact with our cluster, we'll need to supply our cluster's credentials (config file). Since we do not yet have a cluster set up, we'll take care of this in the coming steps.
Now that we have our basic tools ready to go, we can initialize our cluster with DigitalOcean. This can be done with the DigitalOcean control panel, however we'll be using doctl for consistencies sake.
Note: At the time of writing, Kubernetes support for doctl is in beta. As such, the command's syntax may change. Please refer to the doctl documentation if the following commands do not work as intended.
Here are the specs that we want to configure our cluster with:
- Name: kube-cluster
- Region: NYC1
- Version: 1.13.2
- Size: 1 CPU, 2GB Ram
- Node Count: 1
This guide will use these values. However, feel free to change any of these and create the cluster to your own needs.
It is important to note that our Kubernetes cluster must be created in a region that also supports Block Storage Volumes. We'll use Block Storage for Jenkins, as well as Harbor. More on that later.
To create a cluster via the command line in accordance to these specs, we'd run the following:
doctl kubernetes cluster create kube-cluster \
--region nyc1 \
--version 1.13.2-do.0 \
--size s-1vcpu-2gb \
--count 1
Again, feel free to replace any of these parameters to suit your own needs. This will take several minutes to complete. Once complete, the command will have copied the cluster config to ~/.kube/config.
Once the cluster has been initialized, you will be able to download the config file so that kubectl can access the cluster.
doctl kubernetes cluster kubeconfig save kube-cluster
This command will copy the kubeconfig file to ~/.kube/config on your local machine. If you would like to see the contents of the kubeconfig file, you can run the command and replace save with show.
doctl kubernetes cluster kubeconfig show kube-cluster
Naturally, if you opted to name your cluster something other than kube-cluster you would replace the name with your own.
Whether the cluster's config file was copied automatically or manually, we'll need to tell kubectl to use the correct context so that it can access our cluster.
We can use kubectl to view the list of all available contexts.
kubectl config get-contexts
This will return a list of the servers we have in our ~/.kube/config file.
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
do-nyc1-kube-cluster do-nyc1-kube-cluster do-nyc1-kube-cluster-admin
Given that we only have one entry here, that's what we'll use. Set the context to use do-nyc1-kube-cluster.
kubectl config use-context do-nyc1-kube-cluster
Now any commands we use with kubectl will be issued to our DigitalOcean cluster.
Helm is a client side package manager for Kubernetes, while Tiller is the software that runs on the target cluster to manage said packages. In order to set up and run Helm and Tiller, we need to install Helm with Homebrew.
brew install kubernetes-helm
With this installed, we'll be able to issue commands via helm.
We're not quite done however. We still need to set up Tiller on our cluster. To do this we'll need to set up a Service Account, and Cluster Role Binding on our Kubernetes cluster. Run the following to take care of that.
kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
Once these are set up, we can initialize Helm and install Tiller onto our cluster.
helm init --service-account tiller
It will take a few minutes for the resources to become available, so sit tight.
If we're going to be exposing our applications and services on our cluster, we're going to need a way to access them. The way we do this is via an Ingress. In this specific instance, we'll be using an Nginx Ingress.
Note: This will also create a DigitalOcean Load Balancer; a paid service that is offered by DigitalOcean. At the time of writing, this is a base cost of $10/mo.
Installing the Ingress is easy. Using helm, all we have to do is run:
helm upgrade --install nginx-ingress --namespace nginx-ingress stable/nginx-ingress
That's it. Helm will set up everything for us. In order to access our applications we'll need to configure the Ingress. However, that will need to be done on an individual basis, so we'll save that for when we set up an application later.
In order for us to properly utilize our Load Balancer and Nginx Ingress, we need to point the incoming traffic from our domain name to the newly created Load Balancer. This can be done by mapping an A record on our domain name to the IP address of the Load Balancer.
Note: This requires your domain to be managed by DigitalOcean. How to handle this is outside of the scope of this guide. However, for more information please visit DigitalOcean's official documentation on the matter.
We can get Load Balancer information via doctl.
doctl compute load-balancer list -o json
We specify -o json because there is a large amount of information output to the screen regarding the available Load Balancers. Returning the output as JSON makes it easier to read. Simply locate the IP address of your Load Balancer and take note of it. If you want to grab only the IP address of the load balancer, we can filter the results with a tool called jq. This is a tool that allows traversal and manipulation of JSON objects via the command line. It can be installed with the command brew install jq.
doctl compute load-balancer list -o json | jq -r '.[0].ip'
Creating an A record for our domain name can also be done via the command line using doctl. We'll need our domain name, the address that we would like to forward, and the IP address of the Load Balancer we want to send the traffic to.
For example; my domain name is uptilt.io, the address that I want to forward is *.guide, and for the purposes of this guide, the IP address of the Load Balancer that I want to forward traffic to is 178.128.132.173. By forwarding *.guide as a wildcard, anything prefixed to guide.uptilt.io will be sent to the Load Balancer. This includes things like jenkins.guide.uptilt.io, and harbor.guide.uptilt.io.
Let's set up the record now.
doctl compute domain records create uptilt.io \
--record-type A \
--record-name *.guide \
--record-data 178.128.132.173 \
--record-priority 0 \
--record-ttl 1800 \
--record-weight 0
With this command, we're telling doctl to create a record on the uptilt.io domain, with a type of A. The record name is *.guide, and we're forwarding the traffic to the Load Balancer with an IP address of 178.128.132.173. Just like we wanted. Don't forget to replace the information with your own to set everything up properly.
When we host our applications, we'll want to host them with an applicable TLS certificate. Cert Manager is what will manage our certificates. It will issue and renew TLS certificates for our applications automatically so that we won't have to think about it. The Issuer we will be working with is Let's Encrypt. Let's Encrypt offers free TLS certificates and is incredibly easy to set up and use with Cert Manager.
We'll start by installing Cert Manager onto our cluster with helm.
helm upgrade --install cert-manager --namespace kube-system stable/cert-manager
As with the previous installations, it will take a few moments for the applicable resources to become available.
Simply installing the Cert Manager is not enough. We will need to configure Cert Manager to use our designated Issuer. This is done by setting up something called Cluster Issuers. Let's do that now.
We want to specify two Issuers; a staging issuer, and a production issuer. The reason for this is that the production issuer will have rate limits in effect and we do not want to hit the production issuer until we are ready to host our applications in a production environment.
Here's what a Cluster Issuer looks like:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: [PLACE_YOUR_EMAIL_HERE]
http01: {}
privateKeySecretRef:
name: letsencrypt-staging
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [PLACE_YOUR_EMAIL_HERE]
http01: {}
privateKeySecretRef:
name: letsencrypt-prod
We've got two Cluster Issuers here; letsencrypt-staging, and letsencrypt-prod. Both of which need a valid email in which to bind the certificates to. Copy this to a file called cluster-issuers.yaml and replace [PLACE_YOUR_EMAIL_HERE] with your email address.
Note: We'll be using the staging environment for this guide, however when everything is ready for production, it will need to be updated to use the production issuer.
After filling in the email, apply the configuration to the cluster.
kubectl apply -f /path/to/cluster-issuers.yaml
This will initialize the scaffolding we need to set up TLS certificates for our applications. Issuing the certificates will be handled when configuring the Ingress for each application.
The topic of Persistent Volumes (PV) and Persistent Volume Claims (PVC) is something that should be clarified and explained rather than set up all at once. In this guide, we'll need a few Persistent Volumes set up. We'll need a PV for Jenkins so that we can keep all of our configurations. We'll also need to set up multiple PVs for Harbor. The examples in this guide should give enough insight so that PVs can be set up for individual applications outside of the scope of this guide without much of a hassle.
Let's clarify a few things first. Kubernetes can create persistent volumes dynamically. Great, that's good, right? Sure. The downside? All of the PVs created by Kubernetes will have a hash for an ID, as well as for the name. This makes it a bit difficult to distinguish which PV is which.
Say we create a PV for Jenkins using Kubernetes, everything is taken care of and we don't have to worry about a thing. The PV gets created automatically and gets mounted when the container is created. Great, right? Eh... Kind of. Going further, what happens when we need to spin up a new cluster for some reason or another? Our PV is named with a hash and thus we don't know which PV belongs to our Jenkins instance without going through our configs.
PVs by nature are meant to be long term storage. This means outliving our Kubernetes clusters. With this in mind, it would be a good idea to create our PVs on our own beforehand, and then attach them to our clusters as needed. Doing it this way will simplify maintaining our PVs in the future.
Setting up an existing PV however is not as straightforward is it might seem, at least at the moment with DigitalOcean. What we'll need to do is create Block Storage Volume in DigitalOcean. Afterwards in Kubernetes, we'll need a custom PV, as well as a PVC for our existing storage.
Creating a Block Storage Volume can be done from the command line via doctl.
doctl compute volume create my-volume-name \
--region nyc1 \
--fs-type ext4 \
--size 5GiB
Note: It's important to clarify that in order for existing storage to be accessible by a cluster, it must exist in the same region as the cluster.
To use an existing volume in a Kubernetes cluster, we need to create a custom PV, along with a PVC (Persistent Volume Claim) and apply the configuration to the cluster. Here's an example of a yaml file with both:
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-volume-name
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: fbf24f32-15e9-11e9-a0e0-0a58ac1442e8
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-volume-name
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: do-block-storage
There are a couple of key points in this configuration. Let's break it down and go over each of them.
| Key | Explanation |
|---|---|
spec.persistentVolumeReclaimPolicy |
This should be set to Retain so that our volumes are not deleted when they are removed from the cluster. |
spec.capacity.storage |
This must match the size of the target Block Storage Volume in DigitalOcean. |
spec.accessModes |
This must be set to ReadWriteOnce as DigitalOcean does not currently support any other mode. |
spec.csi.fsType |
This must match the filesystem format of the target Block Storage Volume in DigitalOcean. |
spec.csi.volumeHandle |
This is the ID of the target Block Storage Volume in DigitalOcean. |
spec.csi.volumeAttributes |
This must have com.digitalocean.csi/noformat: "true" as an attribute so that the storage driver does not format the data that currently exists on the volume. |
Note: The Volume ID/Handle can be found by running doctl compute volume list. The Volume ID/Handle is not the name of the Block Storage Volume.
| Key | Explanation |
|---|---|
spec.accessModes |
This must be set to ReadWriteOnce as DigitalOcean does not currently support any other mode. |
spec.resources.requests.storage |
This must match the size of the target Block Storage Volume in DigitalOcean. |
It is also important to note that metadata.name must be the same value for both of these resources. (At least a far as I'm aware).
This is more of a general overview of how to handle Persistent Volumes and Persistent Volume Claims in Kubernetes. If this is a bit confusing, sit tight and we'll cover the specifics for setting up our Jenkins instance, as well as Harbor next.
If we're going to be managing a Kubernetes cluster, we'll need a way to manage a CI/CD pipeline. While this guide does not go into specifics of how to implement a CI/CD pipeline, we'll be setting up a popular CI/CD tool; Jenkins.
Note: DigitalOcean charges for Block Storage Volumes. Make sure to check their pricing scheme for the rates that will incur as a result. Currently, at the time of writing, the cost to host the 5GiB Block Storage Volume for Jenkins is $.50/mo.
As outlined in the previous section, we'll need a Persistent Volume to store our Jenkins configuration. Let's go ahead and set that up now.
Create a Block Storage Volume in DigitalOcean named pvc-jenkins, located in nyc1, and give it a size of 5GiB.
doctl compute volume create pvc-jenkins \
--region nyc1 \
--fs-type ext4 \
--size 5GiB
This command will return the information on the newly created volume. Take note of the ID associated with the volume.
ID Name Size Region Filesystem Type Filesystem Label Droplet IDs
688b54e7-2377-11e9-a0e0-0a58ac1442e8 pvc-jenkins 1 GiB nyc1 ext4
With the volume created in DigitalOcean, we'll need to set up a custom PV and PVC so that we may access the volume. Create a file named pvc-jenkins.yaml and paste the following:
apiVersion: v1
kind: Namespace
metadata:
name: jenkins
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc-jenkins
namespace: jenkins
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: [VOLUME_ID_GOES_HERE]
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-jenkins
namespace: jenkins
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: do-block-storage
Note: We specify the namespace jenkins here. This is because we will be installing our Jenkins instance into the jenkins namespace. We want to make sure that our persistent storage is in the same namespace, else we will not be able to access the volume.
Make sure to change [VOLUME_ID_GOES_HERE] with the ID of your own Block Storage Volume. Remember, this can be found by running doctl compute volume list.
Once the file has been created and the volumeHandle updated to reflect your own Block Storage Volume, apply the config with kubectl.
kubectl apply -f /path/to/pvc-jenkins.yaml
Now that we've got our persistent storage configured, let's move on to setting up the actual Jenkins instance. We'll be using helm to install Jenkins, and as such, we need to specify a config file to tweak it to our needs.
Create a jenkins-values.yaml file and paste the following contents.
fullnameOverride: "jenkins"
Master:
ImageTag: "lts-alpine"
ImagePullPolicy: Always
HostName: [YOUR_HOSTNAME_HERE]
ServiceType: ClusterIP
ServicePort: 8080
InstallPlugins:
- kubernetes:1.13.1
- workflow-job:2.31
- workflow-aggregator:2.6
- credentials-binding:1.17
- git:3.9.1
Ingress:
ApiVersion: extensions/v1beta1
Annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
ingress.kubernetes.io/secure-backends: "true"
TLS:
- secretName: jenkins-tls-staging
Persistence:
Enabled: true
ExistingClaim: pvc-jenkins
Size: 5Gi
rbac:
install: true
Let's go over these options so that we can better understand what's going on.
| Key | Explanation |
|---|---|
fullnameOverride |
This is simply what we want to call our Jenkins installation in Kubernetes. |
Master.ImageTag |
We're specifying lts-alpine here because it is a very lightweight package. At the time of writing, the full lts package weighs in at 322MB. The lts-alpine version is a nice 170MB. |
Master.ImagePullPolicy |
We set this to Always so that we don't have to worry about any weird caching issues with images. |
Master.HostName |
This is where you will specify the domain you wish to host Jenkins on. For example, in the scope of this article, I'll be hosting Jenkins at jenkins.guide.uptilt.io. |
Master.ServiceType |
We want to set this to ClusterIP as we're using an external load balancer to take care of serving the application. |
Master.ServicePort |
This is simply the port we wish to use to access Jenkins. Since we're forwarding traffic through a load balancer, we can leave this as the default 8080. |
Master.InstallPlugins |
These are simply the plugins we want to install with our Jenkins instance. Feel free to install whichever plugins you need for your use case. |
Master.Ingress.Annotations |
Here is where we define the annotations needed for our Ingress. These are mainly needed for TLS/SSL support. Take note of certmanager.k8s.io/cluster-issuer. The value we specify for this is letsencrypt-staging. This is telling the ingress to use our staging Cluster Issuer that we set up earlier. We'll get a dummy TLS certificate from Let's Encrypt so that we can make sure our setup works. When using this in a production environment, change the issuer to letsencrypt-prod so you can get a production TLS certificate. |
Master.Ingress.TLS |
We specify the key secretName here to tell our Cluster Issuer where to store the TLS certificate that is generated. This will use Kubernetes' secret store and it will place it in a secret with the name specified here. Naturally, if using a production environment, it would make sense to change the name to something along the lines of jenkins-tls-prod. |
Persistence.Enabled |
This tells Jenkins whether or not to use persistent storage. |
Persistence.ExistingClaim |
If using an existing PVC (we set this up earlier), this is where we specify the name of the PVC. The PVC must be in the same namespace as the Jenkins instance, otherwise it will not find the volume. |
Persistence.Size |
This tells Jenkins the size of the volume. If using an existing volume, set this to the size of the existing volume to avoid conflicts. |
rbac.install |
This tells Jenkins if we would like to use RBAC. This will grant Jenkins privileges to manage certain Kubernetes services. |
With all of this in mind, change Master.HostName to the host name of your choice, additionally tweaking some settings to your liking, and install Jenkins using helm. We must make sure to specify our config file as an override.
helm upgrade --install jenkins --namespace jenkins stable/jenkins -f /path/to/jenkins-values.yaml
After a few minutes, the Jenkins instance will be reachable at the host name specified. As an example, in the scope of this guide, the server is reachable at https://jenkins.guide.uptilt.io. Since we're using a staging TLS certificate, your browser may flag the TLS certificate as "unsafe". Make sure to add the TLS certificate to the list of trusted certificates if this happens.
Now that we can access our Jenkins instance, we need to be able to log into the server. At the bottom of the output from the install command, it says that we need to run a command to obtain our admin password.
printf $(kubectl get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
Run this command and log in using the username admin and the password returned from the command. Once this is taken care of, our Jenkins instance is officially up and running!
In addition to Jenkins, we're also going to be setting up a private Docker and Helm registry. We're going to use an application called Harbor to do this. Harbor is an open source solution for hosting both of these for us on our Kubernetes cluster.
Installing Harbor isn't as intuitive as Jenkins, so let's walk through it step by step.
Note: DigitalOcean charges for Block Storage Volumes. Make sure to check their pricing scheme for the rates that will incur as a result. Currently, at the time of writing, the cost to host all of our Block Storage Volumes for Harbor is $1.30/mo.
Harbor requires a total of five volumes. We need a volume for the registry and chartmuseum, as well as the database, job service, and redis cache.
Let's go ahead and create the volumes using doctl.
doctl compute volume create pvc-harbor-registry --region nyc1 --fs-type ext4 --size 5GiB
doctl compute volume create pvc-harbor-chartmuseum --region nyc1 --fs-type ext4 --size 5GiB
doctl compute volume create pvc-harbor-jobservice --region nyc1 --fs-type ext4 --size 1GiB
doctl compute volume create pvc-harbor-database --region nyc1 --fs-type ext4 --size 1GiB
doctl compute volume create pvc-harbor-redis --region nyc1 --fs-type ext4 --size 1GiB
Once these volumes are created, we need to map the volumes to a PV/PVC in Kubernetes. Create a file named pvc-harbor.yaml and paste the following contents.
# Namespace - Harbor
apiVersion: v1
kind: Namespace
metadata:
name: harbor
---
# PV - Harbor Registry
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc-harbor-registry
namespace: harbor-registry
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: [VOLUME_ID_GOES_HERE]
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
# PVC - Harbor Registry
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-harbor-registry
namespace: harbor-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: do-block-storage
---
# PV - Harbor Chart Museum
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc-harbor-chartmuseum
namespace: harbor-registry
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: [VOLUME_ID_GOES_HERE]
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
# PVC - Harbor Chart Museum
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-harbor-chartmuseum
namespace: harbor-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: do-block-storage
---
# PV - Harbor Job Service
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc-harbor-jobservice
namespace: harbor-registry
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: [VOLUME_ID_GOES_HERE]
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
# PVC - Harbor Job Service
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-harbor-jobservice
namespace: harbor-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: do-block-storage
---
# PV - Harbor Database
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc-harbor-database
namespace: harbor-registry
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: [VOLUME_ID_GOES_HERE]
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
# PVC - Harbor Database
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-harbor-database
namespace: harbor-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: do-block-storage
---
# PV - Harbor Redis
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc-harbor-redis
namespace: harbor-registry
annotations:
pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
storageClassName: do-block-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
csi:
driver: dobs.csi.digitalocean.com
fsType: ext4
volumeHandle: [VOLUME_ID_GOES_HERE]
volumeAttributes:
com.digitalocean.csi/noformat: "true"
---
# PVC - Harbor Redis
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-harbor-redis
namespace: harbor-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: do-block-storage
Don't forget to replace [VOLUME_ID_GOES_HERE] with the corresponding ID applicable to your account. Remember, these can be found using doctl compute volume list. Once that has been taken care of, apply the changes with kubectl.
kubectl apply -f /path/to/pvc-harbor.yaml
Now we can take care of installing Harbor.
Harbor's helm chart is maintained by the community and is not listed in the default stable repository. Instead, to install Harbor we need to clone the git repository and install the chart from our local machine. Before we can install Harbor however, we need to define a config file in a similar way that we did with Jenkins. Start by cloning the repository and setting the active branch to version 1.0.0.
git clone https://github.com/goharbor/harbor-helm && cd harbor-helm
git checkout 1.0.0
Let's define our config file as well. Copy the following into a file named harbor-values.yaml on your local machine.
imagePullPolicy: Always
externalURL: [EXTERNAL_URL_GOES_HERE]
harborAdminPassword: "Harbor12345"
secretKey: "VL6EiEC2uyQN525Q"
expose:
type: ingress
tls:
enabled: true
secretName: "harbor-registry-tls-staging"
notarySecretName: "harbor-notary-tls-staging"
ingress:
hosts:
core: [CORE_URL_GOES_HERE]
notary: [NOTARY_URL_GOES_HERE]
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
persistence:
enabled: true
resourcePolicy: "keep"
persistentVolumeClaim:
registry:
existingClaim: "pvc-harbor-registry"
storageClass: "-"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
chartmuseum:
existingClaim: "pvc-harbor-chartmuseum"
storageClass: "-"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
jobservice:
existingClaim: "pvc-harbor-jobservice"
storageClass: "-"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
database:
existingClaim: "pvc-harbor-database"
storageClass: "-"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
redis:
existingClaim: "pvc-harbor-redis"
storageClass: "-"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
There are a few values here that meed to be changed to match your individual configuration. Let's go over what we're configuring.
| Key | Explanation |
|---|---|
imagePullPolicy |
We set this to Always so that we don't have to worry about any weird caching issues with images. |
externalURL |
This is the URL that you wish to access the registry from. For example, in the scope of this guide, the URL would be https://harbor.guide.uptilt.io. |
harborAdminPassword |
This is the default administrator password that is set when the application is first set up. This can be set to whatever you like, however it would be a good idea to change the administrator password from the application once it is up and running. |
secretKey |
This value is used for encryption and should be set to a random 16 character string. |
expose.type |
This tells the helm chart how we want to expose our registry. We're going with ingress for this one. |
expose.tls.enabled |
This tells the helm chart that we would like to use TLS encryption. |
expose.tls.secretName |
This is the name of the secret where the TLS certificate will be stored. |
expose.tls.notarySecretName |
This is the name of the secret where the notary TLS certificate will be stored. |
expose.ingress.hosts.core |
This is the domain name that the core application will be accessible from, without the protocol prefix. For example, in the scope of this guide, the domain would be harbor.guide.uptilt.io. |
expose.ingress.hosts.notary |
This is the domain name that the notary will be accessible from, without the protocol prefix. For example, in the scope of this guide, the domain would be notary.harbor.guide.uptilt.io. |
expose.ingress.annotations |
These are the annotations that will be passed to the ingress when the application is installed. |
persistence.enabled |
This specifies whether or not we would like to have our data persist beyond the life of the container. |
persistence.resourcePolicy |
This specifies what we would like to do with out persistent storage when the container's lifecycle is over. Naturally, we want this set to keep. |
persistence.persistentVolumeClaim.*.existingClaim |
This is the name of the PVC that we would like to use for the defined resource. The PVC must be in the same namespace as the installation. |
persistence.persistentVolumeClaim.*.storageClass |
This is the storage class that we would like to use if we are to leverage dynamically generated volumes. Since we are using existing volumes, this should be set to - so that it is ignored. |
persistence.persistentVolumeClaim.*.subPath |
This specifies a subdirectory to use to store the data on the volume. For our purposes we do not need this. |
persistence.persistentVolumeClaim.*.accessModes |
This specifies the access mode for the persistent volume. The only value that is valid here is ReadWriteOnce as it is the only value that DigitalOcean supports. |
persistence.persistentVolumeClaim.*.size |
This is the desired size of the volume. Since we are using existing volumes, this must match the size of the volume being used. |
Once everything has been configured as needed, we can proceed onward to installing Harbor.
helm upgrade --install harbor-registry --namespace harbor-registry ./path/to/harbor-helm -f /path/to/harbor-values.yaml
Note: You might notice the ./path/to/harbor-helm argument here. Since we are not using the stable repository, we must specify the path to the helm chart locally. The . prefix specifies to use a local chart, while /path/to/harbor-helm is the location of the git repository we cloned earlier.
This will install Harbor onto our Kubernetes cluster using the values we specified in the configuration section. Harbor takes a few more minutes to spin up than our previous installs, so don't fret.
Whenever Harbor finishes its initialization, we're free to log in using our administrator credentials we set in the config. Now would be a good time to change those as well. To log in to Harbor simply navigate to the externalUrl we specified in the config. For this guide, it would be https://harbor.guide.uptilt.io.
For users who prefer to have a visual representation of their Kubernetes cluster, there's the Kubernetes Dashboard UI. This is an easy one to install and get set up, so don't worry too much.
There are two steps to get going with the Kubeternetes Dashboard, the first is installing it via the helm chart.
helm upgrade --install kubernetes-dashboard --namespace kube-system stable/kubernetes-dashboard --set fullnameOverride="kubernetes-dashboard"
Simple enough. We're not quite done however. In order for us to access the application, we need to use a proxy. We can use the built in Kubernetes proxy to access the dashboard.
kubectl proxy
This will allow access to the Kubernetes cluster via http://localhost:8001. Having access to the cluster via the proxy, we can navigate to http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:https/proxy/ in our browser.
Now that we have our dashboard installed and have access to it via the proxy, we need to grant ourselves access. We're going to go the route of using tokens for access in this guide. To do that, create a file named dashboard-auth.yaml and paste the following.
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
Since there is nothing to configure in this snippet, simply apply the configuration with kubectl.
kubectl apply -f /path/to/dashboard-auth.yaml
This will set up a service account to use for the dashboard which we can use to log in. To get the token, we'll need to run a command.
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}') | grep 'token:' | awk -F " " '{print $2}'
A fairly lengthy command, but this will grab the token we need for the dashboard. If we want to copy it straight to the clipboard, just pipe pbcopy to the end of the command.
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}') | grep 'token:' | awk -F " " '{print $2}' | pbcopy
Paste the returned token into the dashboard and you're good to go!
That's it, we're finished! Pat yourself on the back and take your new Kubernetes cluster for a spin. In this guide, we've set up the required components for a basic Kubernetes cluster. We've gone though what it takes to spin up the cluster using DigitalOcean and have installed our package management software; Helm/Tiller. Our cluster now has an Nginx Ingress to serve our applications, as well as Cert Manager to make sure that we keep TLS encryption up to snuff. Jenkins and Harbor are configured exactly to our needs and can be used however we see fit. We even have persistent storage under control for easy long term management.
There's a lot of moving parts, but when it's broken down into smaller segments it's nowhere near as intimidating as it seems. From here you can use Jenkins and Harbor to set up your own CI/CD pipeline that suits your individual needs, or just dive right in and start deploying whichever applications you desire to the cluster.
Written by Rick Bennett
Last Updated: February 17th, 2019