Skip to content

Instantly share code, notes, and snippets.

@kennyhyun

kennyhyun/microk8s.md

Last active October 14, 2023 12:34
Show Gist options
  • Save kennyhyun/c56c53ff8d18516b126b7aea96912fc5 to your computer and use it in GitHub Desktop.
Save kennyhyun/c56c53ff8d18516b126b7aea96912fc5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
microk8s.md

Microk8s on Oracle Ampere Arm A1

What's that for

Kubernetes cluster for single machine

https://github.com/canonical/microk8s

Great tutorial

https://youtu.be/X48VuDVv0do?si=EXr4SGbll448VGZ4

How to install

Using snap

sudo snap install microk8s

Configuring k8s

commands

  • microk8s help
  • microk8s start
  • microk8s stop
  • microk8s kubectl

aliasing suggestion

try to set in .zshrc

  • mk: MicroK8s
  • kc: KubeCtl

Addons

on when installed

  • dns: CoreDNS
  • ha-cluster: Configure high availability on the current node
  • helm: the package manager for Kubernetes
  • helm3: the package manager for Kubernetes

available

  • cert-manager: Cloud native certificate management
  • community: The community addons repository
  • dashboard: The Kubernetes dashboard
  • host-access: Allow Pods connecting to Host services smoothly
  • hostpath-storage: Storage class; allocates storage from host directory
  • ingress: Ingress controller for external access
  • kube-ovn: An advanced network fabric for Kubernetes
  • mayastor: OpenEBS MayaStor
  • metallb: Loadbalancer for your Kubernetes cluster
  • metrics-server: K8s Metrics Server for API access to service metrics
  • minio: MinIO object storage
  • observability: A lightweight observability stack for logs, traces and metrics
  • prometheus: Prometheus operator for monitoring and logging
  • rbac: Role-Based Access Control for authorisation
  • registry: Private image registry exposed on localhost:32000
  • storage: Alias to hostpath-storage add-on, deprecated

Useful addons

  • minio: AWS s3 compatible storage
  • ingress: For public service
  • dashboard: UI for cluster status

Additional config

Worth to try to see if everything is okay

microk8s kubectl get all -n kube-system

will show some system status and they should be all ready

When it looked not good, try inspect command

$ microk8s inspect
...
WARNING:  IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
The change can be made persistent with: sudo apt-get install iptables-persistent
WARNING:  Docker is installed.
File "/etc/docker/daemon.json" does not exist.
You should create it and add the following lines:
{
    "insecure-registries" : ["localhost:32000"]
}
and then restart docker with: sudo systemctl restart docker

see also kubernetes/kubeadm#193 (comment) and kubernetes/minikube#4350 (comment)

Try it yourself

mongodb and mongo-express (Frontend)

You could try

  • pods
  • services
  • deployment
  • secret
  • config map
  • yaml configuration (infra as a software)

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  mongo-root-username: <base64 string>
  mongo-root-password: <base64 string>

commands

kubectl apply -f secret.yaml
secret/mongodb-secret created

mongodb.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-depl
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
        - name: mongodb
          image: mongo
          ports:
            - containerPort: 27017
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-username
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: g-root-password
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-svc
spec:
  selector:
    app: mongodb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017

commands

$ kubectl apply -f mongo.yaml
deployment.apps/mongodb-depl created
$ kc get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/mongodb-depl-84d7c8b6dd-tpqbd   1/1     Running   0          6m43s

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/kubernetes    ClusterIP   10.152.183.1     <none>        443/TCP     89m
service/mongodb-svc   ClusterIP   10.152.183.203   <none>        27017/TCP   3m47s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mongodb-depl   1/1     1            1           6m43s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/mongodb-depl-84d7c8b6dd   1         1         1       6m43s

mongo-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongo-configmap
data:
  database_url: mongodb-svc

Commands

$ kubectl apply -f mongo-configmap.yaml
configmap/mongo-configmap created

mongoexpress.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-express-depl
  labels:
    app: mongo-express
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo-express
  template:
    metadata:
      labels:
        app: mongo-express
    spec:
      containers:
        - name: mongo-express
          image: mongo-express
          ports:
            - containerPort: 8081
          env:
            - name: ME_CONFIG_MONGODB_ADMINUSERNAME
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-username
            - name: ME_CONFIG_MONGODB_ADMINPASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-password
            - name: ME_CONFIG_MONGODB_SERVER
              valueFrom:
                configMapKeyRef:
                  name: mongo-configmap
                  key: database_url
---
apiVersion: v1
kind: Service
metadata:
  name: mongo-express-svc
spec:
  selector:
    app: mongo-express
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081
      nodeProt: 30000

Commands

$ kubectl apply -f mongoexpress.yaml
deployment.apps/mongo-express-depl created
service/mongo-express-svc created

When all are up

$ kc get all
NAME                                      READY   STATUS    RESTARTS   AGE
pod/mongodb-depl-84d7c8b6dd-b4whl         1/1     Running   0          37s
pod/mongo-express-depl-84f7f67b89-zcqgs   1/1     Running   0          34s

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/kubernetes          ClusterIP      10.152.183.1     <none>        443/TCP          4h54m
service/mongodb-svc         ClusterIP      10.152.183.73    <none>        27017/TCP        37s
service/mongo-express-svc   LoadBalancer   10.152.183.186   <pending>     8081:30000/TCP   34s

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mongodb-depl         1/1     1            1           37s
deployment.apps/mongo-express-depl   1/1     1            1           34s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/mongodb-depl-84d7c8b6dd         1         1         1       37s
replicaset.apps/mongo-express-depl-84f7f67b89   1         1         1       34s

You can connect http://localhost:300000 for mongo-express

That was good enough but

  • what if I want to serve it like https://mongoexpress.example.com/
    • yes, ingress (k8s nginx ingress controller)
  • can I also serve another service to http://app.example.com/ from same node?
    • yes, ingress as well
  • can I also serve another service to http://app.example.com/api from same node?
    • yes, ingress rules have many options
  • can I add and update ssl certificates automatically?
    • yes, install cert-manager
  • where is my database saved? is that persistent? --> stateful set

... To be continued

Raw
part2.md

Microk8s on Oracle Ampere Arm A1 part 2

Use https with automated certificates using Cert Manager

Trying to add/update ssl certificates automatically

and found a good article

https://www.bsmithio.com/post/cert-manager-zerossl/

enable cert-manager addon

$ microk8s enable cert-manager

Will install cert manager and namespace/cert-manager will be created

also you will see some guide

Cert-manager is installed. As a next step, try creating a ClusterIssuer
for Let's Encrypt by creating the following resource:

$ microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: public
EOF

Then, you can create an ingress to expose 'my-service:80' on 'https://my-service.example.com' with:

$ microk8s enable ingress
$ microk8s kubectl create ingress my-ingress     --annotation cert-manager.io/cluster-issuer=letsencrypt     --rule 'my-service.example.com/*=my-service:80,tls=my-service-tls'

Prepare ZeroSSL account and credentials

Before we get started, you'll need a ZeroSSL account Sign Up - ZeroSSL.

After you've created your account, on your ZeroSSL dashboard, navigate to the Developer tab and generate your EAB credentials.

Configure with yaml

zerossl.yaml

apiVersion: v1
kind: Secret
metadata:
  namespace: cert-manager # Must be the namespace cert-manager is installed in
  name: zerossl-eab
stringData:
  secret: <YOUR-HMAC-KEY-HERE>
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: zerossl
spec:
  acme:
    email: <your-email-address>
    server: https://acme.zerossl.com/v2/DV90
    externalAccountBinding:
      keyID: <YOUR-EAB-KID>
      keySecretRef:
        name: zerossl-eab
        key: secret
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: zerossl-account-key
    solvers:
    - http01:
        ingress:
          class: public
$ kubectl apply -f zerossl.yaml
secret/zerossl-eab created
clusterissuer.cert-manager.io/zerossl created

if you dont have enabled ingress yet,

$ microk8s enable ingress

then

$ microk8s kubectl create ingress my-secure-ingress --annotation cert-manager.io/cluster-issuer=zerossl --rule 'mongoexpress.example.com/*=mongo-express-svc:80,tls=mongo-express-svc-tls'
ingress.networking.k8s.io/my-secure-ingress created
$
$ kc describe ingress my-secure-ingress
Name:             my-secure-ingress
Labels:           <none>
Namespace:        default
Address:          127.0.0.1
Ingress Class:    public
Default backend:  <default>
TLS:
  mongo-express-svc-tls terminates mongoexpress.example.com
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  mongoexpress2.yeoyou.net
                            /   mongo-express-svc:80 (10.1.94.20:8081)
Annotations:                cert-manager.io/cluster-issuer: zerossl
Events:                     <none>
$
$ curl -H "host: mongoexpress.example.com" localhost
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
$

or you can also add ingress part in mongoexpress.yml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mongo-express-ingress
  annotations:
    cert-manager.io/cluster-issuer: zerossl
spec:
  ingressClassName: public
  rules:
    - host: mongoexpress.example.com
      http:
        paths:
          - backend:
              service:
                name: mongo-express-svc
                port:
                  number: 8081
            path: /
            pathType: Prefix
  tls:
    - hosts:
      - mongoexpress.example.com
      secretName: mongo-express-svc-tls

this will create/update certificates automatically

Publish your image into Container registry

$ microk8s enable registry

will provide your own docker container registry,

pull base image and build your own base image for your project

and you can publish a new project image to deploy into microk8s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment