Skip to content

Instantly share code, notes, and snippets.

@scottrigby

scottrigby/demo.md

Last active January 24, 2023 20:52
Show Gist options
  • Save scottrigby/0be557cdf4b60e1f6ccf7d9a7332dfdb to your computer and use it in GitHub Desktop.
Save scottrigby/0be557cdf4b60e1f6ccf7d9a7332dfdb to your computer and use it in GitHub Desktop.
Download ZIP
How to Structure GitOps Repos
Raw
demo.md

How to Structure GitOps Repos

  1. Let's get started

    $ tmp=`mktemp -d` && cd $tmp && pwd
/var/folders/7w/tx9gxzkd4p79y2nsj7b5dmdw0000gn/T/tmp.YvRWeerz

  2. Set up simple monorepo directories, preview, and add all in one commit

    $ for env in common stage prod; do
    for app in app1 app2 cert-manager ingress-nginx; do
      mkdir -p $env/$app
      touch $env/$app/.keep
    done
  done
  
$ tree
.
├── common
│   ├── app1
│   ├── app2
│   ├── cert-manager
│   └── ingress-nginx
├── prod
│   ├── app1
│   ├── app2
│   ├── cert-manager
│   └── ingress-nginx
└── stage
    ├── app1
    ├── app2
    ├── cert-manager
    └── ingress-nginx
  
$ git init && git add . && git commit -m "Adding app directories"
[main (root-commit) b6bad6e] Adding app directories
 12 files changed, 0 insertions(+), 0 deletions(-)

  3. Add some example manifests for app1, preview and commit just app1 directory

    $ echo 'apiVersion: v1
kind: Namespace
metadata:
  name: app1' > common/app1/ns.yaml

$ echo 'apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1
  namespace: app1
spec:
  selector:
    matchLabels:
      app: app1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80' > stage/app1/deploy.yaml

$ cp stage/app1/deploy.yaml prod/app1/deploy.yaml 

$ tree
.
├── common
│   ├── app1
│   │   └── ns.yaml
│   ├── app2
│   ├── cert-manager
│   └── ingress-nginx
├── prod
│   ├── app1
│   │   └── deploy.yaml
│   ├── app2
│   ├── cert-manager
│   └── ingress-nginx
└── stage
    ├── app1
    │   └── deploy.yaml
    ├── app2
    ├── cert-manager
    └── ingress-nginx
    
$ git add . && git commit -m "Adding app1 manifests"
[main 2f2202d] Adding app1 manifests
 3 files changed, 44 insertions(+)
 create mode 100644 common/app1/ns.yaml
 create mode 100644 prod/app1/deploy.yaml
 create mode 100644 stage/app1/deploy.yaml

  4. Push the repo

    Could do this either before or after flux bootstrap, which creates the repo for you if your specified repo doesn't already exist.

    $ gh repo create gitops-repos-monorepo --private
✓ Created repository scottrigby/gitops-repos-monorepo on GitHub

$ open https://github.com/scottrigby/gitops-repos-monorepo

$ git remote add origin git@github.com:scottrigby/gitops-repos-monorepo.git
$ git branch -M main
$ git push -u origin main

  5. Botstrap flux components

    $ flux bootstrap github \
  --interval 10s \
  --owner scottrigby --personal \
  --repository gitops-repos-monorepo \
  --branch main \
  --path clusters/stage
Please enter your GitHub personal access token (PAT): 
► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/scottrigby/gitops-repos-monorepo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed sync manifests to "main" ("ec15492995f046c41820c5d58daceb99d1f9e74d")
► pushing component manifests to "https://github.com/scottrigby/gitops-repos-monorepo.git"
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBFbZIryvvxhmP7HCuPcSjYBQ4mkw0D5AHIi37ihiL/ssYvzhZhBXGUrdsgTAslyaEvhroHgVsBFmvZIUubggNX+lJFCdENMyOivmSZytvNrsoDCg0iMAApgxn2bw1hAbGQ==
✔ configured deploy key "flux-system-main-flux-system-./clusters/stage" for "https://github.com/scottrigby/gitops-repos-monorepo"
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("5bf7dc6c62a4b17d770347604aa74618ad245abc")
► pushing sync manifests to "https://github.com/scottrigby/gitops-repos-monorepo.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

    You can see this added the bootstrapped manifests used to run Flux itself

    $ git pull
$ tree
.
├── clusters
│   └── stage
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           └── kustomization.yaml
├── common
│   ├── app1
│   │   └── ns.yaml
│   ├── app2
│   ├── cert-manager
│   └── ingress-nginx
├── prod
│   ├── app1
│   │   └── deploy.yaml
│   ├── app2
│   ├── cert-manager
│   └── ingress-nginx
└── stage
    ├── app1
    │   └── deploy.yaml
    ├── app2
    ├── cert-manager
    └── ingress-nginx

  6. Tell Flux in our stage environment to reconcile the monorepo directory containing manifests common to all environments

    We're going to export this information into a custom resource that Flux understands, and put it inside our monorepo under cluster/stage because this is a configuration for Flux running in our staging environment.

    $ mkdir -p clusters/stage/app1
$ flux create kustomization common \
  --namespace flux-system \
  --source GitRepository/flux-system \
  --path "./common" \
  --interval 10s \
  --prune=true \
  --export > clusters/stage/apps-common-kustomization.yaml

  7. Now going to do the same for the monorepo directory containing manifests specific to staging environment only

    $ flux create kustomization stage \
  --namespace flux-system \
  --source GitRepository/flux-system \
  --path "./stage" \
  --interval 10s \
  --prune=true \
  --export > clusters/stage/apps-stage-kustomization.yaml

    Push that up to Git

    $ git add git add clusters/stage
$ git commit -m "Adding common and staging kustomizations"
$ git push

  8. Check it out

    $ flux get kustomizations -A  
NAMESPACE  	NAME       	REVISION    	SUSPENDED	READY	MESSAGE                        
flux-system	common     	main/5bf7dc6	False    	True 	Applied revision: main/5bf7dc6	
flux-system	flux-system	main/5bf7dc6	False    	True 	Applied revision: main/5bf7dc6	
flux-system	stage      	main/5bf7dc6	False    	True 	Applied revision: main/5bf7dc6

kubectl get deploy -n app1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
app1   1/1     1            1           12m

  9. Now do app2

    $ cp common/app1/ns.yaml common/app2
$ git add common/app2/ns.yaml
$ sed -i 's/app1/app2/g' common/app2/ns.yaml
$ git add -p
diff --git a/common/app2/ns.yaml b/common/app2/ns.yaml
index f39399b..b990c93 100644
--- a/common/app2/ns.yaml
+++ b/common/app2/ns.yaml
@@ -1,4 +1,4 @@
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: app1
+  name: app2
(1/1) Stage this hunk [y,n,q,a,d,e,?]? y

$ cp stage/app1/deploy.yaml stage/app2
$ git add stage/app2/deploy.yaml
$ sed -i 's/app1/app2/g' stage/app2/deploy.yaml
$ git diff
diff --git a/stage/app2/deploy.yaml b/stage/app2/deploy.yaml
index fe85006..c258099 100644
--- a/stage/app2/deploy.yaml
+++ b/stage/app2/deploy.yaml
@@ -1,17 +1,17 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: app1
-  namespace: app1
+  name: app2
+  namespace: app2
 spec:
   selector:
     matchLabels:
-      app: app1
+      app: app2
   minReadySeconds: 5
   template:
     metadata:
       labels:
-        app: app1
+        app: app2
     spec:
       containers:
       - name: nginx
$ cp stage/app2/deploy.yaml prod/app2/deploy.yaml
$ git add .

$ git commit -m "Adding app2 manifests"

$ watch kubectl get deployments.apps -n app2
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
app2   1/1     1            1           59s

    Let's take a look at our monorepo's Git commit history

    $ git log --oneline
8b03a81 (HEAD -> main, origin/main) Adding app2 manifests
5bf7dc6 Add Flux sync manifests
ec15492 Add Flux v0.28.5 component manifests
a7f51a3 Adding app1 manifests
47422e8 Adding app directories

  10. Time to make an example single app repo

    $ tmp=`mktemp -d` && cd $tmp && pwd
/var/folders/7w/tx9gxzkd4p79y2nsj7b5dmdw0000gn/T/tmp.OAXjBFux

$ git clone git@github.com:scottrigby/gitops-repos-monorepo.git gitops-repos-app1
$ cd gitops-repos-app1/

$ mkdir -p src deploy/common deploy/stage deploy/prod
$ touch src/hello-world.py
$ git add src
$ git commit -m "Adding app1 source files"

$ git filter-repo \
    --path src \
    --path-glob '*/app1' \
    --path-rename common/app1:deploy/common \
    --path-rename stage/app1:deploy/stage \
    --path-rename prod/app1:deploy/prod \
    --force

$ tree
tree
.
├── deploy
│   ├── common
│   │   └── ns.yaml
│   ├── prod
│   │   └── deploy.yaml
│   └── stage
│       └── deploy.yaml
└── src
    └── hello-world.py

$ git log --oneline
5ef26b7 (HEAD -> main) App 1 source files
3799600 Adding app1 manifests
6a5e966 Adding app directories

$ gh repo create gitops-repos-app1 --private --source=. --remote=origin
$ open https://github.com/scottrigby/gitops-repos-app1

  11. Temporarily suspend the monorepo kustomizations we're splitting out into app1

    If we don't do this first, we'll have multiple kustomizations competing over the same resources.

    $ flux suspend kustomization -n flux-system common
► suspending kustomization common in flux-system namespace
✔ kustomization suspended

$ flux suspend kustomization -n flux-system stage
► suspending kustomization stage in flux-system namespace
✔ kustomization suspended

$ flux get kustomizations -n flux-system 
NAME       	REVISION    	SUSPENDED	READY	MESSAGE                        
common     	main/8b03a81	True     	True 	Applied revision: main/8b03a81	
flux-system	main/8b03a81	False    	True 	Applied revision: main/8b03a81	
stage      	main/8b03a81	True     	True 	Applied revision: main/8b03a81

  12. Now create a source and kustomizations for our new app1 repo

    For this pattern, the source and kustomizations are usually stored in a separate config repo.

    First, create the secret in the cluster so Flux can access the new private app1 repo:

    $ flux create secret git app1-auth \
∙ --url=ssh://git@github.com/scottrigby/gitops-repos-app1 \
∙ --ssh-key-algorithm=ecdsa \
    --ssh-ecdsa-curve=p521
✚ deploy key: ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBABWl3QM+pGmHni5Ua9o8B5vS0vL59Bwww/Z79TD6HJFxkWEZaKyPDyKCj7YVRgwPBoZBjYVWt71Cg+zjaikIWhGsQA6Gx53buwFaEFXq0jlelrBNF+69vJXkVeWJ5gdImEIDXl6Q984Oae3Dweu09NpFdfNFDFJK2zzsocoqUNBu8k0Dw==

    Copy the deploy key output above and create a read-only GitHub deploy key in your repo.

    open https://github.com/scottrigby/gitops-repos-app1/settings/keys

    Now create the new repo source and kustomizations inside the initial monorepo, which is now our config cluster for the new per-app repos:

    $ flux create source git app1 \
--url ssh://git@github.com/scottrigby/gitops-repos-app1 \    
--branch main \
--secret-ref app1-auth \
--export > clusters/stage/app1/git-source.yaml

$ flux create kustomization app1-common \
  --namespace flux-system \
  --source GitRepository/app1 \
  --path "./deploy/common" \
  --interval 10s \
  --prune=true \
  --export > clusters/stage/app1/common-kustomization.yaml
$ flux create kustomization app1-stage \
  --namespace flux-system \
  --source GitRepository/app1 \
  --path "./deploy/stage" \
  --interval 10s \
  --prune=true \
  --export > clusters/stage/app1/stage-kustomization.yaml
$ git add clusters/stage/app1
$ git commit -m "Creating a source and kustomizations for new app1 repo"
$ git push

$ flux reconcile kustomization -n flux-system app1-stage

$ flux get kustomizations -A
NAMESPACE  	NAME       	REVISION    	SUSPENDED	READY	MESSAGE                        
flux-system	app1-common	main/a136159	False    	True 	Applied revision: main/a136159	
flux-system	app1-stage 	main/a136159	False    	True 	Applied revision: main/a136159	
flux-system	common     	main/6e13fc0	True     	True 	Applied revision: main/6e13fc0	
flux-system	flux-system	main/c1a588c	False    	True 	Applied revision: main/c1a588c	
flux-system	stage      	main/8b03a81	True     	True 	Applied revision: main/8b03a81

  13. Keep paused while repeating for all other apps. Once complete, remove the old kustomizations from the config repo

    $ git rm clusters/stage/apps-common-kustomization.yaml \
    clusters/stage/apps-stage-kustomization.yaml
$ git add .
$ git commit -m "Remove old apps-common and apps-stage kustomizations"
$ git push

    The old kustomizations will automatically be removed from the system

    $ flux get kustomizations -A                                  
NAMESPACE  	NAME       	REVISION    	SUSPENDED	READY	MESSAGE                        
flux-system	app1-common	main/795ba05	False    	True 	Applied revision: main/795ba05	
flux-system	app1-stage 	main/795ba05	False    	True 	Applied revision: main/795ba05	
flux-system	flux-system	main/45b2e03	False    	True 	Applied revision: main/45b2e03

  14. To merge multiple repos from one pattern to another, just reverse the above steps!

  15. Quick recap

    • This demo shows that with flexible tooling and techniques, you can iterate on repo structure – in this example, splitting apart a monorepo into separate per-app repos.
    • You can also do the same with repos per environment and repos per team, as well as reverse this by merging repos together into any of the above patterns. You can use a mix of these on running systems all at the same time.
    • You can do any of the above without downtime.
    • You can do this while retaining your Git commit history, for auditing purposes or just better developer experience 🎉

  16. Clean up by deleting your kind cluster and 2 private demo repos on GitHub (your local temp directories will auto-clean) 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment