Skip to content

Instantly share code, notes, and snippets.

@adiroiban
Last active April 9, 2024 14:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save adiroiban/a3a4444d77d68e372d5578e31548a1f0 to your computer and use it in GitHub Desktop.
Save adiroiban/a3a4444d77d68e372d5578e31548a1f0 to your computer and use it in GitHub Desktop.
GitHub Actions ARC for Windows
kubectl taint node YOUR-WIN-K8S-NODE --overwrite os=windows:NoSchedule
export INSTALLATION_NAME="arc-windows"
export NAMESPACE="arc-runners-windows"
kubectl create namespace "${NAMESPACE}"
kubectl create secret generic pre-defined-secret \
-n "$NAMESPACE" \
--from-literal="github_token=${GITHUB_TOKEN}"
helm install "${INSTALLATION_NAME}" --version "${CHEVAH_ARC_SET_VERSION}" \
--namespace "${NAMESPACE}" \
--create-namespace \
--values infrastructure/k8s/${NAMESPACE}-values.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
#
# Configuration for running the arc-runners-windows GHA runners.
#
#
# We install the runners only on the repo.
#
githubConfigUrl: "https://github.com/YOUR-GITHUB-ORG/YOUR-REPO"
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## We use a PAT stored in a separate secret
githubConfigSecret: pre-defined-secret
# We now have 2 bare-metal servers,
# they should be able to run 9 containers each at a peak.
maxRunners: 18
# We keep one running to check that ARC is working.
# Also, it allows easy access to a windows container for any minor testing.
minRunners: 6
## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
listenerTemplate:
spec:
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
nodeSelector:
'node-role.kubernetes.io/control-plane': ''
containers:
# # Use this section to append additional configuration to the listener container.
# # If you change the name of the container, the configuration will not be applied to the listener,
# # and it will be treated as a side-car container.
- name: listener
# securityContext:
# runAsUser: 1000
# # Use this section to add the configuration of a side-car container.
# # Comment it out or remove it if you don't need it.
# # Spec for this container will be applied as is without any modifications.
# - name: side-car
# image: example-sidecar
## template is the PodSpec for each runner Pod
## For reference:
## https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
spec:
runtimeClassName: windows-2022
nodeSelector:
node.kubernetes.io/windows-build: 10.0.20348
# To debug a bare-metal node, you can schedule all runners on that
# node.
#kubernetes.io/hostname: quise
# Try to spread the pod across all nodes,
# but don't try to hard in order to avoid evicting a pod during a job run.
topologySpreadConstraints:
- maxSkew: 3
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
nodeAffinityPolicy: Honor # or Ignore
nodeTaintsPolicy: Honor # or Ignore
labelSelector:
matchLabels:
actions.github.com/scale-set-name: arc-windows
containers:
- name: runner
image: ghcr.io/YOUR-GITHUB-ORG/arc-runner-windows:2.313.0.x64.1
command: ["run.cmd"]
args: ["--jitconfig", "$(ACTIONS_RUNNER_INPUT_JITCONFIG)"]
env:
- name: CHEVAH_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CHEVAH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# The runner should be updated via the general image.
- name: DISABLE_RUNNER_UPDATE
value: "true"
volumeMounts:
- name: hostedtoolcache
mountPath: /opt/hostedtoolcache
volumes:
- name: hostedtoolcache
hostPath:
# directory location on host
path: "C:/opt/hostedtoolcache"
type: DirectoryOrCreate
# This is the docker file for building our GHA runner with servercore.
#
# Current size about 5.1GB
#
# We build this from Linux using Docker server on Windows.
# The final RUN is called by k8s.
FROM mcr.microsoft.com/windows/servercore:ltsc2022
LABEL maintainer="Chevah Project <dev@chevah.com>"
LABEL name=arc-runner-windows
# The "PLATFORM" argument is created to allow injecting it into the
# build environment.
# In this we can share the build scripts between X64 and ARM64.
ARG GHA_VERSION=2.313.0
ENV GHA_VERSION=$GHA_VERSION
WORKDIR /actions-runner
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop';$ProgressPreference='silentlyContinue';"]
# Get Action runner.
RUN \
Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v${env:GHA_VERSION}/actions-runner-win-x64-${env:GHA_VERSION}.zip -OutFile actions-runner-win.zip ; \
Expand-Archive .\actions-runner-win.zip -DestinationPath . ;\
rm actions-runner-win.zip
# We should be ok with the default PowerShell, no need to update it.
# I left it here in case it's needed.
# RUN \
# Invoke-WebRequest -Uri 'https://aka.ms/install-powershell.ps1' -OutFile install-powershell.ps1; \
# ./install-powershell.ps1 -AddToPath
RUN powershell Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
RUN powershell choco install git.install --params "'/GitAndUnixToolsOnPath'" -y
RUN powershell choco feature enable -n allowGlobalConfirmation
#
# This is used to manually install k8s on a Windows bare metal.
#
# DISM /Online /Enable-Feature /All /FeatureName:Microsoft-Hyper-V
# DISM /Online /Enable-Feature /All /FeatureName:Containers
# DISM /Online /Enable-Feature /All /FeatureName:VirtualMachinePlatform
# DISM /Online /Enable-Feature /All /FeatureName:HypervisorPlatform
Add-WindowsFeature Containers,Hyper-V,Hyper-V-Tools,Hyper-V-PowerShell -Restart -IncludeManagementTools
# At this point it will restart.
# Create cache dir
mkdir -f c:\opt\hostedtoolcache
# Install Containerd
mkdir -f c:\temp
cd c:\temp
mkdir -f containerd
cd containerd
curl.exe -k -L https://github.com/containerd/containerd/releases/download/v1.6.26/cri-containerd-cni-1.6.26-windows-amd64.tar.gz -o containerd-windows-amd64.tar.gz
tar.exe xvf .\containerd-windows-amd64.tar.gz
cd ..
mv containerd $Env:ProgramFiles\containerd
cd $Env:ProgramFiles\containerd\
Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"
.\containerd.exe config default | Out-File config.toml -Encoding ascii
.\containerd.exe --register-service
Start-Service containerd
# Install containerd tools and check Containerd install is ok
cd c:\temp
curl.exe -kL https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.29.0/crictl-v1.29.0-windows-amd64.tar.gz -o crictl.tar.gz
tar xvf crictl.tar.gz
mv crictl.exe "$Env:ProgramFiles\containerd"
# Add ctr and crictl to your path
[Environment]::SetEnvironmentVariable("Path", "$($env:path);C:\Program Files\containerd", [System.EnvironmentVariableTarget]::Machine)
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine")
mkdir -f $env:UserProfile/.crictl/
echo "
runtime-endpoint: npipe:////./pipe/containerd-containerd
image-endpoint: npipe:////./pipe/containerd-containerd
timeout: 2
debug: true
pull-image-on-create: false" | out-file $env:USerProfile/.crictl/crictl.yaml
# Check containerd and crictl
ctr image pull mcr.microsoft.com/windows/nanoserver:ltsc2022
crictl pull mcr.microsoft.com/windows/nanoserver:ltsc2022
mkdir -f c:\k
echo "YOUR K8S
CONFIG FILE HERE
ALL ALL IT" | out-file C:\k\config -Encoding ascii # noqa:text
# Install Calico
#
# Remove default containerd confi
rm 'C:\Program Files\containerd\cni\conf\0-containerd-nat.conf'
Invoke-WebRequest https://github.com/projectcalico/calico/releases/download/v3.27.0/install-calico-windows.ps1 -OutFile install-calico-windows.ps1
.\install-calico-windows.ps1 -KubeVersion 1.28.4 -ServiceCidr 10.96.0.0/12 -DNSServerIPs 10.96.0.10
# Setup Kubelet
C:\CalicoWindows\kubernetes\install-kube-services.ps1
Start-Service -Name "kubelet"
Start-Service -Name "kube-proxy"
# DONE
# At this point check
# $ kubectl get nodes
# To see if it added, and mark the node for no default schedule.
# $ kubectl taint node quise --overwrite os=windows:NoSchedule
# Initial time when you will pull an ServerCore image, it will take a lot,
# as it will download the whole Windows OS.
#
@denny-snapone
Copy link

Do you run it with docker dind mode?

@adiroiban
Copy link
Author

This is using Hyperv+containerd for native Windows containers on bare-metal server.

I have added the script I used to install containerd + k8s + calico on Windows Server.

Calico only works with Windows Server. Don't try on WIndows 11 or any other Desktop version of Windows.


It uses the old Calico install mode...by now, the host process install might be best

@denny-snapone
Copy link

I use EKS, I'm not sure it will work

@adiroiban
Copy link
Author

If you have EKS, you don't have to worry about operating the k8s cluster. AWS does that for you.

In order to operate Actions Runner Controller (ARC) you need to have a good understanding of how k8s works and how GitHub Actions works

Othewise, just go with the GitHub hosted VMs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment