Windows storage guide

cloudprovider-azurefiles.md

Dynamic - PVC

Following https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv.

Create a StorageClass for creating PV dynamically

For now, we can only choose the following Azure storage redundancy for skuName:

• Standard_LRS - standard locally redundant storage (LRS)
• Standard_GRS - standard geo-redundant storage (GRS)
• Standard_RAGRS - standard read-access geo-redundant storage (RA-GRS)

Azure Files currently only work with Standard storage. If you use Premium storage, the volume fails to provision.

Form Storage > Storage Classes:

1. input Name
2. choose Azure File as Provisioner
3. input Standard_LRS as Sku Name
4. optional - set Location (where your Node located) and Storage Account
5. add Mount Options:
   • dir_mode=0777
   • file_mode=0777

Confirm the permission which allow Azure to create the required storage resources

Lauch kubectl from Cluster page:

> k get sa persistent-volume-binder -n kube-system
> k get clusterrole system:azure-cloud-provider -n kube-system
> k get clsuterrolebinding system:azure-cloud-provider -n kube-system

Should we create these RBAC resources when choosing Azure cloud provider?

If not existing, Import YAML from Workloads page of System project:

1. choose Cluster as Import Mode
2. import as below:

   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: system:azure-cloud-provider
   rules:
   - apiGroups: ['']
     resources: ['secrets']
     verbs:     ['get','create']
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: system:azure-cloud-provider
   roleRef:
     kind: ClusterRole
     apiGroup: rbac.authorization.k8s.io
     name: system:azure-cloud-provider
   subjects:
   - kind: ServiceAccount
     name: persistent-volume-binder
     namespace: kube-system

Add the PVC based on the above StorageClass

From Workloads page of Default project, select Volumes tab, click Add Volumes:

1. optional - input Name
2. choose Namespace for deploying workloads
3. select Storage Class as Source
4. select Storage Class which created previously
5. input Capacity
6. expand Customize to select only Many Nodes Read-Write item

Test the PV on Linux workload

From Workloads page of Default project, Deploy a new workload:

1. input Name
2. input ubuntu:xenial as Docker Image
3. choose Run all pods for this worload on Linux nodes only
4. expand Volumes to Use an existing persistent volume (claim) (Add a new persistent volume (claim) is the same as Add the PVC based on the above StorageClass):
   • input Volume Name
   • choose Persistent Volume Claim which created previously
   • mount to /mnt/azure Mount Point
5. Execute Shell to this workload to check the monut point:

   # mount | grep /mnt/azure
   # echo $(hostname) > /mnt/azure/hosts

Test the PV on Windows workload

1. input Name
2. input mcr.microsoft.com/powershell:nanoserver-1809 as Docker Image
3. choose Run all pods for this worload on Windows nodes only
4. expand Volumes to Use an existing persistent volume (claim):
   • input Volume Name
   • choose Persistent Volume Claim which binded on Linux workload
   • mount to /data Mount Point
5. Execute Shell to this workload to check the monut point:

   C:\> pwsh.exe
   C:\> cat c:/data/hosts
   C:\> echo $(hostname) >> c:/data/hosts

6. go back to the Linux workload to check:

   # cat /mnt/azure/hosts

Static - PV

Create an Azure Storage account

Form Azure portal, choose Storage accounts, Add:

1. input the same Resource group as nodes
2. input Storage account name
3. choose Location where the nodes located
4. select Standard as Performance (Azure Files currently only work with Standard storage. If you use Premium storage, the volume fails to provision.)
5. select StorageV2 (gerneral purpose v2) as Account kind
6. select Locally-redundant storage (LRS) as Replication
7. next, next -> create
8. go into this storage resource
   1. find Access keys from Settings of this resource
   2. view the value of Storage account name (use for next step) and one Key (use for next step)
   3. find Files from File service of this resource
   4. add File share:
      • input Name (use for next step)
      • input Quota

Create a Secret to record the Azure Storage account name and Key

From Resources > Secrets page of Default project, Add Secret:

1. input Name (use for next step)
2. select Available to a single namespace as Scope
3. add azurestorageaccountname = the above Storage account name value
4. add azurestorageaccountkey = the above Key value

Test the PV on Linux workload

From Workloads page of Default project, Deploy a new workload:

1. input Name
2. input ubuntu:xenial as Docker Image
3. choose Run all pods for this worload on Linux nodes only
4. expand Volumes to Add an ephemeral volume:
   • select Azure Filesystem as Source
   • expand Source Configuration:
     • input Share Name = the Name of the above created File share
     • input Secret Name which created previously
     • select No as Read Only
   • mount to /mnt/azure Mount Point
5. Execute Shell to this workload to check the monut point:

   # mount | grep /mnt/azure
   # echo $(hostname) > /mnt/azure/hosts

Test the PV on Windows workload

1. input Name
2. input mcr.microsoft.com/powershell:nanoserver-1809 as Docker Image
3. choose Run all pods for this worload on Windows nodes only
4. expand Volumes to Add an ephemeral volume
   • select Azure Filesystem as Source
   • expand Source Configuration:
     • input Share Name = the Name of the above created File share
     • input Secret Name which created previously
     • select No as Read Only
   • mount to /data Mount Point
5. Execute Shell to this workload to check the monut point:

   C:\> pwsh.exe
   C:\> cat c:/data/hosts
   C:\> echo $(hostname) >> c:/data/hosts

6. go back to the Linux workload to check:

   # cat /mnt/azure/hosts

local-iscsi.md

Following https://github.com/Microsoft/K8s-Storage-Plugins/blob/b2dfff4725/flexvolume/windows/sample_yamls/readme.md.

Prerequisites

This example expects there to be a working iSCSI target to connect to. If there isn't one in place then it is possible to setup a software version on Linux by following these guides

• Setup a iSCSI target on Fedora
• Install the iSCSI initiator on Fedora
• Fixed: iscsi returns sense key not ready
• Command for querying iqn: (Get-InitiatorPort).NodeAddress

Create a Windows workload

Import Yaml from Default Project Page:

• Create Secret to record the account for accessing iSCSI application, the type should be microsoft.com/iscsi.cmd

apiVersion: v1
kind: Secret
metadata:
  name: iscsi-secret
data:
  # base64 encode password, for example: username
  node.session.auth.password: c3Ryb25ncGFzc3dvcmQ=
  # base64 encode username, for example: strongpassword
  node.session.auth.username: dXNlcm5hbWU=
type: microsoft.com/iscsi.cmd

• Replace the ISCSI_SERVER_HOST_IP by the iSCSI workload host IP, Replace the ISCSI_TARGET_IQN by the iSCSI IQN

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: windows-iscsi-test
  name: windows-iscsi-test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-iscsi-test
      name: windows-iscsi-test
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - windows-iscsi-test
            topologyKey: "kubernetes.io/hostname"
      nodeSelector:
        beta.kubernetes.io/os: windows
      containers:
      - name: windowsiscsitest
        image: mcr.microsoft.com/powershell:nanoserver-1809
        command:
        - pwsh.exe
        args:
        - -NoLogo
        - -NonInteractive
        - -Command
        - '&{$hs=hostname; echo $hs >> c:/mnt/hosts; do { echo "==========="; cat c:/mnt/hosts; echo "==========="; echo ""; start-sleep -s 10; } while($True)}'
        volumeMounts:
        - name: iscsi-volume
          mountPath: c:/mnt
      volumes:
      - name: iscsi-volume
        flexVolume:
          driver: "microsoft.com/iscsi.cmd"
          fsType: "NTFS"
          secretRef:
            name: "iscsi-secret"
          readOnly: false
          options:
            chapAuthDiscovery:  "false"
            chapAuthSession:  "true"
            targetPortal:  "ISCSI_SERVER_HOST_IP"
            iqn:  "ISCSI_TARGET_IQN"
            lun:  "0"
            authType:  "ONEWAYCHAP"

local-smb.md

Following https://github.com/Microsoft/K8s-Storage-Plugins/blob/b2dfff4725/flexvolume/windows/sample_yamls/readme.md.

Deploy a Samba workload

Import Yaml from Default Project Page:

• Need to expose Samba application via hostNetwork

---
# Source: samba/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: frank-samba
  namespace: default
  labels:
    app: samba
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: samba
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/os
                operator: NotIn
                values:
                - windows
      tolerations:
      - operator: Exists
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: samba
        image: "dperson/samba"
        imagePullPolicy: IfNotPresent
        args:
        - -u
        - "frank;frank123;;users"
        - -p
        - -w
        - "rancher"
        - -g
        - "create mask = 0777"
        - -g
        - "force create mode = 0777"
        - -s
        - "shared;/shared;no;no;no;frank"

        ports:
        - containerPort: 445
          name: smb445
        - containerPort: 139
          name: smb139
        readinessProbe:
          tcpSocket:
            port: 445
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 445
          initialDelaySeconds: 15
          periodSeconds: 20
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi

Create a Linux workload

Import Yaml from Default Project Page:

• Replace the SAMBA_SERVER_HOST_IP by the Samba workload host IP

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: linux-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: linux-test
  template:
    metadata:
      labels:
        app: linux-test
    spec:
      containers:
      - name: linuxtest
        image: usr42/cifs
        securityContext:
          privileged: true
        tty: true
        stdin: true
        command: 
        - /bin/sh
        args:
        - -c
        - "mount -t cifs //SAMBA_SERVER_HOST_IP/shared /mnt -o username=frank,password=frank123; mount | grep /mnt; echo ''; echo $(hostname) >> /mnt/hosts; while true; do echo '==========='; cat /mnt/hosts; echo '==========='; echo ''; sleep 10; done"

Create a Windows workload

Import Yaml from Default Project Page:

• Create Secret to record the account for accessing Samba application, the type should be microsoft.com/smb.cmd

apiVersion: v1
kind: Secret
metadata:
  name: smb-secret
data:
  # base64 encode password, for example: frank123
  password: ZnJhbmsxMjM=
  # base64 encode domain\username, for example: rancher\frank
  # ref: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.security/get-credential?view=powershell-6 (example 7)
  username: cmFuY2hlclxmcmFuaw==
type: microsoft.com/smb.cmd

• Replace the SAMBA_SERVER_HOST_IP by the Samba workload host IP

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: windows-test
  name: windows-test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-test
      name: windows-test
    spec:
      containers:
      - name: windowstest
        image: mcr.microsoft.com/powershell:nanoserver-1809
        command:
        - pwsh.exe
        args:
        - -NoLogo
        - -NonInteractive
        - -Command
        - '&{$hs=hostname; echo $hs >> c:/mnt/hosts; do { echo "==========="; cat c:/mnt/hosts; echo "==========="; echo ""; start-sleep -s 10; } while($True)}'
        volumeMounts:
        - name: smb-volume
          mountPath: c:/mnt
      volumes:
      - name: smb-volume
        flexVolume:
          driver: "microsoft.com/smb.cmd"
          secretRef:
            name: "smb-secret"
          options:
            # source can be in any of the following formats 
            # \\\\servername\\share\\path
            source: "\\\\SAMBA_SERVER_HOST_IP\\shared"