How to Backup AKS Hybrid (Azure Stack HCI AKS) & Kubernetes with Velero and Ubuntu Minio Server

1- How to Backup AKS Hybrid with Velero and Ubuntu Minio Server Part 1.md

Backup Azure Stack HCI AKS (AKS Hybrid)

Introduction

Azure Stack HCI AKS also means AKS Hybrid, doesn't have a native backup solution. When you search about backup solution for this product you will find only Velero Backup Solution in Azure Docs. Backup AKS Hybrid With Velero

Why there is no backup solution except Velero?

AKS hybrid uses akshci CSI drivers and this driver doesn't support Snapshot resource and all other well known backup products use snapshot resources for backup of Kubernetes CRDs.

Velero at this point gives an application level snapshot feature, which is known as Restic.

Velero can make a Volume snapshot backup with Restic service even if your CSI drivers don't support snapshots. That's why we can only use Velero for AKS Hybrid. Don't forget that you can use this method in any K8S distribution if you decided to use Velero Backup on your kubernetes with Minio.

What is the Limitations and Risks

• Velero can make backup only to a object storage
• Velero is a great open source application that has some k8s distributions support, but Microsoft doesn't support if you have some problems with Velero. So you can not open a case to the Vendor. You will focus to community for support.

Get Started

In this article we will move forward in two parts.

1. Create Minio Storage
2. Create Velero in Kubernetes and backup to Minio S3 storage.

PART 1 - Create Minio Storage

As i explain, first you need an object storage. For this you have two main options.

1. Take an object storage from Azure Storage Services
2. Make your own Object Storage.

The first option is the easiest way, but this way you will migrate your data to Azure DCs. Sometimes you may not want to keep your data in Azure DC for some reasons. (Government Regulations, Architecture, Cost, etc...)

So today we are going to select the hard way. We are going to build our object storage, which will make this article attractive. :)

Make your own Storage;

In this step we will use the min.io server storage service. Minio is a virtual storage and this storage gives us S3 storage type.

For this actually we have two options

1. Run minio in k8s
2. Run minio in a Linux Server

Ruining Minio application in k8s services is explained in Azure Docs. They choose this option and there is no any other option in their document. Again, sometimes you may not want to store your backup data in the same storage and in the same environment (your kubernetes backup data storage working on the same physical environment). So because of that you can set your kubernetes storage containers to external storage (in this situation the environment is still the same) or you can create a virtual machine in another environment which is not related to the kubernetes environment.

In this document we are going to build an independent minio server. We will create a minio service in a VMware environment on Ubuntu Server.

• Pros
  • You can back up your data to offside environment.
  • You can back up your Ubuntu server with your existing Vmware backup solution. This means you can backup your s3 storage also.
  • You don't need to worry in case, if you will lose kubernetes cluster or disk. Backup data is still in another environment.
• Cons
  • All data will move on the network.
  • You need to manage extra Ubuntu server.

Create a Minio server on Ubuntu Server;

In this step I assume you know the steps for Linux server installation. I am going to explain how to prepare the data disk for Minio and how to run Minio server.

Let's Begin;

Step 1 - List your block devices.

With this command we are displaying "sd" structure. As you see sda is my system disk and sdb is my second disk which i am going to use as data disk for Minio.

lsblk

Step 2 - Make Primary Partition

I start fdisk from the shell prompt.

run

fdisk /dev/sdb

press n in command line for create partition. press p for select primary partition type in partition number select default. Just hit enter button or type default number for select default.

press l for list system type of partition. You will see that number 83 is Linux type.

Command (m for help): t

Selected partition 1

Hex code (type L to list codes): 83

Changed type of partition 'Linux' to 'Linux' .

To write the changes on disk, run the w command

Command (m for help): w

The partition table has been altered.

Calling ioctl() to re-read partition table

Syncing disks.

Step 3 - Format Partition Once a partition has been created with the parted of fdisk command, format the disk before using it. For this we will use xfsprogs.

run the command

apt install xfsprogs

run mkfs. for list all mkfs format.

Format /dev/sdb1 with xfs format

mkfs.xfs /dev/sdb1

crate a data folder

mkdir /data

Mount data folder to /dev/sdb1

mount /dev/sdb1 /data
mount

now you shoud be able to display /dev/sdb1 Mounted on /data folder with df -h

enter to data folder cd /data and create a test file with touch command

touch hede.txt

If everything is fine go to root with cd - command.

Step 4 - Configure Auto mount in /etc/fstab

vi /etc/fstab

create second line /dev/sdb1 in fstab configuration file and save, exit from vi with wq!

send a mount command one more time. You will see /dev/sdb1 on /data type xfs output end of the line.

Step 5 - Install Minio Server

Firstly, make sure that you are up to date in Ubuntu.

Run

apt list --upgradable

if you have upgradable packages run apt upgrade

now it's time to download Minio Server. Go to minio download website and select your architecture. I am running amd64 Linux.

run

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio

move minio to /usr/local/bin/

mv minio /usr/local/bin/

Step 6- Configure Minio server and storage

• User and Group to does;

#create a system user for minio with no login
useradd --system minio --shell /sbin/nologin

#lock minio system user and set password age
usermod -L minio
chage -E0  minio

• chown the directories with the MinIO user and group chown minio:minio /usr/local/bin/minio

• Configuration volumes and config file;

touch /etc/default/minio
echo 'MINIO_ACCESS_KEY="minio"' >> /etc/default/minio
echo 'MINIO_VOLUMES="/data"' >> /etc/default/minio
echo 'MINIO_OPTS="-C /etc/minio --address :9000"' >> /etc/default/minio
echo 'MINIO_SECRET_KEY="CREATE_A_KEY"' >> /etc/default/minio

#create minio folder in etc
mkdir /etc/minio

#chown the directories with the MinIO user and group
chown minio:minio /etc/minio

• Configuring MinIO with SystemD

wget https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service

Important! minio.service configuration file has default values. In these default values username and groups is minio-user but we used username and group like minio so, we need to change user and username in minio.service config file username group values minio-user to minio. Fort this vi minio.service find user and group which has the value minio-user and change them to minio save and quit wq!

Move minio.service configuration file to systemd directory;

mv minio.service /etc/systemd/system

Do not forget allow port 9000 in ufw

sudo ufw allow 9000

• Start Minio Service and Expose it.

sudo systemctl daemon-reload
sudo systemctl enable minio
sudo systemctl start minio
sudo systemctl status minio

You should be able to see minio service is active and running

open a web browser and go to your minio server web console

http://minioserverinternalIP:9000

Congratulations, you have a minio server! We complete part1. Now we need to setup Velero in kubernetes and backup to Minio server disks using Minio buckets.

2- How to Backup AKS Hybrid with Velero and Ubuntu Minio Server Part 2.md

Backup Azure Stack HCI AKS (AKS Hybrid)

Introduction

In section 1, I explain why I have to use Minio Server and how to configure Ubuntu. We exposed a Minio service now time to use it in kubernetes Velero backup. Have a look if you're not reading the first part. part 1.

Get Started

First, to access your Minio S3 storage bucket kubernetes clusters should have access to Ubuntu server with Port Number TCP 9000.

• Steps
  1. Create a bucket in Minio Server
  2. Install Velero with Helm
  3. Create a backup job in velero

1- Create a bucket in Minio Server

• Go to minio server admin panel with a browser (http://ubuntuInternalIP:9000)
• Enter your user name password (we created it in the first part)
  • Username: minio
  • Password: Your secret.
• Go to Bucket in Web Console
• Press Create Bucket button
• Give a bucket name (keep it simple my name is bucket1 )
• Press Create Bucket

2- Install Velero with Helm

We are using Helm 3 for this installation.

• Add Kubernetes Helm Charts for VMWare-Tanzu

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

You can then run helm search repo vmware-tanzu to see the charts. it should be smiler with this output

$ helm search repo vmware-tanzu
NAME                                            CHART VERSION   APP VERSION     DESCRIPTION                                       
vmware-tanzu/velero                             2.32.3          1.9.3           A Helm chart for velero    

• Prepare helm values and install

First, we need to create a credential secret file for authenticate with Minio Server.

To do this, open a text document, give a file name with no extension (my name is secret1) and fill the inside with this format.

    [default]
    aws_access_key_id=minio
    aws_secret_access_key=YOUR_SECRET_WHICH_WE_CREATED_PART1

save and quit.

I share example from my yml file. Some values are optional, you can change it depending on your architecture. I also write here significant values for Minio just to show.

*--set-file credentials.secretContents.cloud=secret1
--set configuration.backupStorageLocation.bucket=bucket1
--set configuration.backupStorageLocation.config.s3Url=http://YOUR_MINO_SERVER_IP_ADDRESS:9000
--set configuration.volumeSnapshotLocation.name=default
--set configuration.volumeSnapshotLocation.config.region=minio
--set initContainers[0].name=velero-plugin-for-aws
--set initContainers[0].image=velero/velero-plugin-for-aws:v1.4.0
--set initContainers[0].volumeMounts[0].mountPath=/target
--set initContainers[0].volumeMounts[0].name=plugins *

Here is full yml file;

helm install vmware-tanzu/velero \
    --namespace=velero \
    --create-namespace \
    --set-file credentials.secretContents.cloud=secret1  \
    --set configuration.provider=aws \
    --set configuration.backupStorageLocation.name=default \
    --set configuration.backupStorageLocation.bucket=bucket1 \
    --set configuration.backupStorageLocation.config.region=minio \
    --set configuration.backupStorageLocation.config.s3ForcePathStyle=true \
    --set configuration.backupStorageLocation.config.s3Url=http://YOUR_MINO_SERVER_IP_ADDRESS:9000 \
    --set metrics.enabled=true \
    --set metrics.scrapeInterval=10s \
    --set snapshotsEnabled=true \
    --set deployRestic=true \
    --set configuration.defaultVolumesToRestic=true \
    --set configuration.clientQPS=75 \
    --set configuration.clientBurst=100 \
    --set configuration.volumeSnapshotLocation.name=default \
    --set configuration.volumeSnapshotLocation.config.region=minio \
    --set initContainers[0].name=velero-plugin-for-aws \
    --set initContainers[0].image=velero/velero-plugin-for-aws:v1.4.0 \
    --set initContainers[0].volumeMounts[0].mountPath=/target \
    --set initContainers[0].volumeMounts[0].name=plugins \
    --generate-name

Check whether the Velero service is running properly:

kubectl -n velero get pods
kubectl logs deployment/velero -n Velero

If your deployment up and running, you should be able to be ready for backup your kubernetes to Minio bucket.

3- Create a backup job in velero

Run a velero test job

Use the Velero backup create command to create backups to your chosen storage. The following examples use the --default-volumes-to-restic flag, which creates a snapshot of the persistent volumes.

• On-demand backup of all namespaces in your cluster:

velero backup create <BACKUP-NAME> --default-volumes-to-restic

• On-demand backup of a single namespace in your cluster:

velero backup create <BACKUP-NAME> --include-namespaces <NAMESPACE1> --default-volumes-to-restic

While the backup is in progress, you can check the bucket in the Minio administration panel, you should be able to see certain folders and files has been created.

For more Velero commands please take a look; Velero Backup Commands