misho-kr/Architecting with Google Kubernetes Engine: Production.md

## Architecting with Google Kubernetes Engine: Production.md

      
    Raw
  

              Architecting with Google Kubernetes Engine: Production.md
            
          
    Architecting with Google Kubernetes Engine: Production

Learn about Kubernetes and Google Kubernetes Engine (GKE) security; logging and monitoring; and using GCP managed storage and database services from within GKE.
Access Control and Security in Kubernetes and GKE


IAM roles and policies for GKE
Kubernetes RBAC roles and role bindings
Kubernetes pod security policies

1. Kubernetes authentication and authorization

Control who is able to perform what actions to resources in your GKE clusters. Ensure that appropriate access policies are in place to make your cluster and applications more secure.

Two main types of users, Normal users and Kubernetes service accounts
Two ways to authorize what that account can do:

Cloud Identity & Access Management

Permissions to perform operations outside Kubernetes clusters
Who can view or change the configurations of your GKE clusters


Kubernetes role-based access control

Inside Kubernetes clusters, at the cluster and the namespace level
Who can view or change Kubernetes objects inside GKE clusters


Kubernetes service accounts provide an identity for processes in a Pod
GKE supported authentication methods

OpenID connect tokens, only one enabled by default
x509 client certificates
basic authentication using static passwords


2. Cloud IAM: who can do what on which project


A role consists of one or more permissions
A permission compute.instances.get consists of service, resource and a verb
GCP resources are organized hierarchically, starting with the organization
Cloud IAM policies applied at higher levels of a GCP organizational hierarchy are inherited by resources lower down
Primitive Role grants users global, project-level access to all GCP resources within a Project

Viewer, Editor and Owner


Predefined Role provide granular access

GKE Viewer, GKE Developer, GKE Cluster Admin, GKE Admin, GKE Host Service Agent User


Custom Role to specify each individual permission that will make up the role

3. Kubernetes RBAC provide fine grained tools to manage user account permissions


Extends Cloud IAM security by offering control within the cluster
Defines what operations verbs can be executed over which objects resources by who subjects

A subject is a set of users or processes that can make requests to the Kubernetes API
A resource is a set of Kubernetes API objects, such as Pods, Deployments, etc.
A verb is a set of operations that can be performed on resources, such as get, watch, etc.
They are connected by creating two types of RBAC API objects:

Roles defines the objects and permissions
RoleBindings binds Roles to subjects (users) in a namespace


2 types of roles: Role and ClusterRole
Rules are purely additive; there are no deny rules


Not all resources are Namespaced

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: demo-role
rules:
  - apiGroups: [""]
    resource: ["pods"]
    verbs: ["get", "list", "watch"]
---    
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: demo-rolebinding
subjects:
  - kind: User
    name: "joe@example.com"
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: demo-role
  apiGroup: rbac.authorization.k8s.io
4. Kubernetes Control Plane Security


All master components -- API server, etcd database, and controller manager, are managed by Google
Each cluster has its own root certificate authority CA
When new node is created, it is injected with a shared Secret used by its kubelet to submit certificate signing requests to the cluster root CA
Credentials are rotated periodically to improve the security posture

The IP address of the cluster can be rotated too


Protect your metadata

Configure the Cloud IAM service account for the node with minimal permissions
Disable legacy Compute Engine API endpoint
Enable metadata concealment (temporary)

Basically a firewall that prevents Pods from accessing a node’s metadata


5. Pod Security

Control what sort of things a Pod, or more specifically the containers inside a Pod, are allowed to do

Security context is a set of security settings defined in a Pod specification to control:

The use of the host namespace
Networking
Filesystem and Volume types
Whether privileged containers can run
Whether code in a container can escalate to root privileges
Enable Seccomp and limit access to some Linux capabilities
Is enforced by the container runtime


Pod security policy applies security contexts

They are reusable security contexts
Consists of an object and an admission controller
Applied when a Pod is being created or updated


Validating and Mutating controllers

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: demo-psp
spec:
  privileged: false
  allowPrivilegeEscalation: false
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'RunAsAny'
  readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp-clusterrole
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  resourceNames:
  - demo-psp
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: psp-rolebinding
namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp-clusterrole
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts
- kind: ServiceAccount
  name: service@example.com
  namespace: demo

PodSecurityPolicy controller is disabled by default
First define PodSecurityPolicies and then enable the controller
Aditional security measures in Kubernetes

Labs Securing Google Kubernetes Engine with Cloud IAM and Pod Security Policies and code
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted-psp
spec:
  privileged: false  # Don't allow privileged pods!
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: MustRunAsNonRoot
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: restricted-pods-role
rules:
- apiGroups:
  - extensions
  resources:
  - podsecuritypolicies
  resourceNames:
  - restricted-psp
  verbs:
  - use
---
> kubectl apply -f restricted-psp.yaml
> kubectl get podsecuritypolicy restricted-psp
> gcloud info --format='value(config.account)'
> kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin \
  --user $USERNAME_1_EMAIL
> kubectl apply -f restricted-pods-role.yaml
> kubectl descrbe clusterrole restricted-pods-role
> gcloud beta container clusters update $my_cluster --zone $my_zone --enable-pod-security-policy
> kubectl get podsecuritypolicies
> kubectl apply -f privileged-pod.yaml   # as user-1
> kubectl auth can-i use podsecuritypolicy/gce.privileged
> kubectl apply -f unprivileged-pod.yaml # as user-2
> kubectl create clusterrolebinding restricted-pods-binding --clusterrole restricted-pods-role \
  --user @USERNAME_2_EMAIL
> kubectl apply -f unprivileged-pod.yaml
#
> gcloud container clusters update $my_cluster --zone $my_zone --start-credential-rotation
> gcloud container operations list
> gcloud container operations wait operation-1590389893808-0f56aca0 --zone $my_zone
> gcloud container clusters update $my_cluster --zone $my_zone --complete-credential-rotation
Labs Implementing Role-Based Access Control with Google Kubernetes Engine and code
apiVersion: v1
kind: Namespace
metadata:
  name: production
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: production
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "watch"]
--
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: username2-editor
  namespace: production
subjects:
- kind: User
  name: [USERNAME_2_EMAIL]
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
> kubectl apply -f ./my-pod.yaml --namespace=production
> kubectl get pods --namespace=production
> kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin \
  --user USERNAME_1_EMAIL
> kubectl apply -f pod-reader-role.yaml
> kubectl get roles --namespace production
> kubectl apply -f username2-editor-binding.yaml
> kubectl apply -f ./production-pod.yaml
> kubectl get pods --namespace production
> kubectl delete pod production-pod --namespace production
GKE Logging and Monitoring


Create logs for systems monitoring
Monitor your system performance from different vantage points
Create probes for wellness checks on live applications

1. Stackdriver

Labs Implementing Role-Based Access Control with Google Kubernetes Engine and code
Labs Configuring Liveness and Readiness Probes and code
Labs Using Cloud SQL with Google Kubernetes Engine and code