cloud-storage.md

Storage Access methods

we have different types of access methods to a storage:

• Block storage
• File storage
• Object storage
  • offer support for the Amazon S3 and OpenStack Swift , on top of Ceph cluster

Object storage vs Block Storage

	Object S.	Block S.
Performance	Big content,high stream throuhgput	database, transactional data
Geography	multiple regions	high latency if big distance app and storage
Scalability	petabytes or more	limit scability
Analytics	easily organized,retrive	No metadata

Minio

https://min.io/

High Performance cloud Storage service, under apache license

An open source, production solution ready.

100% compatible with AWS S3 API.

Easy and simple

Distributed Storage

http://min.io/pricing

it offer commercial support, but it is not hosted

host could be done on-premise, private cloud

Supports:

• continuous replication for large scale ( disaster recovery )

Ceph

https://www.openshift.com/blog/openshift-container-storage-4-introduction-to-ceph

RGW(web) RBD(block device) CEPHFS(distributed file system) | LIBRADOS (library python, java ) | RADOS Realiable Distributed Object Store

RADOS

Librados

library , hypervisor [librdb] Wrappers Python, and others

Object Storage, S3 ( Ceph RADOS Gateway )

Data protection

Data Distribution

Noobaa

it is a data service provide S3 object-store interface

Create an S3 compatible NooBaa backingstore logs to -- > Noomba

Object Storage solutions

https://www.openio.io/blog/3-recent-studies-confirm-that-object-storage-is-finally-adapted-to-high-performance-use-cases

cloud storage services

• Minio : high performance, kubernetes Native Object Storage
• API compatible with Amazon S3

Kubernetes logs to Elastic Search using Fluentd

https://medium.com/kubernetes-tutorials/cluster-level-logging-in-kubernetes-with-fluentd-e59aa2b6093a Collect logs from many pods

• Log management backends (Elasticsearch, Splunk)
• Big data stores (Hadoop DFS)
• Data archiving (Files, AWS S3)
• PubSub queues (Kafka, RabbitMQ)
• Data warehouses (BigQuery, AWS RedShift)
• Monitoring systems (Datadog)
• Notification systems (email, Slack, etc.)

https://docs.fluentd.org/v/0.12/articles/kubernetes-fluentd

Output destinations

• Backends ( Elastiksearch )
• Data archiving (Files, AWS S3 )
• Notification systems (email, Slack )

Fluentd and MinIO

Use fluentd with MinIO as backend

Kubernetes/Openshift Storage / Volumes

Type of Persistance Volume PVs are implemented as plugins

The plugin connect with a backend

Each backend is a under running cloud storage service.

https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes We have different solutions

• AWS Elastic Block Storage (EBS)
• Local Volume
• Ceph-rbd, ceph-fs --> actually used by LH / Redhat product
• NFS
• more ..

Openshift Forward logs to third party

https://docs.openshift.com/container-platform/4.3/logging/config/cluster-logging-external.html