ppetko/AMQ

## AMQ
## Red Hat AMQ Streams

### Red Hat AMQ Streams is a massively scalable, distributed, and high performance data streaming platform based on the Apache Kafka® project. AMQ Streams provides an event streaming backbone that allows microservices and other application components to exchange data with extremely high throughput and low latency.

### The core capabilities include:
    * A pub/sub messaging model, similar to a traditional enterprise messaging system, in which application components publish and consume events to/from an ordered stream
    * The long term, fault-tolerant storage of events
    * The ability for a consumer to replay streams of events
    * The ability to partition topics for horizontal scalability

### Red Hat AMQ Streams provides a way to run an Apache Kafka cluster on OpenShift in various deployment configurations. See the Red Hat AMQ website for more details about the project.

### What is new in AMQ Streams 1.5
* Support for Apache Kafka 2.5.0
* Built-in Cruise Control support, in Tech Preview, for cluster rebalancing
* New operator metrics and new Grafana dashbaord for operator monitoring
* Improved configurability of TLS
* CORS support in the HTTP Bridge

### Supported Features
    * Manages the Kafka Cluster - Deploys and manages all of the components of this complex application, including dependencies like Apache ZooKeeper® that are traditionally hard to administer.
    * Includes Kafka Connect - Allows for configuration of common data sources and sinks to move data into and out of the Kafka cluster.
    * Topic Management - Creates and manages Kafka Topics within the cluster.
    * User Management - Creates and manages Kafka Users within the cluster.
    * Includes Kafka Mirror Maker 1 and 2 - Allows for morroring data between different Apache Kafka® clusters.
    * Includes HTTP Kafka Bridge - Allows clients to send and receive messages through an Apache Kafka® cluster via HTTP protocol.
    * Cluster Rebalancing - Uses built-in Cruise Control for redistributes partition replicas according to specified goals in order to achieve the best cluster performance.
    * Monitoring - Built-in support for monitoring using Prometheus and provided Grafana dashabords.

### Upgrading your Clusters- The AMQ Streams operator understands how to run and upgrade between a set of Kafka versions. When specifying a new version in your config, check to make sure you are not using any features that may have been removed. See the upgrade guide for more information.
Security

### AMQ Streams supports TLS encryption, authentication, and authorization out of the box. Supported authentication methods are TLS client certificates, SCRAM-SHA mechanism based on username and password, and OAuth authentication.
Monitoring

### AMQ Streams has support for Prometheus metrics. Enable metrics in the custom resources and configure your Prometheus server to scrape the metrics from all Kafka and Zookeeper pods. With the integrated Kafka Exporter you have now also advanced metrics about topics, consumer groups and consumer group lag.
Storage

### An efficient data storage infrastructure is essential to the optimal performance of AMQ Streams. AMQ Streams requires block storage. The use of file storage (for example, NFS) is not recommended.

### AMQ Streams supports three types of data storage:
    * Ephemeral (Recommended for development only)
    * Persistent
    * JBOD (Just a Bunch of Disks, suitable for Kafka only. Not supported in Zookeeper.)
    * AMQ Streams also supports advanced operations such as adding or removing disks in Kafka brokers or resizing the persistent volumes (where supported by the infrastructure).
Documentation

### Documentation for the current release can be found on Red Hat website.
* https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html-single/using_amq_streams_on_openshift/index

## Deployment

### Kafka cluster

```
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  creationTimestamp: '2020-06-24T17:28:04Z'
  generation: 1
  name: my-cluster
  namespace: amq-demo
  resourceVersion: '84722'
  selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/amq-demo/kafkas/my-cluster
  uid: 1a5acfb5-5b1a-41ca-8220-f36e222f6d71
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      log.message.format.version: '2.5'
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
      plain: {}
      tls: {}
    replicas: 3
    storage:
      type: ephemeral
    version: 2.5.0
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
status:
  conditions:
    - lastTransitionTime: '2020-06-24T17:30:09+0000'
      status: 'True'
      type: Ready
  listeners:
    - addresses:
        - host: my-cluster-kafka-bootstrap.amq-demo.svc
          port: 9092
      bootstrapServers: 'my-cluster-kafka-bootstrap.amq-demo.svc:9092'
      type: plain
    - addresses:
        - host: my-cluster-kafka-bootstrap.amq-demo.svc
          port: 9093
      bootstrapServers: 'my-cluster-kafka-bootstrap.amq-demo.svc:9093'
      certificates:
        - |
          -----BEGIN CERTIFICATE-----
          MIIDLTCCAhWgAwIBAgIJAPuoDob0JlW7MA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
          BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjAwNjI0
          MTcyODA0WhcNMjEwNjI0MTcyODA0WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
          FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
          CgKCAQEAuQDNSOFg5MgqtU85sjuXwPYYZhIguIrueDm6wlt4imcxv/ZSwciNabYo
          0LfAAIBb8hlkYWb5//qa6x5tvvtUlrwlZDuc52vur+sphhqE2XAvNEkWK5D1BG/t
          OLSBv7btd+x3fS1/I5UHmxTu+l9/JNq8X/cKsAFovDC4Ldd7cURVQH9EgEpDtNqH
          XgeMlQP+ZkXWLERarcq7EccH7wDnfTa98+JsctLzkC+/wmLV0hXwNUqT+t13Lz2x
          IlWFWHlCA8Sn5owNjqU+SfkHk41d0dJN4PqZJb7Mo2zu+R080AnSP3vMih3s9ZSr
          1gvN8a7bo0eWM21++tG5/5eVx0ScnwIDAQABo1AwTjAdBgNVHQ4EFgQU5LNvastM
          ww5IjDzrlbYVXiJ/bCswHwYDVR0jBBgwFoAU5LNvastMww5IjDzrlbYVXiJ/bCsw
          DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAs3GlFydj8gibErTx7j9U
          if/qS/esbPHYdfJePJfL0GnCsRIIm9ESSWzdXSpyh5F+F8KrGso/mUkfrycaYrAW
          DIqlde181yx9e/HjIBdy+VflbzidaF3QIkxi8uDkVhGRHfbkAIybD/knxhqNy4Sv
          ncVe1pHesMDTsBPz1U+9xWbOHlaDmWLxZrdU9NlUgD2pk70kVcx595GmGdKRKoPY
          fJiFIVmikyqTEM9X9zZKMGsdrUDEVfmGGivBrObzPNYheH3q/GV4UT3haucVULT8
          PAXWoilyQitU0oEBOopxCb0c2j600NGAib01eLqC08P+wo3JzSaRyIJYnt+IX5h1
          3A==
          -----END CERTIFICATE-----
      type: tls
  observedGeneration: 1

```


```
$ oc get all

NAME                                                       READY     STATUS    RESTARTS   AGE
pod/amq-streams-cluster-operator-v1.5.0-7cfd955dc7-g9lkv   1/1       Running   0          52m
pod/my-cluster-entity-operator-545c7d774f-plp4w            3/3       Running   0          39m
pod/my-cluster-kafka-0                                     2/2       Running   0          40m
pod/my-cluster-kafka-1                                     2/2       Running   0          40m
pod/my-cluster-kafka-2                                     2/2       Running   0          40m
pod/my-cluster-zookeeper-0                                 1/1       Running   1          40m
pod/my-cluster-zookeeper-1                                 1/1       Running   0          40m
pod/my-cluster-zookeeper-2                                 1/1       Running   0          40m

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/my-cluster-kafka-bootstrap    ClusterIP   172.30.104.100   <none>        9091/TCP,9092/TCP,9093/TCP   40m
service/my-cluster-kafka-brokers      ClusterIP   None             <none>        9091/TCP,9092/TCP,9093/TCP   40m
service/my-cluster-zookeeper-client   ClusterIP   172.30.163.219   <none>        2181/TCP                     40m
service/my-cluster-zookeeper-nodes    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   40m

NAME                                                  READY     UP-TO-DATE   AVAILABLE   AGE
deployment.apps/amq-streams-cluster-operator-v1.5.0   1/1       1            1           52m
deployment.apps/my-cluster-entity-operator            1/1       1            1           39m

NAME                                                             DESIRED   CURRENT   READY     AGE
replicaset.apps/amq-streams-cluster-operator-v1.5.0-7cfd955dc7   1         1         1         52m
replicaset.apps/my-cluster-entity-operator-545c7d774f            1         1         1         39m

NAME                                    READY     AGE
statefulset.apps/my-cluster-kafka       3/3       40m
statefulset.apps/my-cluster-zookeeper   3/3       40m

```
	## Red Hat AMQ Streams

	### Red Hat AMQ Streams is a massively scalable, distributed, and high performance data streaming platform based on the Apache Kafka® project. AMQ Streams provides an event streaming backbone that allows microservices and other application components to exchange data with extremely high throughput and low latency.

	### The core capabilities include:
	* A pub/sub messaging model, similar to a traditional enterprise messaging system, in which application components publish and consume events to/from an ordered stream
	* The long term, fault-tolerant storage of events
	* The ability for a consumer to replay streams of events
	* The ability to partition topics for horizontal scalability

	### Red Hat AMQ Streams provides a way to run an Apache Kafka cluster on OpenShift in various deployment configurations. See the Red Hat AMQ website for more details about the project.

	### What is new in AMQ Streams 1.5
	* Support for Apache Kafka 2.5.0
	* Built-in Cruise Control support, in Tech Preview, for cluster rebalancing
	* New operator metrics and new Grafana dashbaord for operator monitoring
	* Improved configurability of TLS
	* CORS support in the HTTP Bridge

	### Supported Features
	* Manages the Kafka Cluster - Deploys and manages all of the components of this complex application, including dependencies like Apache ZooKeeper® that are traditionally hard to administer.
	* Includes Kafka Connect - Allows for configuration of common data sources and sinks to move data into and out of the Kafka cluster.
	* Topic Management - Creates and manages Kafka Topics within the cluster.
	* User Management - Creates and manages Kafka Users within the cluster.
	* Includes Kafka Mirror Maker 1 and 2 - Allows for morroring data between different Apache Kafka® clusters.
	* Includes HTTP Kafka Bridge - Allows clients to send and receive messages through an Apache Kafka® cluster via HTTP protocol.
	* Cluster Rebalancing - Uses built-in Cruise Control for redistributes partition replicas according to specified goals in order to achieve the best cluster performance.
	* Monitoring - Built-in support for monitoring using Prometheus and provided Grafana dashabords.

	### Upgrading your Clusters- The AMQ Streams operator understands how to run and upgrade between a set of Kafka versions. When specifying a new version in your config, check to make sure you are not using any features that may have been removed. See the upgrade guide for more information.
	Security

	### AMQ Streams supports TLS encryption, authentication, and authorization out of the box. Supported authentication methods are TLS client certificates, SCRAM-SHA mechanism based on username and password, and OAuth authentication.
	Monitoring

	### AMQ Streams has support for Prometheus metrics. Enable metrics in the custom resources and configure your Prometheus server to scrape the metrics from all Kafka and Zookeeper pods. With the integrated Kafka Exporter you have now also advanced metrics about topics, consumer groups and consumer group lag.
	Storage

	### An efficient data storage infrastructure is essential to the optimal performance of AMQ Streams. AMQ Streams requires block storage. The use of file storage (for example, NFS) is not recommended.

	### AMQ Streams supports three types of data storage:
	* Ephemeral (Recommended for development only)
	* Persistent
	* JBOD (Just a Bunch of Disks, suitable for Kafka only. Not supported in Zookeeper.)
	* AMQ Streams also supports advanced operations such as adding or removing disks in Kafka brokers or resizing the persistent volumes (where supported by the infrastructure).
	Documentation

	### Documentation for the current release can be found on Red Hat website.
	* https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html-single/using_amq_streams_on_openshift/index

	## Deployment

	### Kafka cluster

	```
	apiVersion: kafka.strimzi.io/v1beta1
	kind: Kafka
	metadata:
	creationTimestamp: '2020-06-24T17:28:04Z'
	generation: 1
	name: my-cluster
	namespace: amq-demo
	resourceVersion: '84722'
	selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/amq-demo/kafkas/my-cluster
	uid: 1a5acfb5-5b1a-41ca-8220-f36e222f6d71
	spec:
	entityOperator:
	topicOperator: {}
	userOperator: {}
	kafka:
	config:
	log.message.format.version: '2.5'
	offsets.topic.replication.factor: 3
	transaction.state.log.min.isr: 2
	transaction.state.log.replication.factor: 3
	listeners:
	plain: {}
	tls: {}
	replicas: 3
	storage:
	type: ephemeral
	version: 2.5.0
	zookeeper:
	replicas: 3
	storage:
	type: ephemeral
	status:
	conditions:
	- lastTransitionTime: '2020-06-24T17:30:09+0000'
	status: 'True'
	type: Ready
	listeners:
	- addresses:
	- host: my-cluster-kafka-bootstrap.amq-demo.svc
	port: 9092
	bootstrapServers: 'my-cluster-kafka-bootstrap.amq-demo.svc:9092'
	type: plain
	- addresses:
	- host: my-cluster-kafka-bootstrap.amq-demo.svc
	port: 9093
	bootstrapServers: 'my-cluster-kafka-bootstrap.amq-demo.svc:9093'
	certificates:
	- \|
	-----BEGIN CERTIFICATE-----
	MIIDLTCCAhWgAwIBAgIJAPuoDob0JlW7MA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
	BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjAwNjI0
	MTcyODA0WhcNMjEwNjI0MTcyODA0WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
	FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
	CgKCAQEAuQDNSOFg5MgqtU85sjuXwPYYZhIguIrueDm6wlt4imcxv/ZSwciNabYo
	0LfAAIBb8hlkYWb5//qa6x5tvvtUlrwlZDuc52vur+sphhqE2XAvNEkWK5D1BG/t
	OLSBv7btd+x3fS1/I5UHmxTu+l9/JNq8X/cKsAFovDC4Ldd7cURVQH9EgEpDtNqH
	XgeMlQP+ZkXWLERarcq7EccH7wDnfTa98+JsctLzkC+/wmLV0hXwNUqT+t13Lz2x
	IlWFWHlCA8Sn5owNjqU+SfkHk41d0dJN4PqZJb7Mo2zu+R080AnSP3vMih3s9ZSr
	1gvN8a7bo0eWM21++tG5/5eVx0ScnwIDAQABo1AwTjAdBgNVHQ4EFgQU5LNvastM
	ww5IjDzrlbYVXiJ/bCswHwYDVR0jBBgwFoAU5LNvastMww5IjDzrlbYVXiJ/bCsw
	DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAs3GlFydj8gibErTx7j9U
	if/qS/esbPHYdfJePJfL0GnCsRIIm9ESSWzdXSpyh5F+F8KrGso/mUkfrycaYrAW
	DIqlde181yx9e/HjIBdy+VflbzidaF3QIkxi8uDkVhGRHfbkAIybD/knxhqNy4Sv
	ncVe1pHesMDTsBPz1U+9xWbOHlaDmWLxZrdU9NlUgD2pk70kVcx595GmGdKRKoPY
	fJiFIVmikyqTEM9X9zZKMGsdrUDEVfmGGivBrObzPNYheH3q/GV4UT3haucVULT8
	PAXWoilyQitU0oEBOopxCb0c2j600NGAib01eLqC08P+wo3JzSaRyIJYnt+IX5h1
	3A==
	-----END CERTIFICATE-----
	type: tls
	observedGeneration: 1

	```



	```
	$ oc get all

	NAME READY STATUS RESTARTS AGE
	pod/amq-streams-cluster-operator-v1.5.0-7cfd955dc7-g9lkv 1/1 Running 0 52m
	pod/my-cluster-entity-operator-545c7d774f-plp4w 3/3 Running 0 39m
	pod/my-cluster-kafka-0 2/2 Running 0 40m
	pod/my-cluster-kafka-1 2/2 Running 0 40m
	pod/my-cluster-kafka-2 2/2 Running 0 40m
	pod/my-cluster-zookeeper-0 1/1 Running 1 40m
	pod/my-cluster-zookeeper-1 1/1 Running 0 40m
	pod/my-cluster-zookeeper-2 1/1 Running 0 40m

	NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
	service/my-cluster-kafka-bootstrap ClusterIP 172.30.104.100 <none> 9091/TCP,9092/TCP,9093/TCP 40m
	service/my-cluster-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 40m
	service/my-cluster-zookeeper-client ClusterIP 172.30.163.219 <none> 2181/TCP 40m
	service/my-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 40m

	NAME READY UP-TO-DATE AVAILABLE AGE
	deployment.apps/amq-streams-cluster-operator-v1.5.0 1/1 1 1 52m
	deployment.apps/my-cluster-entity-operator 1/1 1 1 39m

	NAME DESIRED CURRENT READY AGE
	replicaset.apps/amq-streams-cluster-operator-v1.5.0-7cfd955dc7 1 1 1 52m
	replicaset.apps/my-cluster-entity-operator-545c7d774f 1 1 1 39m

	NAME READY AGE
	statefulset.apps/my-cluster-kafka 3/3 40m
	statefulset.apps/my-cluster-zookeeper 3/3 40m

	```