Note: Pseudo code...
Below is some yaml (and no implementation) for Kafka Source and Channel...
We need a ClusterProvisioner
for our Source
e.g. like the container provisioner from Nicolas (see #513):
apiVersion: eventing.knative.dev/v1alpha1
kind: ClusterProvisioner
metadata:
name: container
spec:
reconciles:
group: eventing.knative.dev
kind: Source
and another one for the Channel (e.g. the Kafka one from Sabari in #486)
apiVersion: eventing.knative.dev/v1alpha1
kind: ClusterProvisioner
metadata:
name: kafka
spec:
reconciles:
group: eventing.knative.dev/v1alpha1
kind: Channel
Now, that we have the two Provisioners together....
Next we would need a channel:
apiVersion: eventing.knative.dev/v1alpha1
kind: Channel
metadata:
name: mychannel
spec:
provisioner:
ref:
apiVersion: eventing.knative.dev/v1alpha1
kind: ClusterProvisioner
name: kafka
And we want to connect to Apache Kafka on our source, so we might have:
apiVersion: eventing.knative.dev/v1alpha1
kind: Source
metadata:
name: kafkaevents
namespace: default
spec:
provisioner:
ref:
name: container
arguments:
image: docker.io/mrbean/kafkasource
args:
bootstrapservers: "something.somewhere.com:9092"
subscribe_to_topic: "the_truth"
Now, we need to hook our "service" to the channel:
apiVersion: eventing.knative.dev/v1alpha1
kind: Subscription
metadata:
name: listentochannel
namespace: default
spec:
from:
kind: Channel
apiVersion: eventing.knative.dev/v1alpha1
name: kafka
call:
target:
kind: Service
apiVersion: serving.knative.dev/v1alpha1
name: my-kafka-logger
Question: I am not sure how to actually connect the Channel to the Source, so that the Source can write to the channel, and my ksvc
on the Subscription
can get messages from that populated channel?
The main benefit of channels in a messaging system is to decouple producers from consumers. In other words they aren't supposed to "belong" to either. Producers and consumers should be able to come and go without impacting each other, and if you connect directly - treating the channel as an implementation detail, then you lose that benefit. Also, channels enable many:many relationships between producers and consumers, so if a consumer subscribes directly to just one of many producers sharing a target channel, that can lead to unintended consequences. The channels themselves could advertise via metadata what they have to offer subscribers so that "hunting for which channels" is the right thing to do.
I'm not at all opposed to providing more convenience in higher level CRDs, but I think it's important to first define a solid foundation in the base level CRDs, and that should be motivated by the well-established principles of messaging systems and enterprise integration patterns.