Question.md

Some Source and sone Channel in Knative Eventing

Note: Pseudo code...

Below is some yaml (and no implementation) for Kafka Source and Channel...

We need a ClusterProvisioner for our Source e.g. like the container provisioner from Nicolas (see #513):

apiVersion: eventing.knative.dev/v1alpha1
kind: ClusterProvisioner
metadata:
  name: container
spec:
  reconciles:
    group: eventing.knative.dev
    kind: Source

and another one for the Channel (e.g. the Kafka one from Sabari in #486)

apiVersion: eventing.knative.dev/v1alpha1
kind: ClusterProvisioner
metadata:
  name: kafka
spec:
  reconciles:
    group: eventing.knative.dev/v1alpha1
    kind: Channel

Now, that we have the two Provisioners together....

Next we would need a channel:

apiVersion: eventing.knative.dev/v1alpha1
kind: Channel
metadata:
  name: mychannel
spec:
  provisioner:
    ref:
      apiVersion: eventing.knative.dev/v1alpha1
      kind: ClusterProvisioner
      name: kafka

And we want to connect to Apache Kafka on our source, so we might have:

apiVersion: eventing.knative.dev/v1alpha1
kind: Source
metadata:
  name: kafkaevents
  namespace: default
spec:
  provisioner:
    ref:
      name: container
  arguments:
    image: docker.io/mrbean/kafkasource
    args:
      bootstrapservers: "something.somewhere.com:9092"
      subscribe_to_topic: "the_truth"

Now, we need to hook our "service" to the channel:

apiVersion: eventing.knative.dev/v1alpha1
kind: Subscription
metadata:
  name: listentochannel
  namespace: default
spec:
  from:
    kind: Channel
    apiVersion: eventing.knative.dev/v1alpha1
    name: kafka
  call:
    target:
      kind: Service
      apiVersion: serving.knative.dev/v1alpha1
      name: my-kafka-logger

Question: I am not sure how to actually connect the Channel to the Source, so that the Source can write to the channel, and my ksvc on the Subscription can get messages from that populated channel?

The Source can specify an optional ObjectRef to a channel so that it can send to it.

And I think the subscription.spec.from.name should be “mychannel” to match the name of the channel

The Source can specify an optional ObjectRef to a channel so that it can send to it.

it can, but that field is ignored today.

This should be:

apiVersion: eventing.knative.dev/v1alpha1
kind: Subscription
metadata:
  name: listentochannel
  namespace: default
spec:
  from:
    kind: Source
    apiVersion: eventing.knative.dev/v1alpha1
    name: kafkaevents
  call:
    target:
      kind: Service
      apiVersion: serving.knative.dev/v1alpha1
      name: my-kafka-logger

I am not sure how to actually connect the Channel to the Source, so that the Source can write to the channel, and my ksvc on the Subscription can get messages from that populated channel?

This is a point of contention in the spec proposal.

The current proposal says that the Source should reference the Channel it wants to emit events to, if not specified, then the provisioner must create a channel and emit to it. The Subscription would then be from the kafkaevents source.

My proposal in n3wscott/eventing#4 says that a Source should never be responsible for creating a Channel and should only emit events to an exiting Channel (mychannel in your example). The listentochannel subscription should be from mychannel.

In the example, you have both a Source and Channel backed by Kafka. The example may be more clear if you picked different technologies to back the Source or Channel.

Example source with a ref channel, if it worked:

apiVersion: eventing.knative.dev/v1alpha1
kind: Source
metadata:
  name: kafkaevents
  namespace: default
spec:
  provisioner:
    ref:
      name: container
  channel:
    ref:
      kind: Channel
      apiVersion: eventing.knative.dev/v1alpha1
      name: mychannel
  arguments:
    image: docker.io/mrbean/kafkasource
    args:
      bootstrapservers: "something.somewhere.com:9092"
      subscribe_to_topic: "the_truth"

I think there are two questions here (sorry for jumping in late):

1. Who creates the channel
2. Can you use Source as a 'From' when creating a subscription.

I don't have strong opinions on #1, I can see both being useful.

for #2, I believe it to be very useful to be able to Subscribe to a Source as a user action. We're only dereffing the underlying channel. This makes sense as we create additional constructs like Pipeline where you want to create a Subscription to a Pipeline. You can also Subscribe to another Subscription (if it has the proper interface, if there's no 'Result' Channel, then you can't which makes sense, it doesn't emit events). So, this duck typing I believe to be fundamental property in being able to create a usable system. Otherwise, you're left hunting for which channels belong to where.

The main benefit of channels in a messaging system is to decouple producers from consumers. In other words they aren't supposed to "belong" to either. Producers and consumers should be able to come and go without impacting each other, and if you connect directly - treating the channel as an implementation detail, then you lose that benefit. Also, channels enable many:many relationships between producers and consumers, so if a consumer subscribes directly to just one of many producers sharing a target channel, that can lead to unintended consequences. The channels themselves could advertise via metadata what they have to offer subscribers so that "hunting for which channels" is the right thing to do.

I'm not at all opposed to providing more convenience in higher level CRDs, but I think it's important to first define a solid foundation in the base level CRDs, and that should be motivated by the well-established principles of messaging systems and enterprise integration patterns.

I've added a slightly modified sample here: https://gist.github.com/matzew/b77acef9db9d8331e25a0f629c3d5d00

with suggestions from the above comments (e.g. use different technologies for Source and Channel)