Event Bus Plug.md

Nuxeo Event System

Nuxeo Event System is part of nuxeo-core and provide :

• a way to fire events
  • events are associated with context (user, document)
  • the repository does fire event for pretty much everything it does
  • all services (custom or not) can choose to fire any type of event associated with any type of context
• a way to listen to events
  • synchronously: have a listener executed inside the transaction firing the event
  • asynchronously: run a listener on a bundle of the events that were fired during a transaction

Initially inside Nuxeo, a lot of default processing depends on events and listeners :

• part dublincore meta-data are updated via a listener
• Audit Log is updated via a listener
• Notification System relies on listeners
• ...

From JMS to In-Memory

The whole event system is present since the very beginning of Nuxeo and initially it was relying on the JMS message queuing. For performance and deployment reason, we progressively moved away from JMS and switched to a pure "in memory" message queuing.

From Event Queuing to Job Queuing

In memory message queuing is very efficient, but by definition it is not durable: if you loose the Nuxeo node that produced the event, the even will be lost too.

This problem is particularly visible when it comes to asynchronous processing that is handled by asynchronous listeners (like picture or video conversions that are fired after file is updated).

To overcome this, all asynchronous listeners are actually wrapped into Work that are handled by the WorkManager :

• associate works to queues
• associate threads to queues

Since the work manager can handle "persistence" via the Redis backend, asynchronous process can not longer be "lost" if a Nuxeo nodes is interrupted abruptly.

However, we have to acknowledge the fact that instead of queuing events, we are queuing workers: even if this solves the persistence issue, this is still a pattern that comes with some limitations.

Why we want this to be better

Queuing works instead of messages comes with some constraints :

• there are far more works than events
  • one event can fire several listeners hence several works
• worker are bigger than events
  • worker can be associated with big fat java classes

As a result, it puts more pressure on the WorkManager and because there are more works than events, this is actually more complex to handle than if it was handled at event level.

8.10+ and pluggable event pipe

In order to better address this issue, we have made the event system pluggable so that it becomes possible to contribute different event handling logic.

The new system changes the way the EventBundles are handled:

• rather than being directly wrapped as Work
• they are sent to a EventPipe dispatcher that dispatched the events between several EventPipe
  • each Pipe
    • may do filtering
    • may do queuing in a external system
    • may feed the workmanager

The underlying idea is that EventBundles may be sent to one or more pipe depending on how we want to process them :

• for example for Audit Log: we may want to
  • pre-process the event to extract the required information
  • queue this Audit message
  • consume the messages and write in bulk and bypass the WorkManager
• for async workers: we may want to
  • queue as much as possible the events
  • unqueue them only when a worker slot is available

Current status and future work

The new extension points have been added into the 8.10 release. However :

• Kafka based implementation will be released later
  • we are currently testing scalability on some real life use cases
• we are likely to do changes on the model: see TL-127

More details about the 8.10 extension points

• describe extension points
• give some examples
• link to test implementations