Follow-up to https://twitter.com/avdi/status/716600435021385728

chunked-event-publishing.md

Chunked event publishing - design overview

Class: EventCollector

Roles:

• #enqueue_event(payload, target queue name): stores messages to be sent in a simple array
• #publish() send everything currently enqueued to external queue system (here: RabbitMQ); if the publish fails for any reason, messages are kept in memory for later retry (along with grumbling in logs, etc).
• #clear_events_to_publish! is useful in rollback scenarios
• .thread_singleton_instance accessor with lazy initialization, like ActiveRecord::Base.connection. This allows for easy setup, and persistent state across requests (should flushing fail we can retry on next request on the same thread). The thread_singleton_instance also easy replacement with a simple test implementation, think ActionMailer. Sending actual messages in test is best restricted to a few integration test scenarios.

Class: AutoPublishingEventCollector

Roles:

• ensure a PublishingEventCollector is setup for the current block of code
• call #publish after a block of code has completed successfully
• useful in a trivial Rails middleware, for instance, or in the console

Main patterns of use

• in the Rails console it is usually convenient to autoflush after each enqueued event.
• when handling requests, it makes sense to automatically call #publish after each successful request.
• message consumers (worker processes) are usually explicitly calling #publish at the appropriate times. Most other code paths ignore the existence of queues and events.
• in tests it makes sense to assert on the existence or absence of messages matching specific patterns after some code has run. Having collected the messages with topics in memory these matchers are usually easy to create.

The above was implemented on top of sneakers / RabbitMQ, but I think the pattern is generally useful.