Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

Erlang 20 and RabbitMQ.

RabbitMQ versions before 3.6.11 will not work correctly with OTP-20

When upgrading from OTP-19.x(or earlier) to OTP-20 all the persistent data will be permanently lost!

Although it's possible to run RabbitMQ with OTP-20 from scratch, there will be crashes in queue mirroring and management API.

The breaking change is the new external terms format, which was introduced in R16, but now is used by default. This term format will encode atoms as unicode with a different type tag. Unfortunately there is no way to generate a value in the old format, so it's not easy to perform migrations.

The format change will affect PID decoding and encoding (rabbit_misc:decompose_pid/1) , which will break mirroring, node renaming and management API. The issue was addressed in https://github.com/rabbitmq/rabbitmq-common/commit/4aa1bf4c1450fa8ba60b381c8b295e03fbd17c44

We also use term_to_binary to generate hash IDs for various entities in the system. While some entities are transient, like manegement binadings properties and federation links IDs, some of them are persistent and should not change during upgrade. An umbrella issue for that is https://github.com/rabbitmq/rabbitmq-server/issues/1243

It's been decided to not use term_to_binary to generate hashes in future versions (3.7 and above). Management and federation hash strategies were updated in: https://github.com/rabbitmq/rabbitmq-management-agent/pull/47 , https://github.com/rabbitmq/rabbitmq-management/pull/415 and https://github.com/rabbitmq/rabbitmq-federation/pull/58

As for persistent hashes, they are queue index directory names and STOMP generated queue names. They should be addressed in 3.6.11 Since we don't have upgrade steps for patch releases and cannot rename directories and queue names, we decided to generate the old term_to_binary format for specific types we use. The generation functions are in term_to_binary_compat module.

In STOMP queue generation, a tuple of strings or binaries was used, so the function is called string_and_binary_tuple_2_to_binary and accepts a 2-element tuple of strings or binaries. The issue is https://github.com/rabbitmq/rabbitmq-stomp/issues/115 This is addressed in https://github.com/rabbitmq/rabbitmq-server/pull/1262 and https://github.com/rabbitmq/rabbitmq-stomp/pull/116

Queue index directories is the most sensitive change, since RabbtiMQ will delete all the "unknown" queue index directories. When queue index names change, all the directories become "unknown", so all the persistent data will be lost. Issue https://github.com/rabbitmq/rabbitmq-server/issues/1243

In 3.6 we use compat function queue_name_to_binary to generate the same format of the directory name as in pre-20.

This is addressed in https://github.com/rabbitmq/rabbitmq-server/pull/1246

In 3.7 we decided to generate a queue index name with different algorithm, so we won't be affected with further changes in term_to_binary in future versions. The directories should be renamed during migration step. This will require to generate the old name first using compat function.

This is addressed in https://github.com/rabbitmq/rabbitmq-server/pull/1250

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.