[Questions] Can a consistent hashing exchange be federated?

[YayBurritos]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.1

Erlang version used

27.3.x

Operating system (distribution) used

Ubuntu 24.04.2

How is RabbitMQ deployed?

Kubernetes Operator(s) from Team RabbitMQ

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

I don't believe deployment steps are relevant to the question I'm asking.

Steps to reproduce the behavior in question

Setup

Upstream RabbitMQ Server (Server A):

• 3 node cluster
• One x-consistent-hash exchange with multiple quorum queues bound to the exchange, all with routingKey = 1

Downstream RabbitMQ Server (Server B):

• 3 node cluster
• On downstream (Server B), create federation with upstream (Server A):

  rabbitmqctl set_parameter federation-upstream your-name-here '{"uri":"amqp://{user}:{password}@{upstream-server-a-url}:5672"}

• On downstream (Server B), create policy for exchange federation with upstream (Server A):
  • name: your choice!
  • pattern: pattern to match the x-consistent-hash exchanges on upstream (Server A)
  • apply to: exchanges
  • federation-upstream-set: all
• On downstream (Server B), create multiple quorum queues bound to the federated x-consistent-hash exchange on downstream (Server B), all with routingKey = 1

Setup Observations

• Check consistent hash ring state on upstream (Server A)

  rabbitmqctl eval 'rabbit_exchange_type_consistent_hash:ring_state(<<"/">>, <<"{your-consistent-hash-exchange-name-on-upstream-server-a}">>).'

• Note that the federated exchange from the downstream (Server B) is included in the hash ring.

Testing

After the Upstream & Downstream server setup, publish multiple messages to the x-consistent-hash exchange on the upstream (Server A) using various routing key values (e.g. 1, 10000, 5, 500, etc.).

Testing Observations

• If you publish enough messages to the x-consistent-hash exchange on the upstream (Server A), you may notice (as I did) that some messages end up being routed to the federated exchange on the downstream (Server B).
• With federation, I was expecting messages being published to the x-consistent-hash exchange on the upstream (Server A) to be distributed among the various queues bound to the exchange on the upstream (Server A) - so, normal x-consistent-hash exchange behavior - and, in addition, ALL messages would be sent to the federated exchange on the downstream (Server B) to then be routed as needed by the exchange on the downstream (Server B).
• The current behavior - which may be by design - 1) prevents the downstream (Server B) from acting as a backup/warm standby server for messages; and 2) results in the downstream "stealing" messages (in a sense) that should have instead been routed to a queue on the upstream (Server A).

Questions

1. Is this expected behavior? Or could I have some errors with my federation setup?
2. If this is expected behavior, does this mean that you cannot have a backup/warm standby if you're using x-consistent-hash exchanges?
3. Are there any other approaches for a backup/warm standby that can be used in place of federation?

Thanks for your time!

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

Trying to set up a backup/warm standby RabbitMQ server to receive ALL messages that are published to exchanges on the upstream, regardless of exchange type.

Federation seems to work for direct exchanges, but not for x-consistent-hash exchanges. I haven't tested federation with any other exchange types at this point.

[michaelklishin]

The consistent hashing exchange was never meant to be federated. It is entirely unaware of the state of the ring (or bindings, with the exception of the bindings the plugin manages) on the other side of federation.

Exchange federation is not aware of the details of how each exchange type is implemented. It supports the built-in types because they are fairly straightforward (their entire state is a set of bindings). The CHX is a different beast, it has additional state that the set of bindings does not fully express.

So when exchange federation binds an internally used queue (it has to use something to collect messages flowing through a federated exchange), that will change the CHX ring for the upstream exchange but nothing will change on the downstream because the downstream does not declare/bind any
internal queues, it consumes from the upstream's internal queue and re-publishes what it's consumed
locally.

With an extra internal queue bound to one CHX but not the other, their behaviors will differ because their hash rings are different.

[michaelklishin]

You are rolling your own "warm standby replication" and I know nothing about its assumption. Not every exchange type can realistically be federated, some are explicitly designed to be cluster-local.

Tanzu RabbitMQ has built-in support for warm standby replication, both of the schema and of messages from the queues and streams matched by a policy.

If you want to roll your own WSR solution, ask yourself whether the core team would be open to (or allowed to) helping a non-paying, non-contributing user to build an alternative.