Khepri: can't migrate users from Mnesia

[JJCella]

Describe the bug

When enabling stable feature flags on rabbitmq 4.2.0, mnesia to khepri migration fail with "Tab copier iteration failed" "pattern_matches_no_nodes" error

The error details says that the permissions for a user ("token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4") do not match any node. This user was deleted a long time ago but still persists in the configuration files
Searching through the node files, references to this user are still found

grep -a -H 'token-' /var/lib/rabbitmq/mnesia/rabbit@node/*
...
rabbit_user_permission.DCD
rabbit_user.DCL
rabbit_durable_exchange.DCD

$ rabbitmqctl list_users | grep token
// NO OUTPUT
$ rabbitmqctl list_user_permissions token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4
Listing permissions for user "token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4" ...
Error:
{:no_such_user, "token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4"}

Reproduction steps

$ rabbitmqctl enable_feature_flag all

Enabling all feature flags ...

Error:
{:migration_failure, {:error, {:khepri_mnesia_migration, :converter_mod_error, %{error: {:error, {:EXIT, {:error, {:error, {~c"Tab copier iteration failed", {:error, {:khepri, :keep_while_conditions_not_met, %{node_name: "token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4", node_path: [:rabbitmq, :vhosts, "XXX", :user_permissions, "token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4"], keep_while_reason: {:pattern_matches_no_nodes, [:rabbitmq, :users, "token-0539f1ea-c3f5-ddd9-a1d7-8cce2aa4dfb4"]}}}}}}}}}, converter_mod: {:rabbit_db_m2k_converter, [{:rabbit_vhost, :rabbit_db_vhost_m2k_converter}, {:rabbit_user, :rabbit_db_user_m2k_converter}, {:rabbit_user_permission, :rabbit_db_user_m2k_converter}, {:rabbit_topic_permission, :rabbit_db_user_m2k_converter}, {:rabbit_runtime_parameters, :rabbit_db_rtparams_m2k_converter}, {:rabbit_queue, :rabbit_db_queue_m2k_converter}, {:rabbit_exchange, :rabbit_db_exchange_m2k_converter}, {:rabbit_exchange_serial, :rabbit_db_exchange_m2k_converter}, {:rabbit_route, :rabbit_db_binding_m2k_converter}, {:rabbit_node_maintenance_states, :rabbit_db_maintenance_m2k_converter}, {:mirrored_sup_childspec, :rabbit_db_msup_m2k_converter}, :rabbit_durable_queue, :rabbit_durable_exchange, :rabbit_durable_route, :rabbit_semi_durable_route, :rabbit_reverse_route, :rabbit_index_route, {:rabbit_exchange_type_consistent_hash_ring_state, :rabbit_db_ch_exchange_m2k_converter}, {:x_jms_topic_table, :rabbit_db_jms_exchange_m2k_converter}, {:rh_exchange_table, :rabbit_db_rh_exchange_m2k_converter}]}}}}}

Expected behavior

This user should no longer appear for the migration to proceed normally

Additional context

I tried recreating the user and then deleting it, but the problem persists

[michaelklishin]

@JJCella we need a way to reproduce this behavior. A standard list of Mnesia tables in a directory is not something we can work with. We do not even have any details on the history of this node (from what version it was upgraded, when the feature flags were enabled, whether definitions can be imported concurrently, and so on).

To my knowledge you are the only one reporting a Mnesia to Khepri migration error in close to a couple of years.

@yrigaudeau @MaximeDametis @TomDugue @DamienVicet @suromain @kukinsula FYI, our team really does not appreciate upvoting fests. An hour old issue without any details or prior reports has six reactions. Yeah, right, no doubt that's six different users.

Put that energy into helping the maintainers of a tool you almost certainly get for free, release after release, by providing a way to reproduce this migration failure.

[michaelklishin]

@JJCella "version 3" spans 3.0.0 from Nov 2012 all the way to 3.13.7 in 2024, and a triple digit number of releases between them.

[michaelklishin]

@JJCella I do not see any evidence of "configuration corruption", by which you likely mean "metadata store corruption".

Such "metadata store corruption" is extremely rare to see.

[JJCella]

@michaelklishin Node was created on version 3.8.14 ; "ghost" users were deleted on version 4.0.7 by API
I mean '(metadata store) corruption' in the sense that it contains data that no longer exists; this user should no longer be in the store. The term used is inaccurate, it’s more of a data inconsistency

Mnesia tables were fully synced and I called manually "rabbitmqctl enable_feature_flag all" to start the khepri_db migration ; I think this issue is only surfacing now, but it has been present since those user deletions

[michaelklishin]

Then we need to find a way to reproduce such "incomplete" user deletion on a 3.13.x or 4.x node with Mnesia.

Spending time on 3.8 to 4.0 migration to then migrate to 4.2 is a significant ask in terms of maintainer time, and all those versions are out of community support.

[JJCella]

@michaelklishin Okey ; Indeed; if this issue is not already known, I don’t think an investigation would be worth it for anyone. It may still be present, but it’s probably very difficult to reproduce (we delete many users, but only a few have remained)
Thank you for your time ; We can close this discussion

[michaelklishin]

We do not guess in this community but we have nothing to work with, so here is one hypothesis I have.

It can be that Mnesia was syncing its tables after a rolling node restart and at that moment, the khepri_db flag was enabled by automation.

So Mnesia was temporarily missing some tables or table rows, and the migration library has spotted the missing rows.