[Questions] Missing metrics in per-object endpoint compared to aggregated metrics

[guoard]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.4

Erlang version used

27.3.x

How is RabbitMQ deployed?

Community Docker image

Steps to reproduce the behavior in question

It appears that some metrics present in the aggregated metrics are missing from the per-object metrics.

To verify, you can save the aggregated metrics to a file named metrics and the per-object metrics to a file named per_object, then run the following command to list metrics that exist in the aggregated set but not in the per-object set:

for metric in $(grep -vE "^#" metrics | awk '{print $1}' | cut -d{ -f1 | sort -u); do grep -q "$metric" per_object || echo "$metric"; done

result is:

process_start_time_seconds
rabbitmq_alarms_file_descriptor_limit
rabbitmq_alarms_free_disk_space_watermark
rabbitmq_alarms_memory_used_watermark
rabbitmq_unreachable_cluster_peers_count
telemetry_scrape_encoded_size_bytes_count
telemetry_scrape_encoded_size_bytes_sum

What problem are you trying to solve?

I’m scraping the per-object endpoint and wanted to add an alert for the rabbitmq_unreachable_cluster_peers_count metric. However, this metric isn’t present in the per-object endpoint.
Is this the expected behavior, or could it be a bug?

[mkuratczyk]

it's a judgment call whether per-object endpoint should return everything, or just the per-object metrics. I'm not sure we have a clear decision on this, so there could be that different components (metrics comes from many places) behave differently. We should probably decide and unify the behaviour, but I'm not sure which way. A quick workaround/solution is to scrape both endpoints

[gomoripeti]

Did not test but this might be fixed in 4.2.0 by @mkuratczyk (maybe not intentionally :) ) by a refactoring included in #13895
The PR makes sure that all metrics under /metrics are returned by the /metrics/per-object endpoint as well.

[mkuratczyk]

you're right, seems like this was a part of that reorg. somehow I thought that some of these have been out of scope of what I changed.

Anyway, @guoard please try 4.2.

[guoard]

Thanks! I checked 4.2 and the metrics are all present. Appreciate your help!