Skip to content

Conversation

@adityastic
Copy link

@adityastic adityastic commented Jun 19, 2025

This PR takes over #416

This commit addresses the need to access GCP Monitoring's system_labels metadata
(like machine_type) which are not returned by default through the APIs. By adding
support for metric aggregation configuration, users can now explicitly instruct
the API to include system labels in the response.

Related issues which could use this feature:

Changes:

  • System Labels Access: GCP Monitoring's system_labels metadata (e.g., machine_type)
    are not returned by default through the APIs
  • Metrics Enhancement: Enables better visibility with system labels

New Features:

  • Metric Aggregation Configuration: Add support for specifying custom aggregation
    options per metric prefix using --monitoring.metrics-with-aggregations flag
  • System Labels Integration: Automatically include system labels from Stackdriver
    metadata in exported metrics
  • Enhanced Configuration Format: Support for alignment period, cross-series
    reducer, group-by fields, and per-series aligner configuration

Technical Implementation:

  • Add MetricAggregationConfig struct to handle aggregation configuration
  • Implement aggregation logic in monitoring collector with proper API integration
  • Add JSON unmarshaling for system labels from Stackdriver metadata
  • Extend command-line interface with new aggregation configuration flag

Configuration Format:

The new aggregation configuration uses the format:
metric_name:alignment_period:cross_series_reducer:group_by_fields:per_series_aligner

Example for machine_type access:
compute.googleapis.com/reservation/used:60s:REDUCE_SUM:resource.label."resource_container",resource.label."location",resource.label."reservation_id",metadata.system_labels."machine_type",metric.label."reserved_resource_type":ALIGN_MEAN

Use Case Example:

To expose machine_type dimension for reservation metrics:

--monitoring.metrics-with-aggregations="compute.googleapis.com/reservation/used:60s:REDUCE_SUM:resource.label.\"resource_container\",resource.label.\"location\",resource.label.\"reservation_id\",metadata.system_labels.\"machine_type\",metric.label.\"reserved_resource_type\":ALIGN_MEAN"

Documentation:

  • Update README.md with new configuration option and usage examples

@adityastic
Copy link
Author

@SuperQ mind giving it a review! Thanks!

@terryyanko
Copy link

@adityastic Thank you for this! Would it be possible to grab the user defined labels as part of this PR as well? 🙏 https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeSeries#MonitoredResourceMetadata

@kgeckhart
Copy link
Contributor

👋 @adityastic is this something that requires all the extra configuration for system_labels?

Looking at the docs, https://cloud.google.com/monitoring/alerts/doc-variables#null-values, it seems as though the requirement is that the system labels be used in a filter. Could someone configure the system labels they want directly and we tack on a system_label!="" to the query?

I think we could have a separate conversation regarding introducing support for aggregations/reduce configuration. If we go forward with the current implementation I don't see many people being able to take advantage of system_labels due to the complexity required to configure their support. I would very much like to see a scrape result from your example config to see how the metrics types and names are treated. Applying any of the reduce functions seems like it would dramatically change the context of the metric but that won't be reflected in the metric name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants