Updating otel gateway config sample

[LikeTheSalad]

What does this PR do?

Adding the new elasticapm processor to the rest of signals in the gateway configuration sample.

Why is it important?

The new processor, elasticapm, was created to support all signals, not only traces. However, the outcome is the same as if it only supported traces if we don't encourage users to add it to other signals as well.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @LikeTheSalad? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[theletterf]

@LikeTheSalad I think we should at least backport to 9.2. To which versions can this apply?

[LikeTheSalad]

@LikeTheSalad I think we should at least backport to 9.2. To which versions can this apply?

Good point, I've added the label. Thanks!

[theletterf]

Would be nice to roll this out in 9.2.1 I guess, and first get a review from the Collector team.

[gregkalapos]

After seeing the other discussion from the internally linked repo, I'm not sure we want to do this either.

This would mean that every single log and metric will be enriched with basically apm related fields. So let's say if a service only sends otel logs, and don't really use apm, then we'd still add a bunch of fields, which may not be needed.

@LikeTheSalad what's the exact use-case here?

I know mobile agents rely on logs for lots of things... is that maybe the reason?

Could we maybe handle this by providing a dedicated pipeline for those agents where we enrich logs and metrics, but then keep the general pipeline without this enrichment?

Also.. should we add to both, or would only log enrichment be enough?

[LikeTheSalad]

I know mobile agents rely on logs for lots of things... is that maybe the reason?

Yes. The mobile UI dashboards rely on enriched logs.

Could we maybe handle this by providing a dedicated pipeline for those agents where we enrich logs and metrics, but then keep the general pipeline without this enrichment?

I'm not sure about the deeper details of the EDOT Collector, but I guess as long as there's a way for users to send their mobile telemetry data in a way that logs are also enriched, it works for me.

Also.. should we add to both, or would only log enrichment be enough?

For mobile use cases, logs are enough. And I can't foresee a use case that would need enriched metrics. The reason I added them is because the elasticapm processor supports metrics too, so it seems likely that since we added that support, we should advertise it in the docs.