Problem

As part of #8632, we want receivers to be able to reject incoming requests if the collector is running out of memory limits instead of accepting the data transforming to OTLP and get it rejected by the memory_limiter processor. Scraping receivers would skip the scraping in that situation.

We introduced a memory_limiter extension in #8964, but it has not been applied yet.

#9397 proposes a way to configure a memory_limiter extension explicitly by receivers using httpconfig. Applying only this approach to other receivers would complicate the user configuration interface as they would have to explicitly connect every receiver with a memory_limiter extension.

Proposed solution

We can introduce an option to configure the memory_limiter extension in a way that it's applied to every receiver. Particular receivers can override the default memory limiter and connect to a different one.

Configuration example:

extensions:
  memory_limiter:
    limit_mib: 800
    apply_to_all_receivers: true
  memory_limiter/internal_metrics:
    limit_mib: 1000

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  prometheus:
    memory_limiter: memory_limiter/internal_metrics
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ["localhost:8888"]

processors:
  batch:

exporters:
  debug:
    verbosity: detailed

service:
  extensions: [memory_limiter, memory_limiter/internal_metrics]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [debug]

In this configuration, the default memory_limiter extension is implicitly applied to otlp receiver, and the additional memory_limiter/internal_metrics extension is applied explicitly to the prometheus receiver.

The user cannot define more than one memory_limiter extension with apply_to_all_receivers: true; the config validation will fail.

If memory_limiter extension with apply_to_all_receivers: true is defined, all the receivers used in the pipelines must support the memory_limiter extension. Otherwise, the collector won't start.

Proof of concept: #9590