Nice use of lazy_static! to implement a cache. Although a process can change cgroups, I'm not aware of any container orchestration system which would actually do so, so I agree that a cache is appropriate.

I think this is an appropriate implementation for all existing profilers, but it should be noted that the cgroup membership of the profiler may be different than the cgroup membership of the profiled application. I don't think there's anything to do about that for now, just sort of thinking out loud.

Indeed, this could be problematic for whole host profiling.
Do you have in mind other cases where this could happen ?

Yes, but those scenarios aren't really common any more and our average customer uses a commoditized cloud-based VM with a commoditized cgroup management system (containers), so I wouldn't bother splitting hairs over my dumb edge cases.

Mostly, I was thinking about the exact scenario you mentioned. Reading through your implementation more carefully after my comment, it's clear to me that the current API will work without any modifications. I think what we'd do for the whole-host profiler is something like:

• scrape the container ID for a specific PID
• collect PIDs into disjoint sets according to their container membership
• aggregate call stack information in a pprof for each container
• send each container's export separately, with its own tags
• Anything in a pure-kernel thread, or which cannot otherwise be checked for cgroup membership, can just get associated to an export which will get tagged with the profiler's own container ID

Anyway, nothing to be done about it at this point, we don't even have an end-to-end complete whole-host profiler close to shipping soon. :)

I'm familiar with these regex lines from both the specified source, as well as https://github.com/Qard/container-info/blob/master/index.js. I understand the goal here is to extract an ID which is meaningful for a given container management system, without knowing which system you're using, so that the Datadog Agent can query that system for additional information on the service implemented by the container.

However, it is completely opaque to me which container management systems are hit by which regex under what conditions and what other limitations may be imposed. I don't think it's appropriate to go out and collect all that information in this PR, but I want to note that these regexes are based on conventions of different systems rather than any kind of standard.

Accordingly, would it be appropriate to add a comment explaining where these regexes come from? Right now the significance of this operation is somewhat mysterious/magical.

I am also very confused by these magical regexes.
I found some documentation:

• https://github.com/DataDog/architecture/blob/master/rfcs/apm/agent/containers-tagging/rfc.md
• https://docs.google.com/document/d/1nxc4nN2NWy0w0Sez4FG3jmqfw_FT_VtMroHfqYp6qio/edit#

Each language seems to implement approximately the same process, but still not quite exactly the same.
PHP has some king of precedence of task ID over container ID (https://github.com/DataDog/dd-trace-php/blob/master/components/container_id/container_id.c#L228) because for FarGate we return a Task ID instead of a Container ID. But this precedence is not implemented in other languages and I am not sure it's possible to both have container IDs and task IDs in the same /proc/self/cgroup. @morrisonlevi WDYT ?

• Java has not such precedence, but returns the result of the last matching line: https://github.com/DataDog/dd-trace-java/blob/master/utils/container-utils/src/main/java/datadog/common/container/ContainerInfo.java#L164
• Python returns the result of the first matching line: https://github.com/Yun-Kim/dd-trace-py/blob/1.0-dev/ddtrace/internal/runtime/container.py#L105.
• Go does the same: https://github.com/DataDog/dd-trace-go/blob/v1/internal/container.go#L52

Great information!! Thank you. Unfortunately, now I'm even more confused.

I think there's something significant here about the structure and expectations around /proc/self/cgroup. Formally speaking, a process can participate in a different cgroup for each resource type, and it doesn't really offend my sensibilities to imagine that a container orchestration system would implement resource pools instead of soft/hard caps. In fact, this is a concept we had in HPC schedulers for 20+ years. On the other hand, if a given container management system actually wanted to do this, I imagine it would preserve the integrity of the "container ID" concept by creating a sub-cgroup named with an appropriate ID for any given resource pool... I don't know, I can speculate forever, but all I'm writing here is fictional until proven otherwise.

I guess what I'm saying is, it seems like this mechanism works in a fairly reliable way. I'm not aware of any major complaints from the other tracer/profiler teams. The fact that it works is sort of mysterious to me.

Anyway, I don't think there's anything to be done about any of this, I'm just wasting your time and mine with pointless discussion. Thanks again for sharing those sources, it was very informative for me (although I don't feel any closer to understanding the situation). 8)

Great idea--this is nice because it allows the caller to override the cgroup ID without us having to worry about an API change later.

Oh nice, pretty slick! Let me see if I get this right. If get_container_id() threw an error, then Some(container_id) will try to expand Some(None), which will be false, and the tag will be skipped entirely.

The thing on my mind is, how will the user (or support engineers, like ourselves!) ever figure out the environmental factors that caused a container_id to be omitted? What if we had a special "Datadog-Container-ID-errors" tag (or something) that gave a one-word description of why the "Datadog-Container-ID" couldn't be populated? In the average case, we're sending along ~megabyte of data or more, and only once per minute, and almost always to the local TCP stack, so it's not like header minimization improves much, and it might have significant diagnostic value later.

No need to block on that, just an idea.

If I understand Rust correctly if let Some(container_id) = containerid::get_container_id() is actually a kind of pattern matching. get_container_id returns an Option<&str>, ie. either Some(&str) or None.
if statement will be true only if Some(&str) is returned, this allows to both capture the returned value and do the pattern matching at the same time.

Good remark on the error reporting.
I am not sure that reporting errors through tags is a good idea, but perhaps it would be useful to have some king of local logging to be able to debug issues.
In extract_container_id I tried to distinguish between:

• IO errors: mainly /proc/self/cgroup not present, I don't see how we could fail to read it if it is present, or how it could contain invalid UTF8
• No container ID found (either because we don't run in a container or the layout in /proc/self/cgroup is unknown to us)
  But this distinction is lost with get_container_id.

Ah, indeed. Putting an error in a tag does decouple the error from the source and it does deprive the calling application from doing something about it, but interestingly this is a class of error which is not that useful to report to the profiler, but is very useful to be able to infer from the UX where the data is actually consumed. From a diagnostic perspective, there's this weird interplay between the developer and the user.

Either way, I don't think this needs to be solved, and if it does need to be solved I don't think it needs to be done in this PR. I agree with your classification of the different error modes, and I think the most general issue we'll run into is when the user is running in a system that has cgroup namespaces enabled (as does Ubuntu 21.04 IIRC). In those cases, the host OS will probably tell us more than any particular error tag (if we know to look...)