spec: Linux system packages meta architecture#10
Conversation
|
Out of curiosity for whether AI would find the interface versioning idiomatic, I asked Claude:
Claude: In both .deb and RPM conventions, the trailing 1 is an ABI/major-version slot embedded directly in the package name. It signals “this is version-slot 1 of this component,” and its primary purpose is to allow parallel installation of incompatible major versions on the same system. You see this pattern everywhere: For something like opentelemetry-injector1, the implication is: In .deb land this is especially common for shared libraries (tied to the ELF SONAME), but it’s also used for non-library packages that inject into processes or have ABI-sensitive interfaces — which an OTel injector almost certainly does, since it likely ships a .so that gets LD_PRELOAD-ed or injected via /etc/ld.so.preload. So the 1 is essentially a forward-compatibility escape hatch baked into the name. |
| ├── java/ | ||
| │ └── otel-config.yaml | ||
| ├── nodejs/ | ||
| │ └── otel-config.yaml | ||
| └── dotnet/ | ||
| └── otel-config.yaml |
There was a problem hiding this comment.
Wouldn't it be better to use a single config file?
There was a problem hiding this comment.
Not really, one of the most useful functionalities is going to be the enablement and disablement of instrumentations, and those are language-specific.
There was a problem hiding this comment.
Isn't that already handled given instrumentation config is already split based on language see https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema-docs.md
Further how would the multiple config scenario work with opamp to deploy the config?
There was a problem hiding this comment.
Isn't that already handled given instrumentation config is already split based on language see https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema-docs.md
To my understanding, no, it is not. IIRC @jack-berg also had a remark that it is best to keep the configs for different languages separately as the behavior of trying to disable an instrumentation that does not exist for the SDK is very chancy.
Further how would the multiple config scenario work with opamp to deploy the config?
Assuming SDKs get OpAMP support (which I never heard to be a thing tbh), then they configure to use OpAMP and it works as normal. For the collector, it's a different story, and that already gets today a separate config and separate system package.
There was a problem hiding this comment.
Well it turns out Java does support OpAMP, so I take that back. Let me think about it a bit.
There was a problem hiding this comment.
Are we really needing to recommend a dedicated file per language?
Yes, different languages need different instrumentation opt-in/opt-out lists
Is that not introducing chance of divergence in config ie resource attributes?
It may, but the long-term vision is to have a OpenTelemetry Collector package included in the overall picture and then resource detection would be done there.
Could a single opamp supervisor on a system handle deploying the multiple config files of the same type to different paths?
I don't know.
Will the injector inject the correct config file path if there is multiple?
Yes, the one config file per language.
There was a problem hiding this comment.
Yes, different languages need different instrumentation opt-in/opt-out lists
This is news to me and differs to what the schema/example in the config spec shows https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema-docs.md#experimentalinstrumentation which is that you are allowed to set options for multiple languages in the one file.
Yes, the one config file per language.
I know it would be possible but reading https://github.com/open-telemetry/opentelemetry-injector/blob/main/README.md#environment-modifications it doesn't appear to be existing functionality.
There was a problem hiding this comment.
This is news to me and differs to what the schema/example in the config spec shows https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema-docs.md#experimentalinstrumentation which is that you are allowed to set options for multiple languages in the one file.
I see, I had missed that development.
I know it would be possible but reading https://github.com/open-telemetry/opentelemetry-injector/blob/main/README.md#environment-modifications it doesn't appear to be existing functionality.
Indeed it does not yet, but it is an easy addition if we need it (and Antoine and I are maintainers for the Injector too).
This being said, if one single config file across multiple languages is indeed feasible, maybe that would both solve OpAMP issues and reduce overhead for end users to define e.g. common resource attributes across all languages. I do however suspect that the moment we move to one config filer for all languages, somebody will definitely come up with a very legitimate reason to split some languages out in their deployments, so we should not lock ourselves in.
There was a problem hiding this comment.
I agree we should not lock ourselves in and make it harder to use other otel technologies.
I do however suspect that the moment we move to one config filer for all languages, somebody will definitely come up with a very legitimate reason to split some languages out in their deployments
Thinking about it some more would defining more structure to the distribution block in declarative config help for those scenarios that we could say for this auto-instrumentation use these options? Just like we already do for tracers etc.
There was a problem hiding this comment.
I did some research into the actual state of the instrumentation.<language> sections in the declarative configuration schema and their implementation across SDKs.
The schema does define per-language keys under instrumentation/development (java, js, dotnet, python, cpp, etc.), which would in theory allow a single config file to carry language-specific instrumentation settings. However, the actual implementation status tells a different story:
| SDK | Declarative config | instrumentation.<lang> section |
Actually consumed at runtime |
|---|---|---|---|
| Java | Experimental | Defined & supported | Yes |
| .NET | Experimental | Marked not_applicable in support matrix; .NET has its own file-based config implementation that does not consume the shared schema's instrumentation.dotnet key |
Own impl only |
| Node.js | Experimental | Marked unknown; not implemented in the configuration package — it only handles providers, exporters, and propagators |
No |
| Python | In development | Model defined but never wired up — _sdk.py skips instrumentation entirely |
No (dead field) |
| C++ | Stable | Marked not_applicable |
No |
Summary: Java is the only SDK that actually reads and acts on the language-specific instrumentation section from the shared schema. .NET has its own parallel implementation. Node.js and Python don't support instrumentation configuration from a YAML file at all today.
Given this, I think per-language config files remain the right approach for the system packages for the foreseeable future. If the SDKs converge on a single shared schema for instrumentation in the future, we can revisit, but we'd be designing for a reality that doesn't exist yet.
I did adopt @jack-berg's suggestion to rename the config files from otel-config.yaml to otel-sdk-config.yaml to future-proof against potential collector config conflicts and to make a single-file consolidation easier down the road if it becomes viable.
| A vendor replacement package *must* own the same set of paths as the upstream package it replaces. | ||
| This is what makes `--replaces` work: `dpkg` allows the new package to overwrite files owned by the replaced package. | ||
|
|
||
| ## Vendor Override |
There was a problem hiding this comment.
Thoughts about lowering the barrier to entry for vendors/companies by enabling declarative config to define the auto-instrumentation which is to be used.
The injector would use a path based on declarative if it is defined otherwise uses the conf file.
This would reduce the need for companies to be hosting their own packages.
There was a problem hiding this comment.
I doubt vendors would like to ship a package containing only configurations. All distros I have seen (and build myself over the years), are code plus configurations. But I would be very interested in being proven wrong.
There was a problem hiding this comment.
Vendors wouldn't ship a package containing only configurations and in fact they wouldn't ship an OS package at all. They would only ship their instrumentation package ie npm. The process to install it would be install the official otel package, add the custom instrumentation to your config which is then deployed using opamp to the instances, the opamp agent then installs it on the instances hence lowers the barrier to entry.
There was a problem hiding this comment.
In that case, the OpAMP would simply override the system package's then?
There was a problem hiding this comment.
Correct the agent specified in the config would override the builtin agent.
There was a problem hiding this comment.
Yes vendors are more likely to make a system package as they have a range of customers who would use it hence a public repository/registry works but that is not the only use-case. The other use case would be Companies who build their own auto instrumentation distribution with additional instrumentations.
It could technically be possible for vendors to create independent system packages and have users install them and then modify the configs of the community packages to have those instrumentations additionally activated.
I didn't suggest that and wouldn't recommend it, the suggestion was define in declarative config an override of the auto instrumentation which is to be used.
About activating those additional instrumentations by config, it is expected for users to modify the declarative configurations we ship (which would also be the way to activate OpAMP, by the way), as with pretty much any file living in /etc. But this all would be such a cumbersome process for adopters that I really don't think makes sense to consider.
I think I see where the confusion is, I am invisioning using opamp to configure the system not just the language agent. This enables a non cumbersome solution which is scalable from a few instances to many. The process for users would be the following:
- install opentelemetry system package
- configure opamp address
- the configuration is downloaded from the opamp server
- if config/opamp server defines custom instrumentation it is installed
Hence only 1 thing needs to be configured
There was a problem hiding this comment.
At the moment, OpAMP is simply not part of the vision for these packages. OpAMP is a topic that never came up in the definition of the project. If somebody from @open-telemetry/opamp-go-maintainers and @open-telemetry/opamp-java-maintainers would like to barge in and help out, it is something I'll gladly engage with. But having one file how all language is not sound with respect to instrumentation opt-in and opt-out, which is fundamental when dealing with autoinstrumentations where the user has not upfront control of which are injected.
There was a problem hiding this comment.
OpAMP is simply not part of the vision for these packages. OpAMP is a topic that never came up in the definition of the project.
I don't know why opamp might have been overlooked however the collector couod be considered as an opamp agent hence deploying that introduces opamp. Anyway, I don't particularly see packaging having much interaction with opamp other than deploying the agent/collector. But the key thing for me is not to present to companies that the only option if they want to deploy otel using their own auto-instrumentation is to build their own system package.
The simplest path to address this would be to add a note mentioning something like:
open telemetry does offer alternative method/s ie opamp to install packages ie the auto instrumentation agent onto a target system. however these processes are out of scope of the system packaging and it is their responsibility to ensure that the package they installed will be loaded automatically and they can co-exist with the system package.
That way it is clear other pathways are available.
having one file how all language is not sound with respect to instrumentation opt-in and opt-out, which is fundamental when dealing with autoinstrumentations where the user has not upfront control of which are injected.
The config is already split per language for instrumentation options hence shouldn't be an issue.
There was a problem hiding this comment.
Due to good improvements of the configuration support in SDKs, we went in the end with one configuration file for all languages
There was a problem hiding this comment.
The OpAMP discussion is a bit stuck: open-telemetry/opamp-spec#360
|
|
||
| | Field | DEB | RPM | | ||
| |-------|-----|-----| | ||
| | Architecture | `amd64` or `arm64` | `x86_64` or `aarch64` | |
There was a problem hiding this comment.
Keep in mind, that the latest version supports the same set of systems as the .NET team: https://dotnet.microsoft.com/en-us/download/dotnet
There was a problem hiding this comment.
Acknowledged — we'll track the .NET team's supported runtime versions when building the package.
| | Field | DEB | RPM | | ||
| |-------|-----|-----| | ||
| | Architecture | `amd64` or `arm64` | `x86_64` or `aarch64` | | ||
| | Provides | `opentelemetry-dotnet-autoinstrumentation1` | `opentelemetry-dotnet-autoinstrumentation1` | |
There was a problem hiding this comment.
1 value, 1 version can be breaking change in terms of supported set of .NET versions.
Current version supports .NET 8. 9, 10, In the November we will go to .NET 10 and 11 only (8 and 9 will be no longer supported).
I am not sure if it has any impact on your plans.
There was a problem hiding this comment.
I think here there is some confusion about the package interface version (the 1 in opentelemetry-dotnet-autoinstrumentation1) and the dependencies of the opentelemetry-dotnet-autoinstrumentation (notice, no 1 package), which will then have its own package version (e.g., v1.2.3) and dependencies / conflicts with the actual .NET packages carrying the runtimes.
cpaelzer
left a comment
There was a problem hiding this comment.
Hi,
sorry it ended up being so many comments, but since I come with no context
due to being asked to have a look out of the blue I have only a few "oh you
should change ..." but many "did you mean a or maybe consider b" (like the
comments about how much do you want to be like lib SONAMES) which you
have to decide and then act or ignore. I have mentioned each topic once,
but if you adapt e.g. the dependency graph for one of my comments there might
be chances that you also need to change the pakcage metadata section. But
bringin up the same topic many times would not help after I've seen how many
things I raised on the first few lines.
I hope that it helps you to get this forward, after all even those you
consciously say "no, we meant it to be that way" will be good to be spelled out :-)
Finally: Humans are limited, and so is reviewing a high level spec.
At some point in this evolution you will have first experimental RPM and Deb
builds and I highly recommend to then run lintian in pedantic mode for debs
and for rpms rpmlint/rpminspec/fedora-review (others would know better which to
pick) and be open to adjust the packaging but also the design with all they
throw at you. I'm essentially asking you to plan for that feedback loop
to happen even after we think "yep, seems reasonable" here.
P.S. It was a bit hard to read a spec, to then find "Current POC Gaps" into
"Required Changes to the POC". I'm not sure if I should try to rewrite all I
said :-/ But after a short check I found that most still applies, I can only assume that at some point you will roll up all that is down there plus all the outcome of the discussion into a single description - right?
Co-authored-by: Piotr Kiełkowicz <[email protected]>
Addresses review comment from @Kielek to use the official project name "OpenTelemetry .NET Automatic Instrumentation" consistently.
…interface versions
…ior of the operator
1a26f59 to
6fdb98d
Compare
7927e2b to
6fdb98d
Compare
|
Ongoing PoC at #18 |
…ng#18 Sync the meta-architecture document with the state of the PoC: add the Python auto-instrumentation package throughout, describe the shared valid-as-shipped declarative configuration file and its activation via OTEL_CONFIG_FILE, rename otelinject.conf to injector.conf, reflect the glibc-only .NET bundle, mark the injector PoC gaps as historical, and move the nfpm remark to the implementation notes.
When looking into my notes about the packages architecture (which packages we want to ship, which meta packages, which virtual, dependencies, etc.), I found out I had cut corners in the PoC in terms of implementing the virtual packages for languages, and that I had missed out entirely on future-proofing against "interface" changes among packages. The design document in this PR is intended to fill this gap.
Closes #5