feat(agent): Print plugins source information #16270

neelayu · 2024-12-06T19:11:45Z

Summary

Print the source information for the plugins.

Put this feature behind a flag to preserve backward compatibility in systems relying on Loaded inputs, Outputs strings for validations.
Flag introduced: --print-plugin-config-source Default: false

If required, we can eliminate this flag.

Sample output on telegraf init

2024-12-06T19:05:36Z I! Loaded inputs: cpu disk (2x) internal interrupts mem (2x) net (3x)
+--------------+------------------------------------------------------+
| Name         | Source(s)                                            |
+--------------+------------------------------------------------------+
| net (3x)     | /Users/user-of-unix/telegraf/configs1/config3.conf   |
|              | /Users/user-of-unix/telegraf/configs1/config1.conf   |
|              | /Users/user-of-unix/telegraf/configs1/config2.conf   |
+--------------+------------------------------------------------------+
| disk (2x)    | /Users/user-of-unix/telegraf/configs1/config2.conf   |
|              | /Users/user-of-unix/telegraf/configs1/config2.conf   |
+--------------+------------------------------------------------------+
| mem (2x)     | /Users/user-of-unix/telegraf/configs1/config4.conf   |
|              | /Users/user-of-unix/telegraf/configs1/config2.conf   |
+--------------+------------------------------------------------------+
| internal     | /Users/user-of-unix/telegraf/configs1/config2.conf   |
+--------------+------------------------------------------------------+
| interrupts   | /Users/user-of-unix/telegraf/configs1/config2.conf   |
+--------------+------------------------------------------------------+
| cpu          | /Users/user-of-unix/telegraf/configs1/config5.conf   |
+--------------+------------------------------------------------------+

Checklist

No AI generated code was used in this PR

Related issues

resolves #16269

srebhan · 2024-12-09T18:01:37Z

@neelayu I really like this! Can we please make the flag being something like --print-plugin-config-source or --print-plugin-origin? This would be a bit more speaking I think and avoids the next flag being called --even-newer-plugin-print-behavior. ;-)

Furthermore, please fix the linter issues.

neelayu · 2024-12-10T13:26:39Z

@srebhan Thanks. I have made the changes. The test failure seems unrelated to my change though.

srebhan

Thanks @neelayu for the update. Some more comments from my side.

config/config.go

srebhan · 2024-12-13T10:36:41Z

config/config.go

 // LoadConfigData loads TOML-formatted config data
-func (c *Config) LoadConfigData(data []byte) error {
+func (c *Config) LoadConfigData(data []byte, opts ...cfgDataOption) error {


I think you should just pass the source here. If we ever want to add other flags we can still refactor this later.

The reason for introducing functional ops pattern was to prevent breaking other files calling this func. And there are about 50+ across 20+ files.

I see your point but my fear is that people might forget about adding the source in future changes and we will not notice. You might keep this as is but I really would feel better if we enforce providing a source... ;-)

I am inclined to keep this as-is for this PR. I can open up a second one to enforce the func signature change. Let me know your thoughts.

Please change the function to take a mandatory path or source parameter and unconditionally add the source to the plugin models. The only place that should be conditional should be the printing.

Remember this is a public, exported function and we shouldn't change the signature too often. Having the plugin source in the model is a good thing so please add it.

config/plugin_printer.go

neelayu

Thanks @srebhan

config/plugin_printer.go

config/config.go

neelayu · 2024-12-13T11:01:19Z

config/config.go

 // LoadConfigData loads TOML-formatted config data
-func (c *Config) LoadConfigData(data []byte) error {
+func (c *Config) LoadConfigData(data []byte, opts ...cfgDataOption) error {


The reason for introducing functional ops pattern was to prevent breaking other files calling this func. And there are about 50+ across 20+ files.

srebhan · 2024-12-20T08:53:04Z

@neelayu you need to update the /docs/LICENSE_OF_DEPENDENCIES.md file with the new dependencies' license information. Please check make check-deps for what needs to be done!

neelayu · 2024-12-20T13:37:38Z

@srebhan
I believe I have already added them here https://github.com/influxdata/telegraf/pull/16270/files#diff-2afa363df9dd5dc9cbf8b97542633fda73bf1747a7959b0da02e518a976db745R460

Not sure why it is asking me to remove them again?

--- /tmp/tmp.3FJLxju4ZG/LICENSE_OF_DEPENDENCIES.md      2024-12-16 06:45:42.992042502 +0000
+++ /tmp/tmp.3FJLxju4ZG/HEAD    2024-12-16 06:45:42.992042502 +0000
@@ -233,0 +234 @@
+github.com/jedib0t/go-pretty
@@ -259,0 +261 @@
+github.com/mattn/go-runewidth
@@ -333,0 +336 @@
+github.com/rivo/uniseg
@@ -455,3 +457,0 @@
-github.com/jedib0t/go-pretty
-github.com/mattn/go-runewidth
-github.com/rivo/uniseg


The docs/LICENSE_OF_DEPENDENCIES.md file does not contain the expected entries.

Lines prefixed with '+' should be added to LICENSE_OF_DEPENDENCIES.md and '-'
lines should be removed.

Include a link to the appropriate licenses for any additions.

srebhan · 2024-12-20T15:39:20Z

Seems like you got the positions wrong. The entries are in lexicographic order so you also need to insert the lines in the right places. ;-)

neelayu · 2024-12-20T16:51:52Z

Thanks @srebhan.
Made the changes now

srebhan

@neelayu thanks for your update and sorry for this taking so long (due to holidays). I do have two minor comments in the code and a request to no go for the option pattern for the LoadConfigData function. Let's change the function signature once and remove any room for errors.

config/plugin_printer.go

go.mod

neelayu · 2025-01-16T08:24:29Z

@srebhan I have enforced the func signature. Let me know. Thanks!

telegraf-tiger · 2025-01-16T08:50:09Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

srebhan

Thanks a lot @neelayu!

go.mod

llamafilm · 2025-03-12T16:00:18Z

This is a cool feature, but it's rather hard to read with a very large config file. For example, it prints 819 lines for my config file. Would there be some way to condense the output when you have hundreds of inputs that all come from the same file?

Additionally, when running under systemd with structured log format writing logs to stdout, this hits the journald.conf default limit of 48K per line, so journald breaks the JSON into two lines which breaks any downstream log consumers, making it look like this:

Increasing the journald LineMax might have other implications.

neelayu · 2025-03-12T16:41:22Z

@llamafilm
For the first case, I think we can avoid redundancy. Thanks for pointing it out.
For the second case, I am not sure on how to display a table under a structured log format.

llamafilm · 2025-03-13T01:36:33Z

@neelayu it's actually the same issue, so if you condense the output to avoid redundancy, the output will be short enough to easily fit within a single line.

Hipska · 2025-03-25T09:07:56Z

Wow, totally missed this feature. Nice done!

However, wouldn't it have been nicer to use Unicode box drawing for this?

2024-12-06T19:05:36Z I! Loaded inputs: cpu disk (2x) internal interrupts mem (2x) net (3x)
┌──────────────┬──────────────────────────────────────────────────────────┐
│ Name         │ Source(s)                                                │
╞══════════════╪══════════════════════════════════════════════════════════╡
│ net (3x)     │ /Users/user-of-unix/telegraf/configs1/config3.conf       │
│              │ /Users/user-of-unix/telegraf/configs1/config1.conf       │
│              │ /Users/user-of-unix/telegraf/configs1/config2.conf       │
├──────────────┼──────────────────────────────────────────────────────────┤
│ disk (2x)    │ /Users/user-of-unix/telegraf/configs1/config2.conf (2x)  │
├──────────────┼──────────────────────────────────────────────────────────┤
│ mem (2x)     │ /Users/user-of-unix/telegraf/configs1/config4.conf       │
│              │ /Users/user-of-unix/telegraf/configs1/config2.conf       │
├──────────────┼──────────────────────────────────────────────────────────┤
│ internal     │ /Users/user-of-unix/telegraf/configs1/config2.conf       │
├──────────────┼──────────────────────────────────────────────────────────┤
│ interrupts   │ /Users/user-of-unix/telegraf/configs1/config2.conf       │
├──────────────┼──────────────────────────────────────────────────────────┤
│ cpu          │ /Users/user-of-unix/telegraf/configs1/config5.conf       │
└──────────────┴──────────────────────────────────────────────────────────┘

Using +, | and - looks so out-dated 😉

neelayu · 2025-03-26T11:01:46Z

@Hipska
thanks for the feedback.

This can be achieved. Not sure if we can have a distinct separator for the header.

Hipska · 2025-03-27T09:01:40Z

@neelayu I checked the used module for printing the tables and there is the option to use these unicode symbols (table.StyleLight), but indeed not to have specific heading separator. jedib0t/go-pretty#365

telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Dec 6, 2024

srebhan self-assigned this Dec 9, 2024

srebhan added the area/agent label Dec 9, 2024

neelayu force-pushed the plugin-printer branch from 39f3020 to c885748 Compare December 10, 2024 10:34

srebhan reviewed Dec 13, 2024

View reviewed changes

neelayu commented Dec 13, 2024

View reviewed changes

neelayu force-pushed the plugin-printer branch from 6c83f2d to f066526 Compare December 14, 2024 13:05

neelayu force-pushed the plugin-printer branch from e0ec63d to 13fe6a0 Compare December 20, 2024 16:37

srebhan reviewed Jan 15, 2025

View reviewed changes

config/plugin_printer.go Outdated Show resolved Hide resolved

go.mod Outdated Show resolved Hide resolved

neelayu force-pushed the plugin-printer branch from 13fe6a0 to 487823e Compare January 16, 2025 08:23

srebhan approved these changes Jan 17, 2025

View reviewed changes

srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Jan 17, 2025

srebhan assigned mstrandboge and unassigned srebhan Jan 17, 2025

mstrandboge reviewed Jan 21, 2025

View reviewed changes

go.mod Outdated Show resolved Hide resolved

neelayu added 7 commits January 21, 2025 21:05

Add basic framework

feaccee

Put this behind a flag

bd45158

Add support for secret store

be78258

Minor fixes in table printer

bb7f66f

Changes in tests

6e32b60

Make linter happy

2ec96f8

Rename flag variable

1e78dc6

neelayu added 6 commits January 21, 2025 21:05

Fixing the test by using filepath instead of hardcoded strings

c31b6dc

Address review comments

8b68c00

Add license information

3630a6c

Reorder licenses

cf8617b

address comments and enforce path in function signature

324ee69

move dependency

460515a

neelayu force-pushed the plugin-printer branch from 487823e to 460515a Compare January 21, 2025 15:43

mstrandboge approved these changes Jan 22, 2025

View reviewed changes

mstrandboge merged commit 0ec17e4 into influxdata:master Jan 22, 2025
20 of 24 checks passed

github-actions bot added this to the v1.34.0 milestone Jan 22, 2025

neelayu mentioned this pull request Mar 14, 2025

Plugin source printer becomes too large if same plugin is used multiple times in a single file #16637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(agent): Print plugins source information #16270

feat(agent): Print plugins source information #16270

neelayu commented Dec 6, 2024 •

edited

Loading

srebhan commented Dec 9, 2024

neelayu commented Dec 10, 2024

srebhan left a comment

srebhan Dec 13, 2024

neelayu Dec 13, 2024

srebhan Dec 13, 2024

neelayu Dec 14, 2024

srebhan Jan 15, 2025

neelayu left a comment

neelayu Dec 13, 2024

srebhan commented Dec 20, 2024

neelayu commented Dec 20, 2024

srebhan commented Dec 20, 2024

neelayu commented Dec 20, 2024

srebhan left a comment

neelayu commented Jan 16, 2025

telegraf-tiger bot commented Jan 16, 2025

Artifact URLs

srebhan left a comment

llamafilm commented Mar 12, 2025

neelayu commented Mar 12, 2025

llamafilm commented Mar 13, 2025

Hipska commented Mar 25, 2025

neelayu commented Mar 26, 2025

Hipska commented Mar 27, 2025

feat(agent): Print plugins source information #16270

feat(agent): Print plugins source information #16270

Conversation

neelayu commented Dec 6, 2024 • edited Loading

Summary

Checklist

Related issues

srebhan commented Dec 9, 2024

neelayu commented Dec 10, 2024

srebhan left a comment

Choose a reason for hiding this comment

srebhan Dec 13, 2024

Choose a reason for hiding this comment

neelayu Dec 13, 2024

Choose a reason for hiding this comment

srebhan Dec 13, 2024

Choose a reason for hiding this comment

neelayu Dec 14, 2024

Choose a reason for hiding this comment

srebhan Jan 15, 2025

Choose a reason for hiding this comment

neelayu left a comment

Choose a reason for hiding this comment

neelayu Dec 13, 2024

Choose a reason for hiding this comment

srebhan commented Dec 20, 2024

neelayu commented Dec 20, 2024

srebhan commented Dec 20, 2024

neelayu commented Dec 20, 2024

srebhan left a comment

Choose a reason for hiding this comment

neelayu commented Jan 16, 2025

telegraf-tiger bot commented Jan 16, 2025

Artifact URLs

srebhan left a comment

Choose a reason for hiding this comment

llamafilm commented Mar 12, 2025

neelayu commented Mar 12, 2025

llamafilm commented Mar 13, 2025

Hipska commented Mar 25, 2025

neelayu commented Mar 26, 2025

Hipska commented Mar 27, 2025

neelayu commented Dec 6, 2024 •

edited

Loading