Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retrieve AWS EC2 tags using instance metadata #15789

Closed
kirillmandrygin opened this issue Aug 29, 2024 · 14 comments · Fixed by #15795
Closed

Retrieve AWS EC2 tags using instance metadata #15789

kirillmandrygin opened this issue Aug 29, 2024 · 14 comments · Fixed by #15795
Labels
feature request Requests for new plugin and for new features to existing plugins

Comments

@kirillmandrygin
Copy link

Use Case

I want to add EC2 tags using metadata and I don't want to use the instance role to do this.

AWS Documentation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html

Expected behavior

The AWS EC2 Metadata Processor plugin should support retrieving tags from IMDS

Actual behavior

The aws_ec2 processor plugin doesn't support this

Additional info

No response
@kirillmandrygin kirillmandrygin added the feature request Requests for new plugin and for new features to existing plugins label Aug 29, 2024
@srebhan
Copy link
Member

srebhan commented Aug 29, 2024

For my understanding, do you want the processor to use the instance-metadata endpoint instead of accessing the actual compute instance?

@kirillmandrygin
Copy link
Author

@srebhan currently processors.aws_ec2 supports two options imds_tags (from instance metadata) and ec2_tags (from ec2 through instance profile with ec2:DescribeTags policy). And I think the second option is not required because we have option to retrieve tags from instance metadata, but we can leave it for backward compatibility

@srebhan
Copy link
Member

srebhan commented Aug 29, 2024

Sorry I'm not exactly sure what you are requesting here? To me it sounds like you want to use the imds_tags part of the processor but as you opened an issue I'm clearly missing something here. :-)

@srebhan srebhan added the waiting for response waiting for response from contributor label Aug 29, 2024
@kirillmandrygin
Copy link
Author

I'd like to get instance tags via imds_tags without using ec2_tags and ec2:DescribeTags policy. Have you opened the link I provided in the initial request?

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Aug 29, 2024
@srebhan
Copy link
Member

srebhan commented Aug 29, 2024

That's exactly what should happen if you only specify the imds_tags without using the ec2_tags... What am I missing here?

And yes I've read the document you linked. ;-)

@kirillmandrygin
Copy link
Author

you can not get instance tags from imds_tags, for now you can get instance tags only from ec2_tags

@kirillmandrygin
Copy link
Author

kirillmandrygin commented Aug 29, 2024 •
edited
Loading

for now imds_tags retrieves tags from this url: /latest/dynamic/instance-identity/document/

telegraf/plugins/processors/aws_ec2/ec2.go

Line 146 in f693df8

iido, err := r.imdsClient.GetInstanceIdentityDocument(

https://github.com/aws/aws-sdk-go-v2/blob/main/feature/ec2/imds/api_op_GetInstanceIdentityDocument.go

But I want to get also EC2 instance tags from this path: /latest/meta-data/tags/instance

https://github.com/aws/aws-sdk-go-v2/blob/main/feature/ec2/imds/api_op_GetMetadata.go

@srebhan
Copy link
Member

srebhan commented Aug 29, 2024

I see. Sorry for not getting this earlier! Are you willing to contribute a pull-request?

@kirillmandrygin
Copy link
Author

Probably I didn't explain what I needed in the initial request well and that's what confused you. I apologise and it's great that we've come to an understanding

I'm not sure of my golang skills, but I'll try to add this feature myself next week ;)

@srebhan srebhan mentioned this issue Aug 30, 2024
Merged
1 task
@srebhan
Copy link
Member

srebhan commented Aug 30, 2024

@kirillmandrygin please test the binary in PR #15795, available as soon as CI finished the tests, and let me know if this is what you had in mind. You should use the new metadata_tags setting for your use-case...

@kirillmandrygin
Copy link
Author

kirillmandrygin commented Aug 30, 2024 •
edited
Loading

@srebhan it seems this is not working for me, I can see imds_tags but there is no metadata_tags

telegraf.conf:

[global_tags]
[agent]
  interval = "60s"
  collection_jitter = "0s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false
[[outputs.prometheus_client]]
  listen = ":9281"
  path = "/metrics"
  expiration_interval = "120s"
  collectors_exclude = ["gocollector", "process"]
  string_as_label = true
[[inputs.cpu]]
  percpu = false
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[processors.aws_ec2]]
  imds_tags = [ "accountId", "instanceId", "instanceType", "privateIp", "availabilityZone" ]
  metadata_tags = [ "Name", "Stage" ]
[[processors.template]]
  tag = "instance"
  template = '{{ .Tag "Name" }}-{{ .Tag "privateIp" }}'

telegraf version:

# /usr/bin/telegraf --version
Telegraf 1.32.0-6e912139 (git: pull/15795@6e912139)

tags are available through IMDS:

# curl -H "X-aws-ec2-metadata-token: $TOKEN" ${AWS_API_URL}/latest/meta-data/tags/instance/Name
test-kmandrygin

and there is no any error in debug log:

# /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d --debug
2024-08-30T07:15:58Z I! Loading config: /etc/telegraf/telegraf.conf
2024-08-30T07:15:58Z I! Starting Telegraf 1.32.0-6e912139 brought to you by InfluxData the makers of InfluxDB
2024-08-30T07:15:58Z I! Available plugins: 235 inputs, 9 aggregators, 32 processors, 26 parsers, 62 outputs, 6 secret-stores
2024-08-30T07:15:58Z I! Loaded inputs: cpu
2024-08-30T07:15:58Z I! Loaded aggregators:
2024-08-30T07:15:58Z I! Loaded processors: aws_ec2 template
2024-08-30T07:15:58Z I! Loaded secretstores:
2024-08-30T07:15:58Z I! Loaded outputs: prometheus_client
2024-08-30T07:15:58Z I! Tags enabled: host=**********
2024-08-30T07:15:58Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"***********", Flush Interval:10s
2024-08-30T07:15:58Z D! [agent] Initializing plugins
2024-08-30T07:15:58Z D! [processors.aws_ec2] Initializing AWS EC2 Processor
2024-08-30T07:15:58Z D! [processors.aws_ec2] Initializing AWS EC2 Processor
2024-08-30T07:15:58Z D! [agent] Connecting outputs
2024-08-30T07:15:58Z D! [agent] Attempting connection to [outputs.prometheus_client]
2024-08-30T07:15:58Z I! [outputs.prometheus_client] Listening on http://[::]:9281/metrics
2024-08-30T07:15:58Z D! [agent] Successfully connected to outputs.prometheus_client
2024-08-30T07:15:58Z D! [processors.aws_ec2] cache: size=1000
2024-08-30T07:15:58Z D! [agent] Starting service inputs
2024-08-30T07:16:08Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:18Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:28Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:38Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:48Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:58Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:17:08Z D! [outputs.prometheus_client] Wrote batch of 1 metrics in 87.327µs
2024-08-30T07:17:08Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics

@srebhan
Copy link
Member

srebhan commented Aug 30, 2024 •
edited
Loading

Could you please use the latest version of the binary in the PR and set

[[processors.aws_ec2]]
  log_level = "trace"
  imds_tags = [ "accountId", "instanceId", "instanceType", "privateIp", "availabilityZone" ]
  metadata_tags = [ "Name", "Stage" ]

as this should print out all received metadata key-value pairs (I hope)...

Could you please also provide the output of

# curl -H "X-aws-ec2-metadata-token: $TOKEN" ${AWS_API_URL}/latest/meta-data

@kirillmandrygin
Copy link
Author

@srebhan already provided trace here: #15795 (comment)

2024-08-30T11:34:00Z T! [processors.aws_ec2] received metadata: {values:map[{}:{Results:[{Err:<nil> Retryable:false Retried:false ResponseMetadata:{values:map[]}}]}]}

and please take a look at the output of the meta-data URL:

# curl -H "X-aws-ec2-metadata-token: $TOKEN" ${AWS_API_URL}/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-keys/
reservation-id
security-groups
services/
system
tags/

full path to a tag will be /latest/meta-data/tags/instance/<TAG_NAME>

@kirillmandrygin
Copy link
Author

Hi @srebhan ! It is working good but I have two issues:

  1. Error when I run telegraf with --test:
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x740411b]

goroutine 109 [running]:
github.com/influxdata/telegraf/plugins/processors/aws_ec2.(*AwsEc2Processor).Stop(0xc00252e9c0?)
	/go/src/github.com/influxdata/telegraf/plugins/processors/aws_ec2/ec2.go:166 +0x3b
github.com/influxdata/telegraf/models.(*RunningProcessor).Stop(...)
	/go/src/github.com/influxdata/telegraf/models/running_processor.go:115
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc001fec0c0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:666 +0x172
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors in goroutine 14
	/go/src/github.com/influxdata/telegraf/agent/agent.go:656 +0x3e
  1. Metadata tag added with slash (for example: test_metric{/Name="test-instance"}), is this expected behaviour?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request Requests for new plugin and for new features to existing plugins
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(processors.aws_ec2): Allow to use instance metadata srebhan/telegraf
2 participants
@srebhan @kirillmandrygin