Skip to content

Feature/tags for elasticsearchwriter#10074

Merged
julianbrost merged 3 commits intoIcinga:masterfrom
open-i-gmbh:feature/tags-for-elasticsearchwriter-6837
Apr 10, 2025
Merged

Feature/tags for elasticsearchwriter#10074
julianbrost merged 3 commits intoIcinga:masterfrom
open-i-gmbh:feature/tags-for-elasticsearchwriter-6837

Conversation

@SebastianOpeni
Copy link
Copy Markdown
Contributor

Offers to add additional tags to entries written by the ElasticSearchWriter.

As discussed with Eric at the icinga summit we are submitting this feature. Please revise the code and let us know if there is something we need to improve. otherwise we would be happy for timely merge.

fixes #6837

@cla-bot
Copy link
Copy Markdown

cla-bot bot commented Jun 7, 2024

Thank you for your pull request. Before we can look at it, you'll need to sign a Contributor License Agreement (CLA).

Please follow instructions at https://icinga.com/company/contributor-agreement to sign the CLA.

After that, please reply here with a comment and we'll verify.

Contributors that have not signed yet: @SebastianOpeni

Details
  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Please contact us if you think this is the case.

  • If you signed the CLA as a corporation, your GitHub username may not have been submitted to us. Please reach out to the responsible person in your organization.

@icinga-probot icinga-probot bot added the area/elastic Events to Elasticsearch label Jun 7, 2024
Copy link
Copy Markdown

@ngoeddel-openi ngoeddel-openi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

The CLA should be signed now.

@bobapple
Copy link
Copy Markdown
Member

bobapple commented Jun 7, 2024

@cla-bot check

@cla-bot cla-bot bot added the cla/signed label Jun 7, 2024
Copy link
Copy Markdown
Member

@yhabteab yhabteab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for your contribution! I've just been looking over the code changes and left some inline comments below, but haven't tested it with an Elastic/OpenSearch instance yet. Other than that, can you please rebase this against the current master to fix the GitHub actions. Thanks!

@yhabteab yhabteab added this to the 2.15.0 milestone Sep 13, 2024
@yhabteab yhabteab added the enhancement New feature or request label Sep 13, 2024
@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

Hi thanks for your review :)
I am currently working on your remarks but might have some questions beforehand.

@SebastianOpeni SebastianOpeni force-pushed the feature/tags-for-elasticsearchwriter-6837 branch from 44a55ec to 2f3c2fa Compare September 23, 2024 10:45
}
}
}
fields->Set("tags", tags);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure if tags is the right field for this. Isn't it common that tags is just a simple list instead of key-value pairs? I would set a different field or make it configureable.

Copy link
Copy Markdown
Contributor Author

@SebastianOpeni SebastianOpeni Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The field tags as a dictionary was used because the influxdb-writer is also using it that way.
The functionality would stay the same with a different field name so if you have a preference I could change it accordingly.
(labels would come to mind as a alternative.)

Copy link
Copy Markdown
Contributor Author

@SebastianOpeni SebastianOpeni Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave it some thought and I could also simply allow tags to be of type array. (in addition to dict)
That would change some of the logic we had reviewed until now but if that would be preferred by you, I could also implement it that way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid, in perfdata DB context, they indeed are key-value pairs:

Please show me where we set a field called tags in one of them. Supporting tags in both just means adding key-value pairs, with no reference to a tags field, right?. But here we set the field where I wonder if users of Elasticsearch or similar would expect a list rather than key-value pairs. tag_set sounds more reasonable to me. Anyway, before we just merge this, I'd like to get an informed opinion.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Icingas influxdbcommonwriter this is done here:
https://github.com/Icinga/icinga2/blob/master/lib/perfdata/influxdbcommonwriter.cpp#L253

But tags never reaches Influx DB.

The graphite writer does only allow to configure templates for the host and service name.

Otherwise there would not be a tags field either.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found it as well just after Commenting. 🙈
That was my bad.
I'm very sorry for the confusion.

Copy link
Copy Markdown
Contributor Author

@SebastianOpeni SebastianOpeni Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To prevent confusion about the Type of tags, would you prefer to:

  • Rename the field to tag_set or
  • Add the configured key-values directly into the elasticSearch-Entries as done in the influxdb case?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To prevent confusion about the Type of tags, would you prefer to:

  • Rename the field to tag_set or

I think a tag_set property would be rather confusing in ES/Kibana.

  • Add the configured key-values directly into the elasticSearch-Entries as done in the influxdb case?

This, in contrast, has the advantages:

  • No tags. boilerplate in our ES data
  • Consistency with InfluxDB

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After my changes the configured tags are now added to the root element.
That also allows for tags = ["foo", "bar"] if desired.

Notice:
Since the templated_tags are added after the default fields and the performance Values this change will also allow to overwrite them.

@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

Thanks for your answers in the review.
I tried to implement everything as requested.

@yhabteab
Copy link
Copy Markdown
Member

Apart from #10074 (comment), code wise it should be fine now, I haven't tested it with a actual ElasticSearch instance though, but till I get the time, some of my colleagues might want a look at this as well.

@yhabteab yhabteab requested a review from Al2Klimov September 25, 2024 12:23
}
}
}
fields->Set("tags", tags);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Al2Klimov

This comment was marked as outdated.

Copy link
Copy Markdown
Member

@yhabteab yhabteab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this today with ElasticSearch 7.17.25 and it seems to be working fine 👍🏼. Though , whilst going through the changes again, I just noticed two small things, see inline comments below.

Edit: While you're at it, please rebase this against the master branch to get the GHAs pass.

@SebastianOpeni SebastianOpeni force-pushed the feature/tags-for-elasticsearchwriter-6837 branch from b64788f to 14c47d2 Compare November 26, 2024 17:59
@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

SebastianOpeni commented Dec 3, 2024

I think I worked on all your requested changes, and you also marked them as resolved, but somehow they are still blocking the pipeline 🤔

@yhabteab
Copy link
Copy Markdown
Member

yhabteab commented Dec 3, 2024

I think I worked on all your requested changes, and you also marked them as resolved, but somehow they are still blocking the pipeline 🤔

Hi, yes, thanks for that! Unfortunately, the GitHub actions were changed again with #10253 just after you rebased this against the master branch. You just need to rebase and force push again :)!

@SebastianOpeni SebastianOpeni force-pushed the feature/tags-for-elasticsearchwriter-6837 branch from 14c47d2 to cbe7cc9 Compare December 3, 2024 09:47
@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

Thanks for your fast reply :)
I just pushed the rebased version.
I hope the tests run smoothly :)

@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

SebastianOpeni commented Dec 9, 2024

The checks seem to be not the problem, but rather github does not register that we closed all your previously existing change requests. I think I have to "re-request" a review from you. 🤔

@yhabteab
Copy link
Copy Markdown
Member

The checks seem to be not the problem, but rather github does not register that we closed all your previously existing change requests. I think I have to "re-request" a review from you. 🤔

Hi, sorry for the delay! Unfortunately, there was again some changes made to the GitHub actions :-), and I wanted to rebase and clean up the commit histories for you but I'm not allowed to push it into this PR :(.
Bildschirmfoto 2025-01-24 um 08 54 22

@SebastianOpeni
Copy link
Copy Markdown
Contributor Author

Happy new year and nice to hear from you 😃
Thank you for the effort, I invited you with write permissions I hope that will give you the necessary possibilities :)

@yhabteab yhabteab force-pushed the feature/tags-for-elasticsearchwriter-6837 branch from cbe7cc9 to c78c7de Compare January 24, 2025 10:06
@yhabteab
Copy link
Copy Markdown
Member

I invited you with write permissions I hope that will give you the necessary possibilities :)

Thanks! But it would have been enough to just check the Allow edits by maintainers box.

@yhabteab yhabteab force-pushed the feature/tags-for-elasticsearchwriter-6837 branch from c78c7de to 54cb29e Compare January 24, 2025 10:11
Copy link
Copy Markdown
Member

@yhabteab yhabteab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LFTM!

@yhabteab yhabteab requested a review from lippserd January 24, 2025 14:04
@julianbrost
Copy link
Copy Markdown
Member

@yhabteab Can you please state what you expect @lippserd to review here?

@yhabteab
Copy link
Copy Markdown
Member

yhabteab commented Apr 8, 2025

@yhabteab Can you please state what you expect @lippserd to review here?

Since @lippserd requested a change in #10074 (comment) not to send the tags fields to elastic search, so I wanted to know whether the changed (flattened) version of those matches his expectation.

@julianbrost julianbrost merged commit 8d607d2 into Icinga:master Apr 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/elastic Events to Elasticsearch cla/signed enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support host and service templates to send variables to Elasticsearch

7 participants