Skip to content

Avoid sending deleted events for already completed jobs#133

Merged
mgarabed merged 3 commits into
masterfrom
mg/abort-aborts
Oct 7, 2020
Merged

Avoid sending deleted events for already completed jobs#133
mgarabed merged 3 commits into
masterfrom
mg/abort-aborts

Conversation

@mgarabed

Copy link
Copy Markdown
Contributor

Requirements for Contributing to this repository

  • Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
  • The pull request must only fix one issue at the time.
  • The pull request must update the test suite to demonstrate the changed functionality.
  • After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.

What does this PR do?

In cases where Jenkins is configured to only keep the latest X builds, there will be a deletion event for old builds. These get reported to Datadog as BuildAbortedEvent and can be confusing to the user.

This PR checks if the deleted job has actually completed already, and if so, it will have sent an event with its results previously, so can be ignored.

Description of the Change

Alternate Designs

Possible Drawbacks

It's possible the user may want to see these events, in which case we should perhaps add a config option to allow it? That seems unlikely, but feedback is welcome.

Verification Process

Additional Notes

Release Notes

Review checklist (to be filled by reviewers)

  • Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
  • PR should not have do-not-merge/ label attached.
  • If Applicable, issue must have kind/ and severity/ labels attached at least.

return;
}

// If the build already has a result, this could be a Jenkins cleanup operation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not knowledgeable enough regarding how those deletion happen and how to identify but it seems that we we try to achieve here is to prevent submitting an "AbortedEvent" for builds that are cleaned up.

A second approach could be to check the endTime of the job, if its "old enough" we can consider that this is just a clean up. But that approach could still submit some AbortedEvent if the clean up happens "too fast".

Not saying this approach is better but wanted to make sure we choose the best approach. Your comment mentions that "it could be a Jenkins cleanup operation", we'd rather be confident regarding that assumption before merging. Any thoughts?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that we want to get this right. The potential issue with time gating the results is it will vary wildly depending on how a user has their Jenkins config setup. They could have a limit of 10 builds for something that runs once a day (builds deleted after 10 days), or they could have a recurring job that flushes things within an hour or two. Picking a good default would be the hard part, and may end up causing confusion if some events are reported as Aborted and others are silently dropped.

The thought (possibly flawed) for this approach was that a build which had a status already assigned would have sent an event prior to this one, so its progress would have been recorded at Datadog. Builds which get deleted before reaching that stage seem like the important ones to track.

We could also add an option on whether to send these events and default it to True - it may be that some customers like that "end-to-end" visibility and some would rather clean it up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you validated that approach with a test instance of Jenkins? If so I think this is the best approach to choose and we can move forward with it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, have confirmed it on a local test instance and verified that completed jobs didn't have Aborted events sent when they were deleted.

Next will add a test to this PR to make sure we have the functionality covered. It's a little tricky since the activity spans two existing test classes (BuildAbortedEventTest.java and DatadogBuildListenerTest.java).

}

// If the build already complete, this could be a Jenkins cleanup operation
if (buildData.isCompleted()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: updated this clause to check for the underlying completeBuild flag instead of null. See the breakdown of what's considered complete in the Result class.

ofek
ofek previously approved these changes Oct 6, 2020

@ofek ofek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic appears to be sound 👍

FlorianVeaux
FlorianVeaux previously approved these changes Oct 7, 2020
Comment thread src/test/java/org/datadog/jenkins/plugins/datadog/clients/DatadogClientStub.java Outdated
Co-authored-by: Florian Veaux <[email protected]>
@mgarabed
mgarabed dismissed stale reviews from FlorianVeaux and ofek via f74c534 October 7, 2020 12:44
@mgarabed
mgarabed merged commit 685b75e into master Oct 7, 2020
@mgarabed
mgarabed deleted the mg/abort-aborts branch October 7, 2020 15:58
@mgarabed mgarabed added changelog/Changed Changed features results into a major version bump changelog/Fixed Fixed features results into a bug fix version bump and removed changelog/Changed Changed features results into a major version bump labels Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/Fixed Fixed features results into a bug fix version bump

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants