Skip to content

Add NPE checks for jobs started with scripting#471

Merged
nikita-tkachenko-datadog merged 3 commits into
masterfrom
nikita-tkachenko/npes-fix
Nov 20, 2024
Merged

Add NPE checks for jobs started with scripting#471
nikita-tkachenko-datadog merged 3 commits into
masterfrom
nikita-tkachenko/npes-fix

Conversation

@nikita-tkachenko-datadog

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Nov 14, 2024

Copy link
Copy Markdown
Collaborator

Requirements for Contributing to this repository

  • Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
  • The pull request must only fix one issue at the time.
  • The pull request must update the test suite to demonstrate the changed functionality.
  • After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.

What does this PR do?

Adds guardrails for some NPEs observed in customers' logs.
I could not reproduce the errors, but they're likely caused by pipelines/jobs that are submitted with scripting.
Some of the errors happen inside core Jenkins code, so they had to be caught rather than prevented.

Below are the examples of the 4 exceptions fixed:

2024-11-12 16:13:42.982 +0000: java.lang.NullPointerException: Cannot invoke "org.jenkinsci.plugins.workflow.graph.FlowNode.getId()" because "current" is null
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView.bruteForceScanForEnclosingBlock(StandardGraphLookupView.java:126)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView.findEnclosingBlockStart(StandardGraphLookupView.java:199)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graph.GraphLookupView$EnclosingBlocksIterable$EnclosingBlocksIterator.<init>(GraphLookupView.java:66)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graph.GraphLookupView$EnclosingBlocksIterable.iterator(GraphLookupView.java:100)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.DatadogUtilities.getEnclosingStageNode(DatadogUtilities.java:926)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.model.PipelineStepData.<init>(PipelineStepData.java:135)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.model.PipelineStepData.<init>(PipelineStepData.java:93)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.buildStepData(DatadogGraphListener.java:296)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.processNode(DatadogGraphListener.java:215)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.processNode(DatadogGraphListener.java:194)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.onNewHead(DatadogGraphListener.java:88)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1582)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:164)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1359)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.croak(CpsFlowExecution.java:936)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramFailed(CpsFlowExecution.java:919)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:847)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:719)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:577)
	at hudson.model.RunMap.retrieve(RunMap.java:273)
	at hudson.model.RunMap.retrieve(RunMap.java:65)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:703)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:685)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:579)
	at hudson.model.RunMap.getById(RunMap.java:237)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:968)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:980)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList.resume(FlowExecutionList.java:226)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:217)
	at jenkins.model.Jenkins.<init>(Jenkins.java:1050)
	at hudson.model.Hudson.<init>(Hudson.java:86)
	at hudson.model.Hudson.<init>(Hudson.java:82)
	at hudson.WebAppMain$3.run(WebAppMain.java:248)
2024-11-12 16:13:44.382 +0000: java.lang.NullPointerException: Cannot invoke "org.jenkinsci.plugins.workflow.graph.FlowNode.getId()" because "parent" is null
	at PluginClassLoader for pipeline-rest-api//com.cloudbees.workflow.rest.external.AtomFlowNodeExt.addParentNodeRefs(AtomFlowNodeExt.java:87)
	at PluginClassLoader for pipeline-rest-api//com.cloudbees.workflow.rest.external.AtomFlowNodeExt.create(AtomFlowNodeExt.java:79)
	at PluginClassLoader for pipeline-rest-api//com.cloudbees.workflow.rest.external.ChunkVisitor.makeAtomNode(ChunkVisitor.java:68)
	at PluginClassLoader for pipeline-rest-api//com.cloudbees.workflow.rest.external.ChunkVisitor.atomNode(ChunkVisitor.java:154)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graphanalysis.ForkScanner.fireVisitChunkCallbacks(ForkScanner.java:779)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graphanalysis.ForkScanner.visitSimpleChunks(ForkScanner.java:797)
	at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.graphanalysis.ForkScanner.visitSimpleChunks(ForkScanner.java:661)
	at PluginClassLoader for pipeline-rest-api//com.cloudbees.workflow.rest.external.RunExt.createNew(RunExt.java:323)
	at PluginClassLoader for pipeline-rest-api//com.cloudbees.workflow.rest.external.RunExt.create(RunExt.java:311)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogBuildListener.getRunExtForRun(DatadogBuildListener.java:543)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogBuildListener.onCompleted(DatadogBuildListener.java:319)
	at hudson.model.listeners.RunListener.lambda$fireCompleted$0(RunListener.java:223)
	at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59)
	at jenkins.util.Listeners.notify(Listeners.java:67)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:221)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:645)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:1053)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-11-12 16:13:44.772 +0000: java.lang.NullPointerException
	at java.base/java.util.ArrayDeque.addLast(ArrayDeque.java:303)
	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4966)
	at java.base/java.util.ArrayDeque.copyElements(ArrayDeque.java:329)
	at java.base/java.util.ArrayDeque.addAll(ArrayDeque.java:324)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.traces.DatadogBaseBuildLogic.traverseStages(DatadogBaseBuildLogic.java:122)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.traces.DatadogBaseBuildLogic.getStageBreakdown(DatadogBaseBuildLogic.java:93)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.traces.DatadogWebhookBuildLogic.toJson(DatadogWebhookBuildLogic.java:135)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.traces.write.TraceWriteStrategyImpl.serialize(TraceWriteStrategyImpl.java:52)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.traces.write.AgentTraceWriteStrategy.serialize(AgentTraceWriteStrategy.java:53)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.traces.write.TraceWriter.submitBuild(TraceWriter.java:54)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogBuildListener.onFinalized(DatadogBuildListener.java:417)
	at hudson.model.listeners.RunListener.lambda$fireFinalized$3(RunListener.java:260)
	at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59)
	at jenkins.util.Listeners.notify(Listeners.java:67)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:258)
	at hudson.model.Run.onEndBuilding(Run.java:2007)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:670)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:1053)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-11-12 16:32:43.320 +0000: java.lang.IllegalStateException: Step 63 (Sleep) has no span info.It is possible that CI Visibility was enabled while this step was in progress
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.model.PipelineStepData.<init>(PipelineStepData.java:127)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.model.PipelineStepData.<init>(PipelineStepData.java:93)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.buildStepData(DatadogGraphListener.java:296)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.processNode(DatadogGraphListener.java:215)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.processNode(DatadogGraphListener.java:194)
	at PluginClassLoader for datadog//org.datadog.jenkins.plugins.datadog.listeners.DatadogGraphListener.onNewHead(DatadogGraphListener.java:88)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1582)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:512)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$2(CpsVmExecutorService.java:85)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
	at java.base/java.lang.Thread.run(Thread.java:840)

Description of the Change

Alternate Designs

Possible Drawbacks

Verification Process

Additional Notes

Release Notes

Review checklist (to be filled by reviewers)

  • Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
  • PR should not have do-not-merge/ label attached.
  • If Applicable, issue must have kind/ and severity/ labels attached at least.

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the changelog/Fixed Fixed features results into a bug fix version bump label Nov 14, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog marked this pull request as ready for review November 14, 2024 19:19

@drodriguezhdez drodriguezhdez left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped minor comments

long pauseDurationMillis = 0;
for (StageNodeExt stage : extRun.getStages()) {
pauseDurationMillis += stage.getPauseDurationMillis();
if (extRun != null){

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we drop a log if the extRun is null?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The getRunExtForRun() method that returns it will log Error while getting RunExt along with the exception.

FlowNode node = nodes.poll();
nodes.addAll(node.getParents());
for (FlowNode parent : node.getParents()) {
if (parent != null){

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, how a FlowNode parent can be null :O?

@nikita-tkachenko-datadog nikita-tkachenko-datadog Nov 19, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not reproduce it locally, but I think it has to do with creating job stages dynamically via scripting.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a common situation for sure, maybe not even a "normal" one, as even the code in the Jenkins core starts throwing NPEs.

@nikita-tkachenko-datadog
nikita-tkachenko-datadog merged commit b49a77e into master Nov 20, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog deleted the nikita-tkachenko/npes-fix branch November 20, 2024 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/Fixed Fixed features results into a bug fix version bump

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants