Skip to content

fix: fix BatchSpanProcessor with non-runtime exception#4402

Merged
anuraaga merged 3 commits into
open-telemetry:mainfrom
wallezhang:fix-batch-span-processor-non-runtime-exception
Apr 21, 2022
Merged

fix: fix BatchSpanProcessor with non-runtime exception#4402
anuraaga merged 3 commits into
open-telemetry:mainfrom
wallezhang:fix-batch-span-processor-non-runtime-exception

Conversation

@wallezhang

Copy link
Copy Markdown
Contributor

Fix BatchSpanProcessor worker thread is killed by non-runtime exception.

close #4348

@wallezhang
wallezhang requested a review from a user April 21, 2022 03:27
@wallezhang
wallezhang requested a review from Oberon00 as a code owner April 21, 2022 03:27
@codecov

codecov Bot commented Apr 21, 2022

Copy link
Copy Markdown

Codecov Report

Merging #4402 (09ef450) into main (c2d8f6a) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #4402      +/-   ##
============================================
- Coverage     89.94%   89.91%   -0.04%     
+ Complexity     4908     4815      -93     
============================================
  Files           567      553      -14     
  Lines         15199    14962     -237     
  Branches       1465     1444      -21     
============================================
- Hits          13671    13453     -218     
+ Misses         1058     1043      -15     
+ Partials        470      466       -4     
Impacted Files Coverage Δ
...telemetry/sdk/trace/export/BatchSpanProcessor.java 92.36% <100.00%> (+2.15%) ⬆️
...emetry/sdk/testing/assertj/AttributeAssertion.java 77.77% <0.00%> (-22.23%) ⬇️
...pentelemetry/sdk/autoconfigure/OtlpConfigUtil.java 90.09% <0.00%> (-0.18%) ⬇️
...ava/io/opentelemetry/api/metrics/DefaultMeter.java 100.00% <0.00%> (ø)
...in/java/io/opentelemetry/sdk/metrics/SdkMeter.java 100.00% <0.00%> (ø)
.../io/opentelemetry/api/internal/ValidationUtil.java 100.00% <0.00%> (ø)
...y/sdk/testing/assertj/OpenTelemetryAssertions.java 100.00% <0.00%> (ø)
...lemetry/sdk/testing/assertj/SummaryDataAssert.java
...y/sdk/testing/assertj/AbstractPointDataAssert.java
.../sdk/testing/assertj/HistogramPointDataAssert.java
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2d8f6a...09ef450. Read the comment docs.

@anuraaga anuraaga left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wallezhang

logger.log(Level.FINE, "Exporter failed");
}
} catch (RuntimeException e) {
} catch (Throwable e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should not propagate the exception, the worker thread will be killed as usual if call ThrowableUtil.propagateIfFatal.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes for fatal exceptions like OutOfMemory this needs to happen to let the runtime know of the unrecoverable problem. We always propagateIfFatal in this sort of pattern.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it sounds reasonable.

}
} catch (RuntimeException e) {
logger.log(Level.WARNING, "Exporter threw an Exception", e);
} catch (Throwable t) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test case with a non-runtime exception? I think Mockito will allow throwing a non-runtime exception from a mock even if not declared, or otherwise there is this pattern

https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/common/util/Exceptions.java#L235

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried to add a unit test that made exporter throw a non-runtime exception. However mockito alert that checked exception is invalid for this method!. Thanks for your suggestion, I will try again.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I add a unit test with a non-runtime exception.

when(mockSpanExporter.export(anyList()))
.thenAnswer(
invocation -> {
throw new Exception("No export for you.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah that was even easier than expected, nice

@anuraaga
anuraaga merged commit f36785f into open-telemetry:main Apr 21, 2022
@wallezhang
wallezhang deleted the fix-batch-span-processor-non-runtime-exception branch April 22, 2022 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

opentelemetry-sdk-trace: BatchSpanExporter gets stuck when non-runtime exceptions are thrown (kotlin)

4 participants