Repository navigation
Dynamodb enhanced client keeps spawning sdk-ScheduledExecutor threads #3746
Description
Describe the bug
I use sdk v2 2.19.26 and it keeps spawning threads in a WAITING state.
Thread dump shows thousands of threads in WAITING state like this:
"sdk-ScheduledExecutor-0-0" prio=0 tid=0x0 nid=0x0 waiting on condition
java.lang.Thread.State: WAITING
on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59b572af
at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at [email protected]/java.lang.Thread.run(Thread.java:829)
sdk-ScheduledExecutor is defined in SdkDefaultClientBuilder in sdk2
Expected Behavior
There is no issue with threads
Current Behavior
Thread dump shows thousands of threads in WAITING state
Reproduction Steps
var dynamoDbClient = DynamoDbClient
.builder()
.region(region)
.credentialsProvider(() -> AwsBasicCredentials.create(accessKey, secretKey))
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(DYNAMODB_CLIENT_TIMEOUT)
.retryPolicy(RetryPolicy.defaultRetryPolicy().toBuilder()
.numRetries(DYNAMODB_CLIENT_MAX_ERROR_RETRY)
.build())
.build())
.build();
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.19.26
JDK version used
11
Operating System and version
alpine-zulu
Activity
debora-ito commentedon Feb 16, 2023
@vasylv1 thank you for reaching out.
Having
java.lang.Thread.State: WAITINGin the logs is not necessarily a bad thing, the threads can be inWAITINGstate only temporarily.Are creating one new DynamoDB Client per request, or reusing the client?
We recommend you reuse the same instance of client whenever possible.
vasylv1 commentedon Feb 16, 2023
DynamoDB Client is created only once and then reused by all requests.
That is exactly the problem, the threads are getting stuck in this state and are never released.
Here is grafana chart
The memory and threads in WAITING state go down only when the service is restarted on redeploy.
We started to observe this behavior when we migrated our service from aws sdk 1 to aws sdk 2 + enhanced client.
Here is what we see in memory dumps:
Comparing thread dumps after 3 days:
It looks like the threads are slowly getting stuck in WAITING state and those that are in this state are created by AWS SDK2 with "sdk-ScheduledExecutor" prefix.
debora-ito commentedon Feb 25, 2023
Problem suspect 2 "play.shaded.ahc.io.netty.buffer" is not part of the Java SDK package.
And I can't get any conclusions by looking at the problem suspect 1 and the log snippet you provided.
Can you provide us the heap dump and a self-contained code sample we can use to reproduce the leak?
vasylv1 commentedon Feb 27, 2023
I cannot share the dump file as it was taken from a prod server. Is there anything specific in the dump file that I can give details about? I am open for providing more info if you could specify what is it.
vasylv1 commentedon Feb 27, 2023
My statement: "DynamoDB Client is created only once and then reused by all requests." wasn't true.
The issue is exactly with not closing the client.
debora-ito commentedon Feb 28, 2023
Great, glad you found the cause.
Memory leak reports are usually associated with creating multiple clients and not closing them after they're not needed.
Marking this to close soon. Let us know if you have any other question.
Merge pull request #3746 from aws/staging/6c8562e2-03bc-4111-a73a-f2c…
[Fix](catalog)Resources should be closed when dropping a Catalog. (#5…
[Fix](catalog)Resources should be closed when dropping a Catalog. (ap…