Skip to content

Commit 93fe1e0

Browse files
author
Arpan Mishra
committed
feat: enable session leaks prevention by cleaning up long-running transactions.
1 parent e51c55d commit 93fe1e0

3 files changed

Lines changed: 87 additions & 4 deletions

File tree

google-cloud-spanner/src/main/java/com/google/cloud/spanner/SessionPoolOptions.java

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -348,7 +348,8 @@ static InactiveTransactionRemovalOptions.Builder newBuilder() {
348348
}
349349

350350
static class Builder {
351-
private ActionOnInactiveTransaction actionOnInactiveTransaction;
351+
private ActionOnInactiveTransaction actionOnInactiveTransaction =
352+
ActionOnInactiveTransaction.WARN;
352353
private Duration executionFrequency = Duration.ofMinutes(2);
353354
private double usedSessionsRatioThreshold = 0.95;
354355
private Duration idleTimeThreshold = Duration.ofMinutes(60L);
@@ -575,7 +576,7 @@ public Builder setBlockIfPoolExhausted() {
575576
*
576577
* @return this builder for chaining
577578
*/
578-
Builder setWarnIfInactiveTransactions() {
579+
public Builder setWarnIfInactiveTransactions() {
579580
this.inactiveTransactionRemovalOptions =
580581
InactiveTransactionRemovalOptions.newBuilder()
581582
.setActionOnInactiveTransaction(ActionOnInactiveTransaction.WARN)
@@ -594,7 +595,7 @@ Builder setWarnIfInactiveTransactions() {
594595
*
595596
* @return this builder for chaining
596597
*/
597-
Builder setWarnAndCloseIfInactiveTransactions() {
598+
public Builder setWarnAndCloseIfInactiveTransactions() {
598599
this.inactiveTransactionRemovalOptions =
599600
InactiveTransactionRemovalOptions.newBuilder()
600601
.setActionOnInactiveTransaction(ActionOnInactiveTransaction.WARN_AND_CLOSE)

google-cloud-spanner/src/test/java/com/google/cloud/spanner/SessionPoolOptionsTest.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ public void verifyDefaultInactiveTransactionRemovalOptions() {
129129
InactiveTransactionRemovalOptions inactiveTransactionRemovalOptions =
130130
sessionPoolOptions.getInactiveTransactionRemovalOptions();
131131

132-
assertFalse(sessionPoolOptions.warnInactiveTransactions());
132+
assertTrue(sessionPoolOptions.warnInactiveTransactions());
133133
assertFalse(sessionPoolOptions.warnAndCloseInactiveTransactions());
134134
assertFalse(sessionPoolOptions.closeInactiveTransactions());
135135
assertEquals(0.95, inactiveTransactionRemovalOptions.getUsedSessionsRatioThreshold(), 0.0);

session-and-channel-pool-configuration.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,3 +281,85 @@ This will cause the following to happen internally in the client library:
281281
1. The `TransactionRunner` will automatically commit the transaction if the supplied user code
282282
finished without any errors. The `Commit` RPC that is invoked uses a thread from the default gRPC
283283
thread pool.
284+
285+
### Session Leak
286+
A Spanner object of the Client Library has a limit on the number of maximum sessions. For example the
287+
default value of `MaxSessions` in the Java Client Library is 400. You can configure these values at the time of
288+
Client side Database object creation by passing in the `SessionPoolOptions`. When all the sessions are checked
289+
out of the session pool, every new transaction has to wait until a session is returned to the pool.
290+
If a session is never returned to the pool (hence causing a session leak), the transactions will have to wait
291+
indefinitely and your application will be blocked.
292+
293+
#### Common Root Causes
294+
The most common reason for a session leak is that a transaction was started by the application but
295+
never committed or rolled back. What you should do is simply call commit on the transaction at the
296+
end of your transaction code. Spanner has two types of transactions, read only and read-write
297+
transactions. When we perform a read in a read-write transaction we still need to commit it.
298+
299+
As shown in the example below, the `try-with-resources` block releases the session after it is complete.
300+
If you don't use `try-with-resources` block, unless you explicitly call the `close()` method on all resources
301+
such as `ResultSet` the session is not released back to the pool. If the transaction does not run within
302+
`try-with-resources` block and if we don't close the resources explicitly, we will have a session leak.
303+
304+
```java
305+
DatabaseClient client =
306+
spanner.getDatabaseClient(DatabaseId.of("my-project", "my-instance", "my-database"));
307+
try (ResultSet resultSet =
308+
client.singleUse().executeQuery(Statement.of("select col1, col2 from my_table"))) {
309+
while (resultSet.next()) {
310+
// use the results.
311+
}
312+
}
313+
```
314+
315+
#### Debugging and Resolving Session Leaks
316+
317+
##### Logging
318+
Enabled by default, the logging option shares warn logs when you have exhausted >95% of your session pool.
319+
This could mean two things, either you need to increase the max sessions in your session pool (as the number
320+
of queries run using the client side database object is greater than your session pool can serve) or you may
321+
have a session leak.
322+
323+
To help debug which transactions may be causing this session leak, the logs will also contain stack traces of
324+
transactions which have been running longer than expected. The logs are pushed to a destination based on
325+
how the log exporter is configured for the host application.
326+
327+
``` java
328+
final SessionPoolOptions sessionPoolOptions =
329+
SessionPoolOptions.newBuilder().setWarnIfInactiveTransactions().build()
330+
331+
final Spanner spanner =
332+
SpannerOptions.newBuilder()
333+
.setSessionPoolOption(sessionPoolOptions)
334+
.build()
335+
.getService();
336+
final DatabaseClient client = spanner.getDatabaseClient(databaseId);
337+
338+
// Example Log message to warn presence of long running transactions
339+
// Detected long-running session <session-info>. To automatically remove long-running sessions, set SessionOption ActionOnInactiveTransaction
340+
// to WARN_AND_CLOSE by invoking setWarnAndCloseIfInactiveTransactions() method. <Stack Trace and information on session>
341+
342+
```
343+
##### Automatically clean inactive transactions
344+
When the option to automatically clean inactive transactions is enabled, the client library will automatically spot
345+
problematic transactions that are running for extremely long periods of time (thus causing session leaks) and close them.
346+
The session will be removed from the pool and be replaced by a new session. To dig deeper into which transactions are being
347+
closed, you can check the logs to see the stack trace of the transactions which might be causing these leaks and further
348+
debug them.
349+
350+
``` java
351+
final SessionPoolOptions sessionPoolOptions =
352+
SessionPoolOptions.newBuilder().setWarnAndCloseIfInactiveTransactions().build()
353+
354+
final Spanner spanner =
355+
SpannerOptions.newBuilder()
356+
.setSessionPoolOption(sessionPoolOptions)
357+
.build()
358+
.getService();
359+
final DatabaseClient client = spanner.getDatabaseClient(databaseId);
360+
361+
// Example Log message for when transaction is recycled
362+
// Removing long-running session <Stack Trace and information on session>
363+
```
364+
365+

0 commit comments

Comments
 (0)