What happened?
Constantly hanging connections and threads created by dev tools, holding connections on jdk11 http client.
How can we reproduce the issue?
public static void main(String[] args) throws InterruptedException {
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--ignore-certificate-errors",
"--start-maximized",
"--no-sandbox",
"--dns-prefetch-disable",
"--disable-gpu");
System.setProperty("webdriver.http.factory", "jdk-http-client");
var start = Instant.now();
Duration loopDuration = Duration.ofMinutes(3);
while (Instant.now().minus(loopDuration.toMillis(), ChronoUnit.MILLIS).isBefore(start)) {
WebDriver driver = null;
try {
driver = RemoteWebDriver.builder()
.augmentUsing(new Augmenter())
.oneOf(chromeOptions)
.address(System.getProperty("GRID_URL"))
.build();
try (DevTools devTools = ((HasDevTools) driver).getDevTools()) {
devTools.createSession();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
devTools.send(Network.setExtraHTTPHeaders(new Headers(Map.of("test", Boolean.TRUE.toString()))));
devTools.addListener(Network.requestWillBeSent(), event -> {
});
driver.get("https://cnn.com");
}
} finally {
if (driver != null) {
driver.close();
}
}
TimeUnit.SECONDS.sleep(3);
}
while (true) {
TimeUnit.SECONDS.sleep(10);
}
}
run this code for 5 mins, then using jconsole perform full GC and in thread list you can see left hanging threads even after webdriver session was closed and enough time passed to finish all threads. In production env I can see such threads hanging for days. Same issue is observable on selenium 4.8.3, devtools-v111, chromedriver 111.0.5563.64 Stacktrace of thread is:
Name: HttpClient-20-SelectorManager
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
[email protected]/sun.nio.ch.epoll.wait(Native Method)
[email protected]/sun.nio.ch.epollselectorimpl.doselect(epollselectorimpl.java:118)
[email protected]/sun.nio.ch.selectorimpl.lockanddoselect(selectorimpl.java:129)
- locked sun.nio.ch.Util$2@395d80e0
- locked sun.nio.ch.EPollSelectorImpl@3fb5076d
[email protected]/sun.nio.ch.selectorimpl.select(selectorimpl.java:141)
platform/[email protected]/jdk.internal.net.http.httpclientimpl$selectormanager.run(httpclientimpl.java:889)
hanging connections:
netstat -an | grep gridip
tcp 0 0 myip gridip:4444 ESTABLISHED
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 1 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 1 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
tcp6 0 0 myip gridip:4444 CLOSE_WAIT
Relevant log output
Operating System
Debian 9, Ubuntu 20.04.3
Selenium version
Java 4.8.0, dev tools v109
What are the browser(s) and version(s) where you see this issue?
chrome 109.0
What are the browser driver(s) and version(s) where you see this issue?
ChromeDriver 109.0.5414.74
Are you using Selenium Grid?
Yes, 4.8.0
What happened?
Constantly hanging connections and threads created by dev tools, holding connections on jdk11 http client.
How can we reproduce the issue?
public static void main(String[] args) throws InterruptedException { ChromeOptions chromeOptions = new ChromeOptions(); chromeOptions.addArguments("--ignore-certificate-errors", "--start-maximized", "--no-sandbox", "--dns-prefetch-disable", "--disable-gpu"); System.setProperty("webdriver.http.factory", "jdk-http-client"); var start = Instant.now(); Duration loopDuration = Duration.ofMinutes(3); while (Instant.now().minus(loopDuration.toMillis(), ChronoUnit.MILLIS).isBefore(start)) { WebDriver driver = null; try { driver = RemoteWebDriver.builder() .augmentUsing(new Augmenter()) .oneOf(chromeOptions) .address(System.getProperty("GRID_URL")) .build(); try (DevTools devTools = ((HasDevTools) driver).getDevTools()) { devTools.createSession(); devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty())); devTools.send(Network.setExtraHTTPHeaders(new Headers(Map.of("test", Boolean.TRUE.toString())))); devTools.addListener(Network.requestWillBeSent(), event -> { }); driver.get("https://cnn.com"); } } finally { if (driver != null) { driver.close(); } } TimeUnit.SECONDS.sleep(3); } while (true) { TimeUnit.SECONDS.sleep(10); } } run this code for 5 mins, then using jconsole perform full GC and in thread list you can see left hanging threads even after webdriver session was closed and enough time passed to finish all threads. In production env I can see such threads hanging for days. Same issue is observable on selenium 4.8.3, devtools-v111, chromedriver 111.0.5563.64 Stacktrace of thread is: Name: HttpClient-20-SelectorManager State: RUNNABLE Total blocked: 0 Total waited: 0 Stack trace: [email protected]/sun.nio.ch.epoll.wait(Native Method) [email protected]/sun.nio.ch.epollselectorimpl.doselect(epollselectorimpl.java:118) [email protected]/sun.nio.ch.selectorimpl.lockanddoselect(selectorimpl.java:129) - locked sun.nio.ch.Util$2@395d80e0 - locked sun.nio.ch.EPollSelectorImpl@3fb5076d [email protected]/sun.nio.ch.selectorimpl.select(selectorimpl.java:141) platform/[email protected]/jdk.internal.net.http.httpclientimpl$selectormanager.run(httpclientimpl.java:889) hanging connections: netstat -an | grep gridip tcp 0 0 myip gridip:4444 ESTABLISHED tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 1 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 1 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAIT tcp6 0 0 myip gridip:4444 CLOSE_WAITRelevant log output
Operating System
Debian 9, Ubuntu 20.04.3
Selenium version
Java 4.8.0, dev tools v109
What are the browser(s) and version(s) where you see this issue?
chrome 109.0
What are the browser driver(s) and version(s) where you see this issue?
ChromeDriver 109.0.5414.74
Are you using Selenium Grid?
Yes, 4.8.0