What happened?
When enabling the new jdk-http-client client leads to leaking resources and eventually resulting in 'java.io.IOException: Too many open files' and ' Unable to find a free port' after while. This does not occur if I disable 'System.setProperty("webdriver.http.factory", "jdk-http-client");', the legacy client does not exhibit this behavior (we have run it for many many hours without seeing this once). Therefore, I think the new driver has a resource leak of some type - both related to open files and due to ports.
How can we reproduce the issue?
The easiest way to see this is to run selenium (see below for the details fo how I am running it) and enable the new client in a docker container (see below). Then go and run selenium many times. Use the command 'docker container stats' and monitor the PIDS from that command. Even when including the --init flag in docker run (do 'docker run --init ...') the PIDS grows without limit. This caused by the resource leak, leading to files not being closes (which is why we get 'java.io.IOException: Too many open files' ) and, since selenium tried to get a free port ' Unable to find a free port' due to the ports being exhausted.
I am running in Spring Boot Application in Java 17 running via Docker. I see this after about an hour of running continuously. See below for details:
1) Enabling the client
Add System.setProperty("webdriver.http.factory", "jdk-http-client"); to the main spring boot application. Also add the following to pom.xml:
<properties>
<java.version>17</java.version>
<selenium.constructs.version>4.8.3</selenium.constructs.version>
</properties>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-http-jdk-client</artifactId>
<version>${selenium.constructs.version}</version>
</dependency>
- Client Code:
private List<String> getWebpageInternal(String url) {
WebDriver localWebDriver = null;
try {
Stopwatch stopWatch = Stopwatch.createUnstarted();
stopWatch.start();
localWebDriver = getWebDriver(useFastStrategy);
localWebDriver.get("about:blank");
try {
localWebDriver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(waitMs));
localWebDriver.manage().timeouts().scriptTimeout(Duration.ofMillis(waitMs));
localWebDriver.manage().timeouts().implicitlyWait(Duration.ofMillis(waitMs));
getUrl(localWebDriver, url);
} catch (Exception e) {
// Still try to get the page source - some of the page might have loaded
}
// Note that waiting can take a long time, so for speed we currently put it before the call to stopwatch.stop();
stopWatch.stop();
long lengthWaited = stopWatch.elapsed(TimeUnit.MILLISECONDS);
if (lengthWaited < minWaitMs) {
Thread.sleep(minWaitMs - lengthWaited);
}
String pageSource = getPageSourceSafely(localWebDriver);
if (pageSource == null) {
throw new WebpageScraperTimeoutException("Was not able to get page source=" + url);
}
List<String> pageSources = new ArrayList<>();
pageSources.add(pageSource);
localWebDriver.quit();
return pageSources;
} catch (Exception e) {
LOGGER.error("Got error in MakeNewWebpageScraperVersion3", e);
quitSafely(localWebDriver);
throw new RuntimeException(e);
}
}
private String getPageSourceSafely(WebDriver localWebDriver) {
try {
return localWebDriver.getPageSource();
} catch (Exception e) {
return null;
}
}
private void quitSafely(WebDriver localWebDriver) {
try {
if (localWebDriver == null) {
return;
}
localWebDriver.quit();
} catch (Exception e) {
// nothing to do
LOGGER.info("Got error in quit() call of quitSafely", e);
}
}
private WebDriver getWebDriver(boolean useFastStrategy) {
ChromeDriverService service = new ChromeDriverService.Builder()
.withVerbose(false)
.withSilent(true)
.build();
return new ChromeDriver(service, getChromeOptions(useFastStrategy));
}
private ChromeOptions getChromeOptions(boolean useFastStrategy) {
ChromeOptions chromeOptions = new ChromeOptions();
if (useFastStrategy) {
chromeOptions.setPageLoadStrategy(PageLoadStrategy.EAGER); // was default (Normal) before.
}
// User agent is required because some websites will reject your request if it does not have a user agent
chromeOptions.addArguments(String.format("user-agent=%s", USER_AGENT));
chromeOptions.addArguments("--log-level=OFF");
chromeOptions.addArguments("--headless=new");
List<String> arguments = new LinkedList<>();
arguments.add("--disable-extensions");
arguments.add("--headless");
arguments.add("--disable-gpu");
arguments.add("--no-sandbox");
arguments.add("--incognito");
arguments.add("--disable-application-cache");
arguments.add("--disable-dev-shm-usage");
chromeOptions.addArguments(arguments);
return chromeOptions;
}
- Docker Code:
ARG CHROME_VERSION=111.0.5563.64-1
ADD google-chrome.repo /etc/yum.repos.d/google-chrome.repo
RUN microdnf install -y google-chrome-stable-$CHROME_VERSION \
&& sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome
## ChromeDriver
ARG CHROME_DRIVER_VERSION=111.0.5563.64
RUN microdnf install -y unzip \
&& curl -s -o /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip \
&& unzip /tmp/chromedriver.zip -d /opt \
&& rm /tmp/chromedriver.zip \
&& mv /opt/chromedriver /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& chmod 755 /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& ln -s /opt/chromedriver-$CHROME_DRIVER_VERSION /usr/bin/chromedriver
ENV CHROMEDRIVER_PORT 4444
ENV CHROMEDRIVER_WHITELISTED_IPS "127.0.0.1"
ENV CHROMEDRIVER_URL_BASE ''
EXPOSE 4444
EXPOSE 8080
EXPOSE 5005
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
# For Testing
ENTRYPOINT ["java","-jar", "-Xmx600m","/app.jar"]
### Relevant log output
```shell
For 'Unable to Find a Free Port':
java.lang.RuntimeException: Unable to find a free port
3/27/2023, 6:11:47 PM at org.openqa.selenium.net.PortProber.findFreePort(PortProber.java:62)
3/27/2023, 6:11:47 PM at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:452)
3/27/2023, 6:11:47 PM at monolith.scraping.MakeNewWebpageScraperVersion3.getWebDriver(MakeNewWebpageScraperVersion3.java:194)
### Operating System
Catalina and Ubuntu 22.10
### Selenium version
Java 17
### What are the browser(s) and version(s) where you see this issue?
Chrome
### What are the browser driver(s) and version(s) where you see this issue?
Chromedriver
### Are you using Selenium Grid?
No
What happened?
When enabling the new jdk-http-client client leads to leaking resources and eventually resulting in 'java.io.IOException: Too many open files' and ' Unable to find a free port' after while. This does not occur if I disable 'System.setProperty("webdriver.http.factory", "jdk-http-client");', the legacy client does not exhibit this behavior (we have run it for many many hours without seeing this once). Therefore, I think the new driver has a resource leak of some type - both related to open files and due to ports.
How can we reproduce the issue?
The easiest way to see this is to run selenium (see below for the details fo how I am running it) and enable the new client in a docker container (see below). Then go and run selenium many times. Use the command 'docker container stats' and monitor the PIDS from that command. Even when including the --init flag in docker run (do 'docker run --init ...') the PIDS grows without limit. This caused by the resource leak, leading to files not being closes (which is why we get 'java.io.IOException: Too many open files' ) and, since selenium tried to get a free port ' Unable to find a free port' due to the ports being exhausted.