Skip to content

[🐛 Bug]: Enabling new jdk-http-client leads to 'java.io.IOException: Too many open files' and ' Unable to find a free port'  #11826

@EuclidHeron

Description

@EuclidHeron

What happened?

When enabling the new jdk-http-client client leads to leaking resources and eventually resulting in 'java.io.IOException: Too many open files' and ' Unable to find a free port' after while. This does not occur if I disable 'System.setProperty("webdriver.http.factory", "jdk-http-client");', the legacy client does not exhibit this behavior (we have run it for many many hours without seeing this once). Therefore, I think the new driver has a resource leak of some type - both related to open files and due to ports.

How can we reproduce the issue?

The easiest way to see this is to run selenium (see below for the details fo how I am running it) and enable the new client in a docker container (see below). Then go and run selenium many times. Use the command 'docker container stats' and monitor the PIDS from that command. Even when including the --init flag in docker run (do 'docker run --init ...') the PIDS grows without limit. This caused by the resource leak, leading to files not being closes (which is why we get 'java.io.IOException: Too many open files' ) and, since selenium tried to get a free port ' Unable to find a free port' due to the ports being exhausted.

I am running in Spring Boot Application in Java 17 running via Docker. I see this after about an hour of running continuously. See below for details:
1) Enabling the client
Add System.setProperty("webdriver.http.factory", "jdk-http-client"); to the main spring boot application. Also add the following to pom.xml:

<properties>
		<java.version>17</java.version>
		<selenium.constructs.version>4.8.3</selenium.constructs.version>
	</properties>
<dependency>
		<groupId>org.seleniumhq.selenium</groupId>
		<artifactId>selenium-http-jdk-client</artifactId>
		<version>${selenium.constructs.version}</version>
</dependency>
  1. Client Code:
private List<String> getWebpageInternal(String url) {
        WebDriver localWebDriver = null;
        try {
            Stopwatch stopWatch = Stopwatch.createUnstarted();
            stopWatch.start();
            localWebDriver = getWebDriver(useFastStrategy);
            localWebDriver.get("about:blank");


            try {

                localWebDriver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(waitMs));
                localWebDriver.manage().timeouts().scriptTimeout(Duration.ofMillis(waitMs));
                localWebDriver.manage().timeouts().implicitlyWait(Duration.ofMillis(waitMs));
                getUrl(localWebDriver, url);
            } catch (Exception e) {
                // Still try to get the page source - some of the page might have loaded
            }

            // Note that waiting can take a long time, so for speed we currently put it before the call to stopwatch.stop();
            stopWatch.stop();
            long lengthWaited = stopWatch.elapsed(TimeUnit.MILLISECONDS);

            if (lengthWaited < minWaitMs) {
                Thread.sleep(minWaitMs - lengthWaited);
            }
            String pageSource = getPageSourceSafely(localWebDriver);
            if (pageSource == null) {
                throw new WebpageScraperTimeoutException("Was not able to get page source=" + url);
            }

            List<String> pageSources = new ArrayList<>();
            pageSources.add(pageSource);
            localWebDriver.quit();
            return pageSources;
        } catch (Exception e) {
            LOGGER.error("Got error in MakeNewWebpageScraperVersion3", e);
            quitSafely(localWebDriver);
            throw new RuntimeException(e);
        }
    }

private String getPageSourceSafely(WebDriver localWebDriver) {
        try {
            return localWebDriver.getPageSource();
        } catch (Exception e) {
            return null;
        }
    }

private void quitSafely(WebDriver localWebDriver) {
        try {
            if (localWebDriver == null) {
                return;
            }
            localWebDriver.quit();
        } catch (Exception e) {
                // nothing to do
            LOGGER.info("Got error in quit() call of quitSafely", e);
        }
    }

private WebDriver getWebDriver(boolean useFastStrategy) {
        ChromeDriverService service = new ChromeDriverService.Builder()
                .withVerbose(false)
                .withSilent(true)
                .build();
        return new ChromeDriver(service, getChromeOptions(useFastStrategy));
    }

private ChromeOptions getChromeOptions(boolean useFastStrategy) {
        ChromeOptions chromeOptions = new ChromeOptions();
        if (useFastStrategy) {
            chromeOptions.setPageLoadStrategy(PageLoadStrategy.EAGER); // was default (Normal) before.
        }

        // User agent is required because some websites will reject your request if it does not have a user agent
        chromeOptions.addArguments(String.format("user-agent=%s", USER_AGENT));
        chromeOptions.addArguments("--log-level=OFF");
        chromeOptions.addArguments("--headless=new");
        List<String> arguments = new LinkedList<>();
        arguments.add("--disable-extensions");
        arguments.add("--headless");
        arguments.add("--disable-gpu");
        arguments.add("--no-sandbox");
        arguments.add("--incognito");
        arguments.add("--disable-application-cache");
        arguments.add("--disable-dev-shm-usage");

        chromeOptions.addArguments(arguments);
        return chromeOptions;
    }
  1. Docker Code:
ARG CHROME_VERSION=111.0.5563.64-1
ADD google-chrome.repo /etc/yum.repos.d/google-chrome.repo
RUN microdnf install -y google-chrome-stable-$CHROME_VERSION \
	&& sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome

## ChromeDriver

ARG CHROME_DRIVER_VERSION=111.0.5563.64
RUN microdnf install -y unzip \
	&& curl -s -o /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip \
	&& unzip /tmp/chromedriver.zip -d /opt \
	&& rm /tmp/chromedriver.zip \
	&& mv /opt/chromedriver /opt/chromedriver-$CHROME_DRIVER_VERSION \
	&& chmod 755 /opt/chromedriver-$CHROME_DRIVER_VERSION \
	&& ln -s /opt/chromedriver-$CHROME_DRIVER_VERSION /usr/bin/chromedriver

ENV CHROMEDRIVER_PORT 4444
ENV CHROMEDRIVER_WHITELISTED_IPS "127.0.0.1"
ENV CHROMEDRIVER_URL_BASE ''
EXPOSE 4444

EXPOSE 8080
EXPOSE 5005
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
# For Testing
ENTRYPOINT ["java","-jar", "-Xmx600m","/app.jar"]


### Relevant log output

```shell
For 'Unable to Find a Free Port':

java.lang.RuntimeException: Unable to find a free port
3/27/2023, 6:11:47 PM	at org.openqa.selenium.net.PortProber.findFreePort(PortProber.java:62)
3/27/2023, 6:11:47 PM	at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:452)
3/27/2023, 6:11:47 PM	at monolith.scraping.MakeNewWebpageScraperVersion3.getWebDriver(MakeNewWebpageScraperVersion3.java:194)


### Operating System

Catalina and Ubuntu 22.10

### Selenium version

Java 17

### What are the browser(s) and version(s) where you see this issue?

Chrome

### What are the browser driver(s) and version(s) where you see this issue?

Chromedriver

### Are you using Selenium Grid?

No

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-javaJava BindingsI-defectSomething is not working as intended

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions