[🐛 Bug]: Selenium Grid 4.29 - GoAway Errors

[tgoswami013]

Description

Hi, We are getting random GoAway errors while running Selenium Java Scripts on Selenium Grid Server. This behavior is random, sometimes I am getting single GoAway sometimes I am getting 10+ while running the similar set of scripts.

We are using Selenium and Grid version 4.29.0 and Java version is 19.

Here are the RemoteDriver capabilities we are using. Also I have attached below the server logs when we get GoAway.

Reproducible Code

private WebDriver initRemoteDriver() throws Exception {
        System.setProperty("webdriver.http.factory", "jdk-http-client");
        System.setProperty("jdk.httpclient.http.version", "1.1");
        System.setProperty("webdriver.http.timeout", "180000");
        System.setProperty("jdk.httpclient.http2.disabled", "true");
        System.setProperty("jdk.httpclient.debug", "true");

        DesiredCapabilities desiredCapabilities = new DesiredCapabilities();

        ChromeOptions options = new ChromeOptions();


// Disable GPU (often required for headless stability on Windows/Linux)
//        options.addArguments("--disable-gpu");

// Avoid issues with sandboxing in containerized/grid environments
        options.addArguments("--no-sandbox");

// Prevent crashes or freezes in Docker/Linux grid nodes
        options.addArguments("--disable-dev-shm-usage");

// Disable infobars ("Chrome is being controlled by automated test software")
        options.addArguments("disable-infobars");

// Disable extensions (for clean browser startup)
        options.addArguments("--disable-extensions");

        // Run in headless mode if needed (often useful for CI/CD)
//        options.addArguments("--headless=new"); // or just "--headless" for older Chrome

// Start maximized or set a fixed window size
//        options.addArguments("start-maximized");
// or
        options.addArguments("window-size=1920,1080");

// Use incognito mode (optional, for test isolation)
//        options.addArguments("--incognito");

        options.addArguments("--enable-javascript");
        options.addArguments("--ignore-certificate-errors");

        options.addArguments("--remote-allow-origins=*");

        options.addArguments("--disable-infobars");
        options.setPageLoadStrategy(PageLoadStrategy.NORMAL);  // NONE or NORMAL

        options.setAcceptInsecureCerts(false);
        System.out.println("HTTP Properties: version=" + System.getProperty("jdk.httpclient.http.version") +
                ", http2.disabled=" + System.getProperty("jdk.httpclient.http2.disabled"));

        Map<String, Object> prefs = new HashMap<>();
        prefs.put("profile.default_content_settings.popups", 0);
        prefs.put("download.default_directory", FileDownloadConstants.USER_DIR_TARGET);
        prefs.put("download.prompt_for_download", false);
        prefs.put("download.directory_upgrade", true);
        prefs.put("safebrowsing.enabled", true);
        options.setExperimentalOption("prefs", prefs);
        System.out.println("Download directory in initRemoteDriver(): " + FileDownloadConstants.USER_DIR_TARGET);

        System.out.println("This is remote WebDriver ------ Selenium Grid ");
        desiredCapabilities.setCapability(UtilConstant.SAUCELAB_BROWSERNAME, envParamBean.getBrowserName());
        desiredCapabilities.setCapability(ChromeOptions.CAPABILITY, options);
        return new RemoteWebDriver(new java.net.URL(envParamBean.getGridURL()), desiredCapabilities);

    }

Debugging Logs

06:08:59.931 INFO [LocalNode.newSession] - Session created by the Node. Id: f2eb123b8a18d96d1234567da0a6fe3d, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 134.0.6998.35, chrome: {chromedriverVersion: 134.0.6998.35 (ea6ef4c2ac15..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:40607}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://10.202.202.123:4444/se..., se:cdpVersion: 134.0.6998.35, se:containerName: supchainplanmerch-seleniumg..., se:noVncPort: 7900, se:vnc: ws://10.202.202.123:4444/se..., se:vncEnabled: true, se:vncLocalAddress: ws://10.202.202.123:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
06:14:23.317 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: Build info: version: '4.29.0', revision: '18ae989'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.19.190-2.ph3', java.version: '21.0.6'
Driver info: driver.version: unknown
org.openqa.selenium.WebDriverException: Build info: version: '4.29.0', revision: '18ae989'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.19.190-2.ph3', java.version: '21.0.6'
Driver info: driver.version: unknown
        at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:419)
        at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:54)
        at org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:91)
        at org.openqa.selenium.grid.node.DefaultActiveSession.execute(DefaultActiveSession.java:62)
        at org.openqa.selenium.grid.node.local.SessionSlot.execute(SessionSlot.java:130)
        at org.openqa.selenium.grid.node.local.LocalNode.executeWebDriverCommand(LocalNode.java:728)
        at org.openqa.selenium.grid.node.ForwardWebDriverCommand.execute(ForwardWebDriverCommand.java:48)
        at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:397)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
        at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:87)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:360)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
        at org.openqa.selenium.grid.node.Node.execute(Node.java:287)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:360)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
        at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
        at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
        at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
        at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:49)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.InterruptedException
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:413)

---

ℹ️ Last known working version: 4.29

[selenium-ci]

@tgoswami013, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[github-actions]

⚠️ You reported using Selenium version 4.29, but the latest release is 4.33.

Please verify that this issue still occurs with the latest version. If it no longer applies, you can close this issue or update your comment.

This issue will be marked "awaiting answer" and may be closed automatically if no response is received.

[tgoswami013]

Currently Selenium Grid is not managed by me it is managed at server level by DevOps Team. I can not upgrade the Selenium Grid. Could you please provide some information looking at logs and code, what could be the possible reason for GoAway.

[diemol]

That is usuall an infrastructure issue. You need to correlate the status of the machine with the moment when it happens. What processes are open, if more browsers are open at the same time, what is the load and memory usage, etc. It is hard for us to say where the issue is without you doing a deeper analysis and providing more information.

[VietND96]

For message "GoAway", we had a long discussion before #14258.
And the issue is when Grid deploys with ingress (reverse proxy, or TLS termination) in front. The needed configs should be done at the ingress level.

[tgoswami013]

Thank you @VietND96 I will try this with the DevOps team and update here if fixed.

[tgoswami013]

Hi @VietND96 We have set below paramaters in ingress but still getting same GoAway error

nginx.ingress.kubernetes.io/http2: "false"
nginx.ingress.kubernetes.io/proxy-body-size: 100m
nginx.ingress.kubernetes.io/proxy-connect-timeout: "180"
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"

[tgoswami013]

Here is the complete file

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hidden
namespace: hidden
labels:
app: hidden
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/client-body-buffer-size: 100m
nginx.ingress.kubernetes.io/client-max-body-size: 100m
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Access-Control-Allow-Origin: ";
more_set_headers "Access-Control-Allow-Methods: GET";
nginx.ingress.kubernetes.io/cors-allow-headers: DNT,X-CustomHeader,X-LANG,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-Api-Key,X-Device-Id,Access-Control-Allow-Origin,Authorization,Access-Control-Allow-Methods
nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS
nginx.ingress.kubernetes.io/cors-allow-origin: ''
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/http2: "false"
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-body-size: 100m
nginx.ingress.kubernetes.io/proxy-connect-timeout: "180"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/server-snippet: |
http2_max_header_size 64m;
http2_max_field_size 64m;
spec:
rules:
- host: hidden
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: hidden
port:
number: 4444
tls:
- hosts:
- hidden

[VietND96]

proxy-read-timeout: "600" defines the duration to keep the connection alive with the client.
That means if your test execution took over 600s, connection will be dropped by ingress.

[tgoswami013]

@VietND96 None of the test is taking more than 10 minutes. Is this time for Each test case or for whole suite consiting of all the test cases. Also, few test cases fail in less than 10 seconds on login page with GoAway error.

[VietND96]

Actually, it is the duration since the session was created. If your framework is implemented to reuse the same session across tests in a suite, it would be a problem.

[tgoswami013]

@VietND96 Let me ask the DevOps team to increase it but we are closing browser after every test. Multiple teams are running the grid on same time. I am not sure how they have writting their scripts. For us, we always quit the session after test execution.

[tgoswami013]

@VietND96 We increased the timeout to 30 minutes, which has reduced the number of GOAWAY errors, but they still happen occasionally.

We checked the Selenium Grid session IDs, and confirmed that each test case uses a unique session, so there’s no session sharing issue.

Another thing we noticed: when more teams start using the Grid at the same time, we begin to see more GOAWAY errors — but those teams don’t seem to be affected by them as reported by them to us.

[tgoswami013]

Hey @VietND96 anything else we should try. Please suggest.

[VietND96]

Look at code snippet

        System.setProperty("webdriver.http.factory", "jdk-http-client");
        System.setProperty("jdk.httpclient.http.version", "1.1");
        System.setProperty("webdriver.http.timeout", "180000");
        System.setProperty("jdk.httpclient.http2.disabled", "true");
        System.setProperty("jdk.httpclient.debug", "true");

I am not sure you get these props from which document? Let's remove all these and try below

        System.setProperty("webdriver.httpclient.readTimeout", "1800"); //Unit in seconds
        System.setProperty("webdriver.httpclient.version", "HTTP_1_1");

[tgoswami013]

Hi @VietND96

Thanks a lot, I think below solution is working. I will confirm by running multiple builds. So far No GoAway seen.

    System.setProperty("webdriver.httpclient.readTimeout", "1800"); //Unit in seconds
    System.setProperty("webdriver.httpclient.version", "HTTP_1_1");