Upgrade Kestrel 1.1.0 submit POST not responded on behind NGINX with websocket supported

[thohng]

Our services work fine with Kestrel 1.0.0 and 1.0.1 before. We also use Microsoft.AspNetCore.SignalR.Server 0.1.0

Got issue when upgrade Kestrel to 1.1.0.

Workaround:

• downgrade to Kestrel 1.0.2
• or proxy_set_header Connection $http_connection;

Further investigation:

• Change proxy_set_header Connection keep-alive; then Kestrel 1.1.0 works but websocket not work.
• Ajax POST (json) not get the issue
• ASP.NET pages POST (Form data, Content-Type: application/x-www-form-urlencoded) with content result not get the issue.
• Only ASP.NET pages POST (Form data) with redirect result get the issue.

NGINX 1.10.0 / Ubuntu 16.04:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''   '';
}

map $http_x_forwarded_proto $thescheme {
	default $scheme;
	https https;
}

proxy_pass				http://localhost:5000/;
proxy_set_header Host 			$host;
proxy_set_header X-Real-IP 		$remote_addr;
proxy_set_header X-Forwarded-Host       $http_host;
proxy_set_header X-Forwarded-Proto	$thescheme;
proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
proxy_http_version                      1.1;
proxy_set_header Upgrade                $http_upgrade;
proxy_set_header Connection             "upgrade";

"frameworks": {
    "netcoreapp1.1": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "version": "1.1.0",
          "type": "platform"
        }
      },
      "imports": [
        "dotnet5.6",
        "portable-net45+win8"
      ]
    }
  }
}

[cesarblum]

What behavior are you seeing when using Kestrel 1.1.0? I'm trying to repro your issue but so far haven't been able to.

[cesarblum]

Have you tried using proxy_set_header $http_connection?

[thohng]

I got 504 Gateway Time-out after 60s

Have you tried using proxy_set_header $http_connection?

I did not try that

nginx log
access
1.2.3.4 - - [19/Dec/2016:18:50:07 +0000] "POST /admin/account/login?returnurl=%2Fadmin%2F HTTP/2.0" 504 665 "https://aaa...

error

2016/12/19 18:50:07 [error] 6272#6272: *7943 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 1.1.1.1, server: abc.com, request: "POST /admin/account/login?returnurl=%2Fadmin%2F HTTP/2.0", upstream: "http://127.0.0.1:5000/admin/account/login?returnurl=%2Fadmin%2F", host: "abc.com", referrer: "https://abc.com/admin/account/login?ReturnUrl=%2Fadmin%2F"

[thohng]

What client are you using? This is odd: HTTP/2.0.

Chrome browser

on NGINX listen 443 http2 ssl; => It works before

[cesarblum]

Ok, so you're sending an HTTP/2 request and nginx should be translating that to 1.1 clear text. I'll try that next.

[cesarblum]

But let me know if proxy_set_header $http_connection helps.

[thohng]

Yes, proxy_set_header $http_connection helps work both POST and Websocket.
Thanks @CesarBS !

# proxy_set_header Connection             "upgrade";
proxy_set_header Connection $http_connection;

[cesarblum]

@thosoft I haven't been able to repro your exact issue, unfortunately. A couple points though:

• Microsoft.AspNetCore.SignalR.Server is a deprecated package. You should switch to Microsoft.AspNetCore.SignalR when upgrading to Kestrel 1.1.0.
• On the few occasions that I observed things not working, the reason was that the app I was testing with was making requests to http and ws endpoints, instead of https and wss.

I recommend you review your app and address those points.

[thohng]

@CesarBS I have updated above for more detail map $http_upgrade $connection_upgrade and map $http_x_forwarded_proto $thescheme

• For Microsoft.AspNetCore.SignalR the latest version 0.2.0-rtm-22752 now is on https://dotnet.myget.org/feed/aspnetcore-master/package/nuget/Microsoft.AspNetCore.SignalR.Server . We will consider upgrade SignalR when it ready with .NETCore 1.1.0.
• Because we got issue with redirection twice (https and http) before http://stackoverflow.com/questions/38748989/deploy-asp-net-core-mvc-behind-nginx-that-does-ssl-termination-redirects-to-no/39383668 . We may revisit it again later.

Update Further investigation:

• Change proxy_set_header Connection keep-alive; then Kestrel 1.1.0 works but websocket not work.
• Ajax POST (json) not get the issue
• ASP.NET pages POST (Form data, Content-Type: application/x-www-form-urlencoded) with content result not get the issue.
• Only ASP.NET pages POST (Form data) with redirect result get the issue.

[cesarblum]

Just need to confirm one thing: you see the original issue(s) when using Kestrel 1.1.0 and Microsoft.AspNetCore.SignalR 0.2.0-rtm-22752. Correct?

I think either

proxy_set_header Connection keep-alive;

or

proxy_set_header Connection upgrade;

will always lead to some kind of problem, since you're always forcing the Connection header to have a certain value between nginx and Kestrel (i.e. erasing whatever came from the browser). That's why I recommend you use proxy_set_header Connection $http_connection;. Admittedly though, I'm surprised that nginx's official recommendation is to set it to upgrade if you want to use WebSockets. That's suboptimal for non-WebSocket requests, because Kestrel will always see a Connection header without a keep-alive token and will not handle the request as a keep-alive one, and will close the connection after every response is sent, forcing the browser to open a new connection upon every new request.

[thohng]

Just need to confirm one thing: you see the original issue(s) when using Kestrel 1.1.0 and Microsoft.AspNetCore.SignalR 0.2.0-rtm-22752. Correct?

Confirmed original issue when upgrade Kestrel 1.1.0 with proxy_set_header Connection upgrade; and

"Microsoft.AspNetCore.Server.Kestrel": "1.1.0",
"Microsoft.AspNetCore.SignalR.Server": "0.1.0",
"Microsoft.AspNetCore.WebSockets.Server": "0.1.0",

I just re-test and receive same issue/behavior when upgrade both below or separated

• Microsoft.AspNetCore.SignalR.Server 0.2.0-rtm-22752
• Microsoft.AspNetCore.WebSockets 1.0.0 (replace Microsoft.AspNetCore.WebSockets.Server 0.1.0)

For nginx, if change proxy_set_header Connection keep-alive; => resolve POST 504 issue, but Signalr fallback to EventStream.
/signalr/signalr/negotiate? return TryWebSockets:true but wss://domain.com/signalr/signalr/connect?transport=webSockets& return 400 bad request

[cesarblum]

@thosoft Thanks. Another thing I'm curious about: do you still see the original issue if you use HTTP/1.1 instead of HTTP/2?