Description
Version
v20.10.0
Platform
Linux temp-vps1 6.1.0-13-arm64 #1 SMP Debian 6.1.55-1 (2023-09-29) aarch64 GNU/Linux
Subsystem
http2
What steps will reproduce the bug?
Define a small http2 server that can send a stream, or receive a stream.
const fs = require("fs");
const stream = require("stream");
const http2 = require("http2");
const crypto = require("crypto");
const server = http2.createSecureServer({
key: fs.readFileSync("./localhost-privkey.pem"),
cert: fs.readFileSync("./localhost-cert.pem"),
allowHTTP1: true, // http2 upgrade
settings: { initialWindowSize: 8 * 1024 * 1024}
});
server.on('error', (err) => console.log(err));
const buffer = crypto.randomBytes(256*1024*1024);
server.on("connect", (session) => {
session.setLocalWindowSize(8*1024*1024);
});
server.on("stream", (st, headers) => {
st.respond({
'content-type': 'application/octet-stream',
":status": 200,
});
if (headers[':path'] === "/download") {
stream.Readable.from(buffer).pipe(st);
}
else {
st.on('data', d => st.resume());
st.on('error', console.error);
st.on('end', () => st.close());
}
})
server.listen(8443, () => {
console.log("HTTPS (HTTP/2) listening on 8443 port");
});generate ssl certs:
openssl req -x509 -newkey rsa:2048 -nodes -sha256 \
-subj '//CN=localhost' -keyout localhost-privkey.pem -out localhost-cert.pem
go to a different machine (preferably on the same local network) and run:
curl --http2 -k https://other-machine:8443/download --output /dev/null
dd if=/dev/urandom of=test.bin bs=100M count=4
curl --http2 -k -T test.bin https://chunky.swat.other-machine:8443/upload
observer the difference in speed between up and downstream.
$ curl --http2 -k https://other-machine:8443/download --output /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 256M 0 256M 0 0 356M 0 --:--:-- --:--:-- --:--:-- 356M
$ curl --http2 -k -T test.bin https://other-machine:8443/upload
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 400M 0 0 100 400M 0 11.7M 0:00:33 0:00:33 --:--:-- 11.7M
How often does it reproduce? Is there a required condition?
I've manged to reproduce this with multiple vps on different cloud providers. switching around server and client side.
Loopback never reproduces it, and sometimes if you have 2 machines on the same switch, it also doesn't reproduce.
I suspect it's some kind of window related issue, I've already applied the manual fixes suggested in #38426. Without these the download speed is also this slow.
What is the expected behavior? Why is that the expected behavior?
The performance should be comparable to raw tcp speed (with some lee-way for http2 overhead).
See netcat performance between the 2 machines:
on server:
nc -l -p 5050 > /dev/null
on second machine:
pv -a test.bin| nc other-machine 5050
[ 492MiB/s]
using nodejs as a http2 client has same performance issue
const fs = require("fs");
const http2 = require("http2");
const process = require("process");
const crypto = require("crypto");
const stream = require("stream");
const buffer = crypto.randomBytes(256*1024*1024);
let round = 0;
function upload() {
const client = http2.connect(process.env["TARGET_HOST"] ?? "https://localhost:8443", {
ca: fs.readFileSync("./localhost-cert.pem"),
checkServerIdentity: () => undefined,
settings: { initialWindowSize: 8 * 1024 * 1024}
});
client.once('remoteSettings', settings => {
client.setLocalWindowSize(settings.initialWindowSize ?? (8 * 1024 * 1024));
});
const recv = client.request({":path": "/upload"}, {endStream: false});
recv.on("response", () => {
const start = process.hrtime();
stream.Readable.from(buffer)
.pipe(recv)
.on("end", () => {
const [timeSpendS, timeSpendNS] = process.hrtime(start);
const mbSize = buffer.length / (1024 * 1024);
console.log(
`Uploading to h2 speed (${Math.trunc(mbSize)} MB): ${
mbSize / (timeSpendS + timeSpendNS * 1e-9)
} MB/s`
);
client.close();
round++;
if (round < 3) {
// to give node a fair chance of getting close to curl
// we go throught the upload 3 times (sequentually).
// such that node has had time to JIT etc.
upload();
}
});
});
}
upload()invoke with: TARGET_HOST="https://other-machine:8443" node n2-upload.js
What do you see instead?
Slow upload while the tcp connnection speed would allow for fast upload.
Additional information
I'm closing a previous issue (#50972) I reported, that one got too complex and big. This is a much smaller case and easier to debug, I hope. If there is something I can do to help out, please let me know.
I think this is not a dupe of #38426 as even when you manually setup a big enough window, the receive side is still slow. I suspect there is a call to nghttp2 missing that is not exposed on the nodejs side?