Description
Our application is experiencing unexpected stalls when uploading results back to a server. After some investigation determined that this is likely an implementation bug in HTTP/2 flow control. RFC 7540 section 2 page 5 explicitly states that:
Multiplexing of requests is achieved by having each HTTP request/response exchange associated with its own stream (Section 5). Streams are largely independent of each other, so a blocked or stalled request or response does not prevent progress on other streams.
But its actually very easy to stall the server, sometimes permanently. This bug has denial of service potential.
This issue has been reported before as #40816 and remains open.
What version of Go are you using (go version)?
go1.18.1
Does this issue reproduce with the latest release?
Don't know, reproducible in 1.19.
What did you do?
The program below demonstrates the problem. Three requests are sent by the same client to different URLs on the same server. One of the URLs has a hard-coded 5-second "processing" delay. Each request sends a 10MB dummy payload to the server. Upon executing the program, all three requests will stall for 5 seconds.
In case of #40816, the processing delay was a shared lock which resulted in a server-wide deadlock (denial of service potential).
You'll need to provide your own (self-signed) SSL certificates for the program to work.
package main
import (
"crypto/tls"
"fmt"
"io"
"log"
"net/http"
"os"
"sync"
"time"
"golang.org/x/net/http2"
)
func main() {
server := &http.Server{
Addr: ":12345",
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
log.Printf("receiving post to %s", r.URL.Path)
if r.URL.Path == "/2" {
time.Sleep(time.Second * 5)
}
_, err := io.Copy(io.Discard, r.Body)
if err != nil {
log.Fatal(err)
}
log.Printf("received post to %s", r.URL.Path)
}),
}
go server.ListenAndServeTLS("public.crt", "private.key")
client := &http.Client{
Transport: &http2.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
DisableCompression: true,
AllowHTTP: false,
},
}
time.Sleep(time.Second)
var wg sync.WaitGroup
wg.Add(3)
go post_big_file(&wg, client, 1)
go post_big_file(&wg, client, 2)
go post_big_file(&wg, client, 3)
wg.Wait()
}
func post_big_file(wg *sync.WaitGroup, client *http.Client, index int) {
defer wg.Done()
log.Printf("posting big file %d", index)
file, err := os.Open("/dev/zero")
if err != nil {
log.Fatal(err)
}
defer file.Close()
reader := io.LimitReader(file, 1024*1024*10)
url := fmt.Sprintf("https://localhost:12345/%d", index)
response, err := client.Post(url, "text/plain", reader)
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
_, err = io.Copy(io.Discard, response.Body)
if err != nil {
log.Fatal(err)
}
log.Printf("big file %d posted", index)
}