libcurl using client reports curl error 92 "Stream error in the HTTP/2 framing layer" when receiving a 204 No content reponse from HTTP/2 server to a DELETE request

[zazola]

I did this

Overview

libcurl based C++ client

After having tested HTTP2 POST and GETs against the same server (an h2-hypercorn server using Python's Quart module to generate responses), a DELETE request that gets a 204 No Content response, can't be handled by libcurl in my C++ programmed client. This what I get in my C++ client:

Checking the network traffic capture I see the following:

In the C++ client I tried the following libcurl versions: 7.83, 7.86 and 8.5.0. All of them have the same behavior.

curl CLI tool

I also tried performing the DELETE with the curl command line tool on a:

• WSL2 Ubuntu 22.04.4 LTS distribution
• kernel v:5.15.146.1-microsoft-standard-WSL2
  However in the version 7.86 it performs the DELETE properly, but falling back to HTTP/1.1 instead of HTTP/2 (as specified in the command line).

$ curl -vvv -X DELETE --http2 --cacert /mnt/c/Users/cacert.pem https://localhost:5443/registrations/4b289e40-3830-4122-8468-21d4c1d91679
*   Trying 127.0.0.1:5443...
* Connected to localhost (127.0.0.1) port 5443 (#0)
* ALPN: offers http/1.1
*  CAfile: /mnt/c/Users/cacert.pem
*  CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
*  start date: Feb  5 11:41:50 2024 GMT
*  expire date: Feb  4 11:41:50 2025 GMT
*  common name: localhost (matched)
*  issuer: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> DELETE /registrations/4b289e40-3830-4122-8468-21d4c1d91679 HTTP/1.1
> Host: localhost:5443
> User-Agent: curl/7.86.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 204
< content-type:
< content-length: 0
< date: Thu, 16 May 2024 14:17:48 GMT
< server: hypercorn-h11
<
* Connection #0 to host localhost left intact

Not sure if I'm doing something wrong or if it's some sort of issue in the HTTP2 implementation.

I expected the following

Libcurl being able to handle a seemingly correct HTTP2 204 response without reporting the 92 error.

curl/libcurl version

libcurl, 7.83, 7.86, 8.5.0 (all have same behavior)
curl 7.86

operating system

Linux 21310-1044 5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[icing]

@zazola thanks for the detailed report. I wrote a test case simulating exactly that request against an Apache to send back the HTTP/2 HEADER frame with eos=0 and then a later DATA frame, length=0 and eos=1. This works with curl 8.5.0 and current master branch.

What I see in the wireshark screenshot looks peculiar. The server seems to send the DATA frame inside the TLS shutdown notice record. I am not 100% sure, but that does not look right.

[icing]

How can we move forward? You could add a CURLOPT_DEBUGFUNCTION to your app, and call curl_global_trace('http/2,ssl') to get details of the HTTP/2 and SSL layers.

The command line curl you have does not seem to support HTTP/2. You could built a new one from master (or take it from an image) and test that against your server. There you can configure tracing with --trace-config ids,time,http/2,ssl on the command line.

[zazola]

Hi,

Sorry for the late response, I have been trying to confine more the issue I'm experiencing.
As you said I didn't compile properly the v7.86 curl cli tool I was using (I forgot to use --with-nghttp2 when doing so).
Now I think I have find the border version where (I believe) the issue was fixed, here is the test scenario I prepared to perform tests:

Test scenario

Using curl cli tool in versions: 7.86, 7.87 and 7.88
h2-hypercorn server using Python Quart module

• requitements:

h2==4.1.0
Quart==0.19.4

where I have one single endpoint that checks GET and DELETE requests, both being responding a 204 code answer: test_quart_server.txt (change .txt to .py extension to be able to run it)

Outcome

Version 7.86 and 7.87 seem to have the issue I'm having most of the times (either being request method GET or DELETE). However the issue seem to be fixed in v7.88.0, so I'm going to check my libcurl and nghttp2 dependencies in my C++ client, as I thought being tested in libcurl 7.83, 7.86 and 8.5.0. But I might have made a mistake inclusing v8.5.0 or alike, as it seems to be fixed already from 7.88.0 on.
I'll try that on Monday (20th), will let you know about the final outcome of this particular issue I'm having. Please let me know if you see something I'm not taking into account.

Trace messages:

--trace-config was not recognized by curl, but --trace was, not sure if the outcome would be equivalent.

$ curl -X DELETE --http2  --cacert /mnt/c/Users/certs/cacert.pem https://localhost:5443/registrations/1234-1234-1234 --trace ids,time,http/2,ssl | tee /tmp/response_204_without_error.log

Result:
response_204_without_error.log

curl -X DELETE --http2  --cacert /mnt/c/Users/certs/cacert.pem https://localhost:5443/registrations/1234-1234-1234 --trace ids,time,http/2,ssl | tee /tmp/error_204_.log

Result:
error_92_response_204.log

Correct handling:

$ curl -vvv  -X DELETE --http2 --cacert /mnt/c/Users/certs/cacert.pem https://localhost:5443/registrations/1234-1234-1234
*   Trying 127.0.0.1:5443...
* Connected to localhost (127.0.0.1) port 5443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /mnt/c/Users/certs/cacert.pem
*  CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
*  start date: Feb  5 11:41:50 2024 GMT
*  expire date: Feb  4 11:41:50 2025 GMT
*  common name: localhost (matched)
*  issuer: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: DELETE]
* h2h3 [:path: /registrations/1234-1234-1234]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:5443]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x55b9382537b0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> DELETE /registrations/1234-1234-1234 HTTP/2
> Host: localhost:5443
> user-agent: curl/7.86.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 204
< content-type:
< date: Fri, 17 May 2024 12:59:35 GMT
< server: hypercorn-h2
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host localhost left intact

Erroneous handling:

$ curl -vvv  -X DELETE --http2 --cacert /mnt/c/Users/certs/cacert.pem https://localhost:5443/registrations/1234-1234-1234
*   Trying 127.0.0.1:5443...
* Connected to localhost (127.0.0.1) port 5443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /mnt/c/Users/certs/cacert.pem
*  CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
*  start date: Feb  5 11:41:50 2024 GMT
*  expire date: Feb  4 11:41:50 2025 GMT
*  common name: localhost (matched)
*  issuer: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: DELETE]
* h2h3 [:path: /registrations/1234-1234-1234]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:5443]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x55af248417b0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> DELETE /registrations/1234-1234-1234 HTTP/2
> Host: localhost:5443
> user-agent: curl/7.86.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 204
< content-type:
< date: Fri, 17 May 2024 12:59:40 GMT
< server: hypercorn-h2
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS alert, close notify (256):
* HTTP/2 stream 0 was not closed cleanly before end of the underlying stream
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (92) HTTP/2 stream 0 was not closed cleanly before end of the underlying stream

[bagder]

will let you know about the final outcome of this particular issue I'm having

From our point of view, everything here is solved as it seems fixed in current versions.

[zazola]

Sorry for the late response, yes indeed in the 8.8.0 version everything seems to be solved in libcurl.