About the HTTP Expect: 100-continue header …

~

In a recent project I was fiddling around with implementing an HTTP Server. One of my curl-based tests included a POST request that had a largeish payload.

curl -X POST -d "<somelargepayload>" http://localhost:8080/

Upon executing that command I saw this arrive on the server:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.64.1
Accept: */*
Content-Length: 4096
Expect: 100-continue

As you can see here – and what I didn’t expect – is that the POST body itself isn’t present with that request. Additionally a Expect: 100-continue header got injected somewhere along the road. So where did the POST body go? And what about that new header?

~

👀 Digging into the HTTP specification

Turning to the HTTP spec to see what the Expect header is all about, I found that it is quite a courteous addition to the spec which clients can implement:

The “Expect” header field in a request indicates a certain set of behaviors (expectations) that need to be supported by the server in order to properly handle this request. The only such expectation defined by this specification is 100-continue.

A 100-continue expectation informs recipients that the client is about to send a (presumably large) message body […] This allows the client to wait for an indication [from the server] that it is worthwhile to send the message body before actually doing so […].

[A 100-continue expectation] allows the origin server to immediately respond with an error message, such as 401 (Unauthorized) or 405 (Method Not Allowed), before the client starts filling the pipes with an unnecessary data transfer.

And that’s exactly what happened: curl injected the Expect: 100-continue header and waited for an interim response from the server. This gives the server the time to evaluate whether is should abort the request or not (most likely by evaluating the value of the Content-Length header). After one second of waiting for such an interim response, curl sent over the data in the POST body. Attaboy, curl! 🐶

curl -X POST -d "<somelargepayload>" --expect100-timeout 5 http://localhost:8080/

curl -X POST -d "<somelargepayload>" -H "Expect:" http://localhost:8080/

So, mystery solved right? The POST body is just sent a little later, after curl gives the server the change to prematurely abort the request. Yes, but there’s more to it …

~

🤓 More spec details

Reading further down in the spec I found a few more interesting tidbits:

A client that will wait for a 100 (Continue) response before sending the request message body MUST send an Expect header field containing a 100-continue expectation.

A client that sends a 100-continue expectation is not required to wait for any specific length of time; such a client MAY proceed to send the message body even if it has not yet received a response. Furthermore, since 100 (Continue) responses cannot be sent through an HTTP/1.0 intermediary, such a client SHOULD NOT wait for an indefinite period before sending the message body.

This specifies that a client MAY wait for the server to send a 100 Continue interim response, but that the client may also just wait for an arbitrary amount of time and send over the body without awaiting that interim response. This last part is for compatibility reasons with HTTP/1.0, which does not speak Expect: 100-continue. Again curl did as told: it waited for 1 second and sent over the body.

~

♻️ HTTP 417

Apart from ways of aborting a Expect: 100-continue request (by sending back a 401 or 405) and allowing such a request (by sending back a 100), the spec also provides the server with a way of forcing the client to send everything over in one go:

A client that receives a 417 (Expectation Failed) status code in response to a request containing a 100-continue expectation SHOULD repeat that request without a 100-continue expectation, since the 417 response merely indicates that the response chain does not support expectations (e.g., it passes through an HTTP/1.0 server).

To not have my server keep track of both individual message parts, I decided to implement the 417 behavior. That way I could simply discard the first part of the request (the part with 100-continue expectation), send back the 417, and then read out the entire original HTTP message in one go.

This approach also played nice with the parse_request helper function from Guzzle’s PSR-7 Library which I am using: it takes an entire HTTP message (e.g. headers + body) as its only argument.

~

🐛 curl vs. HTTP 417.

As said, as done. Time to test things out!

curl -X POST -d "" http://localhost:8080/

Again curl sent out a request with a Expect: 100-continue header, and this time I responded with HTTP/1.1 417 Expectation Failed so that curl would do as the spec tells it to do.

Curl however did not behave as … expected (🥁): After waiting for one second it sent over the POST body, just like it did before, even though it had received the 417.

With all the gathered knowledge and this behavior combined I turned to curl’s GitHub repository and opened an issue. Pretty soon after I got a response from curl’s lead developer Daniel Stenberg:

This is indeed a bug. 417 is just a very unusual response code so we’ve never had reason to implement this properly before.

What surprised me even more is that this bug got fixed in no time: two days after reporting it a fix landed and was set out to ship with the 7.69.0 release (which was released early March). Daniel even wrote an extensive blogpost about the 417 header on his blog.

~

Let this be a reminder to everyone to report bugs, no matter how small or rare they are. I think that, as a user of other people’s code, it’s our duty to report so. Being a developer of libraries/stuff myself, it’s one of those thing that I really appreciate, especially when they’re reduced to a minimal test-case and described in detail. Extra bonus points for referring to papers and specifications. And oh, always keep it friendly, that also helps 🙂