TL;DR HTTP clients may send a
Expect: 100-continue header along with
POST requests to warn the server that they’re about to send a large(ish) payload.
At that point the server can:
- Decline, by sending back
405 and successively closing the connection.
- Accept, by sending back
100 Continue, after which the client will send over the payload.
- Ask the client to re-send the original request in an unaltered way (original headers + payload), by replying with a
417 Expectation Failed
The almighty curl is quite a courteous HTTP client and automatically sends the
Expect: 100-continue header when it’s about to send large payloads.
Unfortunately, as I discovered and reported, it
doesn’t play nice when it got sent back a
In a recent project I was fiddling around with implementing an HTTP Server. One of my curl-based tests included a
POST request that had a largeish payload.
curl -X POST -d "<somelargepayload>" http://localhost:8080/
Upon executing that command I saw this arrive on the server:
POST / HTTP/1.1
As you can see here – and what I didn’t expect – is that the
POST body itself isn’t present with that request. Additionally a
Expect: 100-continue header got injected somewhere along the road. So where did the
POST body go? And what about that new header?
👀 Digging into the HTTP specification
Turning to the HTTP spec to see what the
Expect header is all about, I found that it is quite a courteous addition to the spec which clients can implement:
The “Expect” header field in a request indicates a certain set of behaviors (expectations) that need to be supported by the server in order to properly handle this request. The only such expectation defined by this specification is
100-continue expectation informs recipients that the client is about to send a (presumably large) message body […] This allows the client to wait for an indication [from the server] that it is worthwhile to send the message body before actually doing so […].
100-continue expectation] allows the origin server to immediately respond with an error message, such as
401 (Unauthorized) or
405 (Method Not Allowed), before the client starts filling the pipes with an unnecessary data transfer.
And that’s exactly what happened: curl injected the
Expect: 100-continue header and waited for an interim response from the server. This gives the server the time to evaluate whether is should abort the request or not (most likely by evaluating the value of the
Content-Length header). After one second of waiting for such an interim response, curl sent over the data in the
POST body. Attaboy, curl! 🐶
⏱ Sidenote: That one second of waiting is the default timeout curl will wait. You can configure it using the
--expect100-timeout SECONDS option:
curl -X POST -d "<somelargepayload>" --expect100-timeout 5 http://localhost:8080/
📭 Sidenote: To prevent curl from injecting the
Expect: 100-continue header – and have it immediately send the
POST body without warning the server first – you can override the
curl -X POST -d "<somelargepayload>" -H "Expect:" http://localhost:8080/
So, mystery solved right? The
POST body is just sent a little later, after curl gives the server the change to prematurely abort the request. Yes, but there’s more to it …
🤓 More spec details
Reading further down in the spec I found a few more interesting tidbits:
A client that will wait for a
100 (Continue) response before sending the request message body MUST send an Expect header field containing a
A client that sends a
100-continue expectation is not required to wait for any specific length of time; such a client MAY proceed to send the message body even if it has not yet received a response. Furthermore, since
100 (Continue) responses cannot be sent through an
HTTP/1.0 intermediary, such a client SHOULD NOT wait for an indefinite period before sending the message body.
This specifies that a client MAY wait for the server to send a
100 Continue interim response, but that the client may also just wait for an arbitrary amount of time and send over the body without awaiting that interim response. This last part is for compatibility reasons with HTTP/1.0, which does not speak
Expect: 100-continue. Again curl did as told: it waited for 1 second and sent over the body.
Apart from ways of aborting a
Expect: 100-continue request (by sending back a
405) and allowing such a request (by sending back a
100), the spec also provides the server with a way of forcing the client to send everything over in one go:
A client that receives a
417 (Expectation Failed) status code in response to a request containing a
100-continue expectation SHOULD repeat that request without a
100-continue expectation, since the
417 response merely indicates that the response chain does not support expectations (e.g., it passes through an
To not have my server keep track of both individual message parts, I decided to implement the
417 behavior. That way I could simply discard the first part of the request (the part with
100-continue expectation), send back the
417, and then read out the entire original HTTP message in one go.
This approach also played nice with the
parse_request helper function from Guzzle’s PSR-7 Library which I am using: it takes an entire HTTP message (e.g. headers + body) as its only argument.
🐛 curl vs. HTTP
As said, as done. Time to test things out!
curl -X POST -d "" http://localhost:8080/
Again curl sent out a request with a
Expect: 100-continue header, and this time I responded with
HTTP/1.1 417 Expectation Failed so that curl would do as the spec tells it to do.
Curl however did not behave as … expected (🥁): After waiting for one second it sent over the
POST body, just like it did before, even though it had received the
With all the gathered knowledge and this behavior combined I turned to curl’s GitHub repository and opened an issue. Pretty soon after I got a response from curl’s lead developer Daniel Stenberg:
This is indeed a bug. 417 is just a very unusual response code so we’ve never had reason to implement this properly before.
What surprised me even more is that this bug got fixed in no time: two days after reporting it a fix landed and was set out to ship with the 7.69.0 release (which was released early March). Daniel even wrote an extensive blogpost about the
417 header on his blog.
Let this be a reminder to everyone to report bugs, no matter how small or rare they are. I think that, as a user of other people’s code, it’s our duty to report so. Being a developer of libraries/stuff myself, it’s one of those thing that I really appreciate, especially when they’re reduced to a minimal test-case and described in detail. Extra bonus points for referring to papers and specifications. And oh, always keep it friendly, that also helps 🙂