Convert Guzzle requests to curl commands with namshi/cuzzle

The other day the namshi/cuzzle PHP pacakge came in really handy.

This library let’s you dump a Guzzle request to a cURL command for debug and log purposes

This way I could test some things on the CLI, and easily share these tests with all my colleagues, including those without PHP installed.

use Namshi\Cuzzle\Formatter\CurlFormatter;
use GuzzleHttp\Message\Request;

$request = new Request('GET', 'example.local');
$options = [];

echo (new CurlFormatter())->format($request, $options);
// ~> curl example.local -X GET -A 'GuzzleHttp/6.4.1 curl/7.71.1 PHP/7.4.9'

Also comes with a Monolog formatter to easily log the resulting curl commands in your log files. Do keep in mind that you might be leaking sensitive information (passwords/tokens) that way …

Installation per Composer:

composer require namshi/cuzzle

Cuzzle, cURL command from Guzzle requests →

httpstat – curl statistics made simple

httpstat visualizes curl(1) statistics in a way of beauty and clarity.

It is a single file🌟 Python script that has no dependencyπŸ‘ and is compatible with Python 3🍻.

Installation through PiP or HomeBrew:

pip install httpstat
brew install httpstat

Once installed through one of those, you can directly call httpstat:

httpstat https://www.bram.us/

httpstat – curl statistics made simple →

About the HTTP Expect: 100-continue header …

TL;DR HTTP clients may send a Expect: 100-continue header along with POST requests to warn the server that they’re about to send a large(ish) payload.

At that point the server can:

  1. Decline, by sending back 401/405 and successively closing the connection.
  2. Accept, by sending back 100 Continue, after which the client will send over the payload.
  3. Ask the client to re-send the original request in an unaltered way (original headers + payload), by replying with a 417 Expectation Failed

The almighty curl is quite a courteous HTTP client and automatically sends the Expect: 100-continue header when it’s about to send large payloads.

Unfortunately, as I discovered and reported, it doesn’t didn’t play nice when it got sent back a 417 …

~

In a recent project I was fiddling around with implementing an HTTP Server. One of my curl-based tests included a POST request that had a largeish payload.

curl -X POST -d "<somelargepayload>" http://localhost:8080/

Upon executing that command I saw this arrive on the server:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.64.1
Accept: */*
Content-Length: 4096
Expect: 100-continue

As you can see here – and what I didn’t expect – is that the POST body itself isn’t present with that request. Additionally a Expect: 100-continue header got injected somewhere along the road. So where did the POST body go? And what about that new header?

~

πŸ‘€ Digging into the HTTP specification

Turning to the HTTP spec to see what the Expect header is all about, I found that it is quite a courteous addition to the spec which clients can implement:

The “Expect” header field in a request indicates a certain set of behaviors (expectations) that need to be supported by the server in order to properly handle this request. The only such expectation defined by this specification is 100-continue.

A 100-continue expectation informs recipients that the client is about to send a (presumably large) message body […] This allows the client to wait for an indication [from the server] that it is worthwhile to send the message body before actually doing so […].

[A 100-continue expectation] allows the origin server to immediately respond with an error message, such as 401 (Unauthorized) or 405 (Method Not Allowed), before the client starts filling the pipes with an unnecessary data transfer.

And that’s exactly what happened: curl injected the Expect: 100-continue header and waited for an interim response from the server. This gives the server the time to evaluate whether is should abort the request or not (most likely by evaluating the value of the Content-Length header). After one second of waiting for such an interim response, curl sent over the data in the POST body. Attaboy, curl! 🐢

⏱ Sidenote: That one second of waiting is the default timeout curl will wait. You can configure it using the --expect100-timeout SECONDS option:

curl -X POST -d "<somelargepayload>" --expect100-timeout 5 http://localhost:8080/

πŸ“­ Sidenote: To prevent curl from injecting the Expect: 100-continue header – and have it immediately send the POST body without warning the server first – you can override the Expect header:

curl -X POST -d "<somelargepayload>" -H "Expect:" http://localhost:8080/

So, mystery solved right? The POST body is just sent a little later, after curl gives the server the change to prematurely abort the request. Yes, but there’s more to it …

~

πŸ€“ More spec details

Reading further down in the spec I found a few more interesting tidbits:

A client that will wait for a 100 (Continue) response before sending the request message body MUST send an Expect header field containing a 100-continue expectation.

A client that sends a 100-continue expectation is not required to wait for any specific length of time; such a client MAY proceed to send the message body even if it has not yet received a response. Furthermore, since 100 (Continue) responses cannot be sent through an HTTP/1.0 intermediary, such a client SHOULD NOT wait for an indefinite period before sending the message body.

This specifies that a client MAY wait for the server to send a 100 Continue interim response, but that the client may also just wait for an arbitrary amount of time and send over the body without awaiting that interim response. This last part is for compatibility reasons with HTTP/1.0, which does not speak Expect: 100-continue. Again curl did as told: it waited for 1 second and sent over the body.

~

♻️ HTTP 417

Apart from ways of aborting a Expect: 100-continue request (by sending back a 401 or 405) and allowing such a request (by sending back a 100), the spec also provides the server with a way of forcing the client to send everything over in one go:

A client that receives a 417 (Expectation Failed) status code in response to a request containing a 100-continue expectation SHOULD repeat that request without a 100-continue expectation, since the 417 response merely indicates that the response chain does not support expectations (e.g., it passes through an HTTP/1.0 server).

To not have my server keep track of both individual message parts, I decided to implement the 417 behavior. That way I could simply discard the first part of the request (the part with 100-continue expectation), send back the 417, and then read out the entire original HTTP message in one go.

This approach also played nice with the parse_request helper function from Guzzle’s PSR-7 Library which I am using: it takes an entire HTTP message (e.g. headers + body) as its only argument.

~

πŸ› curl vs. HTTP 417.

As said, as done. Time to test things out!

curl -X POST -d "" http://localhost:8080/

Again curl sent out a request with a Expect: 100-continue header, and this time I responded with HTTP/1.1 417 Expectation Failed so that curl would do as the spec tells it to do.

Curl however did not behave as … expected (πŸ₯): After waiting for one second it sent over the POST body, just like it did before, even though it had received the 417.

With all the gathered knowledge and this behavior combined I turned to curl’s GitHub repository and opened an issue. Pretty soon after I got a response from curl’s lead developer Daniel Stenberg:

This is indeed a bug. 417 is just a very unusual response code so we’ve never had reason to implement this properly before.

What surprised me even more is that this bug got fixed in no time: two days after reporting it a fix landed and was set out to ship with the 7.69.0 release (which was released early March). Daniel even wrote an extensive blogpost about the 417 header on his blog.

~

Let this be a reminder to everyone to report bugs, no matter how small or rare they are. I think that, as a user of other people’s code, it’s our duty to report so. Being a developer of libraries/stuff myself, it’s one of those thing that I really appreciate, especially when they’re reduced to a minimal test-case and described in detail. Extra bonus points for referring to papers and specifications. And oh, always keep it friendly, that also helps πŸ™‚

Test an IMAP connection with cURL

Today I needed to debug an IMAP problem. I got reports from a user (whose password I recently rotated) that Outlook wouldn’t connect, even though they had updated the password in Outlook’s settings. Checking things on the server I noticed that the connection to the server was made, but the login attempt always failed.

As I didn’t want to add the account to my local mail client I resorted to the almighty curl to test the new password. Below is how I did that.

🀬 To jump ahead: Eventually it turned out that the newly entered password in Outlook was correct but that Outlook basically ignores it when adjusting it from its settings window. The β€œcorrect” way to change a password in Outlook is to go to the Control Panel instead, and change it from there (in a window that is basically the same as Outlook’s). Makes sense … NOT!

~

Depending of whether you’re using a secured connection or not, use one of these:

After successfully logging in, you should see output similar to this:

Click to expand
*   Trying XXX.XXX.XXX.XXX...
* TCP_NODELAY set
* Connected to domain.tld (XXX.XXX.XXX.XXX) port 993 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL; CN=domain.tld
*  start date: Sep  4 00:00:00 2019 GMT
*  expire date: Sep  3 23:59:59 2020 GMT
*  subjectAltName: host "domain.tld" matched cert's "domain.tld"
*  issuer: C=US; ST=TX; L=Houston; O=cPanel, Inc.; CN=cPanel, Inc. Certification Authority
*  SSL certificate verify ok.
< * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE NAMESPACE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
> A001 CAPABILITY
< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE NAMESPACE LITERAL+ AUTH=PLAIN AUTH=LOGIN
< A001 OK Pre-login capabilities listed, post-login capabilities have more.
> A002 AUTHENTICATE PLAIN dXNlcm5hbWU6cGFzc3dvcmQ=
< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY NAMESPACE LITERAL+ NOTIFY SPECIAL-USE COMPRESS=DEFLATE QUOTA
< A002 OK Logged in
> A003 SELECT INBOX
< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
< * 3270 EXISTS
< * 0 RECENT
< * OK [UNSEEN 3256] First unseen.
< * OK [UIDVALIDITY 1325596071] UIDs valid
< * OK [UIDNEXT 67684] Predicted next UID
< * OK [HIGHESTMODSEQ 68768] Highest
< A003 OK [READ-WRITE] Select completed (0.074 + 0.000 + 0.073 secs).
> A004 SEARCH NEW
< * SEARCH
* SEARCH
< A004 OK Search completed (0.009 + 0.000 + 0.008 secs).
* Connection #0 to host domain.tld left intact
> A005 LOGOUT
< * BYE Logging out
< A005 OK Logout completed (0.001 + 0.000 secs).
* Closing connection 0

~

A note on usernames

If the username you are testing contains an @, you must use its urlencoded counterpart %40.

Alternatively you can also pass in the username:password combination separately using the -u flag

curl -v -u 'user:password' imaps://mailserver.tld/INBOX?NEW

~

Did this help you out? Like what you see?
Thank me with a coffee.

I don't do this for profit but a small one-time donation would surely put a smile on my face. Thanks!

β˜•οΈ Buy me a Coffee (€3)

To stay in the loop you can follow @bramus or follow @bramusblog on Twitter.

What’s the weather like in …? Just cURL it!

wttr-in

To know what the weather is like in a specific (or in the current city) I always Google for β€œ{cityname} weather forecast”. Thanks to wttr.in I can now also get to know that via cURL. Just send a cURL request to the URL and voila:

$ curl wttr.in
Weather for City: Sint-Amandsberg, Belgium

     \   /     Sunny
      .-.      5 – 8 Β°C       
   ― (   ) ―   ↙ 15 km/h      
      `-’      10 km          
     /   \     0.0 mm   

You can also append the name of a specific city to the domain:

$ curl wttr.in/paris
Weather for City: Paris, France

    \  /       Partly cloudy
  _ /"".-.     6 – 9 Β°C       
    \_(   ).   ↙ 17 km/h      
    /(___(__)  10 km          
               0.0 mm     

Since it’s curlable, it’s on the web (ref). Ergo you can also visit wttr.in in your browser.

The actual fetching of the weather data is done by wego, a weather client for the terminal.

wttr.in →
wttr.in Source (GitHub) →
wego Source (GitHub) →

(via)

HTTPie – Command line HTTP client

httpie

HTTPie (pronounced aych-tee-tee-pie) is a command line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. It provides a simple http command that allows for sending arbitrary HTTP requests using a simple and natural syntax, and displays colorized output. HTTPie can be used for testing, debugging, and generally interacting with HTTP servers.

HTTPie β€” A command line HTTP client, a user-friendly cURL replacement →

Guzzle β€” PHP HTTP Client

Guzzle is a framework that includes the tools needed to create a robust web service client, including: Service descriptions for defining the inputs and outputs of an API, resource iterators for traversing paginated resources, batching for sending a large number of requests as efficiently as possible.

<?php
require_once 'vendor/autoload.php';
use Guzzle\Http\Client;

// Create a client to work with the Twitter API
$client = new Client('https://api.twitter.com/{version}', array(
    'version' => '1.1'
));

// Sign all requests with the OauthPlugin
$client->addSubscriber(new Guzzle\Plugin\Oauth\OauthPlugin(array(
    'consumer_key'    => '***',
    'consumer_secret' => '***',
    'token'           => '***',
    'token_secret'    => '***'
)));

echo $client->get('statuses/user_timeline.json')->send()->getBody();
// >>> {"public_gists":6,"type":"User" ...

// Create a tweet using POST
$request = $client->post('statuses/update.json', null, array(
    'status' => 'Tweeted with Guzzle, http://guzzlephp.org'
));

// Send the request and parse the JSON response into an array
$data = $request->send()->json();
echo $data['text'];
// >>> Tweeted with Guzzle, http://t.co/kngJMfRk

Guzzle →