Curl is free open source software that is used for making HTTP(S) requests. It's pre-installed on a variety of non-windows platforms and is supported on almost all operating systems, including Windows. It's particularly useful for debugging APIs, as testers and developers can use identical commands for reproducing issues without the need for screenshots or browsers or any other intermediate technology. Below is a short tutorial to cover most use-cases.
curl www.google.com #returns just the body of the HTTP request
curl -i "www.google.com" #returns the body plus the HTTP headers
curl -v "www.google.com" > /dev/null #(*nix only) HTTP headers without body
curl -k "www.google.com" #ignore HTTPS certificates (useful for unsigned certs)
Note that the "" are not necessary for the commands above, but they are necessary anytime you use query strings. For example, if you were doing this without quotes
curl -k https://www.google.com/?q=linux&gws_rd=ssl #<-- wrong
Bash will interpret the "&" in the URL as a directive to run the command in the background, instead of as part of the command itself. So you have to include quotes instead:
curl -k "https://www.google.com/?q=linux&gws_rd=ssl" #<--right (quotes)
Note that single quotes '' and double quotes "" are generally interchangeable I usually use single quotes when passing POST/PUT requests that have a JSON payload. That being said, if you need to use environment variables within the command, you should use double-quotes.
It doesn't matter what order you include options. This:
curl "https://www.google.com/?q=linux&gws_rd=ssl" -k
is the same as this:
curl -k "https://www.google.com/?q=linux&gws_rd=ssl"
You need to include HTTP headers for certain formats of data (e.g. json) sometimes APIs work without passing headers, but many times they need them in order to process/return the data you are requesting.
headers can be included by doing this:
curl -H "<your header name here>: <your header value here>"
Many times you should tell the target endpoint what content you accept. This is done like this:
curl -H 'Accept: */*' www.google.com
The */* means you accept any content-type that the server returns.
To specify non-query string data, you can use "-d" as an option
curl 'https://<your_site>.com/login' \
-d '{"email":"testcurl@test.com","password":"curlisgreat"}'
By default the HTTP verb is POST when you specify data without the "-X" option
curl -X "PUT" 'https://<your_site>.com/login' \
-d '{"email":"testcurl@test.com","password":"curlisgreat"}'
The above example only works with a POST HTTP verb, and it still returns error messages when we submit data with POST.
curl 'https://<your_site>.com/login' \
-d '{"email":"testcurl@test.com","password":"curlisgreat"}'
[{"type":"email","message":"required"},{"type":"password","message":"required"}]
# ^ response
To fix this, we have to tell the API what the content is that we're sending. We do this with the Content-Type header
curl -H "Content-Type:application/json; charset=UTF-8" \
'https://<your_site>.com/login' \
-d '{"email":"testcurl@test.com","password":"curlisgreat"}' # gives valid response
Another example would be to specify form data as opposed to JSON. This would be done like this:
curl -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" \
'https://<your_site>.com/login' \
-d 'email=testcurl@test.com&password=curlisgreat' # gives valid response as well
Note that when we did this, we had to change the data we passed to not be JSON because we are using a form URL encoded content type.
At some point you're going to need to make requests that require session tokens or cookies. There are options to load cookies from external files and output the cookies returned by requests to external files, but I don't use them for simplicity reasons. I normally just copy them manually into the HEAD of the request if needed. For example if you use "-i" on the login curl request from earlier...
curl -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" \
'https://<your_site>.com/login' \
-d 'email=testcurl@test.com&password=curlisgreat' -si \
| grep -i "<your_site>-session-token"
One of the response headers might be printed as:
Set-Cookie: <your_site>-session-token=<random_md5_hash>; domain=.<your_site>.com; path=/; expires=Fri, 18 Dec 2015 16:11:26 -0000
We will assume that for <your_site>, <your_site>-session-token is what controls the user's session. If we try to make a privileged request without it, as shown below, we will get a 401 unauthorized response:
curl -H "Content-Type: application/json; charset=UTF-8" \
'https://<your_site>.com/update' \
-d '{"first_name":"mr curl", "last_name":"dd", "fiance_first_name":"ms curl",\
"fiance_last_name":"dd", "number_of_guests":"100 - 150",\
"wedding_location":"aruba", "phone_number":"999-999-9999",\
"addresses":[{"address_type":"registry_shipping", "address_1":"195 Broadway",\
"city":"New York", "state":"NY", "zip":"10007"}]}' #doesn't have -H Cookie
{"message":"Must be logged in to make updates"} #401 Unauthorized HTTP status
We have to include the cookie as shown below to make this request:
curl -H "Cookie: <your_site>-session-token=<random_md5_hash>;"\
-H "Content-Type: application/json; charset=UTF-8" \
'https://<your_site>.com/update' \
-d '{"first_name":"mr curl", "last_name":"dd", "fiance_first_name":"ms curl",
"fiance_last_name":"dd", "number_of_guests":"100 - 150",
"wedding_location":"aruba", "phone_number":"999-999-9999",
"addresses":[
{
"address_type":"registry_shipping", "address_1":"195 Broadway",
"city":"New York", "state":"NY", "zip":"10007"
}
]}' #works!!!
When debugging applications and APIs, it's important to be able to duplicate exact requests for developers and other teams that are responsible for APIs when you see errors. This is where you should use the copy as curl command. For example, if I'm debugging Google login functionality here, I open the network tab, trigger the failure I want, and use "copy as curl", then paste the result into my terminal window. The result might look something like this:
curl -X "PUT" 'https://<your_site>.com/login' -H 'Pragma: no-cache' -H 'Origin: https://<your_site>.com' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: no-cache' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: optimizelySegments=%7B%221502791747%22%3A%22gc%22%2C%221504741846%22%3A%22false%22%2C%221504741847%22%3A%22none%22%2C%221507561781%22%3A%22direct%22%7D; optimizelyEndUserId=oeu1447865433126r0.7859691749326885; optimizelyBuckets=%7B%7D; ajs_user_id=null; ajs_group_id=null; connection=average; _ga=GA1.2.929727783.1447865434; ajs_anonymous_id=%221daa9db7-88bf-4d1a-ae07-ddba4cf42c8e%22; _gat=1; mp_94166a6eec563b26dbb53716b06ac02a_mixpanel=%7B%22distinct_id%22%3A%20%221511b81400f40-0b20ca277-10316d51-13c680-1511b814010120%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D' -H 'Connection: keep-alive' -H 'Referer: https://<your_site>.com' --data-binary '{"email":"testcurl@test.com","password":"curlisgreat"}' --compressed
This curl demonstrates what I think would be a failure, but I usually try to reduce the command as much as possible by removing all components that don't contribute to the failure I see, so it's easier for the teams that maintain the API to debug.
In this case, it would look something like this... I would remove all headers besides Content-Type, because login shouldn't require any cookies (this may not always be true for other applications, but it is for this one). The remainder looks something like this:
curl -X "PUT" -H 'Content-Type: application/json; charset=UTF-8' 'https://<your_site>.com/login' --data-binary '{"email":"testcurl@test.com","password":"curlisgreat"}'
I get an HTTP 405 back. If I were the one debugging this, I might try removing the -X "PUT" as well just for fun before logging the issue with the other team.
curl -H 'Content-Type: application/json; charset=UTF-8' 'https://<your_site>.com/login' --data-binary '{"email":"testcurl@test.com","password":"curlisgreat"}'
I would then discover that this gives me a valid response back.
{"first_name":"dd","last_name":"dd","email":"testcurl@test.com","id":"7f479850-f742-413d-97f4-4c9bba335377","addresses":[{"address_1":null,"address_2":null,"city":null,"state":null,"zip":null,"address_type":"account_create"}]}
The issue is that the API endpoint doesn't support PUT here, but only POST.
Thus, if I thought it should support PUT, I would contact the team; otherwise, I would leave it as-is.