Using CURL to call AWS ReST API, signing request with v4 signature

aws-sigv4-ssm-get-parameter.sh

#!/bin/bash

# Source: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

[[ -n "${AWS_ACCESS_KEY_ID}" ]]     || { echo "AWS_ACCESS_KEY_ID required" >&2; exit 1; }
[[ -n "${AWS_SECRET_ACCESS_KEY}" ]] || { echo "AWS_SECRET_ACCESS_KEY required" >&2; exit 1; }

readonly parameterName="SlawekTestParam"

readonly method="POST"
readonly service="ssm"
readonly host="ssm.us-west-2.amazonaws.com"
readonly region="us-west-2"
readonly endpoint="https://${host}/"
readonly contentType="application/x-amz-json-1.1"
readonly amazonTarget="AmazonSSM.GetParameter"
readonly requestParameters="$(printf '{"Name":"%s","WithDecryption":true}' "${parameterName}")"
readonly amazonDate="$(date --utc +'%Y%m%dT%H%M%SZ')"
readonly dateStamp="$(date --utc +'%Y%m%d')"
# readonly amazonDate="20200429T093445Z"
# readonly dateStamp="20200429"

function sha256 {
    echo -ne "$1" | openssl dgst -sha256 -hex
}

function hex {
    echo -ne "$1" | hexdump | sed -e 's/^[0-9a-f]*//' -e 's/ //g' | tr -d '\n'
}

function sign {
    local hexKey="$1"
    local msg="$2"

    echo -ne "${msg}" | openssl dgst -sha256 -mac hmac -macopt "hexkey:${hexKey}"
}

function getSignatureKey {
    local key="$1"
    local dateStamp1="$2"
    local regionName="$3"
    local serviceName="$4"
    local kDate kRegion kService kSigning

    kDate="$(sign "$(hex "AWS4${key}")" "${dateStamp1}")"
    kRegion="$(sign "${kDate}" "${regionName}")"
    kService="$(sign "${kRegion}" "${serviceName}")"
    kSigning="$(sign "${kService}" "aws4_request")"

    echo -ne "${kSigning}"
}

# --- TASK 1: create canonical request ---

readonly canonicalUri="/"
readonly canonicalQueryString=""
readonly canonicalHeaders="content-type:${contentType}\nhost:${host}\nx-amz-date:${amazonDate}\nx-amz-target:${amazonTarget}\n"
readonly signedHeaders="content-type;host;x-amz-date;x-amz-target"
readonly payloadHash="$(sha256 "${requestParameters}")"

readonly canonicalRequest="${method}\n${canonicalUri}\n${canonicalQueryString}\n${canonicalHeaders}\n${signedHeaders}\n${payloadHash}"

# --- TASK 2: create the string to sign ---

readonly algorithm="AWS4-HMAC-SHA256"
readonly credentialScope="${dateStamp}/${region}/${service}/aws4_request"

readonly stringToSign="${algorithm}\n${amazonDate}\n${credentialScope}\n$(sha256 "${canonicalRequest}")"

# --- TASK 3: calculate the signature ---

readonly signingKey="$(getSignatureKey "${AWS_SECRET_ACCESS_KEY}" "${dateStamp}" "${region}" "${service}")"

readonly signature="$(sign "${signingKey}" "${stringToSign}")"

# --- TASK 4: add signing information to the request ---

readonly authorizationHeader="${algorithm} Credential=${AWS_ACCESS_KEY_ID}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}"

# --- SEND REQUEST ---

curl --fail --silent \
    "${endpoint}" \
    --data "${requestParameters}" \
    --header "Accept-Encoding: identity" \
    --header "Content-Type: ${contentType}" \
    --header "X-Amz-Date: ${amazonDate}" \
    --header "X-Amz-Target: ${amazonTarget}" \
    --header "Authorization: ${authorizationHeader}"

Since February 2021, curl supports this with --aws-sigv4.

Nice @maxz, any link where this is documented ?

EDIT: nvm, found it https://curl.se/docs/manpage.html#--aws-sigv4

man, --aws-sigv4 makes no sense how to use. @maxz any examples please

@Moulick it should be along these lines:

curl -X POST "${ENDPOINT}" \
-d $DATA \
--user $AWS_ACCESS_KEY:$AWS_SECRET_KEY \
--aws-sigv4 "aws:amz:${REGION}:${SERVICE}"

However, being said that, I'm currently getting the following response, when I execute such a request.

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

But I'm confident that I'm using correct $AWS_ACCESS_KEY:$AWS_SECRET_KEY

@poborin A simple get worked for me with Amazon ES. Nice. Maybe you got the Service short name wrong ?
If you tell me which service you are trying to hit, I can try to replicate and try to figure out ?

@Moulick, my idiomatic request is

curl --request GET "https://${ES_DOMAIN_ENDPOIN}/my_index_pattern/my_type/_mapping" \
--user $AWS_ACCESS_KEY:$AWS_SECRET_KEY \
--aws-sigv4 "aws:amz:ap-southeast-2:es"

apparently, curl doesn't sign paths with * correctly. my_index_pattern from the example above looks like tempo_data-*

you are right, * in the url does not work, the same url with https://github.com/okigan/awscurl works perfectly. This I think is a bug in curl.

@Moulick I'm trying to use the --aws-signv4 to access the SP-API but I keep getting an invalid signature error.
my Curl request;

curl --location --request GET "https://sellingpartnerapi-eu.amazon.com/reports/2021-06-30/reports?reportTypes=GET_MERCHANT_LISTINGS_ALL_DATA&processingStatuses=&marketplaceIds=A1F83G8C2ARO7P&pageSize=10&createdSince=2021-12-06T07:53:15.435Z&createdUntil=2021-12-10T07:53:15.435Z" --header "x-amz-access-token: Atza|IwxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxZ2iD" --user AxxxxxxxxxxxxxxxxxV:Hxxxxxxxxxxxxxxxxxxxq --aws-sigv4 "aws:amz:eu-west-1:execute-api"

I can perform the same request fine in postman but not curl, is there something obvious I'm doing wrong?

@Gantonmaz I doubt you should be sending --header "x-amz-access-token: Atza|IwxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxZ2iD" as that should be generated by curl based on the --user AxxxxxxxxxxxxxxxxxV:Hxxxxxxxxxxxxxxxxxxxq ?

@Gantonmaz

I can perform the same request fine in postman but not curl, is there something obvious I'm doing wrong?

@Gantonmaz Header name is x-amz-security-token (not x-amz-access-token)

While this is about AWS specific signatures many thanks for the script, as it is a useful model/starting point for generating RFC compliant or other non-AWS specific signatures.

I see following error

{
  "errors" : [ {
    "errorType" : "UnrecognizedClientException",
    "message" : "The security token included in the request is invalid."
  } ]
}%  

I am trying to curl a graphql api . Any ideas ? I have the correct access key and secret . Had to add Authorization Header with correct token but now I get empty Request Body exception, The same request works if used by passing x-api-key

{
  "errors" : [ {
    "message" : "Request body is empty.",
    "errorType" : "MalformedHttpRequestException"
  } ]
}% 

If your AWS authentication involves session tokens, that string goes into a x-amz-security-token header. This is working for me today against an AWS OpenSearch 1.3 cluster.

curl --request GET \
  'https://your-domain-endpoint/_cat/health' \
  --aws-sigv4 aws:amz:us-east-1:es \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
  --header 'Accept: application/json'

awscurl is mentioned above by @Moulick. I've found it very handy for this type of debugging because it picks up your credential from the client creds file (or drops back to botocore's default sequence).

https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-compatible-APIs.html

if you have a device with a TPM (trusted platform module), you can also embed teh AWS_SECRET_ACCESS_KEY into the device and use the tpm to perform the first hmac.

in the end, the aws creds are encoded into hardware and the tpm performs the first step

i have an example of that in go

• https://github.com/salrashid123/aws_hmac

and if you want to use curl and the tpm the first hmac is something like this (you'll have to prepend the "AWS4" to the key though):

• https://github.com/salrashid123/tpm2/tree/master/hmac_import