Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Vault Auth

vault_aws_auth

These are python 2 and 3 snippets showing how to generate headers to authenticate with HashiCorp's Vault using the AWS authentication method. There's also a Ruby implementation which uses version 3 of the AWS SDK for Ruby.

The python scripts look for credentials in the default boto3 locations; if you need to supply custom credentials (such as from an AssumeRole call), you would use the botocore.session.set_credentials method before calling create_client.

The ruby script looks for credentials from the default SDK locations.

Credits

Thanks to @copumpkin for much of the original python 2 implementation (provided privately) on which this was based.

Thanks to @stark525 for starting the python 3 port, on which the python 3 implementation is based.

#!/usr/bin/env python
import botocore.session
from botocore.awsrequest import create_request_object
import json
import base64
def headers_to_go_style(headers):
retval = {}
for k, v in headers.iteritems():
retval[k] = [v]
return retval
def generate_vault_request(role_name=""):
session = botocore.session.get_session()
# if you have credentials from non-default sources, call
# session.set_credentials here, before calling session.create_client
client = session.create_client('sts')
endpoint = client._endpoint
operation_model = client._service_model.operation_model('GetCallerIdentity')
request_dict = client._convert_to_request_dict({}, operation_model)
awsIamServerId = 'vault.example.com'
request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId
request = endpoint.create_request(request_dict, operation_model)
# It's now signed...
return {
'iam_http_request_method': request.method,
'iam_request_url': base64.b64encode(request.url),
'iam_request_body': base64.b64encode(request.body),
'iam_request_headers': base64.b64encode(json.dumps(headers_to_go_style(dict(request.headers)))), # It's a CaseInsensitiveDict, which is not JSON-serializable
'role': role_name,
}
if __name__ == "__main__":
print json.dumps(generate_vault_request('TestRole'))
#!/usr/bin/env ruby
require 'base64'
require 'json'
# Tested against v3
require 'aws-sdk'
class VaultHandler < Seahorse::Client::NetHttp::Handler
def call(context)
req = context.http_request
auth_data = {
"iam_http_request_method" => req.http_method,
"iam_request_url" => Base64.strict_encode64(req.endpoint.request_uri),
"iam_request_body" => Base64.strict_encode64(req.body.read),
"iam_request_headers" => Base64.strict_encode64(JSON.generate(headers(req)))
}
resp = Seahorse::Client::Response.new(context: context, data: auth_data)
resp
end
end
class VaultPlugin < Seahorse::Client::Plugin
handler(VaultHandler, step: :send)
end
# We create our own custom sub-class of Aws::STS::Client so that other calls to STS
# will work as normal (i.e., we don't want to intercept ALL STS requests, JUST this
# request to generate Vault login data)
class VaultStsClient < Aws::STS::Client
set_api(Aws::STS::ClientApi::API)
end
VaultStsClient.add_plugin(VaultPlugin)
sts = VaultStsClient.new()
req = sts.build_request('get_caller_identity')
req.context.http_request.headers['X-Vault-Aws-Iam-Server-Id'] = 'vault.example.com'
auth_data = req.send_request().data
auth_data.each do |key, val|
puts "#{key}=#{val}"
end
#!/usr/bin/env python3
import boto3
import json
import base64
def headers_to_go_style(headers):
retval = {}
for k, v in headers.items():
if isinstance(v, bytes):
retval[k] = [str(v, 'ascii')]
else:
retval[k] = [v]
return retval
def generate_vault_request(role_name=""):
session = boto3.session.Session()
# if you have credentials from non-default sources, call
# session.set_credentials here, before calling session.create_client
client = session.client('sts')
endpoint = client._endpoint
operation_model = client._service_model.operation_model('GetCallerIdentity')
request_dict = client._convert_to_request_dict({}, operation_model)
awsIamServerId = 'vault.example.com'
request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId
request = endpoint.create_request(request_dict, operation_model)
# It's now signed...
return {
'iam_http_request_method': request.method,
'iam_request_url': str(base64.b64encode(request.url.encode('ascii')), 'ascii'),
'iam_request_body': str(base64.b64encode(request.body.encode('ascii')), 'ascii'),
'iam_request_headers': str(base64.b64encode(bytes(json.dumps(headers_to_go_style(dict(request.headers))), 'ascii')), 'ascii'), # It's a CaseInsensitiveDict, which is not JSON-serializable
'role': role_name,
}
if __name__ == "__main__":
print(json.dumps(generate_vault_request('TestRole')))
@Fasian1

This comment has been minimized.

Copy link

@Fasian1 Fasian1 commented Apr 26, 2018

The python implementations have a typo in them.
Line 24/26 should be:
request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId

@joelthompson

This comment has been minimized.

Copy link
Owner Author

@joelthompson joelthompson commented May 17, 2018

Good catch, thanks @Fasian1! Fixed.

@bangpound

This comment has been minimized.

Copy link

@bangpound bangpound commented Jun 15, 2018

@aantono

This comment has been minimized.

Copy link

@aantono aantono commented Aug 9, 2018

Does anyone have a NodeJS variant? I'm struggling in figuring out how to add the custom X-Vault-AWS-IAM-Server-ID header. Using AWS Node.js SDK - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/STS.html#getCallerIdentity-property, but it doesn't look like there is a way to add to the request headers (unless I'm missing something)...

@avoidik

This comment has been minimized.

Copy link

@avoidik avoidik commented Aug 22, 2018

for python you can use hvac library which supports both auth_ec2 and auth_aws_iam

@joelthompson

This comment has been minimized.

Copy link
Owner Author

@joelthompson joelthompson commented Aug 29, 2018

Hey @aantono, check out vault-auth-aws for a NodeJS variant.

@emayssat

This comment has been minimized.

Copy link

@emayssat emayssat commented Apr 2, 2020

The problem with using boto is that it hides how boto generates the signature!
Understanding how the signature is generated is key if you are not using boto!
From the script below, you need extract the sig 4 headers and pass them in the iam_request_headers sent to vault (see at the bottom)

#!/usr/bin/env python3

# Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

# AWS Version 4 signing example

# STS (GetCallerId)

# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a POST request and passes request parameters
# in the body (payload) of the request. Auth information is passed in
# an Authorization header.
import sys, os, base64, datetime, hashlib, hmac
import requests # pip install requests

# ************* REQUEST VALUES *************
method = 'POST'
service = 'sts'
host = 'sts.amazonaws.com'
region = 'us-east-1'
# https://docs.aws.amazon.com/general/latest/gr/sts.html
# endpoint = 'https://sts.us-east-1.amazonaws.com/'
endpoint = 'https://sts.amazonaws.com/'
# POST requests use a content type header. For DynamoDB,
# the content is JSON.
content_type = 'application/x-www-form-urlencoded'

request_parameters = 'Action=GetCallerIdentity&Version=2011-06-15'

# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

def getSignatureKey(key, date_stamp, regionName, serviceName):
    kDate = sign(('AWS4' + key).encode('utf-8'), date_stamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning

# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
    print('No access key is available.')
    sys.exit()

# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amz_date = t.strftime('%Y%m%dT%H%M%SZ')
date_stamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope

# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

# Step 1 is to define the verb (GET, POST, etc.)--already done.

# Step 2: Create canonical URI--the part of the URI from domain to query
# string (use '/' if no path)
canonical_uri = '/'

## Step 3: Create the canonical query string. In this example, request
# parameters are passed in the body of the request and the query string
# is blank.
canonical_querystring = ''

# Step 4: Create the canonical headers. Header names must be trimmed
# and lowercase, and sorted in code point order from low to high.
# Note that there is a trailing \n.
canonical_headers = 'content-type:' + content_type + '\n' + 'host:' + host + '\n' + 'x-amz-date:' + amz_date + '\n' # + 'x-amz-target:' + amz_target + '\n'

# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers include those that you want to be included in the
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = 'content-type;host;x-amz-date'

# Step 6: Create payload hash. In this example, the payload (body of
# the request) contains the request parameters.
payload_hash = hashlib.sha256(request_parameters.encode('utf-8')).hexdigest()

# Step 7: Combine elements to create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash


# ************* TASK 2: CREATE THE STRING TO SIGN*************
# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = date_stamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + '\n' +  amz_date + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()

# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, date_stamp, region, service)

# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()


# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# Put the signature information in a header named Authorization.
authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' +  'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature

# The request can include any headers, but MUST include "host", "x-amz-date",
# "content-type", and "Authorization". Except for the authorization
# header, the headers must be included in the canonical_headers and signed_headers values, as
# noted earlier. Order here is not significant.
# Python note: The 'host' header is added automatically by the Python 'requests' library.
headers = {'Content-Type':content_type,
           'X-Amz-Date':amz_date,
           'Authorization':authorization_header}


# ************* SEND THE REQUEST *************
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + endpoint)



# ************* SEND THE REQUEST *************
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + endpoint)

# Curl note: The 'host' header is added automatically just like python
print('curl --data "{}" -X POST -H "Content-Type: {}" -H "X-Amz-Date: {}" -H "Authorization: {}" {}'.format(
    request_parameters,
    content_type,
    amz_date,
    authorization_header,
    endpoint,
))
r = requests.post(endpoint, data=request_parameters, headers=headers)

print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)

like this

$ cat ./in/curl--aws-login--payload.json; echo
{
    "role": "aws-auth-role--iam-user",
    "iam_http_request_method": "POST",
    "iam_request_url": "aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=",
    "iam_request_body": "QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==",
    "iam_request_headers": {
        "Content-Type": "application/x-www-form-urlencoded",
        "X-Amz-Date": "20200331T234318Z",
        "Authorization": "AWS4-HMAC-SHA256 Credential=AKIAYYYYYYXXXXXXXX/20200331/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=c4ca308d5YYYYYYYYYYYXXXXXXXXX"
    }
}
 
 
# Note that the token to use to access vault is referenced below as the 'client_token'
curl --data @./in/curl--aws-login--payload.json --request POST --silent http://127.0.0.1:8200/v1/auth/aws/login | jq '.'

also remember to modify the code above when you are using the additional and strongly-recommended header seen in boto code

awsIamServerId = 'vault.example.com'
    request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId
@joelthompson

This comment has been minimized.

Copy link
Owner Author

@joelthompson joelthompson commented Apr 2, 2020

Thanks @emayssat for the contribution !

The fact "that it hides how boto generates the signature" is a feature, not a bug :) I was going for simplicity and usability here, rather than trying to expose the raw code, but this is also useful for those who want to understand it more or can't use boto3/botocore for whatever reason.

@rprabakaran

This comment has been minimized.

Copy link

@rprabakaran rprabakaran commented Jun 22, 2020

anyone have java implementation of this generate headers code?

@rprabakaran

This comment has been minimized.

Copy link

@rprabakaran rprabakaran commented Jul 2, 2020

anyone have java implementation of this generate headers code?

I have got this one working in java, if anyone looking for --> https://gist.github.com/rprabakaran/ffd3987b8144b9e7fb776ef44fac1c06

@papovyr

This comment has been minimized.

Copy link

@papovyr papovyr commented Nov 25, 2020

Anyone has example for bash?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment