Skip to content

Instantly share code, notes, and snippets.

@joelthompson
Last active May 21, 2024 17:19
Show Gist options
  • Save joelthompson/378cbe449d541debf771f5a6a171c5ed to your computer and use it in GitHub Desktop.
Save joelthompson/378cbe449d541debf771f5a6a171c5ed to your computer and use it in GitHub Desktop.
Vault Auth

vault_aws_auth

These are python 2 and 3 snippets showing how to generate headers to authenticate with HashiCorp's Vault using the AWS authentication method. There's also a Ruby implementation which uses version 3 of the AWS SDK for Ruby.

The python scripts look for credentials in the default boto3 locations; if you need to supply custom credentials (such as from an AssumeRole call), you would use the botocore.session.set_credentials method before calling create_client.

The ruby script looks for credentials from the default SDK locations.

Credits

Thanks to @copumpkin for much of the original python 2 implementation (provided privately) on which this was based.

Thanks to @stark525 for starting the python 3 port, on which the python 3 implementation is based.

#!/usr/bin/env python
import botocore.session
from botocore.awsrequest import create_request_object
import json
import base64
def headers_to_go_style(headers):
retval = {}
for k, v in headers.iteritems():
retval[k] = [v]
return retval
def generate_vault_request(role_name=""):
session = botocore.session.get_session()
# if you have credentials from non-default sources, call
# session.set_credentials here, before calling session.create_client
client = session.create_client('sts')
endpoint = client._endpoint
operation_model = client._service_model.operation_model('GetCallerIdentity')
request_dict = client._convert_to_request_dict({}, operation_model)
awsIamServerId = 'vault.example.com'
request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId
request = endpoint.create_request(request_dict, operation_model)
# It's now signed...
return {
'iam_http_request_method': request.method,
'iam_request_url': base64.b64encode(request.url),
'iam_request_body': base64.b64encode(request.body),
'iam_request_headers': base64.b64encode(json.dumps(headers_to_go_style(dict(request.headers)))), # It's a CaseInsensitiveDict, which is not JSON-serializable
'role': role_name,
}
if __name__ == "__main__":
print json.dumps(generate_vault_request('TestRole'))
#!/usr/bin/env ruby
require 'base64'
require 'json'
# Tested against v3
require 'aws-sdk'
class VaultHandler < Seahorse::Client::NetHttp::Handler
def call(context)
req = context.http_request
auth_data = {
"iam_http_request_method" => req.http_method,
"iam_request_url" => Base64.strict_encode64(req.endpoint.request_uri),
"iam_request_body" => Base64.strict_encode64(req.body.read),
"iam_request_headers" => Base64.strict_encode64(JSON.generate(headers(req)))
}
resp = Seahorse::Client::Response.new(context: context, data: auth_data)
resp
end
end
class VaultPlugin < Seahorse::Client::Plugin
handler(VaultHandler, step: :send)
end
# We create our own custom sub-class of Aws::STS::Client so that other calls to STS
# will work as normal (i.e., we don't want to intercept ALL STS requests, JUST this
# request to generate Vault login data)
class VaultStsClient < Aws::STS::Client
set_api(Aws::STS::ClientApi::API)
end
VaultStsClient.add_plugin(VaultPlugin)
sts = VaultStsClient.new()
req = sts.build_request('get_caller_identity')
req.context.http_request.headers['X-Vault-Aws-Iam-Server-Id'] = 'vault.example.com'
auth_data = req.send_request().data
auth_data.each do |key, val|
puts "#{key}=#{val}"
end
#!/usr/bin/env python3
import boto3
import json
import base64
def headers_to_go_style(headers):
retval = {}
for k, v in headers.items():
if isinstance(v, bytes):
retval[k] = [str(v, 'ascii')]
else:
retval[k] = [v]
return retval
def generate_vault_request(role_name=""):
session = boto3.session.Session()
# if you have credentials from non-default sources, call
# session.set_credentials here, before calling session.create_client
client = session.client('sts')
endpoint = client._endpoint
operation_model = client._service_model.operation_model('GetCallerIdentity')
request_dict = client._convert_to_request_dict({}, operation_model)
awsIamServerId = 'vault.example.com'
request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId
request = endpoint.create_request(request_dict, operation_model)
# It's now signed...
return {
'iam_http_request_method': request.method,
'iam_request_url': str(base64.b64encode(request.url.encode('ascii')), 'ascii'),
'iam_request_body': str(base64.b64encode(request.body.encode('ascii')), 'ascii'),
'iam_request_headers': str(base64.b64encode(bytes(json.dumps(headers_to_go_style(dict(request.headers))), 'ascii')), 'ascii'), # It's a CaseInsensitiveDict, which is not JSON-serializable
'role': role_name,
}
if __name__ == "__main__":
print(json.dumps(generate_vault_request('TestRole')))
@Fasian1
Copy link

Fasian1 commented Apr 26, 2018

The python implementations have a typo in them.
Line 24/26 should be:
request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId

@joelthompson
Copy link
Author

Good catch, thanks @Fasian1! Fixed.

@bangpound
Copy link

@aantono
Copy link

aantono commented Aug 9, 2018

Does anyone have a NodeJS variant? I'm struggling in figuring out how to add the custom X-Vault-AWS-IAM-Server-ID header. Using AWS Node.js SDK - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/STS.html#getCallerIdentity-property, but it doesn't look like there is a way to add to the request headers (unless I'm missing something)...

@avoidik
Copy link

avoidik commented Aug 22, 2018

for python you can use hvac library which supports both auth_ec2 and auth_aws_iam

@joelthompson
Copy link
Author

Hey @aantono, check out vault-auth-aws for a NodeJS variant.

@emayssat
Copy link

emayssat commented Apr 2, 2020

The problem with using boto is that it hides how boto generates the signature!
Understanding how the signature is generated is key if you are not using boto!
From the script below, you need extract the sig 4 headers and pass them in the iam_request_headers sent to vault (see at the bottom)

#!/usr/bin/env python3

# Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

# AWS Version 4 signing example

# STS (GetCallerId)

# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a POST request and passes request parameters
# in the body (payload) of the request. Auth information is passed in
# an Authorization header.
import sys, os, base64, datetime, hashlib, hmac
import requests # pip install requests

# ************* REQUEST VALUES *************
method = 'POST'
service = 'sts'
host = 'sts.amazonaws.com'
region = 'us-east-1'
# https://docs.aws.amazon.com/general/latest/gr/sts.html
# endpoint = 'https://sts.us-east-1.amazonaws.com/'
endpoint = 'https://sts.amazonaws.com/'
# POST requests use a content type header. For DynamoDB,
# the content is JSON.
content_type = 'application/x-www-form-urlencoded'

request_parameters = 'Action=GetCallerIdentity&Version=2011-06-15'

# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

def getSignatureKey(key, date_stamp, regionName, serviceName):
    kDate = sign(('AWS4' + key).encode('utf-8'), date_stamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning

# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
    print('No access key is available.')
    sys.exit()

# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amz_date = t.strftime('%Y%m%dT%H%M%SZ')
date_stamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope

# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

# Step 1 is to define the verb (GET, POST, etc.)--already done.

# Step 2: Create canonical URI--the part of the URI from domain to query
# string (use '/' if no path)
canonical_uri = '/'

## Step 3: Create the canonical query string. In this example, request
# parameters are passed in the body of the request and the query string
# is blank.
canonical_querystring = ''

# Step 4: Create the canonical headers. Header names must be trimmed
# and lowercase, and sorted in code point order from low to high.
# Note that there is a trailing \n.
canonical_headers = 'content-type:' + content_type + '\n' + 'host:' + host + '\n' + 'x-amz-date:' + amz_date + '\n' # + 'x-amz-target:' + amz_target + '\n'

# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers include those that you want to be included in the
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = 'content-type;host;x-amz-date'

# Step 6: Create payload hash. In this example, the payload (body of
# the request) contains the request parameters.
payload_hash = hashlib.sha256(request_parameters.encode('utf-8')).hexdigest()

# Step 7: Combine elements to create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash


# ************* TASK 2: CREATE THE STRING TO SIGN*************
# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = date_stamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + '\n' +  amz_date + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()

# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, date_stamp, region, service)

# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()


# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# Put the signature information in a header named Authorization.
authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' +  'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature

# The request can include any headers, but MUST include "host", "x-amz-date",
# "content-type", and "Authorization". Except for the authorization
# header, the headers must be included in the canonical_headers and signed_headers values, as
# noted earlier. Order here is not significant.
# Python note: The 'host' header is added automatically by the Python 'requests' library.
headers = {'Content-Type':content_type,
           'X-Amz-Date':amz_date,
           'Authorization':authorization_header}


# ************* SEND THE REQUEST *************
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + endpoint)



# ************* SEND THE REQUEST *************
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + endpoint)

# Curl note: The 'host' header is added automatically just like python
print('curl --data "{}" -X POST -H "Content-Type: {}" -H "X-Amz-Date: {}" -H "Authorization: {}" {}'.format(
    request_parameters,
    content_type,
    amz_date,
    authorization_header,
    endpoint,
))
r = requests.post(endpoint, data=request_parameters, headers=headers)

print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)

like this

$ cat ./in/curl--aws-login--payload.json; echo
{
    "role": "aws-auth-role--iam-user",
    "iam_http_request_method": "POST",
    "iam_request_url": "aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=",
    "iam_request_body": "QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==",
    "iam_request_headers": {
        "Content-Type": "application/x-www-form-urlencoded",
        "X-Amz-Date": "20200331T234318Z",
        "Authorization": "AWS4-HMAC-SHA256 Credential=AKIAYYYYYYXXXXXXXX/20200331/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=c4ca308d5YYYYYYYYYYYXXXXXXXXX"
    }
}
 
 
# Note that the token to use to access vault is referenced below as the 'client_token'
curl --data @./in/curl--aws-login--payload.json --request POST --silent http://127.0.0.1:8200/v1/auth/aws/login | jq '.'

also remember to modify the code above when you are using the additional and strongly-recommended header seen in boto code

awsIamServerId = 'vault.example.com'
    request_dict['headers']['X-Vault-AWS-IAM-Server-ID'] = awsIamServerId

@joelthompson
Copy link
Author

Thanks @emayssat for the contribution !

The fact "that it hides how boto generates the signature" is a feature, not a bug :) I was going for simplicity and usability here, rather than trying to expose the raw code, but this is also useful for those who want to understand it more or can't use boto3/botocore for whatever reason.

@rprabakaran
Copy link

anyone have java implementation of this generate headers code?

@rprabakaran
Copy link

anyone have java implementation of this generate headers code?

I have got this one working in java, if anyone looking for --> https://gist.github.com/rprabakaran/ffd3987b8144b9e7fb776ef44fac1c06

Copy link

ghost commented Nov 25, 2020

Anyone has example for bash?

@parthicseraj
Copy link

for python you can use hvac library which supports both auth_ec2 and auth_aws_iam

Do you have any example using hvac library for aws_iam ?

@joelthompson
Copy link
Author

The hvac docs have some good examples; I'd suggest checking them out: https://hvac.readthedocs.io/en/stable/usage/auth_methods/aws.html#iam-authentication

@srpraharaj
Copy link

"request_dict = client._convert_to_request_dict({}, operation_model)"
The above line throws an exception: ._convert_to_request_dict() missing 1 required positional argument: 'endpoint_url'.
Please help.

@sunchill06
Copy link

sunchill06 commented Dec 28, 2022

@joelthompson
I am trying to use the headers generated by the python3 script in an AWS lambda function but I always get a 400 response. Not sure whats wrong here. Would appreciate any help if someone has achieved this already.

request_body = json.dumps(generate_vault_request())
awsIamServerId = "<vault-addr>/v1/auth/aws/login"
resp = requests.post(url=awsIamServerId, data=request_body)

@santiagoprieto
Copy link

santiagoprieto commented Mar 8, 2023

@srpraharaj did you figure this issue out? Having the same problem. When I add a '' for endpoint it does pass to create the request but then I get this error:

213 | File "/codebuild/output/src965627234/src/provision/secrets/login.py", line 42, in login
214 | sts_request = endpoint.create_request(request_dict, operation_model)
215 | File "/root/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 134, in create_request
216 | self._event_emitter.emit(
217 | File "/root/.local/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
218 | return self._emitter.emit(aliased_event_name, **kwargs)
219 | File "/root/.local/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
220 | return self._emit(event_name, kwargs)
221 | File "/root/.local/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
222 | response = handler(**kwargs)
223 | File "/root/.local/lib/python3.10/site-packages/botocore/signers.py", line 105, in handler
224 | return self.sign(operation_name, request)
225 | File "/root/.local/lib/python3.10/site-packages/botocore/signers.py", line 189, in sign
226 | auth.add_auth(request)
227 | File "/root/.local/lib/python3.10/site-packages/botocore/auth.py", line 424, in add_auth
228 | canonical_request = self.canonical_request(request)
229 | File "/root/.local/lib/python3.10/site-packages/botocore/auth.py", line 365, in canonical_request
230 | cr.append(self.canonical_headers(headers_to_sign) + '\n')
231 | File "/root/.local/lib/python3.10/site-packages/botocore/auth.py", line 300, in canonical_headers
232 | value = ','.join(
233 | File "/root/.local/lib/python3.10/site-packages/botocore/auth.py", line 301, in <genexpr>
234 | self._header_value(v) for v in headers_to_sign.get_all(key)
235 | File "/root/.local/lib/python3.10/site-packages/botocore/auth.py", line 312, in _header_value
236 | return ' '.join(value.split())
237 | AttributeError: 'NoneType' object has no attribute 'split'

@santiagoprieto
Copy link

For future reference, the python3 version needs an update to add the endpoint_url.
For STS this would be 'https://sts.amazonaws.com'

request_dict = client._convert_to_request_dict({}, operation_model, endpoint_url)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment