Skip to content

Instantly share code, notes, and snippets.

@rwdaigle
Forked from mrezentes/SFTP-batch-export.md
Last active July 18, 2016 17:33
Show Gist options
  • Save rwdaigle/20b4aa51025dbad665b7ffb9870843f7 to your computer and use it in GitHub Desktop.
Save rwdaigle/20b4aa51025dbad665b7ffb9870843f7 to your computer and use it in GitHub Desktop.
SFTP Batch Export

Export multiple payment methods in a single, asynchronous, batch call to an SFTP receiver endpoint.

Because the Visa file format is quite unwieldy, and requires encryption, we'll approach this in two phases. First we'll work on getting a test file exported w/ test payment methods and no encryption so you can confirm that your file template is working. Then, we'll add the gpg encryption required by Visa and have it use the production receiver.

1. Create test receiver

Export is part of Spreedly's PMD/receiver functionality, which allows you to send card data to non-gateway APIs. As such, export shares much of its terminology and functionality with deliver, the single-card and synchronous version of export.

First, create a test receiver that will let you specify any hostname (add the sftp auth in the protocol section):

curl https://core.spreedly.com/v1/receivers.json \
  -u 'Ll6fAtoVSTyVMlJEmtpoJV8S:RKOCG5D8D3fZxDSg500IxU2XD4Io5VXmyzdCtTivHFTTSy' \
  -H 'Content-Type: application/json' \
  -d '{
        "receiver": {
          "receiver_type": "test",
          "hostnames": "sftp.cashbackpoint.com",
          "protocol": {"user": "test_user", "password": "test_password"}
        }
      }'

You'll get a receiver record returned, whose token you'll need for the next step.

2. Test export

In this step you'll attempt to export to the receiver created above, using only test payment methods (so be sure to create and retain a few of them in your test environment).

All an export does is add in the PAN data according to the template you provide. More details about how to construct a template can be found in our receiver documentation. Export uses the same approach as deliver, so the documentation for the latter will help during this beta period. You can also refer to our API documentation for export.

Export details

  • If you look in the body element, it contains a template function payment_methods. This is an iterator that exposes each of the payment methods you've asked to be exported. Within the payment_methods block of the template, you can use all of the card variables available in a normal receiver template to access specific card fields (like the number, expiration date etc...).
  • The payment_method_data elements are available for you to pass in metadata to be associated with each payment method that's exported. Since Visa requires a 25-character identifier for each card you can't just use the Spreedly payment method token and will have to maintain a mapping on your side.
  • Since Visa's spec requires the file to have fixed width columns, this makes constructing a template pretty unwieldy. We acknowledge this. If you see opportunities for us to ease this burden for you, let us know (specifically thinking about any template functions we can add to help construct certain fields etc...)
  • We do have a 4kb request limit on all requests. I'm not sure how many payment methods this means for this type of export request, but let us know when/if your requests get rejected and we can see if it's reasonable to adjust this limit.
  • When you submit the export request, it will queue the export job and immediately return. The state of the export transaction record will be pending, indicating that there's work left to be done. You can poll the show transaction endpoint to get the updated tx status (looking for succeeded or failed), or you can specify a callback_url in your export request. When the transaction has completed, Spreedly will post the transaction record to this URL and you can update your system accordingly.

Export request

Here is how an export request looks for Visa's fixed-width file format. Use the receiver token from step 1 in the URL.

We've purposely excluded encryption right now since I want you to be able to see what the resulting file looks like when it ends up on your server.

curl https://core.spreedly.com/v1/receivers/QvCtF4nD6tav60DcztgorcPnTLO/export.json \
  -u 'Ll6fAtoVSTyVMlJEmtpoJV8S:RKOCG5D8D3fZxDSg500IxU2XD4Io5VXmyzdCtTivHFTTSy' \
  -H 'Content-Type: application/json' \
  -d '{
        "export": {
          "payment_method_tokens": ["FJSPfoAM9aavIkdtTZnGKff52l5", "Yj9FPVz1Nz87mDCspfcofBYu5Fy", "SGLYl7swRHjwA5Z3JD5UuBL4ayF"],
          "payment_method_data": {
            "FJSPfoAM9aavIkdtTZnGKff52l5": {"external_cardholder_id": "1111111111111111111111111", "action_code":"A" }, 
            "Yj9FPVz1Nz87mDCspfcofBYu5Fy": {"external_cardholder_id": "2222222222222222222222222", "action_code":"D"},
            "SGLYl7swRHjwA5Z3JD5UuBL4ayF": {"external_cardholder_id": "3333333333333333333333333", "action_code":"U"}
          },
          "callback_url": "https://cashbackpoint.com/api/transaction_callback",
          "url": "sftp://posttestserver.com/filename.txt",
          "body": "0000VISA  VISA  Card Account List Sample File                                                                                                                                                                                                                                  20140101002.0 P2014010120510114VDACCTLISTSAMPLE           I                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              \n{{#payment_methods}}1001VDACCTLISTSAMPLE           {{action_code}}VISA  VISA  {{credit_card_number}}   {{external_cardholder_id}}{{#format_date}}%Y%m%d, {{credit_card_created_at}}{{/format_date}}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n{{/payment_methods}}99990000000003                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          "
        }
      }'

It might be a good idea to use a very simple body template to start with, just to make sure the call and export to the SFTP endpoint work as expected. Perhaps start with something that looks like this?

"body": "Test request"

After we have a successful round trip, then we can start adding in the bits to the body template that will eventually form the file as Visa expects.

3. Encrypt export

Visa requires that the file be encrypted using GnuPGP and their public key. We've exposed a new gpg request function you can use to encrypt some block of text. It takes the public key in PEM form as the first argument, the target identity (Visa) as the second, and the content to encrypt as the third. So the function looks like this in the body template:

{{#gpg}}full pem,recipient identity,file template to encrypt{{/gpg}}

Here is Visa's public key PEM:

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: IPWorks! OpenPGP v9.0
xsBNBFdaDS0BCACPTH1Td1PSUSMeaTDAagncwV18KrpuXSPXWjTgPy9SLHeoKQDt
wxWlIoisyxV1Ex2LQZnidINgNCFzMi26+SqYucm6OFv2bllr91tpk8I0+aeL/XBC
J5DUEbG9JAMyegMyzTz9PjRu4peXV/IUf2/uBJifZyv1bBaARCBXBaHvv+qfJbHx
88QNVo5J7KU8C7MD8hqLxwtqDjHgKtXHGbyscMzJn+ySTueemqhOBI3jst/z9uL2
OuSXeO0DudcLsmp6bVrh3SqpLKiZMbj2GsNcwVA/ikJiriaXOESv2RI/h1j5MjRs
sg9tyJYytuDsqz/rEOVDnqfP5/xpiTX223tjABEBAAHNJHNmaWxlMi1sYS10YXFj
LVBST0QgPG5hd2VzQHZpc2EuY29tPsLAdAQTAQIAHgUCV1oNLwMLAgIEFQoIAgUW
AQIDAAIeAQIXgAIbDwAKCRBPCXevH0QlBoxhB/4o7sG1TOZlxjv9fuK1/Bx9ZjPJ
6zCWGEAMV3li/jDmiNfJUaJOL5ZV8ffLRhgvS6bvKlpdMaRY8FXuJQThGe4T/BU4
HJIP8jPR+x4CHoUHNw1rOVitdkc9y/tWoF7aYAWqcqBBQwqjH9XbSC2XcYbULcl8
j8rVVTbXvJIdnx7u6v9OOeyc6XO7AupV7zjQHE6bdDPnmhyM9Yf+1OkDxuGNywsv
0WP8iB847/ZPmaENvOofIsrbncbztgZu1V2fOJM6JRp0EbctSxP44Mk1K8AKcmx2
MFqkPdKpeZh2bO269TO8fMy82gx6ltzMtms2NrRL3NOWj6suLke7s8K8++JC
=Zp6G
-----END PGP PUBLIC KEY BLOCK-----

So to wrap your body template so it's encrypted, it would look like this:

"body": "{{#gpg}}-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: IPWorks! OpenPGP v9.0\nxsBNBFdaDS0BCACPTH1Td1PSUSMeaTDAagncwV18KrpuXSPXWjTgPy9SLHeoKQDt\nwxWlIoisyxV1Ex2LQZnidINgNCFzMi26+SqYucm6OFv2bllr91tpk8I0+aeL/XBC\nJ5DUEbG9JAMyegMyzTz9PjRu4peXV/IUf2/uBJifZyv1bBaARCBXBaHvv+qfJbHx\n88QNVo5J7KU8C7MD8hqLxwtqDjHgKtXHGbyscMzJn+ySTueemqhOBI3jst/z9uL2\nOuSXeO0DudcLsmp6bVrh3SqpLKiZMbj2GsNcwVA/ikJiriaXOESv2RI/h1j5MjRs\nsg9tyJYytuDsqz/rEOVDnqfP5/xpiTX223tjABEBAAHNJHNmaWxlMi1sYS10YXFj\nLVBST0QgPG5hd2VzQHZpc2EuY29tPsLAdAQTAQIAHgUCV1oNLwMLAgIEFQoIAgUW\nAQIDAAIeAQIXgAIbDwAKCRBPCXevH0QlBoxhB/4o7sG1TOZlxjv9fuK1/Bx9ZjPJ\n6zCWGEAMV3li/jDmiNfJUaJOL5ZV8ffLRhgvS6bvKlpdMaRY8FXuJQThGe4T/BU4\nHJIP8jPR+x4CHoUHNw1rOVitdkc9y/tWoF7aYAWqcqBBQwqjH9XbSC2XcYbULcl8\nj8rVVTbXvJIdnx7u6v9OOeyc6XO7AupV7zjQHE6bdDPnmhyM9Yf+1OkDxuGNywsv\n0WP8iB847/ZPmaENvOofIsrbncbztgZu1V2fOJM6JRp0EbctSxP44Mk1K8AKcmx2\nMFqkPdKpeZh2bO269TO8fMy82gx6ltzMtms2NrRL3NOWj6suLke7s8K8++JC\n=Zp6G\n-----END PGP PUBLIC KEY BLOCK-----,nawes@visa.com,{{#format_text}}%-1000.1000s,0000VISA  VISA  {{#format_text}}%-255.255s,Card Account List Sample File{{/format_text}}20140101002.0 P2014010120510114VDACCTLISTSAMPLE{{/format_text}}\n{{#payment_methods}}{{#format_text}}%-1000.1000s,1001VDACCTLISTSAMPLE           {{action_code}}VISA  VISA  {{#format_text}}%-19.19s,{{credit_card_number}}{{/format_text}}{{external_cardholder_id}}{{#format_date}}%Y%m%d, {{credit_card_created_at}}{{/format_date}}{{/format_text}}\n{{/payment_methods}}{{#format_text}}%-1000.1000s,99990000000003{{/format_text}}{{/gpg}}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment