Access to Ameritrade API
notes that may be helpful for node.js devs
What I've gathered is authorization tokens must be earned first. Authorization tokens are good for three months, once you are setup, you will get two tokens which come as long strings, An access_token and a refresh_token. Access tokens are only valid for 30 minutes so you will need to code to refresh the session using the refresh token frequently. When you refresh, you are given a new access token and that is what you use to "login" or pull from the api for the next period. But in order to get the tokens, you will be required to go through some processes to get a valid token that applies to your app. The following text kind of steers you in that direction...
Making an app
On the api site you will need to make an app, where you pick some random name by using some random numbers. It should be automatically generated but it isnt. For this field item, I simply went to https://www.uuidgenerator.net/ and just grabbed the first 13 digits of a UUID. You won't be able to use the same one twice, so just change a letter or something if you end up deleting the app and starting over. Note that this will be appended by ameritrade with ...@AMER.OAUTHAP so in the future you are going to want to make sure all of that is in the strings you send to the server.
The next thing you will want to do is to get an authorization token, which is going to allow you to login. They only last for thirty minutes so keep that in mind as you are twitching around. In order to get one, you need to get an access code off the ameritrade site which has some quirks to it...
Getting an access code (needs to recur every three months)
First there is a good sleu of information at the following python repo from user @jeog : https://github.com/jeog/TDAmeritradeAPI (scroll to "authentication")
There, you will find the lazy man's way to get this up and running fast, and it is in the name of: https://github.com/jeog/TDAmeritradeAPI/blob/master/tools/get-access-code.html ...Which works famously. Make sure you download it and save it for later. All other attempts without the influence of this html file will burn in twitch hell. Run the html file from your machine after checking the source code for potential security issues like links for phishing.
According to @jeog's docs, the steps are:
- Get an access code. (once)
- Use the access code to get a Credentials object containing access and refresh tokens. (once)
- Pass the Credentials object to the various library calls to get data, access your account etc. - the library will refresh the access token for you when it expires.
My experience was that you load that html file, and you pass your app name, an ameritrade site loads where you log in, and then you are redirected to what looks like a bad page but what you are to do is to copy the entire URL that shows up in the popped up browser window, paste it back into the html page, where it gives you a URI encoded version that you can use to submit on the Ameritrade api site, in order to get your actual access and refresh tokens. (thats a breath)
Getting access and refresh tokens:
https://developer.tdameritrade.com/authentication/apis/post/token-0
The page you are led to is basically a tester page for api calls to make sure your syntax is tight. For the most part follow the instruction son the form. Goal is to get a nice refresh token along with your access token.
here are the fields:
grant_type: authorization_code
refresh_token: leave blank
access_type: offline
code: what you copied from previous HTML file cluster
client_id: <app name you chose earlier>@AMER.OAUTHAP
redirect_uri: https:127.0.0.1
hit send. Note that you will use this form later to test your reset token.
Note that none of this is actually documented on the ameritrade api site... ...losers.
Do yourself a favor and copy the Response text along with the cURL text so you can use the data it gives you and also see what it does. But you basically just received an access_token that is only good for thirty minutes along with a refresh token that you will use in ahalf hour to renew your session. WHILE WE ARE HERE, lets see how to renew the token as well...
Renewing your access token:
SAME FORM https://developer.tdameritrade.com/authentication/apis/post/token-0
grant_type: refresh_token
refresh_token: paste in the refresh token given to you in the previous steps.
access_type: leave blank
code: leave blank
client_id: <app name you chose earlier>@AMER.OAUTHAP
redirect_uri: leave blank
Set permissions for your app
Now that you have gotten your tokens working let's give the app some permission to do something:
https://developer.tdameritrade.com/price-history/apis
click on get price history.
The window is a similar form as compared to the last one. Hell it's sooo bootstrap they all look the same.
Scroll down to the bottom where it says: "Try it out !!" Beneath you will see a little box that says OAuth 2.0 with a link that says "set..." click it and request credentials. Note the ambiguous "Remember credentials for 30 days." What happens if I don't check the box? I have no idea. Once it's authenticated you can play with the various knobs and try to get it to work. Notice here you need to use "Bearer sdojsodufghwoghf" in order to get your token to work, AND it needs to be within thirty minutes of when you made it.
I'd rather see it work in node, but I will admit I spent a good deal of time here scratching my head because of the utter lack of documentation on the site. Watching at the cURL requests as I was trying to figure this out revealed a bug that appears to save your last 30 minute authentication token and submits them both via cURL.
Note that there is some weird syntax in the Authorization block for the request, you must use "Bearer oefngoefgoeyouraccesstokenhere"
Enter Node.js
Caveat: I assume you know a thing or two about node. Probably more than I.
For now, we are just going to manually use the form on the api page to refresh the token, after all we just want to see it work. Later we will automate the process...
Also because curl node package is shit, build it from source
npm install node-libcurl --build-from-source
const curl = new (require( 'curl-request' ))();
const code = "Authorization: Bearer 7odfgDhuLGvyCJEm"; // where text jambalaya is a few hundred characters...
curl.setHeaders([code])
.get('https://api.tdameritrade.com/v1/marketdata/NVDA/pricehistory?&periodType=day&period=1&frequencyType=minute&frequency=1')
.then(({statusCode, body, headers}) => {
console.log(statusCode, body, headers)
})
.catch((e) => {
console.log(e);
});
First try the above code from your computer just to know that it sort of works outside of your world...
A couple pitfalls to notice:
-
Notice that this works because it's a GET request, so keep that in mind.
-
Also, the URI endcoding will doink you in the patootie over and over again. Make sure you know when to (or not to) use it. Will try and annotate proper formats...
Node with request.js
Next is to use the request.js package
const request = require('request');
const headers = {
'Authorization': 'Bearer 7odfgDhuLGvyCJEm'
};
const options = {
headers: headers,
url: 'https://api.tdameritrade.com/v1/marketdata/NVDA/pricehistory?&periodType=day&period=1&frequencyType=minute&frequency=1',
method: 'GET',
};
request(options, function(error, response, body) {
if (response.statusCode === 200){
console.log("theBODY: " + body); // contains candles
}
else {
switch (response.statusCode){
case 400:
console.log('400 Validation problem with the request.');
break;
case 401:
console.log('401 Caller must pass valid credentials in the request body. hint: refresh token');
break;
case 500:
console.log('500 There was an unexpected server error.');
break;
case 403:
console.log('403 Caller doesn\'t have access to the account in the request.');
break;
case 503:
console.log('503 Temporary problem responding.');
break;
default:
console.log('000 Something ain\'t right...');
break;
}
}
Updating the access token after session timeout
Here is a "hello world" version that auto updates the access token when it expires. I haven't tested it in action yet but should be enough to give the gist of what's going on.
const fs = require('fs');
const request = require('request');
const keys = require('../keys/keys');
validatetoken(function(success){
if (success){
const headers = getAuthorizationHeader();
const options = {
headers: headers,
url: 'https://api.tdameritrade.com/v1/marketdata/NVDA/pricehistory?&periodType=day&period=1&frequencyType=minute&frequency=1',
method: 'GET',
};
request(options, function(error, response, body) {
if (response.statusCode === 200){
console.log(body); // contains candles
}
else {
switch (response.statusCode){
case 400:
console.log('400 Validation problem with the request.');
break;
case 401:
console.log('401 hint: refresh token');
refreshToken();
break;
case 500:
console.log('500 There was an unexpected server error.');
break;
case 403:
console.log('403 Caller doesn\'t have access to the account in the request.');
break;
case 503:
console.log('503 Temporary problem responding.');
break;
default:
console.log('000 Something ain\'t right...');
break;
}
}
});
}
});
/*
* determine that existing access_token is valid or not, expires after 30 minutes
* returns true if token is valid, auto refreshes token when expired.
*/
function validatetoken(callback){
// 1. get access token and determine if expired or not.
// read json file
const json = fs.readFileSync('../keys/access_token.json');
const tokenInfo = JSON.parse(json);
// format expires and now to be standard length (in milliseconds)
let expires = +tokenInfo.created_on + (+tokenInfo.expires_in * 1000);
const le = String(expires).length;
let now = Date.now();
const ln = String(now).length;
let diff = ln - le;
if (diff > 0) {
// add diff zeroes to end of string
for (let i = 0; i < diff; i++) {
expires *= 10;
}
}
else if (diff < 0) { // probably unnecessary
// add diff zeroes to end of string
for (let i = 0; i < diff; i++) {
now *= 10;
}
}
if (expires > now) {
// token is still valid
callback(true);
}
else {
// token is invalid, and time to refresh it...
refreshToken(callback);
}
}
function refreshToken(callback) {
console.log("Token appears to be expired... Refreshing");
// 1. get refresh token and determine if expired or not.
// read json file
const json = fs.readFileSync('../keys/refresh_token.json');
const tokenInfo = JSON.parse(json);
// format expires and now to be standard length
let expires = +tokenInfo.created_on + +tokenInfo.refresh_token_expires_in;
const le = String(expires).length;
let now = Date.now();
const ln = String(now).length;
let diff = ln - le;
if (diff > 0) {
// add diff zeroes to end of string
for (let i = 0; i < diff; i++) {
expires *= 10;
}
}
else if (diff < 0) { // probably unnecessary
// add diff zeroes to end of string
for (let i = 0; i < diff; i++) {
now *= 10;
}
}
if (expires > now) {
// 2. request refreshed token
const headers = {
'Content-Type': 'application/x-www-form-urlencoded'
};
const options = {
url: 'https://api.tdameritrade.com/v1/oauth2/token',
method: 'POST',
headers: headers,
form: {
'grant_type': 'refresh_token',
'refresh_token': tokenInfo.refresh_token,
'client_id': keys.client_id,
}
};
request(options, function (error, response, body) {
// 3. now that you have the access token, store it so it persists over multiple instances of the script.
let data = JSON.parse(body);
data.created_on = Date.now();
fs.writeFile("../keys/access_token.json", JSON.stringify(data), function (err) {
if (err) {
return console.log(err);
}
console.log("Access Token updated. Expires in " + data.expires_in + " seconds");
callback(true);
});
});
}
else {
console.log("refresh_token is expired, need to reauthenticate manually.");
callback(false);
}
}
function getAuthorizationHeader(){
const json = fs.readFileSync('../keys/access_token.json');
const tokenInfo = JSON.parse(json);
return {
'Authorization': 'Bearer ' + tokenInfo.access_token
};
}
Parsing the candlestick Data
The api is sort of cryptic and things that should be written aren't. That said, I requested some details about accessing the candles on the API and the response was this:
Response is from Michael Ying, Ameritrade
The Date range from our Get Price History API is in UNIX time (aka EPOCH time). To get historical 30 minute aggregation for the last 6 months, you can use a request like this:
From there, you can choose to aggregate every 2x 30 minute bars to form a 1 hour candle. By design, the Get Price History API will provide the data for the most up to date candle if you specify a time in the future. Here are other examples of the date formatting: https://developer.tdameritrade.com/content/price-history-samples
Using that information I was able to convert the candles into what was required for my model to work:
- most recent candles
- minimum 200 points of data for interval selected.
In some cases, particularly hours, I was unable to just request "hour" candles. I was required to request half hour candles and then divide them by two. Note that if you remove every other candle your open, high and low candles must be averaged in order to get accurate wicks in your display.
Ameritrade provided examples of the date formatting: https://developer.tdameritrade.com/content/price-history-samples
Here is the formulae I am currently using:
// there are some other lines of code here that aren't used in the api calls, but are used in the modeling application I made. They seemed sore of "serving suggestion" relevant, so I left them in
renderChartWithFormulaMulti: function(ticker, formula, info, index, callback){
const oneDay = 86400000;
const oneHour = 3600000;
const oneMinute = 60000;
// code to calculate last candles
const now = new Date();
const future = now.getTime() + oneDay;
// minute candles for 2 days
const formula1minute = now.getTime() - (oneDay*2); // 601 candles
// notice that a request for minutes over three days appears to yield:
// (start to max candles allowed to return)
const formula5minute = now.getTime() - (oneDay*3); // may be buggy mid-day idk 991 / 5 = 198 candles
const formula10minute = now.getTime() - (oneDay*10);
const formula15minute = now.getTime() - (oneDay*30);
const formula1day = now.getTime() - (oneDay*300);
const formula1hour = now.getTime() - (oneHour*1500);
// candle formulae - ameritrade interface
const formulas =
{
// One Minute candles, basic
oneMinute: {
options: {
period: '',
periodType: 'day',
frequency: 1,
frequencyType: 'minute',
endDate: future,
startDate: formula1minute
},
divider:0,
label:"One Minute",
scale: "m"
},
fiveMinute: {
options: {
period: '',
periodType: 'day',
frequency: 1,
frequencyType: 'minute',
endDate: future,
startDate: formula5minute
},
divider:5,
label:"5 Minute / 3 Day",
scale: "m"
},
tenMinute: {
options: {
period: '',
periodType: 'day',
frequency: 10,
frequencyType: 'minute',
endDate: future,
startDate: formula10minute
},
divider:0,
label:"10 Minute / 4 Day",
scale: "m"
},
fifteenMinute: {
options: {
period: '',
periodType: 'day',
frequency: 15,
frequencyType: 'minute',
endDate: future,
startDate: formula15minute
},
divider:0,
label:"15 Minute / 5 Day",
scale: "m"
},
thirtyMinute: {
options: {
period: '',
periodType: 'day',
frequency: 30,
frequencyType: 'minute',
endDate: future,
startDate: formula1hour
},
divider:0,
label:"Thirty Minute / 2 Weeks",
scale: "m"
},
hour: {
options: {
period: '',
periodType: 'day',
frequency: 30,
frequencyType: 'minute',
endDate: future,
startDate: formula1hour
},
divider:2,
label:"Hour / 1 Month",
scale: "h"
},
day: {
options: {
period: '',
periodType: 'month',
frequency: 1,
frequencyType: 'daily',
endDate: future,
startDate: formula1day
},
divider:0,
label:"Day / 6 Months",
scale: "d"
},
week: {
options: {
period: '5',
periodType: 'year',
frequency: 1,
frequencyType: 'weekly',
endDate: '',
startDate: ''
},
divider:0,
label:"1 Week",
scale: "y"
},
month: {
options: {
period: '20',
periodType: 'year',
frequency: 1,
frequencyType: 'monthly',
endDate: '',
startDate: ''
},
divider:0,
label:"1 Month",
scale: "y"
}
};
let selectedFormula = '';
switch(formula){
case '1M':
selectedFormula = formulas.oneMinute;
break;
case '5M':
selectedFormula = formulas.fiveMinute;
break;
case '10M':
selectedFormula = formulas.tenMinute;
break;
case '15M':
selectedFormula = formulas.fifteenMinute;
break;
case '30M':
selectedFormula = formulas.thirtyMinute;
break;
case 'H':
selectedFormula = formulas.hour;
break;
case 'D':
selectedFormula = formulas.day;
break;
case 'W':
selectedFormula = formulas.week;
break;
case 'M':
selectedFormula = formulas.month;
break;
case 'E':
selectedFormula = formulas.exp;
break;
default:
selectedFormula = formulas.day;
}
// call upon the api for data
const coptions = {
ticker: ticker,
period: selectedFormula.options.period,
periodType: selectedFormula.options.periodType,
frequency: selectedFormula.options.frequency,
frequencyType: selectedFormula.options.frequencyType,
endDate: selectedFormula.options.endDate,
startDate: selectedFormula.options.startDate
};
// request the data
candleData.candles(coptions, function(data) {
... and do what you do with the response...
candles:function(options, cb){
auth.validate(function(success){
// todo: expand into additional options
let url = 'https://api.tdameritrade.com/v1/marketdata/' + options.ticker + '/pricehistory?';
url += "&periodType=" + options.periodType;
url += "&period=" + options.period;
url += "&frequencyType=" + options.frequencyType;
url += "&frequency=" + options.frequency;
url += "&endDate=" + options.endDate;
url += "&startDate=" + options.startDate;
url += "&needExtendedHoursData=false";
const headers = auth.authorizationHeader();
const candleOptions = {
headers: headers,
url: url,
method: 'GET',
};
request(candleOptions, function (error, response, body) {
This is an ongoing project and I will update it and eventually make some sort of bloggish thing with it...
- What's Next:
- update authorizations on website
- regenerate new access token after it expires in three months
somehow someway my access token doesn't last 30 minutes.
more like a grand total of 3 seconds. 🤪🤪🤪🤪🤪🤪
don't even know how i managed to achieve that