Skip to content

Instantly share code, notes, and snippets.

@justinmoon

justinmoon/Discussion.md

Last active June 25, 2016 21:01
Show Gist options
  • Save justinmoon/7b9bcbfc8f338da414ae9678ec98016d to your computer and use it in GitHub Desktop.
Save justinmoon/7b9bcbfc8f338da414ae9678ec98016d to your computer and use it in GitHub Desktop.
Download ZIP
See Discussion.md
Raw
Discussion.md

Problem statement

  • When I attmept to exercise a certain API using the main.py script below, the server usually (but not always) sends a 400.
  • When I take a given url that generated a 400, and:
    • Paste it into Chrome, I get a 200!
    • Fire up python REPL and feed it to requests.get, I get a 200! I can do this repeatedly from "for loop" and always get 200's!
    • Open up Chrome debug tools and request if using fetch method, I get a 200!

Observations

  • This API is extremely powerful (contains almost all statistics in history of NBA) but entirely undocumented.
  • The 400's don't contain a response body -- see line 60 of main.py.
  • The main.py script is very flaky. I've seen it only generate 400's, only generate 200's, and generate a mix. Have yet to notice a pattern ...
  • I've attempted the same exact operation from node and gotten very similar results.

Potential Culprits

  • Ratelimit?
    • See the ratelimit.py repl output below ... I can hit the same URL one hundred times and only get 200s. So a ratelimit isn't always applied.
  • Some arcane string formatting that is fixed in the process of printing to STDOUT?
Raw
from_browser.js
// URL from line 181 of "main.py OUTPUT" above. It was a 400.
// If I run this from the chrome debug window's console, I get a 200
var url = 'http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612764'
fetch(url, {
method: 'get'
}).then(function(response) {
console.log(response)
}).catch(function(err) {
console.log(err)
});
Raw
from_py_repl.py
In [1]: import requests
In [2]: url = 'http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612764'
In [3]: requests.get(url)
Out[3]: <Response [200]>
Raw
main.py
import requests
from urllib.parse import urlencode
teams = [{'team': 'Atlanta Hawks', 'teamId': '1610612737'},
{'team': 'Boston Celtics', 'teamId': '1610612738'},
{'team': 'Brooklyn Nets', 'teamId': '1610612751'},
{'team': 'Charlotte Hornets', 'teamId': '1610612766'},
{'team': 'Chicago Bulls', 'teamId': '1610612741'},
{'team': 'Cleveland Cavaliers', 'teamId': '1610612739'},
{'team': 'Dallas Mavericks', 'teamId': '1610612742'},
{'team': 'Denver Nuggets', 'teamId': '1610612743'},
{'team': 'Detroit Pistons', 'teamId': '1610612765'},
{'team': 'Golden State Warriors', 'teamId': '1610612744'},
{'team': 'Houston Rockets', 'teamId': '1610612745'},
{'team': 'Indiana Pacers', 'teamId': '1610612754'},
{'team': 'Los Angeles Clippers', 'teamId': '1610612746'},
{'team': 'Los Angeles Lakers', 'teamId': '1610612747'},
{'team': 'Memphis Grizzlies', 'teamId': '1610612763'},
{'team': 'Miami Heat', 'teamId': '1610612748'},
{'team': 'Milwaukee Bucks', 'teamId': '1610612749'},
{'team': 'Minnesota Timberwolves', 'teamId': '1610612750'},
{'team': 'New Orleans Pelicans', 'teamId': '1610612740'},
{'team': 'New York Knicks', 'teamId': '1610612752'},
{'team': 'Oklahoma City Thunder', 'teamId': '1610612760'},
{'team': 'Orlando Magic', 'teamId': '1610612753'},
{'team': 'Philadelphia 76ers', 'teamId': '1610612755'},
{'team': 'Phoenix Suns', 'teamId': '1610612756'},
{'team': 'Portland Trail Blazers', 'teamId': '1610612757'},
{'team': 'Sacramento Kings', 'teamId': '1610612758'},
{'team': 'San Antonio Spurs', 'teamId': '1610612759'},
{'team': 'Toronto Raptors', 'teamId': '1610612761'},
{'team': 'Utah Jazz', 'teamId': '1610612762'},
{'team': 'Washington Wizards', 'teamId': '1610612764'}]
def seasons():
season = 2016
while season > 2013:
yield '{first}-{second}'.format(first=season, second=str(season-1)[-2:])
season -= 1
def team_ids():
return [team['teamId'] for team in teams]
def team_game_log_url(team_id, season_type, season):
query = urlencode({'TeamID': team_id, 'SeasonType': season_type, 'Season': season})
return 'http://stats.nba.com/stats/teamgamelog?{}'.format(query)
def urls():
return [
team_game_log_url(team_id, 'Regular Season', season)
for season in seasons()
for team_id in team_ids()
]
def sync():
all_urls = urls()
for url in all_urls:
res = requests.get(url)
print(url)
print(res.text() or res.status_code)
if __name__ == '__main__':
sync()
Raw
main.py OUTPUT
$ python req.py
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612737
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612738
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612751
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612766
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612741
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612739
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612742
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612743
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612765
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612744
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612745
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612754
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612746
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612747
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612763
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612748
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612749
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612750
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612740
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612752
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612760
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612753
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612755
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612756
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612757
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612758
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612759
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612761
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612762
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2016-15&SeasonType=Regular+Season&TeamID=1610612764
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612737
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612738
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612751
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612766
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612741
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612739
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612742
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612743
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612765
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612744
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612745
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612754
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612746
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612747
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612763
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612748
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612749
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612750
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612740
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612752
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612760
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612753
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612755
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612756
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612757
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612758
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612759
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612761
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612762
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2015-14&SeasonType=Regular+Season&TeamID=1610612764
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612737
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612738
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612751
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612766
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612741
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612739
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612742
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612743
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612765
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612744
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612745
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612754
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612746
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612747
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612763
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612748
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612749
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612750
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612740
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612752
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612760
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612753
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612755
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612756
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612757
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612758
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612759
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612761
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612762
<Response [400]>
http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612764
<Response [400]>
Raw
ratelimit.py
In [1]: import requests
In [2]: url = 'http://stats.nba.com/stats/teamgamelog?Season=2014-13&SeasonType=Regular+Season&TeamID=1610612764'
In [3]: for i in range(100): print(requests.get(url))
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment