Skip to content

Instantly share code, notes, and snippets.

@drbh
Created February 22, 2021 16:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save drbh/4d0e69dee81e8e1ca73fd5644960322a to your computer and use it in GitHub Desktop.
Save drbh/4d0e69dee81e8e1ca73fd5644960322a to your computer and use it in GitHub Desktop.
fetch all emsi skills with pandas and requests in Python3
import pandas as pd
import requests
CLIENT_ID = "<CLIENT_ID>"
CLIENT_SECRET = "<CLIENT_SECRET>"
CLIENT_SCOPE = "emsi_open"
VERSION = "latest"
url = "https://auth.emsicloud.com/connect/token"
payload = f"client_id={CLIENT_ID}&client_secret={CLIENT_SECRET}&grant_type=client_credentials&scope={CLIENT_SCOPE}"
headers = {"content-type": "application/x-www-form-urlencoded"}
response = requests.request("POST", url, data=payload, headers=headers)
data = response.json()
token = data.get("access_token")
url = f"https://emsiservices.com/skills/versions/{VERSION}/skills"
headers = {"authorization": f"Bearer {token}"}
response = requests.request("GET", url, headers=headers)
skill_response = response.json()
df = pd.DataFrame.from_dict(skill_response.get("data"))
df2 = pd.json_normalize(df["type"])
result = pd.concat([df, df2], axis=1)
del result["type"]
result.to_csv("skills.csv", index=None)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment