Skip to content

Instantly share code, notes, and snippets.

@paulgb
Last active January 31, 2020 22:40
Show Gist options
  • Save paulgb/5265767 to your computer and use it in GitHub Desktop.
Save paulgb/5265767 to your computer and use it in GitHub Desktop.
Convert the Yelp Academic dataset from JSON to CSV files with Pandas.
'''
Convert Yelp Academic Dataset from JSON to CSV
Requires Pandas (https://pypi.python.org/pypi/pandas)
By Paul Butler, No Rights Reserved
'''
import json
import pandas as pd
from glob import glob
def convert(x):
''' Convert a json string to a flat python dictionary
which can be passed into Pandas. '''
ob = json.loads(x)
for k, v in ob.items():
if isinstance(v, list):
ob[k] = ','.join(v)
elif isinstance(v, dict):
for kk, vv in v.items():
ob['%s_%s' % (k, kk)] = vv
del ob[k]
return ob
for json_filename in glob('*.json'):
csv_filename = '%s.csv' % json_filename[:-5]
print 'Converting %s to %s' % (json_filename, csv_filename)
df = pd.DataFrame([convert(line) for line in file(json_filename)])
df.to_csv(csv_filename, encoding='utf-8', index=False)
@emredjan
Copy link

emredjan commented Sep 5, 2018

I wrote another one that works with the 2018 version of the dataset. In theory it should work with any arbitrary dataset as long as they're structured as one json object per line:
https://gist.github.com/emredjan/6fffc4f696d2201d1e3697b783f9590b

Give it the directory where the json files reside from the command line, it should do the trick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment