Convert the Yelp Academic dataset from JSON to CSV files with Pandas.

convert.py

'''
Convert Yelp Academic Dataset from JSON to CSV

Requires Pandas (https://pypi.python.org/pypi/pandas)

By Paul Butler, No Rights Reserved
'''

import json
import pandas as pd
from glob import glob

def convert(x):
    ''' Convert a json string to a flat python dictionary
    which can be passed into Pandas. '''
    ob = json.loads(x)
    for k, v in ob.items():
        if isinstance(v, list):
            ob[k] = ','.join(v)
        elif isinstance(v, dict):
            for kk, vv in v.items():
                ob['%s_%s' % (k, kk)] = vv
            del ob[k]
    return ob

for json_filename in glob('*.json'):
    csv_filename = '%s.csv' % json_filename[:-5]
    print 'Converting %s to %s' % (json_filename, csv_filename)
    df = pd.DataFrame([convert(line) for line in file(json_filename)])
    df.to_csv(csv_filename, encoding='utf-8', index=False)

Getting an error for Sep 2017

File "F:/Yelp/yelp_dataset/yelp_dataset/jsontocsv_business.py", line 10, in convert
for k, v in ob.items():

RuntimeError: dictionary changed size during iteration

I also got "RuntimeError: dictionary changed size during iteration", while using the above code to open the Round 11 business.json file. For the review.json file, I didn't get the error message but the code ran for almost an hour and nothing happened. Finally, I had to terminate the execution. Any advice to get around these issues is highly appreciated.

I wrote another one that works with the 2018 version of the dataset. In theory it should work with any arbitrary dataset as long as they're structured as one json object per line:
https://gist.github.com/emredjan/6fffc4f696d2201d1e3697b783f9590b

Give it the directory where the json files reside from the command line, it should do the trick.