Skip to content

Instantly share code, notes, and snippets.

@asgillmor
Created May 10, 2018 00:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save asgillmor/3b9a94db9f957b69a7a1215fcba4714a to your computer and use it in GitHub Desktop.
Save asgillmor/3b9a94db9f957b69a7a1215fcba4714a to your computer and use it in GitHub Desktop.
def main(sc, args):
# ...
# read input logs
business_rdd = sc.textFile(path_finder('yelp', 'business')).map(parse_line)
# ...
# ...
# apply logic
eligible_biz_list_rdd = get_eligible_business_list_rdd(business_rdd)
biz_id_to_locations_rdd = get_business_id_to_location(unique_biz_ids, business_rdd)
# ...
# ...
# write output
output_rdd.saveAstextFile(output_path)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment