This gist is created to store the atom configuration:
Rafael Castillo kingkastle
kingkastle
/ pandas_to_spark.py
Created
January 28, 2020 13:47
— forked from zaloogarcia/pandas_to_spark.py
Script for converting Pandas DF to Spark's DF
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pyspark.sql.types import * | |
| # Auxiliar functions | |
| # Pandas Types -> Sparks Types | |
| def equivalent_type(f): | |
| if f == 'datetime64[ns]': return DateType() | |
| elif f == 'int64': return LongType() | |
| elif f == 'int32': return IntegerType() | |
| elif f == 'float64': return FloatType() | |
| else: return StringType() |
kingkastle
/ README.md
Last active
October 3, 2017 12:02
automatic update by http://atom.io/packages/sync-settings
kingkastle
/ README.md
Last active
August 29, 2015 14:17
This project is part of the Udacity Data Analyst Nanodegree Project 5, with the aim of sharing knowledge with others. This is a dimplejs example based on the flight data from RITA that you can find at:
http://stat-computing.org/dataexpo/2009/the-data.html
The graph represents the evolution in terms of number of flights per year versus total distance accumulated per airline per year for the period between 1988 to 2008.
Check it out in b.locks: http://bl.ocks.org/kingkastle/raw/7188565a417968331fda/