canimus/sparker.py

Created March 20, 2020 18:55

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/canimus/824b7e3ed11655373a34e8f853c5cfbb.js"></script>
Save canimus/824b7e3ed11655373a34e8f853c5cfbb to your computer and use it in GitHub Desktop.

Download ZIP

Spark custom separator reader

Raw

sc.binaryFiles(new_file).values().flatMap(lambda x: x.decode("iso-8859-1").split(chr(172))).map(lambda x: x.split(chr(171)))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment