Skip to content

Instantly share code, notes, and snippets.

@mitliagkas

mitliagkas/pyspark-vector-accumulator.py

Created August 6, 2014 05:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mitliagkas/a7654843efeb379763f8 to your computer and use it in GitHub Desktop.
Save mitliagkas/a7654843efeb379763f8 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark Vector Accumulator Class
Raw
pyspark-vector-accumulator.py
# Taken almost verbatim from PySpark's doctest
from pyspark.accumulators import AccumulatorParam
class VectorAccumulatorParam(AccumulatorParam):
def zero(self, value):
return [0.0] * len(value)
def addInPlace(self, val1, val2):
for i in xrange(len(val1)):
val1[i] += val2[i]
return val1
xnew = sc.accumulator([0.0]*p, VectorAccumulatorParam())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment