A list of commonly used words as a broadcast variable

TextAnalyser.scala

class TextAnalyser(val sc: SparkContext, ...) {
  ...
  // Instance variable
  val _commonWords = sc.broadcast(TextAnalyser.loadCommonWords())
  ...
  // In a worker thread
  def analyse(rdd: RDD[String]): TextStats = {
    // To prevent the whole class to be 'sucked' into serialization
    val commonWords = _commonWords
    ...
    // Like accumulators a Broadcast is a wrapper, the 'value' method provides access to the actual data.
    .filter(!commonWords.value.contains(_))  // Filter out all too common words
    ...