Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
A groovyfied version of GroovySparkPi
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.examples
import groovy.transform.CompileStatic
import groovy.util.logging.Log
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.api.java.function.Function
import org.apache.spark.api.java.function.Function2
/**
* Computes an approximation to pi
* Usage: GroovySparkPi [slices]
*/
@CompileStatic
final class GroovySparkPi {
static void main(String[] args) throws Exception {
def sparkConf = new SparkConf().setAppName("GroovySparkPi")
def jsc = new JavaSparkContext(sparkConf)
int slices = (args.length == 1) ? Integer.parseInt(args[0]) : 2
int n = 100000 * slices
def dataSet = jsc.parallelize(0..<n, slices)
def mapper = {
double x = Math.random() * 2 - 1
double y = Math.random() * 2 - 1
return (x * x + y * y < 1) ? 1 : 0
}
int count = dataSet
.map(mapper as Function<Integer, Integer>)
.reduce({int a, int b -> a + b} as Function2<Integer, Integer, Integer>)
println "Pi is roughly ${4.0 * count / n}"
jsc.stop()
}
}
@pditommaso
Copy link

pditommaso commented Jun 15, 2016

OK. I've got the problem. The class must implement Serializable (obviously!). The other examples worked because it was just a static method (run).

AFAIK it is possible to specify the script superclass, but I don't remember how to do that.

@pditommaso
Copy link

pditommaso commented Jun 15, 2016

For the records. It's enough to create a superclass of Script implementing Serializable and use the BaseScript annotation. An example here:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment