Skip to content

Instantly share code, notes, and snippets.

View HyukjinKwon's full-sized avatar

Hyukjin Kwon HyukjinKwon

558 followers · 25 following
View GitHub Profile
@HyukjinKwon
HyukjinKwon / repartition-reproduce.scala
Last active March 25, 2016 15:39
[Spark][Scala] repartition issue reproduce
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
case class Custom(a: Int, b: String)
// Set parallelism level as 2
val conf = new SparkConf().setAppName("RepartitionIssue").setMaster("local[2]")
val sc = new SparkContext(conf)
val data = Seq(Custom(1, "a"), Custom(2, "b"))
@HyukjinKwon
HyukjinKwon / output.err
Last active May 20, 2016 04:22
[SPARK][R] test output (stderr) on Windows 7 32bit
[Stage 0:> (0 + 0) / 4]
[Stage 1:=============================> (2 + 2) / 4]
[Stage 1:============================================> (3 + 1) / 4]
[Stage 2:> (0 + 2) / 4]
[Stage 2:==============> (1 + 2) / 4]
[Stage 2:=============================> (2 + 2) / 4]
@HyukjinKwon
HyukjinKwon / output.msg
Created May 20, 2016 04:22
[SPARK][R] test output (stdout) on Windows 7 32bit
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
The following objects are masked from 'package:stats':
@HyukjinKwon
HyukjinKwon / output01.msg
Created May 25, 2016 13:17
[SPARK][R] test output (stdout) on Windows 7 32bit (fixed)
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
The following objects are masked from 'package:stats':
@HyukjinKwon
HyukjinKwon / output02.msg
Created May 26, 2016 03:19
[SPARK][R] test output (stdout) on Windows 7 32bit (fixed_2)
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
The following objects are masked from 'package:stats':
@HyukjinKwon
HyukjinKwon / output03.msg
Created May 26, 2016 08:02
[SPARK][R] test output (stdout) on Windows 7 32bit (fixed_3)
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
The following objects are masked from 'package:stats':
@HyukjinKwon
HyukjinKwon / output04.msg
Created August 22, 2016 11:55
[SPARK][R] test output (stdout) on Windows 7 32bit
Warning message:
package 'testthat' was built under R version 3.3.1
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
@HyukjinKwon
HyukjinKwon / output05.msg
Created August 22, 2016 12:59
[SPARK][R] test output (stdout) on Windows 7 32bit
Warning message:
package 'testthat' was built under R version 3.3.1
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
@HyukjinKwon
HyukjinKwon / output.msg
Last active September 5, 2016 13:52
[SPARK][R] test output (AppVeyor) https://ci.appveyor.com/project/HyukjinKwon/spark/build/66-SPARK-17339-test
[00:42:08] Failed -------------------------------------------------------------------------
[00:42:08] 1. Error: spark.glm save/load (@test_mllib.R#168) ------------------------------
[00:42:08] java.io.IOException: No FileSystem for scheme: C
[00:42:08]
[00:42:08] at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
[00:42:08]
[00:42:08] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
[00:42:08]
[00:42:08] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
[00:42:08]
@HyukjinKwon
HyukjinKwon / output.msg
Created September 5, 2016 13:43
[SPARK][R] test output https://ci.appveyor.com/project/HyukjinKwon/spark/build/69-SPARK-17339-after
[00:41:11] Failed -------------------------------------------------------------------------
[00:41:11] 1. Error: spark.lda with text input (@test_mllib.R#655) ------------------------
[00:41:11] org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:/projects/spark/R/lib/SparkR/tests/testthat/data/mllib/sample_lda_data.txt;
[00:41:11]
[00:41:11] at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:376)
[00:41:11]
[00:41:11] at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:365)
[00:41:11]
[00:41:11] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
[00:41:11]