Skip to content

Instantly share code, notes, and snippets.

@lordpretzel

lordpretzel/>.txt

Created June 11, 2020 19:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lordpretzel/9b5780615338610f58b60aa7d228b43b to your computer and use it in GitHub Desktop.
Save lordpretzel/9b5780615338610f58b60aa7d228b43b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
>.txt
Running sbt
[info] Loading global plugins from /Users/lord_pretzel/.sbt/1.0/plugins
[info] Loading settings for project mimir-caveats-build from metals.sbt ...
[info] Loading project definition from /Users/lord_pretzel/Documents/workspace/mimir-caveats/project
[info] Loading settings for project mimir-caveats from build.sbt ...
[info] Set current project to mimir-caveats (in build file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/)
[info] sbt server started at local:///Users/lord_pretzel/.sbt/1.0/server/3b9af470c5023f52927d/sock
sbt:mimir-caveats> testOnly org.mimirdb.caveats.LogicalPlanRangeSpec -- ex "certain inputs.aggregation - no group-by - aggregtion functions only"
[info] LogicalPlanRangeSpec
[info] DataFrame Range Annotations
[info] Certain inputs
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/lord_pretzel/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.12/3.0.0-preview2/spark-unsafe_2.12-3.0.0-preview2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
REWRITING PLAN OPERATOR: Aggregate [avg(cast(A#14 as double)) AS X#824]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
REWRITING PLAN OPERATOR: RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
========================================
REWRITE OPERATOR TYPE LEAF NODE
========================================
--------------------------
REWRITTEN OPERATOR:
--------------------------
'Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, 'A AS __CAVEATS_A_LB#832, 'A AS __CAVEATS_A_UB#833, 'B AS __CAVEATS_B_LB#834, 'B AS __CAVEATS_B_UB#835, 'C AS __CAVEATS_C_LB#836, 'C AS __CAVEATS_C_UB#837]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
----------------------------------------
EXPR: avg(cast(A#14 as double)) AS X#824
GB: None
trace: true
----------------------------------------
EXPR: avg(cast(A#14 as double))
GB: None
trace: true
----------------------------------------
EXPR: sum(cast(A#14 as double))
GB: None
trace: true
===========> BG EQUALS: true
GROUP BY: None
----------------------------------------
EXPR: cast(A#14 as double)
GB: None
trace: true
----------------------------------------
EXPR: A#14
GB: None
trace: true
sum(CASE WHEN (`__CAVEATS_ROW_LB` > 0) THEN (CAST(`__CAVEATS_A_LB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_LB` AS DOUBLE) < 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END) ELSE least(0.0D, (CAST(`__CAVEATS_A_LB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_LB` AS DOUBLE) < 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END)) END)
sum(CASE WHEN true THEN (CAST(`A` AS DOUBLE) * `__CAVEATS_ROW_BG`) ELSE 0.0D END)
sum(CASE WHEN (`__CAVEATS_ROW_LB` > 0) THEN (CAST(`__CAVEATS_A_UB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_UB` AS DOUBLE) > 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END) ELSE greatest(0.0D, (CAST(`__CAVEATS_A_UB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_UB` AS DOUBLE) > 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END)) END)
----------------------------------------
EXPR: count(1)
GB: None
trace: true
===========> BG EQUALS: true
GROUP BY: None
--------------------------
REWRITTEN OPERATOR:
--------------------------
'Aggregate [CASE WHEN (sum('__CAVEATS_ROW_BG) = 0) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast('A as double) * '__CAVEATS_ROW_BG) ELSE 0.0 END) / cast(sum('__CAVEATS_ROW_BG) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE least(0.0, (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum('__CAVEATS_ROW_UB) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE greatest(0.0, (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum('__CAVEATS_ROW_UB) as double)) END AS __CAVEATS_X_UB#843]
+- 'Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, 'A AS __CAVEATS_A_LB#832, 'A AS __CAVEATS_A_UB#833, 'B AS __CAVEATS_B_LB#834, 'B AS __CAVEATS_B_UB#835, 'C AS __CAVEATS_C_LB#836, 'C AS __CAVEATS_C_UB#837]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
is already annotated? false
base schema: StructType(StructField(X,DoubleType,true))
row encoder StructType(StructField(X,DoubleType,true), StructField(__CAVEATS_ROW_LB,IntegerType,false), StructField(__CAVEATS_ROW_BG,IntegerType,false), StructField(__CAVEATS_ROW_UB,IntegerType,false), StructField(__CAVEATS_X_LB,DoubleType,true), StructField(__CAVEATS_X_UB,DoubleType,true))
================================================================================
FINAL
================================================================================
============================== QUERY EXECUTION (PLANS) ==============================
== Parsed Logical Plan ==
'Aggregate [CASE WHEN (sum('__CAVEATS_ROW_BG) = 0) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast('A as double) * '__CAVEATS_ROW_BG) ELSE 0.0 END) / cast(sum('__CAVEATS_ROW_BG) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE least(0.0, (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum('__CAVEATS_ROW_UB) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE greatest(0.0, (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum('__CAVEATS_ROW_UB) as double)) END AS __CAVEATS_X_UB#843]
+- 'Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, 'A AS __CAVEATS_A_LB#832, 'A AS __CAVEATS_A_UB#833, 'B AS __CAVEATS_B_LB#834, 'B AS __CAVEATS_B_UB#835, 'C AS __CAVEATS_C_LB#836, 'C AS __CAVEATS_C_UB#837]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
== Analyzed Logical Plan ==
X: double, __CAVEATS_ROW_LB: int, __CAVEATS_ROW_BG: int, __CAVEATS_ROW_UB: int, __CAVEATS_X_LB: double, __CAVEATS_X_UB: double
Aggregate [CASE WHEN (sum(cast(__CAVEATS_ROW_BG#830 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast(A#14 as double) * cast(__CAVEATS_ROW_BG#830 as double)) ELSE 0.0 END) / cast(sum(cast(__CAVEATS_ROW_BG#830 as bigint)) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE least(0.0, (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum(cast(__CAVEATS_ROW_UB#831 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE greatest(0.0, (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(__CAVEATS_ROW_UB#831 as bigint)) as double)) END AS __CAVEATS_X_UB#843]
+- Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833, B#15 AS __CAVEATS_B_LB#834, B#15 AS __CAVEATS_B_UB#835, C#16 AS __CAVEATS_C_LB#836, C#16 AS __CAVEATS_C_UB#837]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
== Optimized Logical Plan ==
Aggregate [CASE WHEN (sum(1) = 0) THEN 0.0 ELSE (sum((cast(A#14 as double) * 1.0)) / cast(sum(1) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(1) = 0) THEN 0.0 ELSE (sum((cast(__CAVEATS_A_LB#832 as double) * 1.0)) / cast(sum(1) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum(1) = 0) THEN 0.0 ELSE (sum((cast(__CAVEATS_A_UB#833 as double) * 1.0)) / cast(sum(1) as double)) END AS __CAVEATS_X_UB#843]
+- Project [A#14, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833]
+- RelationV2[A#14] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
== Physical Plan ==
*(1) HashAggregate(keys=[], functions=[sum(1), sum((cast(A#14 as double) * 1.0)), sum((cast(__CAVEATS_A_LB#832 as double) * 1.0)), sum((cast(__CAVEATS_A_UB#833 as double) * 1.0))], output=[X#842, __CAVEATS_ROW_LB#838, __CAVEATS_ROW_BG#839, __CAVEATS_ROW_UB#840, __CAVEATS_X_LB#841, __CAVEATS_X_UB#843])
+- *(1) HashAggregate(keys=[], functions=[partial_sum(1), partial_sum((cast(A#14 as double) * 1.0)), partial_sum((cast(__CAVEATS_A_LB#832 as double) * 1.0)), partial_sum((cast(__CAVEATS_A_UB#833 as double) * 1.0))], output=[sum#855L, sum#856, sum#857, sum#858])
+- *(1) Project [A#14, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833]
+- BatchScan[A#14] CSVScan Location: InMemoryFileIndex[file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv], ReadSchema: struct<A:string>
============================== SCHEMA ==============================
StructType(StructField(X,DoubleType,true), StructField(__CAVEATS_ROW_LB,IntegerType,false), StructField(__CAVEATS_ROW_BG,IntegerType,false), StructField(__CAVEATS_ROW_UB,IntegerType,false), StructField(__CAVEATS_X_LB,DoubleType,true), StructField(__CAVEATS_X_UB,DoubleType,true))
============================== RESULT ==============================
+---+----------------+----------------+----------------+--------------+--------------+
| X|__CAVEATS_ROW_LB|__CAVEATS_ROW_BG|__CAVEATS_ROW_UB|__CAVEATS_X_LB|__CAVEATS_X_UB|
+---+----------------+----------------+----------------+--------------+--------------+
|1.0| 1| 1| 1| 1.0| 1.0|
+---+----------------+----------------+----------------+--------------+--------------+
================================================================================
QUERY
================================================================================
Aggregate [CASE WHEN (sum(cast(__CAVEATS_ROW_BG#830 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast(A#14 as double) * cast(__CAVEATS_ROW_BG#830 as double)) ELSE 0.0 END) / cast(sum(cast(__CAVEATS_ROW_BG#830 as bigint)) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE least(0.0, (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum(cast(__CAVEATS_ROW_UB#831 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE greatest(0.0, (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(__CAVEATS_ROW_UB#831 as bigint)) as double)) END AS __CAVEATS_X_UB#843]
+- Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833, B#15 AS __CAVEATS_B_LB#834, B#15 AS __CAVEATS_B_UB#835, C#16 AS __CAVEATS_C_LB#836, C#16 AS __CAVEATS_C_UB#837]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
================================================================================
RESULT
================================================================================
+---+----------------+----------------+----------------+--------------+--------------+
| X|__CAVEATS_ROW_LB|__CAVEATS_ROW_BG|__CAVEATS_ROW_UB|__CAVEATS_X_LB|__CAVEATS_X_UB|
+---+----------------+----------------+----------------+--------------+--------------+
|1.0| 1| 1| 1| 1.0| 1.0|
+---+----------------+----------------+----------------+--------------+--------------+
================================================================================
QUERY
================================================================================
Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833, B#15 AS __CAVEATS_B_LB#834, B#15 AS __CAVEATS_B_UB#835, C#16 AS __CAVEATS_C_LB#836, C#16 AS __CAVEATS_C_UB#837]
+- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
================================================================================
RESULT
================================================================================
+---+----+----+----------------+----------------+----------------+--------------+--------------+--------------+--------------+--------------+--------------+
| A| B| C|__CAVEATS_ROW_LB|__CAVEATS_ROW_BG|__CAVEATS_ROW_UB|__CAVEATS_A_LB|__CAVEATS_A_UB|__CAVEATS_B_LB|__CAVEATS_B_UB|__CAVEATS_C_LB|__CAVEATS_C_UB|
+---+----+----+----------------+----------------+----------------+--------------+--------------+--------------+--------------+--------------+--------------+
| 1| 2| 3| 1| 1| 1| 1| 1| 2| 2| 3| 3|
| 1| 3| 1| 1| 1| 1| 1| 1| 3| 3| 1| 1|
| 2|null| 1| 1| 1| 1| 2| 2| null| null| 1| 1|
| 1| 2|null| 1| 1| 1| 1| 1| 2| 2| null| null|
| 1| 4| 2| 1| 1| 1| 1| 1| 4| 4| 2| 2|
| 2| 2| 1| 1| 1| 1| 2| 2| 2| 2| 1| 1|
| 4| 2| 4| 1| 1| 1| 4| 4| 2| 2| 4| 4|
+---+----+----+----------------+----------------+----------------+--------------+--------------+--------------+--------------+--------------+--------------+
================================================================================
QUERY
================================================================================
RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv
================================================================================
RESULT
================================================================================
+---+----+----+
| A| B| C|
+---+----+----+
| 1| 2| 3|
| 1| 3| 1|
| 2|null| 1|
| 1| 2|null|
| 1| 4| 2|
| 2| 2| 1|
| 4| 2| 4|
+---+----+----+
14:44:52.231 [Executor task launch worker for task 167] ERROR org.apache.spark.executor.Executor - Exception in task 5.0 in stage 38.0 (TID 167)
java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344)
at org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
at org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)
at org.apache.spark.sql.Row.isNullAt(Row.scala:204)
at org.apache.spark.sql.Row.isNullAt$(Row.scala:204)
at org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_1$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340)
... 19 common frames omitted
14:44:52.231 [Executor task launch worker for task 173] ERROR org.apache.spark.executor.Executor - Exception in task 11.0 in stage 38.0 (TID 173)
java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344)
at org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
at org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)
at org.apache.spark.sql.Row.isNullAt(Row.scala:204)
at org.apache.spark.sql.Row.isNullAt$(Row.scala:204)
at org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340)
... 19 common frames omitted
14:44:52.253 [task-result-getter-3] ERROR o.a.spark.scheduler.TaskSetManager - Task 11 in stage 38.0 failed 1 times; aborting job
[error] ! certain inputs.aggregation - no group-by - aggregtion functions only
[error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 38.0 failed 1 times, most recent failure: Lost task 11.0 in stage 38.0 (TID 173, 192.168.0.7, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080
[error] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344)
[error] at org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
[error] at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
[error] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[error] at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[error] at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
[error] at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
[error] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[error] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[error] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[error] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[error] at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
[error] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] at java.base/java.lang.Thread.run(Thread.java:830)
[error] Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
[error] at org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)
[error] at org.apache.spark.sql.Row.isNullAt(Row.scala:204)
[error] at org.apache.spark.sql.Row.isNullAt$(Row.scala:204)
[error] at org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)
[error] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source)
[error] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
[error] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340)
[error] ... 19 more
[error]
[error] Driver stacktrace: (DAGScheduler.scala:1989)
[error] org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1989)
[error] org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1977)
[error] org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1976)
[error] org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1976)
[error] org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:956)
[error] org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:956)
[error] org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:956)
[error] org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2206)
[error] org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2155)
[error] org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2144)
[error] org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[error] org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:758)
[error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
[error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
[error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
[error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
[error] org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
[error] org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[error] org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[error] org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[error] org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
[error] org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:365)
[error] org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3482)
[error] org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2812)
[error] org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3472)
[error] org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100)
[error] org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
[error] org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
[error] org.apache.spark.sql.Dataset.withAction(Dataset.scala:3468)
[error] org.apache.spark.sql.Dataset.collect(Dataset.scala:2812)
[error] org.mimirdb.utility.Bag$.apply(Bag.scala:41)
[error] org.mimirdb.test.DataFrameMatchers.dfBagEquals(DataFrameMatchers.scala:14)
[error] org.mimirdb.test.DataFrameMatchers.dfBagEquals$(DataFrameMatchers.scala:12)
[error] org.mimirdb.caveats.LogicalPlanRangeSpec.dfBagEquals(LogicalPlanRangeSpec.scala:21)
[error] org.mimirdb.test.DataFrameMatchers.$anonfun$beBagEqualsTo$2(DataFrameMatchers.scala:48)
[error] org.mimirdb.caveats.LogicalPlanRangeSpec.$anonfun$annotBagEqualToDF$1(LogicalPlanRangeSpec.scala:79)
[error] org.mimirdb.caveats.LogicalPlanRangeSpec.annotBagEqualToDF(LogicalPlanRangeSpec.scala:79)
[error] org.mimirdb.caveats.LogicalPlanRangeSpec.$anonfun$new$8(LogicalPlanRangeSpec.scala:366)
[error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344)
[error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
[error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[error] org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
[error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
[error] org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)
[error] org.apache.spark.sql.Row.isNullAt(Row.scala:204)
[error] org.apache.spark.sql.Row.isNullAt$(Row.scala:204)
[error] org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
[error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340)
[error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
[error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[error] org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
[error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
[error] CAUSED BY
[error] java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079
[error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080 (ExpressionEncoder.scala:344)
[error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344)
[error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
[error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[error] org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
[error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
[error] org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)
[error] org.apache.spark.sql.Row.isNullAt(Row.scala:204)
[error] org.apache.spark.sql.Row.isNullAt$(Row.scala:204)
[error] org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
[error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340)
[error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
[error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[error] org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
[error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
[error] CAUSED BY
[error] java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 (rows.scala:174)
[error] org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)
[error] org.apache.spark.sql.Row.isNullAt(Row.scala:204)
[error] org.apache.spark.sql.Row.isNullAt$(Row.scala:204)
[error] org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
[error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340)
[error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350)
[error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
[error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[error] org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441)
[error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444)
[info] TIP inputs
[info] Caveated inputs
[info] Total for specification LogicalPlanRangeSpec
[info] Finished in 8 seconds, 245 ms
[info] 1 example, 0 failure, 1 error
[error] Error: Total 1, Failed 0, Errors 1, Passed 0
[error] Error during tests:
[error] org.mimirdb.caveats.LogicalPlanRangeSpec
[error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 16 s, completed Jun 11, 2020, 2:44:57 PM
sbt:mimir-caveats>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment