Skip to content

Instantly share code, notes, and snippets.

@mingwei-li
Created August 4, 2020 18:34
Show Gist options
  • Save mingwei-li/b0d444aa2d5778c3eb369f5c31f7bcda to your computer and use it in GitHub Desktop.
Save mingwei-li/b0d444aa2d5778c3eb369f5c31f7bcda to your computer and use it in GitHub Desktop.
hyperspace - explain
=============================================================
Plan with indexes:
=============================================================
Project [name#11]
+- Filter (isnotnull(id#10) && (id#10 = 1))
<----+- FileScan parquet [id#10,name#11] Batched: true, Format: Parquet, Location: InMemoryFileIndex[file:/Users/mingwli/Dev/lib/hadoop-2.9.2/bin/spark-warehouse/indexes/index/v__=0], PartitionFilters: [], PushedFilters: [IsNotNull(id), EqualTo(id,1)], ReadSchema: struct<id:int,name:string>---->
=============================================================
Plan without indexes:
=============================================================
Project [name#11]
+- Filter (isnotnull(id#10) && (id#10 = 1))
<----+- FileScan csv [id#10,name#11] Batched: false, Format: CSV, Location: InMemoryFileIndex[hdfs://localhost:9000/mingwli/customers.csv], PartitionFilters: [], PushedFilters: [IsNotNull(id), EqualTo(id,1)], ReadSchema: struct<id:int,name:string>---->
=============================================================
Indexes used:
=============================================================
index:file:/Users/mingwli/Dev/lib/hadoop-2.9.2/bin/spark-warehouse/indexes/index/v__=0
=============================================================
Physical operator stats:
=============================================================
+-----------------+-------------------+------------------+----------+
|Physical Operator|Hyperspace Disabled|Hyperspace Enabled|Difference|
+-----------------+-------------------+------------------+----------+
| *Scan csv| 1| 0| -1|
| *Scan parquet| 0| 1| 1|
| Filter| 1| 1| 0|
| Project| 1| 1| 0|
|WholeStageCodegen| 1| 1| 0|
+-----------------+-------------------+------------------+----------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment