Skip to content

Instantly share code, notes, and snippets.

@1ambda
Created December 25, 2021 00:43
Embed
What would you like to do?
== Physical Plan ==
InMemoryTableScan (1)
+- InMemoryRelation (2)
+- * ColumnarToRow (4)
+- Scan parquet (3)
(1) InMemoryTableScan
Output [5]: [listing_id#20, listing_url#21, listing_name#22, listing_summary#23, listing_desc#24]
Arguments: [listing_id#20, listing_url#21, listing_name#22, listing_summary#23, listing_desc#24]
(2) InMemoryRelation
Arguments: [listing_id#20, listing_url#21, listing_name#22, listing_summary#23, listing_desc#24], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@286e1e85,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) ColumnarToRow
+- FileScan parquet [listing_id#20,listing_url#21,listing_name#22,listing_summary#23,listing_desc#24] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/home/1ambda/airbnb_listings_parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<listing_id:int,listing_url:string,listing_name:string,listing_summary:string,listing_desc:...
,None)
(3) Scan parquet
Output [5]: [listing_id#20, listing_url#21, listing_name#22, listing_summary#23, listing_desc#24]
Batched: true
Location: InMemoryFileIndex [file:/home/1ambda/airbnb_listings_parquet]
ReadSchema: struct<listing_id:int,listing_url:string,listing_name:string,listing_summary:string,listing_desc:string>
(4) ColumnarToRow [codegen id : 1]
Input [5]: [listing_id#20, listing_url#21, listing_name#22, listing_summary#23, listing_desc#24]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment