Skip to content

Instantly share code, notes, and snippets.

@1ambda

1ambda/df-cache-sql-lazy.py

Created December 25, 2021 07:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save 1ambda/92ba767271ec93b4425b60fd5f69f8c6 to your computer and use it in GitHub Desktop.
Save 1ambda/92ba767271ec93b4425b60fd5f69f8c6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
df-cache-sql-lazy.py
== Physical Plan ==
Scan In-memory table RAW_CACHED_LAZY (1)
+- InMemoryRelation (2)
+- * Filter (4)
+- InMemoryTableScan (3)
+- InMemoryRelation (4)
+- * ColumnarToRow (6)
+- Scan parquet (5)
(1) Scan In-memory table RAW_CACHED_LAZY
Output [5]: [listing_id#1349, listing_url#1350, listing_name#1351, listing_summary#1352, listing_desc#1353]
Arguments: [listing_id#1349, listing_url#1350, listing_name#1351, listing_summary#1352, listing_desc#1353]
(2) InMemoryRelation
Arguments: [listing_id#1349, listing_url#1350, listing_name#1351, listing_summary#1352, listing_desc#1353], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@df15d03,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Filter (isnotnull(listing_id#0) AND (listing_id#0 >= 20000000))
+- InMemoryTableScan [listing_id#0, listing_url#1, listing_name#2, listing_summary#3, listing_desc#4], [isnotnull(listing_id#0), (listing_id#0 >= 20000000)]
+- InMemoryRelation [listing_id#0, listing_url#1, listing_name#2, listing_summary#3, listing_desc#4], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) ColumnarToRow
+- FileScan parquet [listing_id#10,listing_url#11,listing_name#12,listing_summary#13,listing_desc#14] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/home/1ambda/airbnb_listings_parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<listing_id:int,listing_url:string,listing_name:string,listing_summary:string,listing_desc:...
,Some(RAW_CACHED_LAZY))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment