Skip to content

Instantly share code, notes, and snippets.

@yihua

yihua/gist.scala Secret

Created April 7, 2023 23:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yihua/6eb11ce3f888a71935dbf21c77199a48 to your computer and use it in GitHub Desktop.
Save yihua/6eb11ce3f888a71935dbf21c77199a48 to your computer and use it in GitHub Desktop.
Download ZIP
Duplicates are deleted with Simple Index (default) in Apache Hudi
Raw
gist.scala
An example demonstrating the following scenario with the default Simple Index:
(1) Write duplicates to a Hudi table from two commits, so that primary key “499fc337-98ca-4e8c-b00b-0ea1a94c1397” exists
in two data files: “f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet”
and “36e2439e-119d-4904-98d5-f77f7d9dd5e3-0_0-68-186_20230406141555922.parquet”;
(2) Delete the record;
(3) After the deletion, the primary key does not exist anymore in the table, both records are deleted,
and the two data files are re-written (“f0124bf2-5554-4952-91cd-bd70306d17fc-0_1-134-351_20230406141601520.parquet”,
and “36e2439e-119d-4904-98d5-f77f7d9dd5e3-0_0-134-350_20230406141601520.parquet”).
org.apache.hudi#hudi-spark3.3-bundle_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-5da83ca2-1d66-4445-ae5e-d6dcfa95990a;1.0
confs: [default]
found org.apache.hudi#hudi-spark3.3-bundle_2.12;0.13.0 in central
:: resolution report :: resolve 120ms :: artifacts dl 3ms
:: modules in use:
org.apache.hudi#hudi-spark3.3-bundle_2.12;0.13.0 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 1 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-5da83ca2-1d66-4445-ae5e-d6dcfa95990a
confs: [default]
0 artifacts copied, 1 already retrieved (0kB/4ms)
23/04/06 14:14:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.1.101:5555
Spark context available as 'sc' (master = local[8], app id = local-1680815661894).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.1
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.hudi.QuickstartUtils._
import org.apache.hudi.QuickstartUtils._
scala> import scala.collection.JavaConversions._
import scala.collection.JavaConversions._
scala> val tableName = "hudi_trips_cow"
tableName: String = hudi_trips_cow
scala> val basePath = "/tmp/hudi_trips_cow"
basePath: String = /tmp/hudi_trips_cow
scala> val dataGen = new DataGenerator
dataGen: org.apache.hudi.QuickstartUtils.DataGenerator = org.apache.hudi.QuickstartUtils$DataGenerator@223973a9
scala> val inserts = convertToStringList(dataGen.generateInserts(10))
inserts: java.util.List[String] = [{"ts": 1680502605737, "uuid": "499fc337-98ca-4e8c-b00b-0ea1a94c1397", "rider": "rider-213", "driver": "driver-213", "begin_lat": 0.4726905879569653, "begin_lon": 0.46157858450465483, "end_lat": 0.754803407008858, "end_lon": 0.9671159942018241, "fare": 34.158284716382845, "partitionpath": "americas/brazil/sao_paulo"}, "partitionpath": "americas/brazil/sao_paulo"}, {"ts": 1680502615086, "uuid": "1bdb514d-2a9d-4da1-94a8-205b9b5f612a", "rider": "rider-213", "driver": "driver-213", "begin_lat": 0.6100070562136587, "begin_lon": 0.8779402295427752, "end_lat": 0.3407870505929602, "end_lon": 0.5030798142293655, "fare": 43.4923811219014, "partitionpath": "americas/brazil/sao_paulo"}, "partitionpath": "americas/brazil/sao_paulo"}, {"ts":...
scala> val df = spark.read.json(spark.sparkContext.parallelize(inserts))
df: org.apache.spark.sql.DataFrame = [begin_lat: double, begin_lon: double ... 8 more fields]
scala> df.write.format("hudi").
| option("hoodie.datasource.write.precombine.field", "ts").
| option("hoodie.datasource.write.recordkey.field", "uuid").
| option("hoodie.datasource.write.partitionpath.field", "partitionpath").
| option("hoodie.table.name", tableName).
| mode("overwrite").
| save(basePath)
scala> spark.read.format("hudi").load(basePath).count()
res1: Long = 10
scala> spark.read.format("hudi").load(basePath).select("_hoodie_record_key").distinct().count()
res2: Long = 10
scala> spark.read.format("hudi").load(basePath).orderBy("_hoodie_partition_path", "_hoodie_record_key").show(false)
+-------------------+---------------------+------------------------------------+------------------------------------+------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key |_hoodie_partition_path |_hoodie_file_name |begin_lat |begin_lon |driver |end_lat |end_lon |fare |rider |ts |uuid |partitionpath |
+-------------------+---------------------+------------------------------------+------------------------------------+------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
|20230406141538687 |20230406141538687_0_1|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet|0.6100070562136587 |0.8779402295427752 |driver-213|0.3407870505929602 |0.5030798142293655 |43.4923811219014 |rider-213|1680502615086|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_0_0|499fc337-98ca-4e8c-b00b-0ea1a94c1397|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet|0.4726905879569653 |0.46157858450465483|driver-213|0.754803407008858 |0.9671159942018241 |34.158284716382845|rider-213|1680502605737|499fc337-98ca-4e8c-b00b-0ea1a94c1397|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_0_2|d28072ed-ae11-4673-91ab-9b2fbf51d95e|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet|0.0750588760043035 |0.03844104444445928|driver-213|0.04376353354538354|0.6346040067610669 |66.62084366450246 |rider-213|1680248683730|d28072ed-ae11-4673-91ab-9b2fbf51d95e|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_1_4|11378ff4-0ecb-4478-94dc-d987fd1aaeb2|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet|0.11488393157088261|0.6273212202489661 |driver-213|0.7454678537511295 |0.3954939864908973 |27.79478688582596 |rider-213|1680666965860|11378ff4-0ecb-4478-94dc-d987fd1aaeb2|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_0|74bce67e-ecdf-4be0-8919-f0ad52af7901|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet|0.21624150367601136|0.14285051259466197|driver-213|0.5890949624813784 |0.0966823831927115 |93.56018115236618 |rider-213|1680468167807|74bce67e-ecdf-4be0-8919-f0ad52af7901|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_2|b8877b9d-a548-4897-99d3-cc6adb11887e|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet|0.8742041526408587 |0.7528268153249502 |driver-213|0.9197827128888302 |0.362464770874404 |19.179139106643607|rider-213|1680576258401|b8877b9d-a548-4897-99d3-cc6adb11887e|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_3|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet|0.5731835407930634 |0.4923479652912024 |driver-213|0.08988581780930216|0.42520899698713666|64.27696295884016 |rider-213|1680345848780|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_1|f9189d99-3cfd-430f-a14c-8061af637113|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet|0.1856488085068272 |0.9694586417848392 |driver-213|0.38186367037201974|0.25252652214479043|33.92216483948643 |rider-213|1680609826450|f9189d99-3cfd-430f-a14c-8061af637113|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_2_0|a5a3839c-df92-4ae8-8538-34140d392036|asia/india/chennai |f462c8c8-d8ed-4fa5-834e-7bae4b5c0ab4-0_2-15-73_20230406141538687.parquet|0.40613510977307 |0.5644092139040959 |driver-213|0.798706304941517 |0.02698359227182834|17.851135255091155|rider-213|1680699144106|a5a3839c-df92-4ae8-8538-34140d392036|asia/india/chennai |
|20230406141538687 |20230406141538687_2_1|fb7d3210-0664-4ddf-8079-df6688d75ffb|asia/india/chennai |f462c8c8-d8ed-4fa5-834e-7bae4b5c0ab4-0_2-15-73_20230406141538687.parquet|0.651058505660742 |0.8192868687714224 |driver-213|0.20714896002914462|0.06224031095826987|41.06290929046368 |rider-213|1680252927299|fb7d3210-0664-4ddf-8079-df6688d75ffb|asia/india/chennai |
+-------------------+---------------------+------------------------------------+------------------------------------+------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
scala> val df2 = spark.read.json(spark.sparkContext.parallelize(inserts.slice(0, 3)))
df2: org.apache.spark.sql.DataFrame = [begin_lat: double, begin_lon: double ... 8 more fields]
scala> df2.write.format("hudi").
| option("hoodie.datasource.write.operation", "insert").
| option("hoodie.datasource.write.precombine.field", "ts").
| option("hoodie.datasource.write.recordkey.field", "uuid").
| option("hoodie.datasource.write.partitionpath.field", "partitionpath").
| option("hoodie.table.name", tableName).
| option("hoodie.parquet.small.file.limit", "0").
| mode("append").
| save(basePath)
scala> spark.read.format("hudi").load(basePath).count()
res5: Long = 13
scala> spark.read.format("hudi").load(basePath).select("_hoodie_record_key").distinct().count()
res6: Long = 10
scala> spark.read.format("hudi").load(basePath).orderBy("_hoodie_partition_path", "_hoodie_record_key").show(false)
+-------------------+---------------------+------------------------------------+------------------------------------+-------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key |_hoodie_partition_path |_hoodie_file_name |begin_lat |begin_lon |driver |end_lat |end_lon |fare |rider |ts |uuid |partitionpath |
+-------------------+---------------------+------------------------------------+------------------------------------+-------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
|20230406141538687 |20230406141538687_0_1|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet |0.6100070562136587 |0.8779402295427752 |driver-213|0.3407870505929602 |0.5030798142293655 |43.4923811219014 |rider-213|1680502615086|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |
|20230406141555922 |20230406141555922_0_1|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |36e2439e-119d-4904-98d5-f77f7d9dd5e3-0_0-68-186_20230406141555922.parquet|0.6100070562136587 |0.8779402295427752 |driver-213|0.3407870505929602 |0.5030798142293655 |43.4923811219014 |rider-213|1680502615086|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_0_0|499fc337-98ca-4e8c-b00b-0ea1a94c1397|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet |0.4726905879569653 |0.46157858450465483|driver-213|0.754803407008858 |0.9671159942018241 |34.158284716382845|rider-213|1680502605737|499fc337-98ca-4e8c-b00b-0ea1a94c1397|americas/brazil/sao_paulo |
|20230406141555922 |20230406141555922_0_0|499fc337-98ca-4e8c-b00b-0ea1a94c1397|americas/brazil/sao_paulo |36e2439e-119d-4904-98d5-f77f7d9dd5e3-0_0-68-186_20230406141555922.parquet|0.4726905879569653 |0.46157858450465483|driver-213|0.754803407008858 |0.9671159942018241 |34.158284716382845|rider-213|1680502605737|499fc337-98ca-4e8c-b00b-0ea1a94c1397|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_0_2|d28072ed-ae11-4673-91ab-9b2fbf51d95e|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_0-15-71_20230406141538687.parquet |0.0750588760043035 |0.03844104444445928|driver-213|0.04376353354538354|0.6346040067610669 |66.62084366450246 |rider-213|1680248683730|d28072ed-ae11-4673-91ab-9b2fbf51d95e|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_1_4|11378ff4-0ecb-4478-94dc-d987fd1aaeb2|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.11488393157088261|0.6273212202489661 |driver-213|0.7454678537511295 |0.3954939864908973 |27.79478688582596 |rider-213|1680666965860|11378ff4-0ecb-4478-94dc-d987fd1aaeb2|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_0|74bce67e-ecdf-4be0-8919-f0ad52af7901|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.21624150367601136|0.14285051259466197|driver-213|0.5890949624813784 |0.0966823831927115 |93.56018115236618 |rider-213|1680468167807|74bce67e-ecdf-4be0-8919-f0ad52af7901|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_2|b8877b9d-a548-4897-99d3-cc6adb11887e|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.8742041526408587 |0.7528268153249502 |driver-213|0.9197827128888302 |0.362464770874404 |19.179139106643607|rider-213|1680576258401|b8877b9d-a548-4897-99d3-cc6adb11887e|americas/united_states/san_francisco|
|20230406141555922 |20230406141555922_1_0|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|0624770a-9225-49e9-a76d-9e1fcd7266be-0_1-68-187_20230406141555922.parquet|0.5731835407930634 |0.4923479652912024 |driver-213|0.08988581780930216|0.42520899698713666|64.27696295884016 |rider-213|1680345848780|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_3|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.5731835407930634 |0.4923479652912024 |driver-213|0.08988581780930216|0.42520899698713666|64.27696295884016 |rider-213|1680345848780|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_1|f9189d99-3cfd-430f-a14c-8061af637113|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.1856488085068272 |0.9694586417848392 |driver-213|0.38186367037201974|0.25252652214479043|33.92216483948643 |rider-213|1680609826450|f9189d99-3cfd-430f-a14c-8061af637113|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_2_0|a5a3839c-df92-4ae8-8538-34140d392036|asia/india/chennai |f462c8c8-d8ed-4fa5-834e-7bae4b5c0ab4-0_2-15-73_20230406141538687.parquet |0.40613510977307 |0.5644092139040959 |driver-213|0.798706304941517 |0.02698359227182834|17.851135255091155|rider-213|1680699144106|a5a3839c-df92-4ae8-8538-34140d392036|asia/india/chennai |
|20230406141538687 |20230406141538687_2_1|fb7d3210-0664-4ddf-8079-df6688d75ffb|asia/india/chennai |f462c8c8-d8ed-4fa5-834e-7bae4b5c0ab4-0_2-15-73_20230406141538687.parquet |0.651058505660742 |0.8192868687714224 |driver-213|0.20714896002914462|0.06224031095826987|41.06290929046368 |rider-213|1680252927299|fb7d3210-0664-4ddf-8079-df6688d75ffb|asia/india/chennai |
+-------------------+---------------------+------------------------------------+------------------------------------+-------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
scala> val deletes = dataGen.generateDeletes(df2.collectAsList().slice(0, 1))
deletes: java.util.List[String] = [{"ts": "1680502605737","uuid": "499fc337-98ca-4e8c-b00b-0ea1a94c1397","partitionpath": "americas/brazil/sao_paulo"}]
scala> val deleteDf = spark.read.json(spark.sparkContext.parallelize(deletes))
deleteDf: org.apache.spark.sql.DataFrame = [partitionpath: string, ts: string ... 1 more field]
scala> deleteDf.write.format("hudi").
| option("hoodie.datasource.write.operation", "delete").
| option("hoodie.datasource.write.precombine.field", "ts").
| option("hoodie.datasource.write.recordkey.field", "uuid").
| option("hoodie.datasource.write.partitionpath.field", "partitionpath").
| option("hoodie.table.name", tableName).
| mode("append").
| save(basePath)
scala> spark.read.format("hudi").load(basePath).count()
res9: Long = 11
scala> spark.read.format("hudi").load(basePath).select("_hoodie_record_key").distinct().count()
res10: Long = 9
scala> spark.read.format("hudi").load(basePath).orderBy("_hoodie_partition_path", "_hoodie_record_key").show(false)
+-------------------+---------------------+------------------------------------+------------------------------------+--------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key |_hoodie_partition_path |_hoodie_file_name |begin_lat |begin_lon |driver |end_lat |end_lon |fare |rider |ts |uuid |partitionpath |
+-------------------+---------------------+------------------------------------+------------------------------------+--------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
|20230406141538687 |20230406141538687_0_1|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_1-134-351_20230406141601520.parquet|0.6100070562136587 |0.8779402295427752 |driver-213|0.3407870505929602 |0.5030798142293655 |43.4923811219014 |rider-213|1680502615086|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |
|20230406141555922 |20230406141555922_0_1|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |36e2439e-119d-4904-98d5-f77f7d9dd5e3-0_0-134-350_20230406141601520.parquet|0.6100070562136587 |0.8779402295427752 |driver-213|0.3407870505929602 |0.5030798142293655 |43.4923811219014 |rider-213|1680502615086|1bdb514d-2a9d-4da1-94a8-205b9b5f612a|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_0_2|d28072ed-ae11-4673-91ab-9b2fbf51d95e|americas/brazil/sao_paulo |f0124bf2-5554-4952-91cd-bd70306d17fc-0_1-134-351_20230406141601520.parquet|0.0750588760043035 |0.03844104444445928|driver-213|0.04376353354538354|0.6346040067610669 |66.62084366450246 |rider-213|1680248683730|d28072ed-ae11-4673-91ab-9b2fbf51d95e|americas/brazil/sao_paulo |
|20230406141538687 |20230406141538687_1_4|11378ff4-0ecb-4478-94dc-d987fd1aaeb2|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.11488393157088261|0.6273212202489661 |driver-213|0.7454678537511295 |0.3954939864908973 |27.79478688582596 |rider-213|1680666965860|11378ff4-0ecb-4478-94dc-d987fd1aaeb2|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_0|74bce67e-ecdf-4be0-8919-f0ad52af7901|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.21624150367601136|0.14285051259466197|driver-213|0.5890949624813784 |0.0966823831927115 |93.56018115236618 |rider-213|1680468167807|74bce67e-ecdf-4be0-8919-f0ad52af7901|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_2|b8877b9d-a548-4897-99d3-cc6adb11887e|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.8742041526408587 |0.7528268153249502 |driver-213|0.9197827128888302 |0.362464770874404 |19.179139106643607|rider-213|1680576258401|b8877b9d-a548-4897-99d3-cc6adb11887e|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_3|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.5731835407930634 |0.4923479652912024 |driver-213|0.08988581780930216|0.42520899698713666|64.27696295884016 |rider-213|1680345848780|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|
|20230406141555922 |20230406141555922_1_0|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|0624770a-9225-49e9-a76d-9e1fcd7266be-0_1-68-187_20230406141555922.parquet |0.5731835407930634 |0.4923479652912024 |driver-213|0.08988581780930216|0.42520899698713666|64.27696295884016 |rider-213|1680345848780|d3cd3330-2a17-4c24-bb6e-10a854291a50|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_1_1|f9189d99-3cfd-430f-a14c-8061af637113|americas/united_states/san_francisco|8c0ecf64-e9bb-44c2-ab25-14a9b6d639ae-0_1-15-72_20230406141538687.parquet |0.1856488085068272 |0.9694586417848392 |driver-213|0.38186367037201974|0.25252652214479043|33.92216483948643 |rider-213|1680609826450|f9189d99-3cfd-430f-a14c-8061af637113|americas/united_states/san_francisco|
|20230406141538687 |20230406141538687_2_0|a5a3839c-df92-4ae8-8538-34140d392036|asia/india/chennai |f462c8c8-d8ed-4fa5-834e-7bae4b5c0ab4-0_2-15-73_20230406141538687.parquet |0.40613510977307 |0.5644092139040959 |driver-213|0.798706304941517 |0.02698359227182834|17.851135255091155|rider-213|1680699144106|a5a3839c-df92-4ae8-8538-34140d392036|asia/india/chennai |
|20230406141538687 |20230406141538687_2_1|fb7d3210-0664-4ddf-8079-df6688d75ffb|asia/india/chennai |f462c8c8-d8ed-4fa5-834e-7bae4b5c0ab4-0_2-15-73_20230406141538687.parquet |0.651058505660742 |0.8192868687714224 |driver-213|0.20714896002914462|0.06224031095826987|41.06290929046368 |rider-213|1680252927299|fb7d3210-0664-4ddf-8079-df6688d75ffb|asia/india/chennai |
+-------------------+---------------------+------------------------------------+------------------------------------+--------------------------------------------------------------------------+-------------------+-------------------+----------+-------------------+-------------------+------------------+---------+-------------+------------------------------------+------------------------------------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment