Sivabalan Narayanan nsivabalan
nsivabalan
/ Hudi spark datasource CRUD
Created
July 30, 2023 03:55
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./bin/spark-shell --packages org.apache.hudi:hudi-spark2.4-bundle_2.11:0.13.1 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' | |
| import org.apache.hudi.QuickstartUtils._ | |
| import scala.collection.JavaConversions._ | |
| import org.apache.spark.sql.SaveMode._ | |
| import org.apache.hudi.DataSourceReadOptions._ | |
| import org.apache.hudi.DataSourceWriteOptions._ | |
| import org.apache.hudi.config.HoodieWriteConfig._ | |
| import org.apache.hudi.common.model.HoodieRecord |
nsivabalan
/ asf-site update
Created
July 29, 2023 18:24
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/website/releases/older-releases.md b/website/releases/older-releases.md | |
| index 595c644147f..5362c12d642 100644 | |
| --- a/website/releases/older-releases.md | |
| +++ b/website/releases/older-releases.md | |
| @@ -549,9 +549,9 @@ users to migrate to. Release (0.12.3) is the latest 0.12 release. | |
| * This release (0.12.2) does not introduce any new table version, thus no migration is needed if you are on 0.12.0. | |
| * If migrating from an older release, please check the migration guide from the previous release notes, specifically | |
| - the upgrade instructions in [0.6.0](/releases/older-releases#migration-guide-for-this-release), | |
| - [0.9.0](/releases/release-0.9.0#migration-guide-for-this-release), [0.10.0](/releases/release-0.10.0#migration-guide), |
nsivabalan
/ MDT rollback patch
Created
July 10, 2023 19:31
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java | |
| index c4323081a30..8287faf3ae4 100644 | |
| --- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java | |
| +++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java | |
| @@ -23,6 +23,7 @@ import org.apache.hudi.avro.model.HoodieCleanPartitionMetadata; | |
| import org.apache.hudi.avro.model.HoodieIndexPartitionInfo; | |
| import org.apache.hudi.avro.model.HoodieIndexPlan; | |
| import org.apache.hudi.avro.model.HoodieRestoreMetadata; | |
| +import org.apache.hudi.avro.model.HoodieRestorePlan; | |
| import org.apache.hudi.avro.model.HoodieRollbackMetadata; |
nsivabalan
/ global simple dups
Created
March 18, 2023 05:34
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.hudi.QuickstartUtils._ | |
| import scala.collection.JavaConversions._ | |
| import org.apache.spark.sql.SaveMode._ | |
| import org.apache.hudi.DataSourceReadOptions._ | |
| import org.apache.hudi.DataSourceWriteOptions._ | |
| import org.apache.hudi.config.HoodieWriteConfig._ | |
| import org.apache.hudi.common.model.HoodieRecord |
nsivabalan
/ mor with clustering
Last active
March 17, 2023 17:49
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.hudi.QuickstartUtils._ | |
| import scala.collection.JavaConversions._ | |
| import org.apache.spark.sql.SaveMode._ | |
| import org.apache.hudi.DataSourceReadOptions._ | |
| import org.apache.hudi.DataSourceWriteOptions._ | |
| import org.apache.hudi.config.HoodieWriteConfig._ | |
| import org.apache.hudi.common.model.HoodieRecord | |
| val tableName = "hudi_trips_cow" |
nsivabalan
/ gist:7ed9aabc65e8de536b07390e6e872b87
Last active
February 24, 2023 09:15
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| git diff | |
| diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java | |
| index e2de9c7633..ede050b1b4 100644 | |
| --- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java | |
| +++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java | |
| @@ -33,8 +33,11 @@ import org.apache.hudi.client.heartbeat.HeartbeatUtils; | |
| import org.apache.hudi.client.utils.TransactionUtils; | |
| import org.apache.hudi.common.HoodiePendingRollbackInfo; | |
| import org.apache.hudi.common.config.HoodieCommonConfig; | |
| +import org.apache.hudi.common.config.HoodieMetadataConfig; |
nsivabalan
/ run book
Last active
January 12, 2023 20:58
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./bin/spark-shell --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --jars ~/Documents/personal/projects/nov26/hudi/packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.13.0-SNAPSHOT.jar | |
| // spark-shell | |
| import org.apache.hudi.QuickstartUtils._ | |
| import scala.collection.JavaConversions._ | |
| import org.apache.spark.sql.SaveMode._ | |
| import org.apache.hudi.DataSourceReadOptions._ |
nsivabalan
/ gist:1fcfcaf19a916342aa74775960533cd9
Created
August 30, 2022 15:17
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ITTestHoodieSanity.testRunHoodieJavaAppOnMultiPartitionKeysMORTable | |
| https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/11006/logs/40 | |
| 2022-08-29T07:56:42.2376427Z Logging initialized using configuration in file:/opt/hive/conf/hive-log4j2.properties Async: true | |
| 2022-08-29T07:56:42.2376830Z OK | |
| 2022-08-29T07:56:42.2377102Z Time taken: 1.399 seconds | |
| 2022-08-29T07:56:42.2377235Z | |
| 2022-08-29T07:56:42.3504813Z [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 785.047 s <<< FAILURE! - in org.apache.hudi.integ.ITTestHoodieSanity |
nsivabalan
/ dStream write to parquet
Last active
August 26, 2022 20:46
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark._ | |
| import org.apache.spark.streaming._ | |
| import org.apache.spark.sql.SaveMode._ | |
| val sc = spark.sparkContext | |
| val ssc = new StreamingContext(sc, Seconds(1)) | |
| val inputPath = "/tmp/inputDir/" |
nsivabalan
/ Spark streaming write to hudi
Created
August 24, 2022 18:09
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.4 \ | |
| --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --driver-memory 8g --executor-memory 9g --jars ~/Documents/personal/projects/nov26/hudi/packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.11.0-SNAPSHOT.jar | |
| import org.apache.spark.sql.SparkSession | |
| import org.apache.spark.sql.functions.{col, from_json,to_json,struct} | |
| import org.apache.spark.sql.avro.to_avro | |
| import org.apache.spark.sql.types.{IntegerType, StringType, LongType, StructType} | |
| import java.time.LocalDateTime |
NewerOlder