Skip to content

Instantly share code, notes, and snippets.

@nfarah86
Last active July 20, 2023 16:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nfarah86/a448eec50d2b394932c2bcdf6349e53a to your computer and use it in GitHub Desktop.
Save nfarah86/a448eec50d2b394932c2bcdf6349e53a to your computer and use it in GitHub Desktop.
import org.apache.hudi.client.validator.SparkPreCommitValidator
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.Row
class CustomPreCommitValidator extends SparkPreCommitValidator {
override def validateRecordsBeforeAndAfter(before: Dataset[Row],
after: Dataset[Row], partitionsAffected: Set[String]): Unit = {
// Custom validation logic
// Perform data quality checks, apply business rules, etc.
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment