Skip to content

Instantly share code, notes, and snippets.

@findepi

findepi/iceberg-dev-cheat-sheet.md

Last active July 24, 2023 01:15
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save findepi/da2ea85d095eaa0f3cfdcbb311ff59ed to your computer and use it in GitHub Desktop.
Save findepi/da2ea85d095eaa0f3cfdcbb311ff59ed to your computer and use it in GitHub Desktop.
Download ZIP
Raw
iceberg-dev-cheat-sheet.md

full build, like mvn install

# See knownSparkVersions in gradle.properties for current list Spark versions
./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 -x test -x integrationTest build publishToMavenLocal

# ... or pull the list automatically
./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal

reformat code

./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) spotlessApply

run a single Spark test class

./gradlew :iceberg-spark:iceberg-spark-extensions:test --tests TestRemoveOrphanFilesProcedure

# or with non-default Spark version
./gradlew -DsparkVersions=3.2 :iceberg-spark:iceberg-spark-extensions:test --tests TestRemoveOrphanFilesProcedure

run individual Spark test methods

# Note that * at the end of the method name is required
./gradlew -DsparkVersions=3.3 :iceberg-spark:iceberg-spark-extensions:test \
    --tests TestRemoveOrphanFilesProcedure.testRemoveOrphanFilesWithStatisticFiles\* \
    --tests TestRemoveOrphanFilesAction3.testRemoveOrphanFilesWithStatisticFiles\*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment