Skip to content

Instantly share code, notes, and snippets.

@vaskokj

vaskokj/gist:0d1a5e2602a112f0c02a48aad51f7f58 Secret

Created January 31, 2023 20:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vaskokj/0d1a5e2602a112f0c02a48aad51f7f58 to your computer and use it in GitHub Desktop.
Save vaskokj/0d1a5e2602a112f0c02a48aad51f7f58 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
./spark-3.3.1-bin-hadoop3/bin/spark-submit --class io.treeverse.clients.GarbageCollector \
--packages org.apache.hadoop:hadoop-aws:3.3.2 \
--master spark://localhost:7077 \
-c spark.hadoop.lakefs.api.url=http://mylakeFS:8000/api/v1 \
-c spark.hadoop.lakefs.api.access_key=<lakeFSCredentials> \
-c spark.hadoop.lakefs.api.secret_key=<lakeFSCredentials> \
-c spark.hadoop.fs.s3a.access.key=<AWSCredentials> \
-c spark.hadoop.fs.s3a.secret.key=<AWSCredentials> \
-c spark.hadoop.fs.s3a.session.token=<AWSCredentials> \
-c spark.hadoop.fs.s3a.endpoint=http://bucket.vpce-<redacted>-<redacted>.s3.us-gov-west-1.vpce.amazonaws.com \
-c spark.hadoop.fs.s3a.endpoint.region=us-gov-west-1 \
lakefs-spark-client-312-hadoop3-assembly-0.6.0.jar \
myproject us-gov-west-1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment