To be able to use custom endpoints with the latest Spark distribution, one needs to add an external package (hadoop-aws). Then, custum endpoints can be configured according to docs.
bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.7.2
tobilg
/ custom_s3_endpoint_in_spark.md
Last active
May 22, 2023 14:57
Description on how to use a custom S3 endpoint (like Rados Gateway for Ceph)