Skip to content

Instantly share code, notes, and snippets.

@saintmalik
Created October 16, 2023 14:22
Show Gist options
  • Save saintmalik/0daab4755dbdcdaec0e0348cc8272700 to your computer and use it in GitHub Desktop.
Save saintmalik/0daab4755dbdcdaec0e0348cc8272700 to your computer and use it in GitHub Desktop.
first get the file object names and append the s3 url with it then use the follow the command
imgdir=$1
urllist=$2
mkdir $imgdir
# Initialize a counter for incremental filenames
count=1
# Read URLs from the urllist file and download files
while IFS= read -r url; do
filename="file${count}.png"
curl "$url" --output "$imgdir/$filename"
((count++))
done < $urllist
echo "Finish downloading"
# deduplicate the urls and remove the headers in the file
# use the command `nohup bash download_images.sh img/ urllist.txt &> download.log &` to run it on the background
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment