Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
bash wget - check if file exists at url before downloading
#!/bin/bash
# simple function to check http response code before downloading a remote file
# example usage:
# if `validate_url $url >/dev/null`; then dosomething; else echo "does not exist"; fi
function validate_url(){
if [[ `wget -S --spider $1 2>&1 | grep 'HTTP/1.1 200 OK'` ]]; then echo "true"; fi
}
@deadlocks

This comment has been minimized.

Copy link

deadlocks commented Nov 13, 2014

Really helpful. Thanks!!!

@sawarineko

This comment has been minimized.

Copy link

sawarineko commented Dec 18, 2015

Ty <3

@bigretromike

This comment has been minimized.

Copy link

bigretromike commented Jul 15, 2016

Yup still great code :-)

@itsmanishagarwal

This comment has been minimized.

Copy link

itsmanishagarwal commented Aug 13, 2016

Thanks !

@lalet

This comment has been minimized.

Copy link

lalet commented Sep 22, 2016

Thank you. The script is really helpful !

@Deevad

This comment has been minimized.

Copy link

Deevad commented Nov 5, 2016

Thank you!

@fg6

This comment has been minimized.

Copy link

fg6 commented Apr 10, 2017

Thank you! Very helpful!

@ivadt

This comment has been minimized.

Copy link

ivadt commented Nov 8, 2017

The function validate_url() was helpful for me, but the example usage didn't work as expected - I had to change

 if `validate_url $url >/dev/null`; then dosomething; else echo "does not exist"; fi

to

if `validate_url $url`; then dosomething; else echo "does not exist"; fi

otherwise it always went in the "else" statement even for valid URLs

Anyway, thanks for the example!

@wangyoucao577

This comment has been minimized.

Copy link

wangyoucao577 commented Dec 26, 2017

Really helpful. Thanks!!!

@googlebe

This comment has been minimized.

Copy link

googlebe commented Jan 9, 2018

but if the remote url is https,the function is not working?

@radupearsica

This comment has been minimized.

Copy link

radupearsica commented Jan 29, 2018

Adapted the code to FTP server availability check, and for checking if the file is on server. For HTTP replace '220' or '150' with 'HTTP/1.1 200 OK'.

#!/bin/bash

Response code: Service ready for new user (220)

Response code: 150 Here comes the directory listing / opening binary data connection

url="ftp://path/to/file.something"

function validate_url(){
if [[ wget -S --spider $url 2>&1 | grep '220' ]]; then exit_status=$?; fi
if [[ $exit_status == 0 ]]; then
echo "FTP location exists"
elif [[ $exist_status == 1 ]]; then
echo "FTP location not available"
fi
}
validate_url

@julekgwa

This comment has been minimized.

Copy link

julekgwa commented Apr 30, 2018

It works but it's not reliable validate_url 'www.bing.com/az/hprichbg/rb/MaryLouWilliams_EN-US11937645356_1920x1200.jpg', echo's true yet the file doesn't exists.

add no redirection to wget --max-redirect=0 -S --spider $url

@HSarode-Compumatrice

This comment has been minimized.

Copy link

HSarode-Compumatrice commented May 31, 2018

awesome!!

@azhar22k

This comment has been minimized.

Copy link

azhar22k commented Dec 18, 2018

Very nice solution and useful. Here is how I used it in the boolean format

function validate_url(){
  if [[ `wget -S --spider $1  2>&1 | grep 'HTTP/1.1 200 OK'` ]]; then
    return 0
  else
    return 1
  fi
}

if validate_url $endpoint; then
    # Do something when exists
  else
    # Return or print some error
fi
@WillChaos

This comment has been minimized.

Copy link

WillChaos commented May 31, 2019

Thanks, works great.
itz-azhar's example made this a 2 second addition to my script.

ta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.