Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
bash wget - check if file exists at url before downloading
#!/bin/bash
# simple function to check http response code before downloading a remote file
# example usage:
# if `validate_url $url >/dev/null`; then dosomething; else echo "does not exist"; fi
function validate_url(){
if [[ `wget -S --spider $1 2>&1 | grep 'HTTP/1.1 200 OK'` ]]; then echo "true"; fi
}
@deadlocks

This comment has been minimized.

Copy link

@deadlocks deadlocks commented Nov 13, 2014

Really helpful. Thanks!!!

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Dec 18, 2015

Ty <3

@bigretromike

This comment has been minimized.

Copy link

@bigretromike bigretromike commented Jul 15, 2016

Yup still great code :-)

@itsmanishagarwal

This comment has been minimized.

Copy link

@itsmanishagarwal itsmanishagarwal commented Aug 13, 2016

Thanks !

@lalet

This comment has been minimized.

Copy link

@lalet lalet commented Sep 22, 2016

Thank you. The script is really helpful !

@Deevad

This comment has been minimized.

Copy link

@Deevad Deevad commented Nov 5, 2016

Thank you!

@fg6

This comment has been minimized.

Copy link

@fg6 fg6 commented Apr 10, 2017

Thank you! Very helpful!

@ivadt

This comment has been minimized.

Copy link

@ivadt ivadt commented Nov 8, 2017

The function validate_url() was helpful for me, but the example usage didn't work as expected - I had to change

 if `validate_url $url >/dev/null`; then dosomething; else echo "does not exist"; fi

to

if `validate_url $url`; then dosomething; else echo "does not exist"; fi

otherwise it always went in the "else" statement even for valid URLs

Anyway, thanks for the example!

@wangyoucao577

This comment has been minimized.

Copy link

@wangyoucao577 wangyoucao577 commented Dec 26, 2017

Really helpful. Thanks!!!

@googlebe

This comment has been minimized.

Copy link

@googlebe googlebe commented Jan 9, 2018

but if the remote url is https,the function is not working?

@radupearsica

This comment has been minimized.

Copy link

@radupearsica radupearsica commented Jan 29, 2018

Adapted the code to FTP server availability check, and for checking if the file is on server. For HTTP replace '220' or '150' with 'HTTP/1.1 200 OK'.

#!/bin/bash

Response code: Service ready for new user (220)

Response code: 150 Here comes the directory listing / opening binary data connection

url="ftp://path/to/file.something"

function validate_url(){
if [[ wget -S --spider $url 2>&1 | grep '220' ]]; then exit_status=$?; fi
if [[ $exit_status == 0 ]]; then
echo "FTP location exists"
elif [[ $exist_status == 1 ]]; then
echo "FTP location not available"
fi
}
validate_url

@julekgwa

This comment has been minimized.

Copy link

@julekgwa julekgwa commented Apr 30, 2018

It works but it's not reliable validate_url 'www.bing.com/az/hprichbg/rb/MaryLouWilliams_EN-US11937645356_1920x1200.jpg', echo's true yet the file doesn't exists.

add no redirection to wget --max-redirect=0 -S --spider $url

@HSarode-Compumatrice

This comment has been minimized.

Copy link

@HSarode-Compumatrice HSarode-Compumatrice commented May 31, 2018

awesome!!

@azhar22k

This comment has been minimized.

Copy link

@azhar22k azhar22k commented Dec 18, 2018

Very nice solution and useful. Here is how I used it in the boolean format

function validate_url(){
  if [[ `wget -S --spider $1  2>&1 | grep 'HTTP/1.1 200 OK'` ]]; then
    return 0
  else
    return 1
  fi
}

if validate_url $endpoint; then
    # Do something when exists
  else
    # Return or print some error
fi
@WillChaos

This comment has been minimized.

Copy link

@WillChaos WillChaos commented May 31, 2019

Thanks, works great.
itz-azhar's example made this a 2 second addition to my script.

ta.

@jbijota

This comment has been minimized.

Copy link

@jbijota jbijota commented Apr 22, 2020

Really cool, thanks!

@axet

This comment has been minimized.

Copy link

@axet axet commented Jun 9, 2020

@gluttony38

This comment has been minimized.

Copy link

@gluttony38 gluttony38 commented Oct 22, 2020

Hello, and thanks for this,

Anyway, I simplified the code before using it, there is no need to grep since wget returns 0 (OK) if file exists and there is no error, like authentication error for example:

function validate_url()
{
    wget --spider $1
    return $?
}

And if you want no output from wget (always hide output instead of putting >/dev/null when calling validate_url):

function validate_url()
{
    wget --spider $1 >/dev/null 2>&1
    return $?
}

And call it with:

if validate_url http://www.google.com; then
   echo Code if OK here
else
   echo Code if NOK here
fi

And I though I had to do some extra dev since I need to give a user and password and another parameter but it works too with for example (put all parameter inside "" is important else $1 in wget -S --spider $1 will only be the part before first space, URL then in example):

if validate_url "http://www.my_server_with_authentification.com/path/to/file --user=me --password=pwd --no-check-certificate"; then
   echo Code if OK here
else
   echo Code if NOK here
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.