Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Wget Curl comparison and our Approach

We have changed our check_url implementation from curl to wget for three reasons

  1. It gives us the ability to retry on connection refused. Compare the command we used previously with curl:
curl -sSfL --retry 3 URL

to the current one:

wget --retry-connrefused --no-check-certificate -T 60 URL

The retry-connrefused triggers a retry whenever the connection is closed after it was already established to the server. This behaviour is for example seen when applications on Heroku take too long to boot and Heroku terminates the request. Then the connection is simply closed.

From the wget manpage:

--retry-connrefused
           Consider "connection refused" a transient error and try again.  Normally Wget gives up on a URL when it is unable to connect to the site because failure to connect is taken as a sign that the server is not running at all
           and that retries would not help.  This option is for mirroring unreliable sites whose servers tend to disappear for short periods of time.

You can try it yourself by running nc -l 5000 and then curl -sSfL --retry 3 localhost:5000. Now kill the nc process and see how curl dies, but wget --retry-connrefused --no-check-certificate -T 60.

Curls connect-timeout is not helpful here, as it specifically only works until the connection is established which isn't and has never been the problem for us.

  1. Curl retries on HTTP Status 500 or other transient errors, but doesn't retry on connection refused as wget does. We do prefer the error to happen on connection refused. We have implemented a wrapper around wget that retries up to 3 times if it fails for any 500 Status.

  2. Wget in its default configuration provides the more helpful output during a build. It shows when the connection is established and what is currently happening. Of course something we could probably achieve with curl as well, but I like to use what is already available.

@eric-hu

This comment has been minimized.

Copy link

@eric-hu eric-hu commented Nov 13, 2017

Curl now supports --retry --retry-connrefused for this behavior. I'm not sure what version this begins with, but they merged it in November 2016:

curl/curl#1064

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment