Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Socrata HTTP Gripes
% curl -sIL 'https://data.acgov.org/api/geospatial/8e4s-7f4v?method=export&format=Original'
HTTP/1.1 405 Method Not Allowed
Server: nginx
Date: Tue, 13 Jan 2015 01:52:36 GMT
Connection: keep-alive
Access-Control-Allow-Origin: *
X-Error-Code: method_not_allowed
X-Error-Message: HEAD not allowed
X-Socrata-Region: production
Age: 0
% curl -sIL https://data.sfgov.org/download/kvej-w5kb/ZIPPED%20SHAPEFILE
HTTP/1.1 302 Found
Server: nginx
Date: Tue, 13 Jan 2015 01:51:36 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=Edge,chrome=1
Cache-Control: no-cache
X-Request-Id: a79b2f973b7f46e4292a4dd1ca6b555f
Set-Cookie: _socrata_session_id=BAh7BkkiD3Nlc3Npb25faWQGOgZFRiIlNmQ5NGQ2YTNiN2I4Y2ZjN2UyMzRjN2Y0MGMwODM5YjU%3D--d8c873bdb9117ef5c10134a8ebf188095cfeea17; path=/; HttpOnly; secure
Set-Cookie: logged_in=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Location: http://apps.sfgov.org/datafiles/view.php?file=sfgis/eas_addresses_with_units.zip
Status: 302
Vary: Accept-Encoding,User-Agent
X-Socrata-Region: production
Age: 0
HTTP/1.1 200 OK
Date: Tue, 13 Jan 2015 01:51:37 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.10
Pragma: public
Expires: 0
Cache-Control: must-revalidate, post-check=0, pre-check=0
Content-Disposition: attachment; filename=eas_addresses_with_units.zip;
Content-Transfer-Encoding: binary
Content-Length: 8300106
Content-Type: application/download
@chrismetcalf

This comment has been minimized.

Copy link

@chrismetcalf chrismetcalf commented Feb 19, 2015

@migurski - This just finally crossed my desk. Hopefully I can help, or at least we can commiserate :)

In the former case, you're right, we don't support the HTTP HEAD method. This is a big request of mine and I'm pushing our engineering team to include it our big platform release in a few months. It's something that I'd make a lot of use of too.

Along with that, we also need to provide you the correct content-types. Even if you did a full GET on that URL, you'd still just get application/octet-stream, which is pretty much useless, unless you want to look at the Content-Disposition header too. Unfortunately the proper mime type for a zipped Shapefile is still "application/zip", but at least that'd get you one step closer. We're going to fix that too.

In the latter case, that's actually an "External Dataset" that San Francisco has provided - basically it's just a link in the catalog. By the time you get redirected to apps.sfgov.org, it's unfortunately out of our hands and returning the proper content type is something that they'd need to fix. I've asked if our CS team can reach out to the city to let them know about that one, but you might have channels of your own too.

@dedan

This comment has been minimized.

Copy link

@dedan dedan commented Feb 24, 2015

I currently try to download datasets from SFgov using CURL or WGET. I receive a file, but when I try to unzip it, it converts into a .cpgz file, which converts into a .zip file, which converts into a .cpgz file, ...

Is this related to the above mentioned problem?

@migurski

This comment has been minimized.

Copy link
Owner Author

@migurski migurski commented Mar 21, 2015

Thanks @chrismetcalf, very useful background! I’ll see if I can follow up with SF to get a better HTTP response out of them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.