Skip to content

Instantly share code, notes, and snippets.

@timtomch
Last active June 7, 2021 14:10
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save timtomch/34bc2c4c9666cf9bf011 to your computer and use it in GitHub Desktop.
Save timtomch/34bc2c4c9666cf9bf011 to your computer and use it in GitHub Desktop.
OpenRefine operations to get minimum price data from ISBNdb.com -- Edit: no longer working due to ISBNdb API changes
[
{
"op": "core/column-addition-by-fetching-urls",
"description": "Create column ISBNdb Price at index 2 by fetching URLs based on column pISBN using expression grel:\"http://isbndb.com/api/v2/json/YOURKEY/prices/\"+toString(floor(value))",
"engineConfig": {
"mode": "row-based",
"facets": []
},
"newColumnName": "ISBNdb Price",
"columnInsertIndex": 2,
"baseColumnName": "pISBN",
"urlExpression": "grel:\"http://isbndb.com/api/v2/json/YOURKEY/prices/\"+toString(floor(value))",
"onError": "set-to-blank",
"delay": 5000
},
{
"op": "core/text-transform",
"description": "Text transform on cells in column ISBNdb Price using expression grel:if(toNumber(value.parseJson()[\"result_count\"])>0,sort(forEach(value.parseJson()[\"data\"],resp,resp.parseJson()[\"price\"]))[0],Null)",
"engineConfig": {
"mode": "row-based",
"facets": []
},
"columnName": "ISBNdb Price",
"expression": "grel:if(toNumber(value.parseJson()[\"result_count\"])>0,sort(forEach(value.parseJson()[\"data\"],resp,resp.parseJson()[\"price\"]))[0],Null)",
"onError": "keep-original",
"repeat": false,
"repeatCount": 10
}
]
@timtomch
Copy link
Author

This JSON can be applied to any OpenRefine project, by using the "Replaying Operations" feature.

Some things to consider

  1. This looks for a column called pISBN for ISBNs to search for. Change accordingly if the column containing ISBNs in your project has a different name.
  2. You will need to register with ISBNdb in order to get an API key. Replace the string YOURKEY above with the key you will receive.
  3. These operations do not apply any facets. They will therefore happily process all the rows in your file. If you have a large file, this is likely to exceed the number of calls to the API you are allowed to make in a day. Consider splitting your file first, e.g. by using facets.

What do these operations do?

The first operation queries the ISBNdb Prices API, searching by the ISBN provided in the input column (pISBN). The JSON data returned is then stored into a new column called ISBNdb Price.

The second operation processes the returned JSON data to compute the minimum price returned for each title. If the API call returned no result, the value is set to Null for this row.

Have fun!

@lmccoll44
Copy link

lmccoll44 commented Jun 3, 2021

Hello,

Is this JSON still valid? I'm getting a column returned with the name ISBNdb Price, but in each cell is this:

`

<title>301 Moved Permanently</title>

301 Moved Permanently

`

I did manually look up a few ISBNs in the ISBNdb search interface and it did return results. Thank you for any help you can give!

@timtomch
Copy link
Author

timtomch commented Jun 7, 2021

Hi. This is an old project and I haven't used ISBNdb in a while. It's possible that the API has moved to a different URI structure, which would explain the 301 error. Looking at the documentation, it seems that the API now lives at https://api2.isbndb.com and also is using HTTP authentication to control access, so the API key itself won't be enough.

This means you'd likely have to edit the API endpoint URI on line 15, but also will need to add HTTP authentication headers.
It looks like ISBNdb is no longer offering free access tiers to the service, so I can't test it out, sorry.

If you find a way to fix the issue, I'd be happy to update the gist!

@lmccoll44
Copy link

lmccoll44 commented Jun 7, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment