sul-exhibits-bib-201703.md

SUL Exhibits Bibliographies

For background, see this GitHub issue -- it has the UX design in there.

User Action/Data Required

1. The exhibit users are required to manage Zotero bibliography libraries as the database for an exhibit item's bibliography information.
2. They are also required to manage the "synchronization" between their exhibit and Zotero (via the Exhibit Settings > Bibliography).

Library

Our application makes the following assumptions about the Zotero library:

1. That the Zotero library is made public (so that we don't need to authenticate to use the Zotero API).
2. That the Zotero library items are tagged with zero or more druids (so we can associate druids with their bibliographies).
3. That the Zotero library does not use nested collections or items (aka "relations").
4. That there is a single Zotero library per exhibit

User/Group Identity

The "Service Endpoint" will always be:

• https://api.zotero.org/users/#{USER_ID} or
• https://api.zotero.org/groups/#{GROUP_ID}

So, the data required is the Zotero user id (or group id). We need to know whether the type of the Zotero ID is a user or a group, so we'll need a type pulldown for User or Group in the UX (or alternatively, we can require that they use a Zotero group).

To get this id the exhibit user will need to copy it from the Zotero "API" section of the "Settings" interface. (Though the group ID may only be available in the Zotero URL -- we will need to document this.)

This is how to make the (entire) library public on the Zotero dashboard (via "Settings"), and the "Feeds/API" tab is where you can copy the user id:

Tagging Bibliography Items

The exhibit user associates an exhibit item (key = druid; e.g., aa111bb2222) to zero or more Zotero bibliography items by adding "tags" to the bibliography items in the Zotero UI. The tags must contain the exhibit item keys (e.g., aa111bb2222). Here's what the Zotero UI looks like for adding tags to a bibliography item:

NOTE the tags can include the druid in parenthesis if desired -- for example, tag = My favorite book (aa111bb2222).

Requirements for Bibliography service (internal to our application)

Zotero has a public Web API. Note that I couldn't find an existing Ruby gem that supports the API portions that we need.

Retrieving all biblography items for a given tag in a Zotero library

This is how you use the Zotero API to retrieve the first page of items for a given tag (which is a druid value in our case) with the styling information:

bibliography = []
url = "https://api.zotero.org/users/#{USER_ID}/items" + '?' +
      URI.encode_www_form(v: 3,
                          format: :json,
                          tag: 'aa111bb2222', # the exhibit item's druid
                          include: :bib,      # only return the styling
                          start: 0,
                          limit: 100)
open(url) do |conn|
  bibliography += JSON.parse(conn.read)
end

# (optional) remove data we don't care about
bibliography.collect! do |item|
  item.keep_if {|k,_v| %w(meta bib).include?(k) }
end

NOTES

• the example retrieves only the first page, but to retrieve all items, you need to use pagination with the start and limit parameters.
• the example uses the include=bib parameter so that we download the Zotero styling for each item -- the default styling is chicago author-date which is what we want.
• the example uses the format=json API because some of our bibliographies will be greater than 150 items, which is the hard limit for a format=bib request.

Retrieving all of the tags in a bibliography

The Zotero API provides for a simple API call to fetch all of the tags in a user's library -- /users/1/tags.

Sorting the bibliography

The Zotero API does not support Author-Date sorting when using the search-by-tag API. So, we will need to sort the bibliography by author-date ourselves.

The API provides for a meta hash that has both the author and date, so sorting can be done like so:

def author_date(item)
  [item['meta']['creatorSummary'], item['meta']['parsedDate']].join
end
bibliography.sort! do |x, y|
  author_date(x) <=> author_date(y)
end

Zotero service outages

We can live with Zotero service outages (e.g., 503 response codes). We need only display a message to the user saying the bibliography is unavailable at this time.

Rendering the bibliography

The heading for the bibliography is an option in the settings so we need to get that from the database.

The Zotero API provides the styling as HTML, so we need only add our own custom CSS entries if desired (for csl-bib-body and csl-entry). The styling should be in Chicago Author-Date.

<div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Abby, Jane. &#x201C;My Thesis,&#x201D; 2010.</div>
</div>
<div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Allen, John. <i>Another Book</i>, 2017.</div>
</div>
<div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Doe, Jane. <i>Yet Another Book</i>, 2001.</div>
</div>
<div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Doe, Jane. <i>My Book</i>, 2017.</div>
</div>

Documentation Needed

• User guide for how to manage a Zotero library for Spotlight exhibits
• User guide for the "Sync with Zotero" workflow

Technical design for SUL-Exhibits codebase

• Need "Sync with Zotero" button in Settings > Bibliography that fires a job that renders all the bibliographies for all items. This functionality would:

   For all items in the exhibit,
     Retrieve all bibliography entries for a given exhibit item (may be 100s)
     Sort the entries by Author-Date
     Store the bibliography as HTML (using the styling from Zotero) in the Solr document
     If there's no bibliography entries for this item, then Store an empty string (or delete?) the Solr field

Open Issues

1. NO: Can this be developed as a plugin?
2. NO: Should we do realtime Zotero API calls on item display, or cache/refresh that data?
3. See ticket: Need revised UX wire-frame for admin UI to address only user/group id.
4. How/where do we get test fixtures for a Zotero bibliography?
5. YES: Can we make this into a bibliography "indexing" mechanism where it refreshes the bibliography data for all exhibit items at the click of a button in the Exhibit Settings? That way we don't have to change the viewers or add a route -- it would just be another solr field.
6. YES Does the bibliography text need to be searchable? That is, if I enter a keyword that matches a bibliography, should I see the matching exhibit items?
7. See UX What information/feedback do we need to give the curator for the "Sync with Zotero" button? A status bar? A message? Statistics on the bibliographies? The last time the bibliography was sync'd? others?
8. OK Need to test that a Zotero tag can include a typical title with a druid in parens.

Example Code

See https://gist.github.com/drh-stanford/d8e0080c47f0d37acc8456eca57a279f

This all looks pretty good. A few comments.

Open Issues

1. I don't think we should build this as a plugin. Given that this is modeling a workflow that we're defining for the content creators using this feature, this doesn't seem to me to be generally applicable. If there is a desire by others to have this kind of behavior, the code will be open sourced, and we can turn it into a plugin of the use-case presents itself.
2. In my initial thoughts about this, I'm imagining that we'll fetch this data via an ajax call to a local controller and cache that data (for x days) so we don't have to re-fetch frequently (perhaps @BLABritton can weigh in on how often these are updated and how often we'll want to bust the cache).
3. I think this should be relatively straight forward actually. I'm sure @ggeisler can give us input.

Spotlight integration

Given that this doesn't really appear to be generally applicable, I don't think we really need to worry about it there (however; we may want to make a change there that can allow us to tap into the admin menu system). I believe that at this point, this code would be best served to live in the exhibits codebase. As an interim solution, we can override the partial that renders the admin menu to add our link. Once we know how we want to inject this link into the menu, this may give us a better idea of how this change could be accomplished without overrides (or we can if we're happy).

This gives the code that we need to write in Exhibits a bit of a shape. We will need to (initially at least) override the admin menu partial in exhibits (to add a link to our new admin UI). We will need a model/controller/view for the admin/configuration side of things. We will also need a model/controller/view/js for fetching the data on the UI side. Sprinkle a little i18n in there for our labels and defaults and we're pretty much good to go.

Just a quick addition to this: Parker Library has a group already (id: 1051392). I am in the process of populating that group's library with all of the current Parker Bibliography (not yet tagged) which we can use to test with when ready.

For @jkeck's question about #2: these are likely to be updated on a monthly basis. The Parker team will do periodic additions to the bibliography and then do a one-time re-fetch. Could potentially cache for a long time, breaking the cache only on re-fetch?