Generally, to encapsulate the logic for getting the best possible thumbnail for a resource, given the user's current known auth status for those resources. Anyone using manifesto or some other library that implements this can just call getThumbnail(..) and not worry about the details. Most of the time auth is not an issue, but if it is, they still don't need to worry about the details, getThumbnail(..) will figure it out and return the best thumbnail the user can currently see (which may be no thumbnail). Crucially, getThumbnail() assumes that the best thumbnail is the FASTEST one, within reason; that viewers favour speed over a precise size because they can scale an image in the browser.
Specifically, to help viewers generate a "field" of thumbnails as quickly as possible. Mirador, UV and other viewers all show large numbers of thumbnails at once. This can give a poor user experience, especially if viewers ask for arbitrary image sizes which are unlikely to be cached anywhere. Users will have a better experience if the viewer asks for thumbnails that are more likely to be returned by intermediate caches rather than trigger a new image server operation. IIIF allows you to provide an explicit thumbnail property, which could be an image resource or an image service. A getThumbnail(..) call will try to give you a thumbnail from the thumbnail property of the resource if possible. Manifest publishers can help this algorithm by including some image service details in the manifest, so that it avoids a request for each info.json.
Additionally, to return the user's current auth status on the image resources that the thumbnail represents, so that the viewer can provide visual clues to the user (e.g., a padlock overlay) even if a thumbnail is available:
Here we see 3 different statuses:
- no access and no thumbnail
- thumbnail, but you're not authed for the image - if you click on it you will be asked for credentials
- all fine
(not shown and maybe same as "all fine" - access controlled but you have an active token for the login service)
If the resources are subject to auth, this kind of display is only practical for manifests that include auth hints about their image resources. We don't recommend following info.json for each image to find this out. Maybe that's an optional parameter for getThumbnail(..), a viewer could opt to follow info.json for manifests of less that 20 canvases, or only for currently visible thumbnails (thumbnails in the viewport - although this could be many hundreds on a large display). Such decisions are not the concern of this algorithm.
If a viewer is using this algorithm, you can help by providing explicit sizes in the optional sizes array on an image service. This can be declared on any level of compliant service, it's not just for static images:
| Property | Description |
|---|---|
| sizes | "...or simply as a hint that requesting an image of this size may result in a faster response." |
You can also provide an explicit thumbnail service on your resource, which can include a sizes array directly in the manifest. If any of the resources are subject to auth, you can also include auth service information in the manifest. This doesn't mean you have to repeat the full auth services on each image; once asserted they can be declared by URI alone. See IIIF/api#526 (comment).
Some examples are provided in this manifest: (link to demonstration manifest - placeholder http://library-uat.wellcomelibrary.org/iiif/b17564980/manifest)
- No explicit thumbnail, no image service. This is the presentation API at its simplest. The algorithm can only return you the image resource.
- No explicit thumbnail, image resource with an image service. The algorithm must select an image from the image service, favouring explicit sizes if they are provided
- etc...
(Highlight best practice - explicit thumbnail property on canvas, auth services by @id)
A very simple implementation of getThumbnail() on a resource would accept no parameters and return a URL. However, we need more than this to meet the requirements above.
Different viewers have different thumbnail needs at different times. The UV wants to request fairly large thumbnails because it allows the user to scale them up with a slider. A collection browsing application might want very small thumbnails to simulate a file system explorer. IIIF allows the publisher to provide a thumbnail service, offering a range of thumbnail sizes. A viewer needs to be able to specify the size of the thumbnail it wants.
There are two problems with this. Firstly, unless the resource has a thumbnail service that conforms to Image API profile level 1 or 2, they can't ask for any arbitrary image size. Secondly this would defeat the caching that we're trying to encourage. The getThumbnail(..) function should take a hint parameter, so a viewer can specify the size of the thumbnail it prefers, and some indication of leeway the algorithm can afford. If I would like a thumbnail 300px wide with 200px leeway, the algorithm could return me one that's 400px wide because that's the closest match it can make from a level 0, fixed size thumbnail service. If I want a thumbnail 346px wide with no leeway, the algorithm could detect that no such thumbnail is available on the provided fixed size thumbnail service. If the resource is a canvas, the algorithm could then go to the first unauthed image service that belongs to the canvas and return a URL for that precise image size. This is an extreme case; callers are encouraged to specify size range they will accept to give the algorithm more chance of matching one from a sizes array. Modern browsers scale images smoothly. For a thumbnail, it is better to get an image straight away that may be slightly larger than you need, than wait longer for one that is exactly the size you want.
A caller might provide the ultimate leeway - I don't care how big the thumbnail is, just give me one! This could return thumbnails for which the algorithm cannot determine the size, such as a bare URL:
"thumbnail": "/my-thumb.jpg"The parameters of getThumbnail need to convey a desired size, and a permitted range within which the actual size may fall. This and other possible settings suggest a thumbnailOptions object to carry the various values.
Assumption: viewers want to confine the thumbnail to a box. A viewer doesn't want to calculate the precise height and width it needs to ask for each image, that's what getThumbnail should be doing. The size passed to getThumbnail is to be interpreted as a bounding box for the thumbnail.
Often this bounding box will be square - but that doesn't mean we want to force a square image. However, IIIF 2.1 explicitly allows a server to return a square image (see http://iiif.io/api/image/2.1/#region), and getThumbnail should support that. If enough information is available in the manifest it could also be used to generate square thumbnails from image servers that don't support the square region syntax (i.e., most of them at time of writing).
Given the above discussion, here's a typical invocation that my used in generating a field of thumbnails like this:
I don't want to force square thumbnails, but I don't know anything in advance about the shape of the images. In this case I still enforce a square bounding box, and this may be fairly common, but a bounding box can be any rectangle.
var thumbnailOptions = {
square: false,
size: { width: 300, height: 300 },
minimum: { width: 200, height: 200 },
maximum: { width: 500, height: 500 },
followInfoJson: false
};
var thumbInfo = getThumbnail(thumbnailOptions);| thumbnailOptions | |
|---|---|
| square | Force a square thumbnail if possible. This will use the IIIF Image API 2.1 square region if available and if necessary. It might alternatively result in a constructed IIIF Image API URL that will return the correct square region. Note - will leave out logic for square=true from first draft. |
| size | A bounding box that getThumbnail will try to match as closely as possible. |
| minimum | The smallest bounding box for which a fixed size thumbnail is acceptable. If the algorithm can't obtain a thumbnail larger than this size (>=) it should fall back to constructing an Image API URL for the requested size. If null, you don't mind. |
| maximum | The largest bounding box for which a fixed size thumbnail is acceptable. If the algorithm can't obtain a thumbnail smaller than this size (<=) it should fall back to constructing an Image API URL for the requested size. If null, you don't mind. |
| followInfoJson | If there is insufficient information in the manifest to return an image that we are confident will work, the URL returned will be speculative (might result in a broken image). This is especially relevant for authed images where the manifest has no hints about the auth services. However, it's preferable to be fast and we assume content is open. When generating fields of thumbnails, expect to set this to false. |
Although we want the URL of the thumbnail back, we need more than this to drive the kind of UI seen in the first image.
The object returned from getThumbnail(..) is a thumbInfo:
{
url: "http://example.org/iiif/identifier/full/400,300/0/default.jpg",
width: 400,
height: 300
}This is a simple return value when no auth information is available to the algorithm (either because the resources are not access controlled, or that the manifest doesn't mention any auth services and followInfoJson was false in the thumbnailOptions). The URL is not guaranteed to work, but we can expect it to unless the images are access controlled. Typical IIIF open content doesn't need anything more than this.
If the manifest includes auth service information, then we can return an extra authInfo object. The presence of this object does not guarantee it is accurate about access, because the information in the manifest may be wrong and it might not be following info.json links. But it is reasonable to build user interface based on the values returned.
If the resource we called getThumbnail(..) on was a canvas, and auth services are declared in the manifest (or followInfoJson was true) then we can also return information about the auth status of the user for each of the images of the canvas. This isn't whether the user can see the thumbnail - it's whether the user can make requests to the image services belonging to the canvas the thumbnail represents. This is required to build the padlock overlays shown in the first example image.
{
url: "http://example.org/iiif/thumbs/identifier/full/400,300/0/default.jpg",
width: 400,
height: 300,
// source: "http://example.org/iiif/thumbs/identifier",
authInfo: {
betterThumbnailAvailable: false,
images : [
{
id: "http://example.org/images/identifier.jpg",
// canViewResource: true, // later we could also return this
hasImageService: true,
canUseImageService: true
}
]
}
}This shows the typical case where the canvas has one image.
Note: certain possible complex scenarios have been swept under the carpet at the expense of keeping the authInfo object simple. Images resources (not services) might themselves be subject to access control, and it might be different from the access control on the image service... It's pointless to try and bake all this into getThumbnail(..) because although possible, such scenarios will be rare and not a good idea in general. Viewers should expect to handle them, because they can always fall back to having to determine image statuses on demand when the user tries to interact with each image. The getThumbnail(..) function should aim to cover 99% of IIIF manifests, not all possible ones.
Note: I did think about returning, for each image in the array, the image service id, the login service id and the current auth token the viewer has stored for this service (and maybe its TTL too), as we would have looked at all of these to determine the value of hasImageService and canUseImageService. But it was too much bloat, the viewer can get these all again when it needs them.
| thumbInfo | |
|---|---|
| url | The URL of the thumbnail. This might not always be a IIIF Image API - the thumbnail property might be an image resource without a service, and still pass the bounding box requirements. |
| width, height | The actual dimensions of the image returned. Sometimes this information is in the url property, but not always. It's also possible for getThumbnail(..) to return a URL and not know these dimensions, if you didn't specify a minimum or maximum and a plain URL was on offer as a thumbnail property. So these properties might be missing. |
| source | (Don't know if we need this) The @id or value of the resource in the manifest that was used to generate url. Might be helpful but maybe not. For example, the @id of a thumbnail image service, the @id of an image resource, the URI of a plain thumbnail. |
| authInfo | Extra information that could be used to present visual clues about access control. The absence of an authInfo may mean that the viewer doesn't know about access control on the resource, not that there is no access control. If there is no authInfo it should go ahead and load the thumbnail, because it has no access to any more information about whether it will succeed. |
| authInfo | |
|---|---|
| betterThumbnailAvailable | This thumbnail can be upgraded if the user interacts with a login service. *Note - I have this as a boolean, but its value could be the login service the user needs to interact with. Or it could stay a boolean and the viewer looks at source for a login service. That's problematic because the better thumbnail might be available from a different resource in the manifest from the current source of this thumbnail. We could just not return this at all. Does it help the viewer to know this? * |
| images | Only returned when getThumbnail(..) was called on a canvas (which is of course the primary use case). For each image that annotates the canvas (usually only one) we give the viewer enough info to generate hints in the user interface. |
| image | |
|---|---|
| id | The @id of the image annotation, so the viewer can match it to the manifest |
| hasImageService | Whether this image annotation has a IIIF image service. As mentioned above we'll stay silent on images that are just image resources to keep things simple |
| canUseImageService | Whether the current user can request images from that service, as of "now" - i.e., if true, the current user has a valid, unexpired auth token in the viewer's auth token cache for a login service asserted on the image service (there may be more than one). This is where the optimisation of asserting auth services in the manifest pays off, because unless followInfoJson is set to true, a viewer cannot know this unless the manifest carries the information. The viewer is expected to keep track of tokens it acquires, and servers are expected to give a time-to-live on the auth token that corresponds to a cookie auth token. A viewer is free to reacquire an auth token for a service whenever it wants to update this cache, which may renew the ttl (or may not, it's up to the server). |
One calling pattern a viewer might want to use is to call getThumbnail with the desired thumbnail size, and if that call fails to return a thumbnail, call it again without any parameters. Then you'll at least get something. This is not expensive (unless info.json is being dereferenced), you're just processing the manifest.
Notes:
squareis currently ignored completely. We'll get to that later.- The algorithm tries to get a good match from the thumbnail property, if supplied. The thumbnail property could be a plain URI, an image resource, or an image resource with a IIIF image service. It STRONGLY FAVOURS getting a thumbnail from the thumbnail property. Unless the resource is a canvas, it can't get it from anywhere else.
- If we can't get the required match from the thumbnail property and the resource is a canvas, then try the image annotations of the canvas (what the UV does now - it attempts a thumb from the first).
var thumb = resource.getThumbnail(options);
Pseudocode - needs to be refactored into units
getThumbnail(options) // returns thumbInfo
{
var thumbInfo; // unassigned
if (resource.thumbnail) {
1) shortcut if caller doesn't care about size:
IF options.size, options.minimum and options.maximum are all empty,
and resource.thumbnail is a plain URI or is an array that contains
a plain URI:
thumbInfo = { url: (the plain URI or first plain URI) }
CONTINUE
2) Any other values of size, minimum and maximum require knowledge of image
dimensions, which requires an Image Resource. We will ignore multiple image resources -
if the manifest publisher has several thumbnails they should state that as a service
with a sizes array.
var thumbImageResource = (first image resource in thumbnail property/array)
thumbInfo = getThumbfromImageResource(thumbImageResource, options);
It's possible that thumbInfo is still null (no thumbnail available). It might still get
populated in the next section of code.
}
if(resource is canvas) {
Here we do two things
1) try to populate the thumbnail url, width and height if we haven't managed to get one so far.
2) report on the auth status of each image if we know it.
We should not return anything in authInfo unless we KNOW it (but we can assume we know
from manifest hints alone if followInfoJson=false)
var images = []; // use this to hold the hint information about each image in the canvas
for each imageResource in canvas.images
{
if(thumbInfo is null) {
// We haven't been able to get a thumbnail yet
thumbInfo = getThumbfromImageResource(imageResource, options);
// still might be null, but might get populated next time round the loop
}
if (imageResource has IIIF image service) {
var loginServices = (get declared login service(s) for service)
if(no loginServices found AND followInfoJson) {
loginServices = (dereference info.json and check for services - but don't do any auth with user)
}
if(one or more loginServices) {
// we only say anything about an image if we know it has auth services.
var image = {
id: imageResource.@id,
hasImageService = true, // In our first, simpler impl of this logic this will always be set to true here
canUseImageService = hasValidToken(loginServices)
}
images.push(image);
}
}
} // end for each imageResource
if(images.length > 0)
{
if(thumbInfo is null) // still possible !
{
// need to create one to hold the images
thumbInfo = {}
}
if(thumbnail.authInfo is null)
{
thumbInfo.authInfo = { images: images}
}
else
{
thumbnail.authInfo.images = images;
}
}
}
RETURN thumbInfo
}
A regular IIIF image resource, which may or may not include an IIIF Image Service. We allow for the possibility that the resource and the service could both have auth services Note: this mechanism is how we'll provide open small thumbs but closed large thumbs: the small thumb is the thumbnail resource and the larger thumbs are its service.
getThumbfromImageResource(imageResource, options) // return null if no thumbnail possible
{
var imageService = imageResource.imageService // (the first service property with a recognised IIIF image service profile)
// it's possible the image resource does not have a service
// we need the profile, which MUST be present in the manifest, and the sizes array, which might not be.
if((followInfoJson is true) AND (imageService is not null) AND (sizes array not in imageService OR auth services not in imageService))
{
imageService = (dereference info.json into what we hope is a more detailed service object)
}
// only Wellcome will have this at the moment - it's an IxIF thing but necessary for protecting image resources
var knownLoginServicesForResource = (gather login services from resource itself - this is the binary auth which MUST be in the manifest)
var canSeeResource = knownLoginServicesForResource.length=0 OR hasValidToken(knownLoginServicesForResource);
// more common, probably
var knownLoginServicesForService = (gather login services from imageService) // in any sane implementation they would be the same as the resource services
var canUseService = knownLoginServicesForService.length=0 OR hasValidToken(knownLoginServicesForService);
var bestThumb; // (a thumbInfo object)
var bestThumbUserCanSee; // (a thumbInfo object)
int currentScore = -1;
// first see what the resource itself scores - assuming the manifest publisher has included its sizes
if(imageResource.width > 0 && imageResource.height > 0)
{
var imageResourceSize = new { width: imageResource.width, height: imageResource.height }
currentScore = getMatchingScore(imageResourceSize, options);
if(score >= 0) {
bestThumb = new {
url: imageResource,
width: imageResource.width,
height: imageResource.height
}
if(canSeeResource) {
bestThumbUserCanSee = bestThumb;
}
}
}
// See if we can get a better size from the sizes array
if(imageService is not null AND imageService has a sizes array) {
for each size in imageResource.imageService.sizes
{
var score = getMatchingScore(size, options);
if(score >= 0 and score <= currentScore)
{
bestThumb = new {
url: (construct canonical image url from imageService @id and size),
width: size.width,
height: size.height
}
if(canUseService) {
bestThumbUserCanSee = bestThumb;
}
}
}
}
// at this stage we've used up our suggested thumbnails, are we able to return anything yet?
if(bestThumbUserCanSee is not null)
{
if(bestThumbUserCanSee != bestThumb) {
bestThumbUserCanSee.authInfo = new { betterThumbnailAvailable: true }
}
RETURN bestThumbUserCanSee;
}
// if we haven't got a thumbnail so far, try to the service directly
if(imageService has a profile that supports sizeByForcedWh, or explicit "supports" is listed for sizeByForcedWh )
{
// TODO - we can be more subtle and allow for missing heights or widths independently rather than the whole size
// This allows the viewer to have a bounding height or bounding width alone, rather than a bounding box.
// We can then use /w,/ or /,h/ syntaxes (assuming the service supports the relevant ops)
// for now we'll assume that if a size is given it has w and h, and we need the server to support /!w,h/ (sizeByForcedWh)
var sizeToUse = options.size
if(options.size is not specified)
sizeToUse = (options.maximum || options.minimum) // favour maximum
if(sizeToUse is null) { // still null!
sizeToUse = (default thumbnail size from config)
}
if(canUseService) {
RETURN new {
url: (construct canonical url from imageService.id and sizeToUse),
width: sizeToUse.width,
height: sizeToUse.height
}
} else {
// return an empty thumb, with a hint
RETURN new { authInfo: new { betterThumbnailAvailable: true } }
}
}
// TODO: As a last resort, we could try to use the smallest possible tile as the thumbnail.
This would allow a thumbnail to be returned for completely static level 0 image pyramids even when an
explicit thumbnail has not been provided.
We need to generate the canonical URL for this tile. Do we know its dimensions?
}
The lowest possible positive score is the best match. A negative result is NOT a match. 0 is a perfect match.
getMatchingScore(sizeToTest, options)
{
var minimum, maximum, reqdSize;
minimum = options.minimum || { width: 0, height: 0 };
maximum = options.maximum || (maximum from config);
reqdSize = options.size || maximum;
// first try to eliminate this size
if(sizeToTest.width < minimum.width || sizeToTest.height < minimum.height)
RETURN -1;
if(sizeToTest.width > maximum.width || sizeToTest.height > maximum.height)
RETURN -1;
// sizeToTest is in the permitted range
dWidth = abs(sizeToTest.width - reqdSize.width);
dHeight = abs(sizeToTest.height - reqdSize.height);
// This is where this is naive; very wide or very tall images may not give the results we want
// RETURN dWidth + dHeight; // is that actually better?
RETURN dWidth * dHeight;
}
hasValidToken(loginServices) {
// loginServices will usually be length=1
RETURN (user has current valid token for at least one of supplied loginServices, from viewer token cache)
}
