Rendering thumbnails might be the activity that puts most stress on servers if not given some care both on server and in viewer.
We have to remember that IIIF consumers are under no obligation to do what you want them to do. It’s not necessarily your viewer looking at the endpoints. While you can arrange a careful honourable agreement between client and server if it’s your viewer looking at your server, other viewers won’t know the rules – or may choose not to play by them.
An uncaring viewer app takes no notice of any hints, recommended sizes or explicit thumbnails declared in the manifest and goes straight to the first image service it finds for a canvas and requests an image in the size it thinks appropriate for a thumbnail. It doesn’t know whether this is an optimum size for the server to produce, or whether the server has this cached.
This is what the our Viewer is doing currently. If generated images are cached then over time the performance of thumbnails for an already “warmed up” work would be no different for the "uncaring" client than for a polite client that only hit advertised thumbnail URIs, because in neither case would the image server be troubled; a reverse proxy somewhere would serve up the image. But in the case of a never-before-viewed work (something LIKELY in a library scenario), a viewer that can show a screen full of thumbnails may unleash 100 image requests each of which involves requesting a thumbnail generated from a different source image – 100 JP2s or TIFFs would be opened for processing. An Image Server like IIPImage can handle 100 tile requests for different tiles from the SAME image a lot more efficiently than it can handle 100 tile requests from different source images. Even a cluster of image servers is going to have its work cut out to load and extract the thumbnails from 100 source images.
And what if you have hundreds of concurrent users looking at different works?
In a small collection of endpoints, or a large collection of single image items (like an art gallery) this problem is not so significant. But a National Library exposing vast swathes of its catalogue in this way is going to encounter huge numbers of requests that it won't have cached, or hasn't seen for years.
To make it easier for clients and servers to adopt sensible policies, IIIF allows the manifest to give clues to the viewer about thumbnails. Of course, a viewer is under no obligation to pay any attention, but we have to assume that unless they are intent on a deliberate denial of service attack, viewers want to give their users the best possible experience. The speed at which a screen full of thumbnails gets displayed is a big part of the user experience. So a client viewer should take account of what the manifest suggests. Also, explicit thumbnails can give a user a specific image – if there were multiple images on a canvas how is the viewer to know the appropriate one to generate a thumbnail from?
IIIF gives several ways to help us.
Before we even get to the “thumbnail” property, we can add information to a service description that lists suggested sizes:
{
"@context" : "http://iiif.io/api/image/2/context.json",
"@id" : "http://www.example.org/image-service/abcd1234/1E34750D-38DB-4825-A38A-B60A345E591C",
"protocol" : "http://iiif.io/api/image",
"width" : 6000,
"height" : 4000,
"sizes" : [
{"width" : 150, "height" : 100},
{"width" : 600, "height" : 400},
{"width" : 3000, "height": 2000}
],What this actually means is up to the server. But it is supposed to convey to a viewer the following:
A set of height and width pairs the client should use in the size parameter to request complete images at different sizes that the server has available. This may be used to let a client know the sizes that are available when the server does not support requests for arbitrary sizes, or simply as a hint that requesting an image of this size may result in a faster response. A request constructed with the w,h syntax using these sizes must be supported by the server, even if arbitrary width and height are not.
In our case this could mean that we’ve pre-warmed the cache with these images, or have some other way of serving them faster than an arbitrary size. In the case of our Viewer’s sliding-scalable thumbnails, we could decide that we request the 600x400 image as the thumbnail and then scale it in the client.
The viewer is still using the raw image service for its thumbnails, but it’s at least taking note of its suggestion. But IIIF does have an explicit thumbnail property:
A small image that depicts or pictorially represents the resource that the property is attached to, such as the title page, a significant image or rendering of a canvas with multiple content resources associated with it. It is RECOMMENDED that a IIIF Image API service be available for this image for manipulations such as resizing. Usage:
- A manifest SHOULD have a thumbnail image that represents the entire object or work.
- A sequence MAY have a thumbnail and SHOULD have a thumbnail if there are multiple sequences in a single manifest. Each of the thumbnails SHOULD be different.
- A canvas MAY have a thumbnail and SHOULD have a thumbnail if there are multiple images or resources that make up the representation.
- A content resource MAY have a thumbnail and SHOULD have a thumbnail if it is an option in a choice of resources.
In our case we’re most concerned with thumbnails on Canvases, but any node may have a thumbnail. For instance thumbnails on Ranges could be used to build a visual table of contents.
A thumbnail may be a straightforward image URI:
"thumbnail": "http://www.example.org/images/book1-page1/thumb.jpg"Or it could be an IIIF image service in its own right:
"thumbnail": {
"@id": "http://www.example.org/images/book1-page1/full/80,100/0/default.jpg",
"service": {
"@context":"http://iiif.io/api/image/2/context.json",
"@id":"http://www.example.org/images/book1-page1",
"profile":"http://iiif.io/api/image/2/level1.json"
}
}To protect its own resources a manifest could provide a simple URI for a thumbnail property, maybe pointing to some pre-prepared images residing in Amazon S3 buckets or on a CDN. This would be safest, but runs the risk of being too limiting – a viewer might want to implement variable size thumbnails, and finding the supplied thumbnails too small for this purpose bypass the thumbnail property and fetch them straight from the image service.
If the alternative is to reproduce the image service on the thumbnail property, then we haven’t really gained anything.
One option here is to still use an IIIF image service as the thumbnail property, but make it a Level0 compliant profile and supply a sizes array. From this service it would be legitimate to throw an error for any request that isn’t one of our preferred sizes.
{
"@id":"http://www.example.org/iiif/book1/canvas/p1",
"@type":"sc:Canvas",
"label":"p. 1",
"height":1000,
"width":750,
"images": [
{
"@type":"oa:Annotation",
"motivation":"sc:painting",
"resource":{
"@id":"http://www.example.org/iiif/book1/res/page1.jpg",
"@type":"dctypes:Image",
"format":"image/jpeg",
"service": {
"@context": "http://iiif.io/api/image/2/context.json",
"@id": "http://www.example.org/images/book1-page1",
"profile":"http://iiif.io/api/image/2/level1.json"
},
"height":2000,
"width":1500
},
"on":"http://www.example.org/iiif/book1/canvas/p1"
}
],
"thumbnail": {
"@id": "http://www.example.org/images/book1-page1/full/200,150/0/default.jpg",
"service": {
"@context":"http://iiif.io/api/image/2/context.json",
"@id":"http://www.example.org/images/book1-page1-thumbservice",
"profile": [
"http://iiif.io/api/image/2/level0.json",
{
"formats" : [ "jpg" ],
"qualities" : [ "color" ],
"supports" : [ "sizeByWhListed" ]
}
"width" : 2000,
"height" : 1500,
"sizes" : [
{"width" : 200, "height" : 150},
{"width" : 400, "height" : 300},
{"width" : 800, "height" : 600}
],
}
}In this example of a canvas, the canvas’s image service is a fully level 2 compliant service. But the thumbnail is only obliged to return colour jpegs in the sizes listed, and is entitled to throw an error if asked for anything else.
This may be valid, but is it too complex?
In our Viewer, are we making it IIIF compliant in the sense that it will handle anything we know the British Library are going to throw at it, or are we going to ensure that it handles ALL the spec, that it will be able to deal with anything ANY IIIF 2.0 compliant server gives it, and furthermore is clever enough to look for thumbnail optimisations like the one above?
Here's where a relatively complex algorithm could be hidden behind an IIIF API:
var thumbUri = canvas.getThumbnail(preferredSize);
This is a condensation of some issues we've been throwing around for Mirador as well. We handle it by trying to be gentle on the server where we can, like lazy-loading thumbnails as they ought to become visible, but these are all still major bottlenecks.
The Seadragon people have described a "deep zoom collections" support, and that community is likely to jump on that now that the Stanford/Harvard-funded collections work has been merged back to master. The deep zoom collection does a number of things, but one function addresses this problem in a way we've been discussing speculatively for Mirador/IIIF.
The major overhead for thumbs, as you point out, is the sheer number of requests required, compounded by the need to do separate disc reads for each of potentially hundreds of thumbnails from potentially thousands of clients within a very short time (<200ms). Web developers, applying techniques used in 2D game programming, have solved this kind of problem in the past with "spritesheets", which are basically one very large image made up of many smaller ones whose coordinates are described somehow. This allows the client to download all images in a single request for a single, medium-sized image. The client can then move the image around and crop it as necessary.
There is no mention of anything like this in the image API, but it would be very useful for the kind of viewers that the community wants, which should allow the user to quickly navigate the whole object. It might go a certain distance toward addressing the problems you raise, but of course, not every server would support it, and we would still want a way to manage the problem without a construct like "collections" or "spritesheets".