- Implement IIIF read in Cineast - GSoC 2021
With the tremendous increase of video recording devices and the resulting abundance of digital video, finding a particular video sequence in ever-growing collections is a major challenge. Existing approaches to retrieve videos mostly still rely on text-based retrieval techniques to find desired sequences. vitrivr is an open source full-stack content-based multimedia retrieval system with a focus on video. Its modular architecture makes it easy to also search audio, images, 3D Models and structured data.
The International Image Interoperability Framework (IIIF) describes a standard to interact with images that are being adopted by an increasing number of museums, archives and other institutions that store large collections of digital images. The aim of this project is to make vitrivr compatible with the APIs specified by IIIF so that it can be used to interact with any IIIF capable data sources easily. The primary goal of this project is to allow Cineast to download files from an IIIF compliant server and extract features through these files using the extraction service. Cineast is a multi-feature content-based multimedia retrieval engine. It is capable of retrieving images, audio- and video sequences as well as 3d models based on edge or colour sketches, sketch-based motion queries and example objects. Cineast is written in Java and uses CottontailDB as a storage backend. Cottontail DB is a column store aimed at multimedia retrieval. It allows for classical boolean as well as vector-space retrieval, i.e., nearest-neighbours lookup, as used in similarity search. implementing IIIF support will open up Cineast to a multitude of IIIF compliant servers such as Mirador, TIFY and IIPMooViewer.
All source code is available in the vitrivr/cineast and vitrivr/vitrivr-ng repositories on GitHub.
- Tanmay Singal
- Florian Spiess - Mentor
- Luca Rossetto - Mentor
Most of my work in the project was focused on downloading images in Cineast from IIIF compliant servers during the extraction process. To achieve this, I implemented compatibility with Image API 2.1.1, Image API 3.0 and Presentation API 2.1.1 servers. Cineast now supports performing extraction on IIIF resources hosted on compatible media servers. During the feature extraction process, Cineast will automatically download all the resources specified in the extraction config file to a folder on the local filesystem and perform extraction on it.
