Skip to content

Instantly share code, notes, and snippets.

@singaltanmay
Last active August 18, 2021 10:20
Show Gist options
  • Save singaltanmay/24006ed15ad899bdd83be0e93bf5a491 to your computer and use it in GitHub Desktop.
Save singaltanmay/24006ed15ad899bdd83be0e93bf5a491 to your computer and use it in GitHub Desktop.
GSoC Work Report - 2021

Implement IIIF read in Cineast - GSoC 2021

Table of Contents

About the organisation

With the tremendous increase of video recording devices and the resulting abundance of digital video, finding a particular video sequence in ever-growing collections is a major challenge. Existing approaches to retrieve videos mostly still rely on text-based retrieval techniques to find desired sequences. vitrivr is an open source full-stack content-based multimedia retrieval system with a focus on video. Its modular architecture makes it easy to also search audio, images, 3D Models and structured data.

About the project

The International Image Interoperability Framework (IIIF) describes a standard to interact with images that are being adopted by an increasing number of museums, archives and other institutions that store large collections of digital images. The aim of this project is to make vitrivr compatible with the APIs specified by IIIF so that it can be used to interact with any IIIF capable data sources easily. The primary goal of this project is to allow Cineast to download files from an IIIF compliant server and extract features through these files using the extraction service. Cineast is a multi-feature content-based multimedia retrieval engine. It is capable of retrieving images, audio- and video sequences as well as 3d models based on edge or colour sketches, sketch-based motion queries and example objects. Cineast is written in Java and uses CottontailDB as a storage backend. Cottontail DB is a column store aimed at multimedia retrieval. It allows for classical boolean as well as vector-space retrieval, i.e., nearest-neighbours lookup, as used in similarity search. implementing IIIF support will open up Cineast to a multitude of IIIF compliant servers such as Mirador, TIFY and IIPMooViewer.

Code

All source code is available in the vitrivr/cineast and vitrivr/vitrivr-ng repositories on GitHub.

Team

  • Tanmay Singal
  • Florian Spiess - Mentor
  • Luca Rossetto - Mentor

Project Structure and Features

Most of my work in the project was focused on downloading images in Cineast from IIIF compliant servers during the extraction process. To achieve this, I implemented compatibility with Image API 2.1.1, Image API 3.0 and Presentation API 2.1.1 servers. Cineast now supports performing extraction on IIIF resources hosted on compatible media servers. During the feature extraction process, Cineast will automatically download all the resources specified in the extraction config file to a folder on the local filesystem and perform extraction on it.

Project Structure Diagram

Features

Overview

Screenshot 2021-08-18 at 3 33 03 PM

Configuring an IIIF Job

Screenshot 2021-08-18 at 3 33 09 PM

Extraction Command

Screenshot 2021-08-18 at 3 33 15 PM

Image API

Screenshot 2021-08-18 at 3 33 19 PM

Presentation API

Screenshot 2021-08-18 at 3 33 24 PM

Change Discovery API

Screenshot 2021-08-18 at 3 33 28 PM

IIIFMetaDataExtractor

Screenshot 2021-08-18 at 3 33 33 PM

Vitrivr-NG

Screenshot 2021-08-18 at 3 33 37 PM

Vitrivr-NG OpenSeaDragon Viewer

Screenshot 2021-08-18 at 3 33 40 PM

Vitrivr-NG IIIF Metadata

Screenshot 2021-08-18 at 3 33 43 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment