Vitrivr Setup

vitrivr-setup.md

vitrivr / cineast setup guide

This guide is for developers interested in contributing to vitrivr's retrieval engine Cineast.

Quick setup

1. Have a JDK (Java 1.8 or above) ready. We recommend OpenJDK 11.

2. Quick Setup of Cineast:

3. Clone cineast, checkout the multi-module branch and update the submodules

$> git clone https://github.com/vitrivr/cineast.git && cd cineast \
&& git checkout multi-module && git submodule update --init --recursive

2. Build the executable JARS

$> ./gradlew clean generateProto fatJar

3. Quick Setup of CottontailDB - vitrivr's storage layer

4. Clone cottontaildb and

$> git clone https://github.com/ppanopticon/cottontaildb.git && cd cottontaildb \
&& git submodule update --init --recursive

2. Build cottontaildb and generate the distribution

$> ./gradlew clean generateProto distZip

3. Extract the distribution

$> unzip build/distribution/cottontaildb-1.0-SNAPSHOT.zip && \
mv cottontaildb-1.0-SNAPSHOT/* . && rm -r cottontaildb-1.0-SNAPSHOT

4. Start cottontaildb

$> ./bin/cottontaildb config.json

Windows users beware! Adopt the configuration for windows beforehand by changing the forceUnmapMappedFiles property to true.

Prerequisites

• Cineast is written in Java and hence, a JDK of at least version 8 is required. We recommend the OpenJDK 11.
• Have an IDE. We recommend using the amazing IntelliJ IDEA.
• Git

Basic Setup (of Development Environment)

1. Clone cineast and navigate to its location:

$> git clone https://github.com/vitrivr/cineast.git && cd cineast

2. Switch to the multi-module branch:

$> git checkout multi-module

3. Initialise and update the submodules for the storage components:

$> git submodule update --init --recursive

4. Open build.gradle with your IDE. This requires the IDE supports [Gradle].
5. Generate the protobuf bindings using Gradle:

$> ./gradlew clean generateProto

6. Build Cineast using Gradle:

$> ./gradlew clean fatJar

This will create two JARS:

• cineast-api/build/libs/cineast-api-X-full.jar - the JAR for the REST / Websocket API
• cineast-runtime/build/libs/cineast-runtime-X-full.jar - the JAR for the CLI (including the REST / Websocket API)

7. You also need a storage layer for the usage of cineast. We recommend to use cottontaildb. Grab cottontaildb from github and set up its submodules. We assume you still are in cineast's folder hence, move one level up:

$> cd ..
$> git clone https://github.com/ppanopticon/cottontaildb.git && git submodule update --init --recursive

8. Build the database Server using Gradle

$> ./gradlew clean generateProto distZip

It is sufficient to unzip the generated distribution and locate it in the current directory:

$> unzip build/distribution/cottontaildb-1.0-SNAPSHOT.zip && mv cottontaildb-1.0-SNAPSHOT/* . && rm -r cottontaildb-1.0-SNAPSHOT

This way, you are ready to use that freshly built cottontaildb instance. 9. Now you should have set up everything. Be sure to follow the next steps to verify your setup is compete.

Setup vitrivr for extraction / usage

1. Follow the steps above for setting up vitrivr (i.e. cineast and cottontaildb)
2. Start a cottontaildb server. For this, you have to navigate to your cottontaildb folder

$> ./bin/cottontaildb config.json

Only for Windows users: Edit the config.json file by setting the forceUnmapMappedFiles field to true and use the Windows dist: bin/cottontaildb.bat 3. Navigate back to cineast and prepare the storage

$> java -jar cineast-runtime/build/libs/cineast-runtime-2.5-full.jar cineast.json setup

4. Now you are ready to extract a collection using a jobfile.

TODOS

• Cover cineast.json
• Cover job.json
• Prepare small collection for extraction -- lsc jobfile?

• The guide currently covers the multi-module branch. As this guide is a preparation for 3.0, it should probably talk about master
• You can already clone submodules so you don't have to init afterwards. I'd keep the command if people have already checked out the repo.
• I don't think you need to generate proto-files with cineast, the generated classes should already be in the repo.
• Re: Extraction. The setup command only creates the tables which are present in the cineast.json config. This relates to vitrivr/cineast#79, which would solve this issue.
• I'd more clearly separate development and usage. For users, we could think about having docker-containers with pre-built data examples (e.g. a subset of V3C1) or just distribute cottontail data-folders.