shared-indexes.md

Shared indexes with Coder

TLDR: Shared indexes work with Coder + JetBrains Projector. To quickly prototype your project with shared indexes, you can generate them and host them as a dev URL on your workspace, for other workspaces to try.

After testing, you'll need to upload the contents of /home/coder/indexes/generate-output to a file server and use that as the index location, or include them in the Coder image's filesystem. Dev URLs are used simply as a test. You'll need to regenerate the metadata with cdn-layout-tool with the proper destination as well.

Generating a shared index in a Coder workspace:

Sourced from this tutorial: https://www.jetbrains.com/help/go/shared-indexes.html#configure-index-update

# For testing, we'll use a Dev URL as the "file server" for the indexes
# Change stable.cdr.dev to your Coder deployment
# This will not work unless your deployment has public dev URLs enabled
INDEX_SERVER_URL=https://9999-$CODER_ENVIRONMENT_NAME-$CODER_USERNAME.stable.cdr.dev

# Edit these with information relevant to your project
REPO_URL=https://github.com/kubernetes/kubernetes.git
PROJECT_NAME=kubernetes
IDE=goland

# Create public dev url for port 9999
coder urls create $CODER_ENVIRONMENT_NAME 9999 --access public

# Clone a project
git clone $REPO_URL $HOME/$PROJECT_NAME

# Set up folders
mkdir -p /home/coder/indexes
mkdir $HOME/indexes/ide-config
mkdir $HOME/indexes/ide-log
mkdir $HOME/indexes/ide-system

# Create shared index
$IDE dump-shared-index project --output=$HOME/indexes/generate-output --tmp=$HOME/indexes/temp --project-dir=$HOME/$PROJECT_NAME --project-id=$PROJECT_NAME

# Requires Java installed on the workspace
# Install, if you don't have it in the image
sudo apt-get install -y openjdk-11-jre

# Use the cdn-layout-tool for generating metadata
wget https://packages.jetbrains.team/maven/p/ij/intellij-shared-indexes-public/com/jetbrains/intellij/indexing/shared/cdn-layout-tool/0.8.60/cdn-layout-tool-0.8.60.zip
sudo apt-get install -y unzip # install unzip if not present in workspace
unzip cdn-layout-tool-0.8.60.zip

cdn-layout-tool-0.8.60/bin/cdn-layout-tool --url=$INDEX_SERVER_URL --indexes-dir=$HOME/indexes/generate-output

# Tell the IDE to use the shared index
touch $HOME/$PROJECT_NAME/intellij.yaml
cat <<EOF >>$HOME/$PROJECT_NAME/intellij.yaml
sharedIndex:
  project:
    - url: $INDEX_SERVER_URL/project/$PROJECT_NAME
EOF

# Start an example http server to simulate "hosting" this shared index
cd $HOME/indexes/generate-output
python3 -m http.server 9999

# Ensure you have the "Shared Project Indexes" plugin installed
# (GoLand and non-community JetBrains IDEs have it pre-installed)
# and open the project folder.

# For the Kubernetes project,
# this resulted in a 56% decrease in loading time.
# (22 seconds vs 50 seconds)

Testing from another workspace

1. Make the dev URL for the workspace public.
2. Copy the intellij.yaml file into another workspace before opening the project in IntelliJ. You could also commit it, but again, the Dev URL is just for testing.

Adding shared indexes into the image

The example above simulates an external "file server" hosting the indexes. When the Coder image is built, indexes could also be generated (via the script) and copied into the image. During runtime, the IDE could "download" the indexes from its own filesystem using the file:// schema.

During testing this led to a 60% faster indexing (versus 56% from a dev URL / server on the network). This significantly depends on the shared index size and networking speed. It does removes the need to upload the indexes anywhere :)

Note: I haven't done this in a Dockerfile yet. Here's how it might work:

# Dockerfile

FROM openjdk:11 AS indexer
# Generate shared index here
ARG INDEXES_SERVER_URL=file:///var/tmp/indexes
# e.g RUN create-shared-index.sh

FROM codercom/enterprise-goland:ubuntu

COPY --from=indexer /root/indexes /var/tmp/indexes

In the project's intellij.yaml file, simply reference the indexes dir with the file:// schema. I have tested this and it works as long as the user has read permissions to the dir.

sharedIndex:
  project:
    - url: file:///var/tmp/indexes

If the image is built regularily (e.g weekly), it will lead more performant indexing as it uses a more recent copy of the project, requiring fewer updates locally.