Setting up spotlight search using elasticsearch and Samba on TrueNAS 12

0_gistfile1.md

IMPORTANT: This is not safe! This currently requires root login over SSH.

Step 0: Enable SSH password login for root on TrueNAS. This is a huge security risk, so only do this if the system is completely firewalled from the open internet, and if there is no mission-critical data on the NAS.

First off, you will need a host running docker. Maybe eventually I will figure out a way to run this all natively, but that time is not now.

Create a folder each for your elasticsearch data and fscrawler configuration. They are labeled $ES_DATA and $FSCRAWL_CONFIG in this write-up.

Next you will need an fscrawler configuration file. This file will be placed at $FSCRAWL_CONFIG/$JOBNAME/_settings.yaml. I have included an example below, but you will need to fill in the details for your own system. I disabled the index_content settings to reduce space usage, but that can be changed for your own usage. You can also enable PDF OCR if desired.

Now deploy the docker-compose file below with the details filled in for your environment. Once deployed you should be able to run docker-compose logs -f fscrawler and see that it has started indexing.

Finally, add the auxiliary parameters to Services/SMB -> Auxiliary Parameters and add spotlight = yes to each Samba share individually.

_settings.yaml

---
name: "$JOBNAME"
fs:
  url: "$JOB_DIR"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: false
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "http://elasticsearch:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
server:
  hostname: "$TRUENAS_HOST"
  port: 22
  username: "root"
  password: "$PASSWORD"
  protocol: "ssh"

Auxiliary Parameters

spotlight backend = elasticsearch
elasticsearch:address = $DOCKERHOST
elasticsearch:port = 9200

docker-compose.yml

version: 2.1

services:
  elasticsearch:
    image: elasticsearch:7.9.3
    restart: unless-stopped
    ports:
      - 9200:9200
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    environment:
      discovery.type: single-node
    volumes:
      - $ES_DATA:/usr/share/elasticsearch/data
  kibana:
    image: kibana:7.9.3
    ports:
      - 5601:5601
  fscrawler:
    image: toto1310/fscrawler
    volumes:
      - $FSCRAWL_CONFIG:/usr/share/fscrawler/config
    depends_on:
      - elasticsearch
    command:
      - fscrawler
      - --config_dir
      - /usr/share/fscrawler/config
      - $JOBNAME