DatalayerPluginFeatureRequest.md

Title: Add Custom Headers Configuration for DLaaS Data Layer Plugin Services

Current Behavior: DLaaS uses a job queue pattern to do work on the local machine. Jobs are queued up in SQS, and any number of workers can join/leave the system. Additionally, each individual worker has multiple parallel threads dedicated to processing jobs from the job queue.

To utilize the existing system and its paradigm, the Datalayer Uploader plugin was developed, creating a serverless API with the correct datalayer uploader interface. The API receives the store and filenames from the datalayer and creates a job in the job queue that maps to an S3 uploader job in the worker. The API will queue up the job, and the next available worker thread will pick it up and upload the required files to S3 to be served through a CDN.

The system has shown to work as expected; however, the configuration in the datalayer's config.yaml only accepts the base host of the plugin interface. The original intention was that a localhost service would be running. The DLaaS system handles the uploading of the files locally, but the job queuing mechanism is still external.

There is a security concern in that there is no way to tell the datalayer plugin to use an Authorization header when communicating with the API. This means that to use it, we have to make the API unauthenticated and open to the public. This creates an attack vector where someone could spam the job queue and seize up the entire system.

Proposed Solution: A simple fix would be to allow the config.yaml for the datalayer to include a configuration that enables the plugin to include custom headers with the request. The specific header needed is x-api-key.

Please consider adding a new configuration to specify custom headers for the plugin services to enhance the security of the system. The config.yaml could be updated as follows:

data_layer:
  client_timeout: 15
  database_path: data_layer/db/data_layer_CHALLENGE.sqlite
  downloaders:
    - url: http://localhost:9456
      headers:
        x-api-key: your-api-key-here
    - url: http://localhost:3145
      headers:
        x-api-key: your-api-key-here
...
  uploaders:
    - url: https://plugin.datalayer.storage
      headers:
        x-api-key: your-api-key-here

This updated configuration example allows specifying custom headers for each uploader and downloader, enhancing the security and flexibility of the DLaaS system.