Title: Add Custom Headers Configuration for DLaaS Data Layer Plugin Services
Current Behavior: DLaaS uses a job queue pattern to do work on the local machine. Jobs are queued up in SQS, and any number of workers can join/leave the system. Additionally, each individual worker has multiple parallel threads dedicated to processing jobs from the job queue.
To utilize the existing system and its paradigm, the Datalayer Uploader plugin was developed, creating a serverless API with the correct datalayer uploader interface. The API receives the store and filenames from the datalayer and creates a job in the job queue that maps to an S3 uploader job in the worker. The API will queue up the job, and the next available worker thread will pick it up and upload the required files to S3 to be served through a CDN.
The system has shown to work as expected; however, the configuration in the datalayer's config.yaml
only accepts the base host of the plugin interface. The original intention was that a localhost service