Meltano + Kubernetes :: Production Deployment

meltano_prod.md

Meltano + Kubernetes :: Production Deploy

This guide assumes:

• Custom Singer taps and targets are complete and tested
• The production environment is GCP

The following process is to create a Meltano application that contains the desired extractors and loaders, and then deploy it on GCP Kubernetes Engine.

Environment Requirements:

• A Google Cloud Platform Project
  • Google Cloud Platform
• Google Cloud SDK
  • Installing Google Cloud SDK  |  Cloud SDK Documentation
• Docker
  • Docker Desktop for Mac and Windows | Docker

Pull Meltano Docker Image

docker pull meltano/meltano:latest-python-3.8

Create a Meltano Project

If you’re using Docker, don’t forget to mount the current working directory, and then run the init command to create a new project:

docker run -v $(pwd):/projects -w /projects meltano/meltano:latest-python3.8 init my-meltano-project

When using the Meltano image the meltano command is the entry point, and therefore not required to run Meltano commands.

Add Custom Sources to Meltano

After a Meltano project has been created, add any required custom Singer taps and targets (extractors/loaders). To add a custom extractor or loader use the --custom flag .

# ensure that interactive mode is enabled so that Meltano can ask you additional questions about the plugin and get your answers over STDIN
# run commands from Meltano project root

# For custom taps

docker run --interactive -v $(pwd):/project -w /project meltano/meltano:latest-python3.8 add --custom extractor tap-name

# For custom targets

docker run --interactive -v $(pwd):/project -w /project meltano/meltano:latest-python3.8 add --custom loader target-name

Add Native Sources to Meltano

Meltano has some native sources available that can be discovered and added if they are included in your Meltano project. The links below include the current complete list of extractors and loaders.

CLI Reference | Meltano

# This command discovers all: extractors, loaders, and models

meltano discover all

Sources (Extractors) | Meltano

meltano discover extractors

Destinations (Loaders) | Meltano

meltano discover loaders

If necessary, add native sources to the Meltano project (examples included):

# Add a native extractor

docker run -v $(pwd):/project -w /project meltano/meltano:latest-python3.8 add extractor tap-gitlab

# Add a native loader

docker run -v $(pwd):/project -w /project meltano/meltano:latest-python3.8 add loader target-snowflake

Configure Meltano Plugins

If the custom extractors and loaders in the previous step were added with the --interactive flag it is likely many configuration options in meltano.yml have been set already. For native plugins the following section will help to get them configured correctly.

::IMPORTANT ASSUMPTION::

For the remainder of this guide it will assume the Docker command in the code block below precedes any meltano command from within the Meltano project root directory:

docker run -v $(pwd):/projects -w /projects meltano/meltano:latest-python3.8

[Optional]

Create a .zshrc alias or function with the above Docker command to save time because it will be used often.

::ASSUMPTION SECTION END::

After a plugin is installed in Meltano check what settings it supports:

meltano config <plugin> list

To update any settings from the list:

meltano config <plugin> set <setting> <value>

To display the current config settings for a plugin:

meltano config <plugin>