aaronsteers

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://github.com/airbytehq/airbyte/blob/master/airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml",
    "title": "AirbyteProtocol",
    "type": "object",
    "description": "AirbyteProtocol structs",
    "version": "0.3.2",
    "properties": {
        "airbyte_message": {
            "$ref": "#/definitions/AirbyteMessage"

aaronsteers

# First install Python 3.9. This can be installed from Windows Store or using winget.

# Make sure key libraries/tools are updated

pip install --upgrade wheel
pip install --upgrade setuptools
pip install --upgrade pip

# Install the C++ Build Tools v14 or greater:

aaronsteers

{{! Meltano Docstring Template - Same as Google but delegates type hinting to the function signature 
    Save locally and use with this VS Code extension: 
    https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring }}
{{summaryPlaceholder}}

{{extendedSummaryPlaceholder}}

{{#parametersExist}}
Args:
{{#args}}

aaronsteers

# First step: download and install Winget using the latest from GitHub releases here: https://github.com/microsoft/winget-cli/releases

As of Aug 25, 2021: https://github.com/microsoft/winget-cli/releases/download/v1.0.11692/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle

# Confirm winget is installed
winget --help

# How to search:
winget search VS Code

aaronsteers

Privacy Policy

No data is shared externally. Each person is responsible for creating and managing their own API tokens and secrets.

aaronsteers

# first navigate to brew.sh to install homebrew
# compare below with https://docs.dataops.tk/setup/mac

# Note: The 'cask' keyword is no longer needed for gui tools: https://stackoverflow.com/a/66081492
#       Only if there's a name conflict do you need the new (and optional) `--cask` argument during install

brew install visual-studio-code

# Install pyenv for managine multiple python versions:
brew install pyenv

aaronsteers

# Instructions:
#
# 1. Save this file as `{root}/.github/workflows/cicd.yml`.
# 2. Set the `PYPI_USER` and `PYPI_PASS` secrets in your GitHub CI config.
# 3. Update the `PYPACKAGE_NAME` declaration to the name you want to use in PyPi.
#
# Public Gist: https://gist.github.com/aaronsteers/f3eec4db94d3db1de78b76b3318e9e33

name: Python CI/CD
env:

aaronsteers

locals {
  # Directories start with "C:..." on Windows; All other OSs use "/" for root.
  is_windows = substr(pathexpand("~"), 0, 1) == "/" ? false : true
}
resource "null_resource" "cli_command" {
  provisioner "local-exec" {
    # Keep as-is. This ensures windows always uses PowerShell, linux/mac use their default shell.
    interpreter = local.is_windows ? ["PowerShell", "-Command"] : []
    # TODO: Replace the below with the windows and linux variants
    command     = local.is_windows ? "sleep 60" : "sleep 60"

aaronsteers

There's no easy way to go from a full S3 path to the isolated bucket and key path, so I created the below snippet to do this.

  bucket = split("/", split("//", var.s3_path_to_lambda_zip)[1])[0]
  key = join("/", slice(
    split("/", split("//", var.s3_path_to_lambda_zip)[1]),
    1,
    length(split("/", split("//", var.s3_path_to_lambda_zip)[1]))
  ))

aaronsteers

choco install python3

pip install black awscli azure boto3 docker fire psutil s3fs xmlrunner pandas joblib junit-xml pyyaml bandit flake8 pycodestyle pydocstyle pylama prospector mypy

# On non-windows:

pip install pyspark pyhive[hive]