davehowell

# THE WSL2 with Ubuntu 20.04 and bash 5.0 edition. Mainly for VSCode usage
# This is just the customized parts 

# if you can only use /usr/bin/python3 because public apt repo is blocked
alias python='python3'

#create a python venv within project dir using current python
#   python -m venv .venv
 
alias vnv='source ./.venv/bin/activate'

davehowell

## Compute Engine
MIG - managed instance groups (for autoscaling)
- lift n shift, where you don't want to refactor

## Kubernetes Engine GKE
- managed, cloud-native K8s
- supports complex configurations
- capable, but complex, e.g. complex manifests
- autopilot helps scale-on-demand

davehowell

Using Redhat UBI with Python 3.11 and Oracle drivers

https://developers.redhat.com/blog/2020/03/24/red-hat-universal-base-images-for-docker-users

• Will we have EL7, or EL8?

probably 8. Note that 9 is also available but beta so don't use

• We need the oracle python client:

davehowell

AWS Re:invent recap Feb 2023

Redshift

Aurora to Redshift serverless “zero ETL” transaction replication

• MySQL flavour only
• Redshift serverless only
• Create an “integration” on the Aurora side
• Create database from integration on redshift side
• Cannot replicate deletes

davehowell

Responses to questions

Also notebooks here https://github.com/databricks-academy/data-engineering-with-databricks-english

Lakehouse (MANDATORY)

What is Lakehouse?

davehowell

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
import org.slf4j.{Logger, LoggerFactory}

object Main {
  val logger: Logger = LoggerFactory.getLogger(this.getClass)

  private lazy val sparkConf: SparkConf = new SparkConf()
    .setMaster("local[*]")

davehowell

version: 2
jobs:
  terraform-validate:
    docker:
      - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
    steps:
      - checkout
      - run:
          name: Terraform Validate
          command: |

davehowell

Use the Docker host's ssh socket and known hosts cache

docker run -it -v $PWD:/dbt_tmp -w /dbt_tmp
-v $SSH_AUTH_SOCK:/ssh_agent_sock
-v "$HOME/.ssh/known_hosts":"/etc/ssh/ssh_known_hosts"
-e SSH_AUTH_SOCK=/ssh_agent_sock
-e DBT_PROFILES_DIR

buildkit buildx secret credential security

https://docs.docker.com/engine/reference/commandline/buildx_build/#secret

davehowell

https://realpython.com/intro-to-pyenv/

install pyenv

brew update brew install pyenv

Add pyenv init to shell profile

#zsh

davehowell

#!/bin/bash

# Add these functions to your bash_profile or bash_rc
# Dependencies:
#   - awscli - AWS command line client
#   - jq - Command-line JSON processor

function cfn-validate-template() {
for template in $@
do