zhukovgreen

All code is available here: https://github.com/zhukovgreen/protocols-demo

zhukovgreen

Pointing multiple spark versions in the same project

• databrics-connect:

conda create --name databricks-connect python=3.10 -y
conda activate databricks-connect
pip install databricks-connect==13.*
export _SITE="$(python -c 'import sys;print(sys.path[-1])')"
export SPARK_HOME="$_SITE/pyspark"
export PYSPARK_PYTHON="$(which python)"
export PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON

zhukovgreen

1. Ensure you have java installed (in my case it is version 19). But you can install fro example 17

$ java -version || brew install --cask zulu17
openjdk version "19.0.1" 2022-10-18
OpenJDK Runtime Environment (build 19.0.1+10-21)
OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)

2. Download and extract spark with the hadoop

zhukovgreen

import os
import pathlib
import re
from functools import singledispatch
from typing import Any, Dict

import yaml

GROUP_CONTAINING_VAR_NAME_RE = "var"
STRING_WITH_VARS_RE = re.compile(

zhukovgreen

import sys
import typing

import classes


Paper = typing.NewType("Paper", str)
# RecycledPaper = typing.NewType("RecycledPaper", str)

zhukovgreen

import pytest


def longest_chain_of_words(
    inputs: tuple[str, ...],
) -> int:
    """Find the longest chain of words within an array matching the rule.

    The rule is:

zhukovgreen

class MyTransformation(BaseTransformation):
    def implementation(metastore) -> DataFrame:
        table_x = metastore.get("table_x")
        table_y = metastore.get("table_y")

        # do some transformation
        return some_result_dataframe

zhukovgreen

import functools
import inspect

from typing import Any, Callable, Dict

import pytest


class feature:
    fixtures: Dict[str, Callable[[], Any]] = {}

zhukovgreen

import org.scalatest.wordspec.AnyWordSpec

import scala.math.*

class test_different_samples extends AnyWordSpec {
  /**Calculates square root of x*/
  def square_root(x: Double, precision: Double = 0.01): Double = {
    require(x >= 0)
    def inner(min: Double, max: Double): Double = {
      val guess = (min + max) / 2

zhukovgreen

[tool.poetry]
name = "zhukovgreen111"
version = "0.1.0"
description = ""
authors = ["ZhukovGreen <zhukovgreen@icloud.com>"]

[tool.poetry.dependencies]
python = "^3.8"
docker-compose = "^1.25.3"
ipython = {git = "https://github.com/ipython/ipython"}