AbdealiLoKo

#!/usr/bin/env sh
# . "$(dirname -- "$0")/_/husky.sh"  # - uncomment if using husky

# Get the current commit message file
COMMIT_MSG_FILE=$1
SOURCE_MSG=$2

# Ref: https://git-scm.com/docs/githooks#_prepare_commit_msg
# SOURCE_MSG is the source where the commit messsage is taken from
# - EMPTY    -> No source commit message present `git commit -a`

AbdealiLoKo

AbdealiLoKo

AbdealiLoKo

"""
Simple benchmark to check if withColumn() is faster or select() is faster
Conflusion: select() is faster than withColumn() in a for loop as lesser dataframes are created
"""

import datetime
import findspark; findspark.init(); import pyspark
spark = pyspark.sql.SparkSession.builder.getOrCreate()

for ncol in [10, 100, 1000, 2000, 5000]:

AbdealiLoKo

AbdealiLoKo

Test SQLAlchemy application

Slides: https://docs.google.com/presentation/d/1SMtBILSrqt9SWK5BnEKe4ivGTHItrVjgCEB0AUhGeyM

Setup

$ python -m venv venv
$ venv/bin/pip install sqlalchemy colorama

AbdealiLoKo

Better commit messages

The 3 common ways for developers to document information about their work is:

1. Comments
   • When is this written: When the developer wants something to be clearly and immediately visible to all other developers
   • When is this found: As soon as other developers are reading code, they will find these comments
2. Commit messages
   • When is this written: When the developer wants to explain the work involved in them making a change. Why a change was made, explanation of the
   • When is this found: When other developers dig a big deeper on why or when a change was made - they will find these commit messages
3. Tech Documentation

AbdealiLoKo

# Example:
# PYJAVA_LIB=jpype venv/bin/python pyjava.py

import os
from datetime import datetime
from jpmml_evaluator import _package_classpath

lib = os.environ.get('PYJAVA_LIB')
assert lib is not None, 'Set env var PYJAVA_LIB to py4j/jnius/jpype'

AbdealiLoKo

# Write a spark DataFrame into a single CSV files (to open with Excel/other tools easily)
# Save the file to S3

import s3fs
import pyspark.sql.functions as F  # noqa: N812

def spark_to_csv(spark_df, out_path):
    """
    Save the file in part files with spark and then append them together

AbdealiLoKo

name: Python Tests

jobs:
  build:
    runs-on: ubuntu-18.04
    strategy:
      max-parallel: 2
      matrix:
        python-version: [3.5, 3.6, 3.7]