sonhmai

#basic python code for working with POSGRESQL through  psycopg2



import psycopg2

conn = psycopg2.connect(database="testdb", user="postgres", password="pass123", host="127.0.0.1", port="5432")
print("Opened database sucessfully")
cur = conn.cursor()

sonhmai

com.sun.management:type=DiagnosticCommand
com.sun.management:type=HotSpotDiagnostic
java.lang:name=CodeCacheManager,type=MemoryManager
java.lang:name=Code Cache,type=MemoryPool
java.lang:name=Compressed Class Space,type=MemoryPool
java.lang:name=G1 Eden Space,type=MemoryPool
java.lang:name=G1 Old Generation,type=GarbageCollector
java.lang:name=G1 Old Gen,type=MemoryPool
java.lang:name=G1 Survivor Space,type=MemoryPool
java.lang:name=G1 Young Generation,type=GarbageCollector

sonhmai

#!/bin/bash
export PYTHON_VERSION=3.6.5
export PYTHON_DOWNLOAD_URL=https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz

sudo apt update
sudo apt install --no-install-recommends -y \
    software-properties-common build-essential \
    libssl-dev libreadline-dev libbz2-dev libsqlite3-dev zlib1g-dev \
    python-minimal

sonhmai

# saves credentials

$ git config credential.helper store
$ git push http://example.com/repo.git
Username: <type your username>
Password: <type your password>

sonhmai

class pyspark.sql.DataFrame

people = spark.read.parquet("...")
department = spark.read.parquet("...")

people.filter(people.age > 30) \
      .join(department, people.deptId == department.id) \
      .groupBy(department.name, "gender") \
      .agg({"salary": "avg", "age": "max"})

sonhmai

# XGBClassifier
param_dist = {'objective': 'binary:logistic', 'n_estimators': 2}
clf = xgb.XGBClassifier(**params_dist)

sonhmai

psql -U postgres

postgres=# SHOW config_file;

sonhmai

Python relative imports in AWS Lambda fail with attempted relative import with no known parent package

The Problem

In AWS Lambda if I attempt an explicit relative import like this

.
├── lambda_file.py
└── example.py

sonhmai

# BENCHMARK-----------------------------------------------------

#1. Rust kafka-benchmark (https://github.com/fede1024/kafka-benchmark)
# must create topic with 6 partitions first
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 6--topic [scenario]
# replace scenario with one in file kafka-benchmark/config/base_producer.yaml