borislitvak

borislitvak

Setup PySpark and its S3 connection

• Create an environment with your favorite venv manager, ala: conda create --name etl_spark_standalone python=3.8 && activate etl_spark_standalone
• pip install -r requirements.txt. Note: This installs pyspark.

Make a choice

• Do you want standalone spark/hadoop (any version you want) or
• Built-in Spark that comes with pip installs ala pip install pyspark==3.0.1

borislitvak

# Key point: Stomp.py must get the key files, does not work without them. Even stomp.py test_ssl unit test does not work without them.

# First: Generate the keys in cmdline. 
# openssl req -newkey rsa:2048 -nodes -keyout privateKey.key -x509 -days 365 -out certificate.pem

# I am not a Python expert, so don't bash me on this :)
# Based on https://forums.aws.amazon.com/thread.jspa?messageID=885374

import ssl
import stomp

borislitvak

Entity Framework: What steps are needed for EF4 entities to coexist with EF6 ones in the same dll?

In this case, EF4 entities/context are generated by Legacy ObjectContext codegen, DbFirst. The new EF6 entities/context are generated by T4 templates, DbFirst as well.

While there are some hindrances to this transformation, they can be overcome as long as any third party libraries don't use EntityFramework.dll. Let's dive into these details.

Third party libraries with EF4/ObjectContext

The libraries that I've seen are: