shravan-kuchkula

Installing postgres and connecting to it using jupyter notebook

Goal: I want a postgres environment wherein I can create some tables and insert some values into those tables and run some queries. This allows me to practice writing SQL queries to understand some concepts.

STEP 1: First make sure you install homebrew (if you don't already have it)

The first step is to install homebrew - package manager for mac.

Run command: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

The first part of the output shows what will be installed under /usr/local/

shravan-kuchkula

Using Docker and docker-compose to manage Apache Airflow on mac

Using our beloved docker and docker-compose, we can very quickly bring up an Apache Airflow instance on our mac.

Contents of docker-compose.yml

About the only thing you need to customize in this docker-compose.yml file is the volumes section. This will tell docker to map the given directory containing your Airflow DAGs/plugins to container file system.

version: '3'
services:

shravan-kuchkula

How to install geopandas library on macOS?

I suggest you create a brand new environment before starting.

Ensure that when you create a conda env, you don't use default packages. You will thank yourself later.

  conda create --no-default-packages -n geo_pandas python=3.7
  conda activate geo_pandas
  conda config --env --add channels conda-forge
  conda config --env --set channel_priority strict (This is optional)
  

shravan-kuchkula

Conda is a powerful package manager and environment manager that you use with command line commands at the Anaconda Prompt for Windows, or in a Terminal window for macOS or Linux.

Common Commands:
a. To see a list of all your environments, type:
conda info --envs

$conda info --envs
# conda environments:
#
base                  *  /Users/Shravan/anaconda3

shravan-kuchkula

import os
files = os.listdir(os.getcwd())
[os.replace(file, file.replace(" ", "_")) for file in files]

shravan-kuchkula

import pymongo

connection = pymongo.MongoClient("mongodb://localhost")

db = connection.school
people = db.people
people.drop()


def insert(peopleList):

shravan-kuchkula

# To insert the data in the people collection use 'insertOne.py' that I have given earlier

# Display the contents of the people collection
> db.people.find()
{ "_id" : ObjectId("59a36310fc2b2b3d843af1b6"), "name" : "Smith", "age" : 30, "profession" : "hacker" }
{ "_id" : ObjectId("59a36310fc2b2b3d843af1b7"), "name" : "Jones", "age" : 35, "profession" : "baker" }
{ "_id" : ObjectId("59a36310fc2b2b3d843af1b8"), "name" : "Alice" }
{ "_id" : ObjectId("59a36310fc2b2b3d843af1b9"), "name" : "Bob" }
{ "_id" : ObjectId("59a36310fc2b2b3d843af1ba"), "name" : "Charlie" }
{ "_id" : ObjectId("59a36310fc2b2b3d843af1bb"), "name" : "Dave" }

shravan-kuchkula

<?xml version="1.0" encoding="UTF-8"?>
<TLSales xmlns="http://tlsales.com/namespace" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tlsales.com/namespace /tmp/msds7330/customer.xsd">
	<Address>
		<Name>Addr1</Name>
		<Street>123 Main St.</Street>
		<City>Seattle</City>
	</Address>
	<Customer OneTime="true" Regular="false" Senior="false" Discount="5">
		<First>John Q.</First>
		<Last>Public</Last>

shravan-kuchkula

# Import the pandas package
import pandas as pd

# Turn list of lists into list of dicts: list_of_dicts
# https://gist.github.com/shravan-kuchkula/6ca3054d5ec0549e1759d739b3d47513
list_of_dicts = [lists2dict(feature_names, sublist) for sublist in row_lists]

# Turn list of dicts into a dataframe: df
df = pd.DataFrame(list_of_dicts)

shravan-kuchkula

# Define lists2dict()
def lists2dict(list1, list2):
    """Return a dictionary where list1 provides
    the keys and list2 provides the values."""

    # Zip lists: zipped_lists
    zipped_lists = zip(list1, list2)

    # Create a dictionary: rs_dict
    rs_dict = dict(zipped_lists)