Skip to content

Instantly share code, notes, and snippets.

View anjijava16's full-sized avatar
💭
Awesome

Anjaiah Methuku anjijava16

💭
Awesome
View GitHub Profile
Incoming/Sources/Inputs: S3 Files (CSV,Parquet) and RDBMS (Oracle,MySQL ) Data
Languages: Python,Shell Script
Cluster : AWS EMR (It is Like Hortnworks)
Linux : Linux server (It is EC2)
Processing :
1. Hive SQL
# Setting PATH for Python 3.8
# The original version is saved in .zprofile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
# Hadoop
export HADOOP_HOME=/Users/welcome/Desktop/hadoop/hadoop-3.2.1/
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
# Virtual Environmnet
1. There are other great third-party tools for creating virtual environments, such as
1. virtualenv
2. conda
3. Pipenv
4. Poetry
# Step1
Jump to beginning of a line – Command+Left Arrow
Jump to end of a line – Command+Right Arrow
Jump to beginning of current word – Option+Right Arrow
Jump to end of current word – Option+Right Arrow
To backspace on a Mac, press the fn and Delete keys, as shown below
command +Space ==> QUick search
Control + LeftArrow ==> Left Open one
Control+ RightArrow ==> Right Opne one
command + c==> COPY
command + p ==> Past
vi .zprofile
# Setting PATH for Python 3.8
# The original version is saved in .zprofile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
# Hadoop
export HADOOP_HOME=/Users/welcome/Desktop/hadoop/hadoop-3.2.1/
export HADOOP_INSTALL=$HADOOP_HOME
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
# Hadoop setup here
1. https://blog.contactsunny.com/data-science/installing-hadoop-on-the-new-m1-pro-and-m1-max-macbook-pro
# Hive Setup Here
1. https://dbmstutorials.com/hive/hive-setup-on-mac.html
welcome@Anjaiahs-MacBook-Pro ~ % more .zprofile
# Setting PATH for Python 3.8
# The original version is saved in .zprofile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
export PATH
welcome@Anjaiahs-MacBook-Pro ~ %
import csv
from datetime import datetime
INPUT_FILE_PATH = 'C:/Tech_Learn_welcome/Python_Utils/FastAPI/sai_workspace/fast_api_welcome/files/input.csv'
OUTPUT_FILE_PATH = 'C:/data/python_output/input.csv'
HEADER_SKIP = True
OUTPUT_HEADER = 'id,first,last,ssn,address,firstname_lastname,process_date'
def read_file():
mkdir delab_f
cd delab_f
python3 -m venv delab-venv
source delab-venv/bin/activate
pip install jupyterlab