ranjitiyer

Hadoop Streaming step

{
  "jobFlowId": "j-1JKGNMYXT59DP",
  "steps": [
    {
      "name": "EMRHadoopLogPushingTest",
      "actionOnFailure": "CANCEL_AND_WAIT",
      "hadoopJarStep": {
        "jar": "command-runner.jar",

ranjitiyer

# submit a step to run a command on a cluster
AWS_PROFILE=integ-preprod aws emr add-steps \
--cluster-id j-32M91WVU0V1OR \
--steps Name="Create table",ActionOnFailure=CONTINUE,Jar=command-runner.jar,Args=[beeline,-u,jdbc:hive2://ec2-52-91-119-206.compute-1.amazonaws.com:10000,-n,admin,-p,admin,-e,"create table d417f3333(id INT, value STRING);"] --region us-east-1 --endpoint https://emr-preprod-us-east-1.amazon.com --no-verify

ranjitiyer

db = 'test.db'
table = 'Table'

releaseLabel = "releaseLabel"
applicationString = "applicationString"
account = "account"
creationDate = "creationDate"

def create_table(overwrite: bool):
  if overwrite and table_exists(table):

ranjitiyer

time.asctime([t])
t is response of gmtime() or localtime()

time.clock()

time.clock_gettime(clk_id) > float

time.clock_gettime_ns(clk_id) > float

# Returns the response in this format

ranjitiyer

{
  "id": 56,
  "title": "Secret Garden",
  "description": "Mary Lennox is a spoiled, middle-class, self-centred child who has been recently orphaned. She is accepted into the quiet and remote country house of an uncle, who has almost completely withdrawn into himself after the death of his wife. Mary gradually becomes drawn into the hidden side of the house: why does she hear the crying of a unseen child? Why is there an overgrown, walled garden, its door long locked? (Summary by Peter)",
  "num_sections": 27,
  "sections": {
    "1": {
      "number": 1,
      "name": "01 - There Is No One Left",
      "url": "https://ia600204.us.archive.org/15/items/secret_garden_version2_librivox/secretgarden_01_burnett_64kb.mp3"

ranjitiyer

 git log --oneline | awk '{print $1}' | xargs git show | grep "diff" | awk '{print $3}' | sort | uniq -c | sort -nr | more

ranjitiyer

git log --oneline --graph --decorate --all

ranjitiyer

export PS1="\@ \w $(if [[ -e .gitignore ]]; then git branch | head -n 1 | awk '{print $2}'; fi) > "

Prints time in am/pm, the current working directory and the checked out gitbranch if pwd is a git repo
01:39 PM ~/work/emr/sparkmagic jupyter >

ranjitiyer

[[Title >>https://aws.amazon.com/blogs/big-data/running-jupyter-notebook-and-jupyterhub-on-amazon-emr]]


Review Vanguard requirements
========
{{toc numbered="true" /}}

**bold**
* item
** subitem

ranjitiyer

#
# Source https://www.linuxtrainingacademy.com/linux-commands-cheat-sheet/
#

– SYSTEM INFORMATION
# Display Linux system information
uname -a

# Display kernel release information
uname -r