phani111

#!/usr/bin/env groovy
import groovy.json.JsonSlurper

def nexus_server_base_url = 'http://nexus:8081/service/siesta/rest/beta/components'

def repository = 'maven-public'

def base_url = "${nexus_server_base_url}?repositoryId=${repository}"

def jsonSlurper = new JsonSlurper()

phani111

#!/bin/bash

# This script will update a new workspace created on Cloud9 IDE with the latest packages. 
# In order to use it, create a update_workspace.sh file in your C9 Workspace and then make it executable using the command
# `touch update_workspace.sh && chmod +x update_workspace.sh`. 
# Now you can open the updata_workspace.sh file and copy/paste this full script, save and close. 
# Run the command `./update_workspace.sh` to execute the script.
#
# Alternatively you can use this command to download and make this script executable from github
# wget -O update_workspace.sh  https://gist.githubusercontent.com/aubort/836888f8aaeeeff75024c87e9c9199f0/raw && chmod +x update_workspace.sh

phani111

import argparse, datetime, logging

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions


class GetTimestampFn(beam.DoFn):
  """Prints element timestamp"""
  def process(self, element, timestamp=beam.DoFn.TimestampParam):

phani111

the basic MSCK REPAIR TABLE table-name was not working for me. but this was

import boto3
import os
import uuid
import pytz
import datetime

def make_athena_client():
    athena_client = boto3.client('athena',

phani111

This gist includes hive ql scripts to create an external partitioned table for Syslog
generated log files using regex serde; 
Usecase:  Count the number of occurances of processes that got logged, by year, month,
day and process.
 
Includes:
---------
Sample data and structure:           01-SampleDataAndStructure 
Data download:                       02-DataDownload
Data load commands:                  03-DataLoadCommands

phani111

NOTE: This is a question I found on StackOverflow which I’ve archived here, because the answer is so effing phenomenal.

---

Q: How can I make a chain of function decorators in Python?

---

If you are not into long explanations, see [Paolo Bergantino’s answer][2].

phani111

#!/bin/bash -e
#
# Copyright 2014 Tony Burns
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#

phani111

sudo add-apt-repository ppa:webupd8team/sublime-text-3;
sudo apt-get update;
sudo apt-get install sublime-text-installer;
sudo ln -s /usr/lib/sublime-text-3/sublime_text /usr/local/bin/sublime;

phani111

var data = sc.parallelize(Seq(("A", 2), ("A", 4), ("B", 2), ("Z", 0), ("B", 10)))
// data: org.apache.spark.rdd.RDD[(java.lang.String, Int)] = ParallelCollectionRDD[31] at parallelize at <console>:12

val avgValue = data.mapValues((_, 1)
        .reduceByKey((x, y) => (x._1 + y._1, x._2 + y._2))
        .mapValues{ case (sum, count) => (1.0 * sum) / count }
        .collectAsMap()
// avgValue: scala.collection.Map[java.lang.String,Double] = Map(Z -> 0.0, B -> 6.0, A -> 3.0)