Skip to content

Instantly share code, notes, and snippets.

View evidanary's full-sized avatar

evidanary evidanary

13 followers · 2 following
View GitHub Profile
@evidanary
evidanary / instant-search.html
Created May 7, 2012 00:00
Real time instant search
<style>
.loc {
color:gray;
font-style:italic;
font-size:11px;
}
</style>
<script type='text/javascript' src='https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js'></script>
<script type='text/javascript'>
$(document).ready(function() {
@evidanary
evidanary / gist:6418644
Last active December 22, 2015 04:39
Monitor Hadoop periodically to check if there is a change in the number of data nodes. Also checks if the utilized space on the cluster exceeds a thresold DFS_USED_PERCENT_THRESHOLD. Exits with error code = 1 when it is not able to run the dfsadmin -report command. Also prints the tmp directory space as this often gets filled up if running bad q…
#!/usr/bin/env ruby
# Checks Hadoop and alerts if there is a change in data nodes
require 'yaml'
DFS_USED_PERCENT_THRESHOLD = 75
error_flag = 0
report = `hdfs dfsadmin -report`
# Datanodes available: 17 (28 total, 11 dead)
@evidanary
evidanary / gist:7651363
Created November 26, 2013 00:25
this should be in conf/activemq.xml
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
@evidanary
evidanary / pwgen.rb
Created December 20, 2013 18:57 — forked from lpar/pwgen.rb
#!/usr/bin/env ruby
# encoding: UTF-8
# Simple password generation in Ruby.
#
# Generate reasonably secure random passwords of any chosen length,
# designed to be somewhat easy for humans to read and remember.
# Each password has a capitalized letter and a digit.
#
# Example:
@evidanary
evidanary / json_schema_extractor.rb
Last active August 29, 2015 13:56
Extract Schema from a JSON row. Generates a hive table. Flattens based on {"event" => ....., "properties" = > ....}
# This utilityi extracts schema of a JSON object
# It returns the keys of the JSON object separated by commas
# It also returns a create table statement in Hive
require '/var/lib/jenkins/scripts/lib/hive_utils.rb'
require 'json'
# This function should probably be in hive_utils.rb
def create_table_string(params)
header = "CREATE TABLE #{params[:hive_schema]}.#{params[:hive_table]}("
@evidanary
evidanary / mixpanel_extract.rb
Last active August 29, 2015 13:56
Class file to download mixpanel data and convert to CSV
require 'mixpanel_client'
require 'csv'
require 'multi_json'
require 'net/http'
class MixpanelExtract
attr_reader :filename, :request_uri
def initialize(config, for_date)
@client = Mixpanel::Client.new(config[:mixpanel_auth])
@evidanary
evidanary / mixpanel_download_events.rb
Created February 24, 2014 22:27
This gets mixpanel JSON data and uploads to a table in HIVE
# Download Mixpanel Data
require '../lib/hive_utils.rb'
require 'yaml'
require 'fileutils'
require File.expand_path(File.dirname(__FILE__) + '/mixpanel_extract.rb')
# TODO use thor here to take the date from the commandline
def print_usage_and_exit
puts %q[Usage:
@evidanary
evidanary / gist:9242371
Last active August 29, 2015 13:56
Thrift Monitoring for storm topologies. gen-rb folder has thrift generated ruby wrappers for talking to storm cluster
$:.push('/home/yranadive/gen-rb')
# This file monitors a set of storm topologies and generates
# 1. Zenoss friendly status message. e.g. topo1:1 topo2:0 (1=active, 0=inactive)
# 2. Mails the inactive topologies
# Usage: ruby storm_monitor.rb [mail]
# Optional argument "mail" on the command line will mail the report
require 'rubygems'
require 'thrift'
require 'nimbus'
@evidanary
evidanary / gist:593b6c3039545ab73f8f
Created July 17, 2014 18:33
Hive Last modification time of a table using webhdfs gem
irb> require 'webhdfs'
irb> client = WebHDFS::Client.new('hadoop-nn', 50070)
irb> fl = client.list('/user/hive/warehouse/database.db/tablename/')
irb> DateTime.strptime(fl.collect {|x| x['modificationTime']}.max.to_s, '%M')
@evidanary
evidanary / gist:47f55a5b7bcc559896f7
Created July 31, 2014 06:46
NerdTree and vim-rails .vimrc file for osx
execute pathogen#infect()
call pathogen#helptags()
filetype off
syntax on
filetype plugin indent on
map <C-n> :NERDTreeToggle<CR>
autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTreeType") && b:NERDTreeType == "primary") | q | endif