Rishi Barve rishibarve
rishibarve
/ BigQueryGeohashEncode.sql
Created
October 15, 2018 06:08
— forked from killerbees/BigQueryGeohashEncode.sql
Big Query STD SQL Gist for Geohash Encode
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #standardSQL | |
| CREATE TEMPORARY FUNCTION geohashEncode(latitude FLOAT64, logitude FLOAT64, precision FLOAT64) | |
| RETURNS STRING | |
| LANGUAGE js | |
| AS """ | |
| var Geohash = {}; | |
| /* (Geohash-specific) Base32 map */ | |
| Geohash.base32 = '0123456789bcdefghjkmnpqrstuvwxyz'; | |
| lat = Number(latitude); |
rishibarve
/ Multiprocessing Python With Monitoring
Created
August 30, 2018 06:54
Code for multiprocessing python with feature to save work in middle to skip in case of failure. Helpful for processing large number of small files in a environment where failures are common.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import multiprocessing | |
| # split a list into evenly sized chunks | |
| def chunks(l, n): | |
| return [l[i:i+n] for i in range(0, len(l), n)] | |
| def do_job(job_id, data_slice): | |
| processed = [] | |
| for i, item in enumerate(data_slice): |
rishibarve
/ NotNullSetterTemplate
Created
August 10, 2018 06:02
Template for generating setter with checking not null in IntelliJ Idea
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #set($paramName = $helper.getParamName($field, $project)) | |
| #if($field.modifierStatic) | |
| static ## | |
| #end | |
| void set$StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project)))($field.type $paramName) { | |
| if($paramName != null){ | |
| #if ($field.name == $paramName) | |
| #if (!$field.modifierStatic) | |
| this.## | |
| #else |
rishibarve
/ gist:8320684b372e830c0ddf367d996caf71
Created
May 16, 2017 07:08
Remove comments in hive query (and optionally do clean formatting)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #Requires sqlparse library to be installed | |
| import sqlparse | |
| def remove_hive_comments( query, format = False): | |
| if format is True: | |
| qry = sqlparse.format(query, reindent=True) | |
| new_qry = ''' | |
| ''' | |
| for line in iter(qry.splitlines()): | |
| index = line.find('--') | |
| line1 = line |