Skip to content

Instantly share code, notes, and snippets.

View karthikbgl's full-sized avatar

Karthik Ravindra karthikbgl

  • New York, NY
View GitHub Profile
@karthikbgl
karthikbgl / spark_dataframe_to_single_file_csv_using_hdfs.py
Created March 8, 2018 00:31
Saves a spark dataframe into a single csv/delimited file efficiently. Assumes the file storage to be hdfs
import subprocess
def write_to_local_fs(df):
"""
This method writes to local filesystem efficiently without using coalesce or repartition.
The idea is to persist data in cluster format in hdfs (or whatever file storage) and write to local file system.
Write header to the file in the local file system
:param: df: the dataframe being sent as argument
"""
@karthikbgl
karthikbgl / regex_matcher.js
Last active July 22, 2017 14:04
Regex pattern matches using jquery
(function($) {
$.fn.regexFinder = function(text) {
/*
Phone number formats
XXX-XXX-XXXX
XXX.XXX.XXXX
XXX XXX XXXX
*/
var cc_re = /(?:4[0-9]{12}(?:[0-9]{3})?|[25][1-7][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})/g;
var date_re = /\d{2}\/\d{4}/g;
@karthikbgl
karthikbgl / combinations_substring.py
Created October 24, 2015 04:46
All possible combinations of substrings
import itertools
def yieldSubset(s):
return sorted(list(set([''.join(c) for i in range(len(s)+1) for c in itertools.combinations(s, i) if c])))
@karthikbgl
karthikbgl / smallest_multiple.py
Created October 24, 2015 04:44
Find smallest multiple
import fractions
def smallest_multiple( n):
return reduce(lambda x, y: x*y/fractions.gcd(x, y), range(2, n+1))
@karthikbgl
karthikbgl / console_log_ie_fix.js
Last active August 29, 2015 14:25
console.log messages in code are bad for Internet Explorer <= 9. This plugin would avoid the application from crashing when console.log messages are present in the javascript code.
;(function () {
// console.log statements cause issues in Internet Explorer <= 9.
// This piece of code prevents the application to fail
// Source for console: https://developer.mozilla.org/en-US/docs/Web/API/Console
var fns = [
"assert",
"count",
"debug",
"dir",
"dirxml",
@karthikbgl
karthikbgl / handler_500.py
Created June 25, 2015 01:00
Override the standard 500 error page to display the stack trace if certain conditions are met. The advantage of overriding the server error handler is, it keeps everything else intact - example all the error emails, etc.
from django.views.debug import technical_500_response
from django.views.defaults import server_error
def server_error_handler(request):
ip_address = request.META.get("HTTP_X_FORWARDED_FOR", request.META.get("REMOTE_ADDR"))
if ip_address in settings.ALLOWED_IPS_FOR_DEBUG_SCREEN: #or request.user.is_staff
return technical_500_response(request, *sys.exc_info())
return server_error(request)
@karthikbgl
karthikbgl / seconds_to_string
Created October 23, 2014 15:36
Time elapsed in seconds
def secondsToStr(t):
'''
This snippet converts the time elapsed in
seconds to HH:MM:SS.mmm
Example:
>>> secondsToStr(123456)
'34:17:36.000'
'''
@karthikbgl
karthikbgl / functools_partials.py
Last active August 29, 2015 14:07
Usage of partials
'''
This is an example of functools.partials
Typical usage is when you have a function that needs multiple parameters,
of which one is a constant, you can use a partial.
Example:
Return only the values in an iterable which match the regex 'c.t' where . is a single character.
Input:
['cat', 'cut', 'abc', 'def']
@karthikbgl
karthikbgl / migrations.py
Last active August 29, 2015 14:06
Editing south migrations to run custom sql
# -*- coding: utf-8 -*-
import datetime
from south.db import db
from south.v2 import DataMigration
from django.db import models
class Migration(DataMigration):
def forwards(self, orm):
db.start_transaction()
@karthikbgl
karthikbgl / sunsigns.py
Last active August 29, 2015 14:06
Sunsigns models and create view using django
#models.py
CAPRICORN = 'Capricorn'
AQUARIUS = 'Aquarius'
PISCES = 'Pisces'
ARIES = 'Aries'
TAURUS = 'Taurus'
GEMINI = 'Gemini'
CANCER = 'Cancer'
LEO = 'Leo'