Skip to content

Instantly share code, notes, and snippets.

View geeknam's full-sized avatar

Nam Ngo geeknam

View GitHub Profile
@robertdale
robertdale / describe.groovy
Last active September 27, 2019 07:10
JanusGraph Schema Describe Command
// This can be imported via ./bin/gremlin.sh -i describe.groovy
// A variable 'graph' must be defined with a JanusGraph graph
// Run it as a plugin command ':schema'
// :schema describe
//
import org.janusgraph.graphdb.database.management.MgmtLogType
import org.codehaus.groovy.tools.shell.Groovysh
import org.codehaus.groovy.tools.shell.CommandSupport
@sjudeng
sjudeng / jg_tp33_cdh512.md
Last active November 6, 2022 12:54
Testing OLAP using JanusGraph with TinkerPop 3.3.0 and Spark 2.2 on Yarn (Cloudera)
@dusenberrymw
dusenberrymw / spark_tips_and_tricks.md
Last active February 8, 2023 05:11
Tips and tricks for Apache Spark.

Spark Tips & Tricks

Misc. Tips & Tricks

  • If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
  • Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
  • Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the
@mmazzarolo
mmazzarolo / Appfile
Created May 17, 2016 11:27
Simple Fastlane setup for React-Native (Android - iOS)
# iOS
app_identifier "com.myapp.app" # The bundle identifier of your app
apple_id "me@gmail.com" # Your Apple email address
team_id "1234ABCD" # Developer Portal Team ID
# Android
json_key_file "./google-play-api-secret.json" # Path to the json secret file - Follow https://github.com/fastlane/supply#setup to get one
package_name "com.myapp.app" # Your Android app package
@giwa
giwa / file0.txt
Last active March 27, 2020 11:31
Install hive on Mac with Homebrew ref: http://qiita.com/giwa/items/dabf0bb21ae242532423
$ brew update
$ brew install hive
@epicserve
epicserve / factories.py
Created October 3, 2012 19:37
Example Factory-boy (https://github.com/dnerdy/factory_boy) factory that uses a file field.
from django.template.defaultfilters import slugify
from django.contrib.sites.models import Site
from django.core.files import File
from taggit.models import Tag
from .models import Photo
import factory
import os
TEST_MEDIA_PATH = os.path.join(os.path.dirname(__file__), 'tests', 'test_media')
TEST_PHOTO_PATH = os.path.join(TEST_MEDIA_PATH, 'test_photo.png')
@jpennell
jpennell / fabfile.py
Created September 18, 2012 01:15
Fabric fabfile for Django/Heroku App
from fabric.api import env, local, require
def deploy():
"""fab [environment] deploy"""
require('environment')
maintenance_on()
push()
syncdb()
migrate()
@mikeyk
mikeyk / gist:1329319
Created October 31, 2011 22:56
Testing storage of millions of keys in Redis
#! /usr/bin/env python
import redis
import random
import pylibmc
import sys
r = redis.Redis(host = 'localhost', port = 6389)
mc = pylibmc.Client(['localhost:11222'])
A unit test helper library for App Engine.
Note that this is currently COMPLETELY UNTESTED. Consider it demo code only.
This library aims to make it easier to unit-test app engine apps and libraries
by handling the creation and registration of service stubs and so forth for you.
It also provides a custom implementation of the Capability service that allows
you to specify what capabilities you want it to report as disabled, and it wraps
all stubs in a wrapper that will throw a CapabilityDisabledError if you attempt
to use a disabled service or method.