gggordon

Spark Tips & Tricks

Misc. Tips & Tricks

• If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
• Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
• Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the

gggordon

Deploying a subfolder to GitHub Pages

Sometimes you want to have a subdirectory on the master branch be the root directory of a repository’s gh-pages branch. This is useful for things like sites developed with Yeoman, or if you have a Jekyll site contained in the master branch alongside the rest of your code.

For the sake of this example, let’s pretend the subfolder containing your site is named dist.

Step 1

Remove the dist directory from the project’s .gitignore file (it’s ignored by default by Yeoman).

gggordon

#Steps to install latest Laravel, LEMP on AWS Ubuntu 14.4 version. This tutorial is the improvised verision of this tutorial on Digitalocean based on my experience.

Install PHP 7 on Ubuntu

Run the following commands in sequence.

sudo apt-get install -y language-pack-en-base
sudo LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install zip unzip

gggordon

# bash <(curl -s https://gist.github.com/drye/5387341/raw/ec72cddfe43ec3d39c91a3c118cb68ab14a049f8/enable_dnsmasq_on_osx.sh)

# ----------------------
# installing dnsmasq and enable daemon
# ----------------------
brew install dnsmasq
sudo cp -v $(brew --prefix dnsmasq)/homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons
# ----------------------
# adding resolver for vbox domain
# ----------------------

gggordon

Never touch your local /etc/hosts file in OS X again

To setup your computer to work with *.dev domains, e.g. project.dev, awesome.dev and so on, without having to add to your hosts file each time.

Requirements

• Homebrew
• Mountain Lion

Install

gggordon

make a Virtual Router on Ubuntu 13.04

Run this in a terminal:

sudo apt-get install hostapd

Then, open a text editor program, for example gedit. Copy the following into it.

• http://wireless.kernel.org/en/users/Documentation/hostapd#Establishing_Baseline_for_Configuration

gggordon

Introduction

• C-a == Ctrl-a
• M-a == Alt-a

General

:q        close
:w        write/saves
:wa[!]    write/save all windows [force]
:wq       write/save and close