devendrasr

2018 Update: Easiest option is Justin's repo and image

Just run this from your Mac terminal and it'll drop you in a container with full permissions on the Moby VM. This also works for Docker for Windows for getting in Moby Linux VM (doesn't work for Windows Containers).

docker run -it --rm --privileged --pid=host justincormack/nsenter1

more info: https://github.com/justincormack/nsenter1

---

devendrasr

How to install virtualenv:

Install pip first

sudo apt-get install python3-pip

Then install virtualenv using pip3

sudo pip3 install virtualenv 

devendrasr

import base64
import gzip

def decompress(encoded_gz_data: bytes):
    gzip_string = base64.b64decode(encoded_gz_data)
    return gzip.decompress(gzip_string)
  
  # Test - 
  # gz_data = b"H4sIAAAAAAAAAO1WbW/iRhD+K5E/H8Rv2dcEipao0hCRUBF5z05RShxV7DCtvrW6+....<your input string here.>"
  # print(decompress(gz_data))

devendrasr

Spark Tips & Tricks

Misc. Tips & Tricks

• If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
• Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
• Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the

devendrasr

TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

devendrasr

➜  dist kafka-topics --list --zookeeper localhost:2181
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Error occurred during initialization of VM
java.lang.Error: Properties init: Could not determine current working directory.
	at java.lang.System.initProperties(Native Method)
	at java.lang.System.initializeSystemClass(System.java:1166)

devendrasr

curl -v -F "@file=template-document-hx.doc" host:port/path/to/server

devendrasr

# Add alias for zsh shell
echo "alias json='python -mjson.tool'" >> ~/.zshrc

# Add alias to bash shell
echo "alias json='python -mjson.tool'" >> ~/.bashrc

devendrasr

//Usage - toBinaryString("\xd4!")) ==> 110101000010000100100000

func toBinaryString(str string) string {
	bitstring := ""
	for i := 0; i < len(str); i++ {
		for bit := 7; bit >= 0; bit-- {
			set := (str[i]>>uint(bit))&1 == 1
			if set {
				bitstring += "1"
			} else {

devendrasr

package utils

import (
	"runtime"

	"github.com/Sirupsen/logrus"
)

// DecorateRuntimeContext appends line, file and function context to the logger
func DecorateRuntimeContext(logger *logrus.Entry) *logrus.Entry {