dksifoua

from pyspark.sql.types import StringType
from pyspark.sql.functions import udf

maturity_udf = udf(lambda age: "adult" if age >=18 else "child", StringType())

df = sqlContext.createDataFrame([{'name': 'Alice', 'age': 1}])
df.withColumn("maturity", maturity_udf(df.age))

dksifoua

# Create a file /etc/systemd/system/zookeeper.service and add it this content
==============================================================================================
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=dimitri_sifoua
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties

dksifoua

$ sudo useradd kafka -m
The -m flag ensures that a home directory will be created for the user.
This home directory, /home/kafka, will act as our workspace directory for executing commands in the sections below.

Set the password using passwd:
$ sudo passwd kafka

Add the kafka user to sudo group
$ sudo adduser kafka sudo

dksifoua

## SSH
> gcloud compute ssh spark-cluster-m --zone=us-east1-c --ssh-flag="-D" --ssh-flag="-N" --ssh-flag="10000"

The flag -D is to allow dynamic port forwardinig
The flag -N is to instruct gcloud to not open a remote shell
The flag 10000 is the port on which we want to open the ssh connection


## Start new browser session that uses the SOCKS proxy through the ssh tunnel created.
> "DIR of chrome.exe" "http://spark-cluster-m:8080" --proxy-server="socks5://localhost:10000" --host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE localhost" --user-data-dir=/tmp/spark-cluster-m

dksifoua

__all__ = ['EMO_UNICODE', 'UNICODE_EMO', 'EMOTICONS', 'EMOTICONS_EMO']

EMOTICONS = {
    u":‑\)": "Happy face or smiley",
    u":\)": "Happy face or smiley",
    u":-\]": "Happy face or smiley",
    u":\]": "Happy face or smiley",
    u":-3": "Happy face smiley",
    u":3": "Happy face smiley",
    u":->": "Happy face smiley",