Skip to content

Instantly share code, notes, and snippets.

Avatar
🎯
Focusing

Sam Bessalah samklr

🎯
Focusing
View GitHub Profile
@samklr
samklr / Job.scala
Created Jun 27, 2017
Spark Streaming Websocket Receiver
View Job.scala
val ssc = new StreamingContext("local", "datastream", Seconds(15))
// create InputDStream
ssc.registerInputStream(stream)
// interact with stream
ssc.start()
@samklr
samklr / install-proto.sh
Created Apr 20, 2015
Install Protobuf debian ...
View install-proto.sh
#! /bin/bash
wget https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz
tar xzf protobuf-2.6.1.tar.gz
cd protobuf-2.6.1
sudo apt-get update
sudo apt-get install build-essential
sudo ./configure
sudo make
sudo make check
sudo make install
View redis-osx.sh
#!/bin/bash
brew install redis
ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents # Enable Redis autostart
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist # load Redis server
# homebrew.mxcl.redis.plist contains reference to redis.conf file location: /usr/local/etc/redis.conf
redis-server /usr/local/etc/redis.conf # Start Redis server using configuration file, Ctrl+C to stop
@samklr
samklr / OffsetMngHbase.scala
Last active Mar 31, 2020
Offset Management on HBase
View OffsetMngHbase.scala
/*
Save offsets for each batch into HBase
*/
def saveOffsets(TOPIC_NAME:String,GROUP_ID:String,offsetRanges:Array[OffsetRange],
hbaseTableName:String,batchTime: org.apache.spark.streaming.Time) ={
val hbaseConf = HBaseConfiguration.create()
hbaseConf.addResource("src/main/resources/hbase-site.xml")
val conn = ConnectionFactory.createConnection(hbaseConf)
val table = conn.getTable(TableName.valueOf(hbaseTableName))
val rowKey = TOPIC_NAME + ":" + GROUP_ID + ":" +String.valueOf(batchTime.milliseconds)
@samklr
samklr / ipython-nbserver.py
Last active Mar 26, 2020
Install Ipython Notebook on a VM and Launch it as a server in a Cloud Platform. Here, in Google Compute Engine.
View ipython-nbserver.py
##### Install a lot of stuff first #####
$sudo apt-get update
##install python
$ wget http://09c8d0b2229f813c1b93-c95ac804525aac4b6dba79b00b39d1d3.r79.cf1.rackcdn.com/Anaconda-2.0.1-Linux-x86_64.sh
$ sudo bash anaconda........sh
##install necessary libs
$ sudo apt-get install -y python-matplotlib python-tornado ipython ipython-notebook python-setuptools python-pip
@samklr
samklr / ansible-gce.sh
Last active Jan 11, 2020
Ansible GCE setup
View ansible-gce.sh
#! /bin/sh
### Must have Gcloud sdk installed and configured
###Create a micro instance as ansible master
gcloud compute --project $PROJECT_NAME instances create "ansible" --zone "us-central1-b" --machine-type "f1-micro" --network "default" --maintenance-policy "MIGRATE" --scopes "https://www.googleapis.com/auth/userinfo.email" "https://www.googleapis.com/auth/compute" "https://www.googleapis.com/auth/devstorage.full_control" --tags "http-server" "https-server" --no-boot-disk-auto-delete
###or a centos like in the tutorial
gcloud compute --project $PROJECT_NAME instances create "ansible-master" --zone "us-central1-b" --machine-type "g1-small" --network "default" --maintenance-policy "MIGRATE" --scopes "https://www.googleapis.com/auth/userinfo.email" "https://www.googleapis.com/auth/compute" "https://www.googleapis.com/auth/devstorage.full_control" --tags "http-server" "https-server" --image "https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20140926" --no-boot-disk-auto-de
View kamon.conf
play.modules.enabled += "com.samklr.KamonModule"
kamon {
environment {
service = "my-svc"
}
jaeger {
@samklr
samklr / hdfs-mesos.md
Last active Dec 5, 2019
Setup HDFS on Mesos, Run Spark Cluster dispatcher via Marathon
View hdfs-mesos.md

Setup Mesos-DNS

Scripts for setting up

sudo mkdir /etc/mesos-dns
sudo vi /etc/mesos-dns/config.json
View golang_job_queue.md
View StreamsDSLAndProcessorExample.java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
You can’t perform that action at this time.