Skip to content

Instantly share code, notes, and snippets.

View djamelz's full-sized avatar

Djamel Zouaoui djamelz

15 followers · 0 following
View GitHub Profile
@djamelz
djamelz / custom_deser.scala
Created September 29, 2016 12:28
custom deserialization with json4s for date in string format to timestamp (long)
import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat
import org.json4s._
import org.json4s.native.JsonMethods._
val datePattern = "yyyy-MM-dd hh:mm:ss.SSSSSS"
val s = """{"timestamp": "2016-09-29 11:31:13.247772", "domain": "d1", "filePath": "s3://..."}"""
case class Event(domain : String, filePath : String, timestamp : Long)
@djamelz
djamelz / emr_wait.py
Created October 12, 2015 21:35
Sync waiting for the TERMINATED status of an EMR cluster use as : python emr_wait.py us-east-1 j-XXXXXXXXXXXX
#!/usr/bin/python
import boto.emr
import time
import sys
region = sys.argv[1]
clusterId = sys.argv[2]
def getStatus(cluster_id):
@djamelz
djamelz / aliasSwitching.py
Created October 12, 2015 21:09
Elasticsearch Alias switching in python => restore the new one, close the latest and delete the olders (based on name versioning index_name_xxx, index_name_xxy for index name and index_name for alias)
from elasticsearch import Elasticsearch
es = Elasticsearch(hosts=['es-host:9200'], timeout=600) #long timeout for the restore operation
snapshotName='index_name_20151012140502'
alias = es.indices.get_alias('index_name').keys()[0] if es.indices.exists_alias(name='index_name') else ''
indices = filter(lambda x: x.startswith('index_name'), es.indices.get_aliases())
indices.sort()
print 'restore repository'
es.snapshot.restore(repository='repository_name', snapshot=snapshotName, body ='{ "ignore_unavailable": "true", "include_global_state": false}', wait_for_completion=True)
if (len(alias) > 1):
@djamelz
djamelz / Ec2Utils.scala
Created October 7, 2015 22:28
Play with ec2 and java aws api.... Not the best way to do with scala (no future, tail rec vs. while...) but useful to have sync way to pop ec2 instances and kill them
import com.amazonaws.auth.{BasicAWSCredentials, InstanceProfileCredentialsProvider}
import com.amazonaws.services.ec2.AmazonEC2Client
import com.amazonaws.services.ec2.model._
import com.amazonaws.util.Base64
trait Ec2Utils extends Logging {
val AWS_ACCESS_KEY_ID: String
val AWS_SECRET_ACCESS_KEY: String
val instanceProfile = false
@djamelz
djamelz / es.sh
Last active October 6, 2015 12:25
Script to install es 1.1 on ec2 ubuntu ami (sudo es.sh)
#!/bin/sh
cd /mnt/
apt-get -y update
apt-get -y install openjdk-7-jre
apt-get -y install zip
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.1.1.zip && unzip elasticsearch-1.1.1.zip && /mnt/elasticsearch-1.1.1/bin/plugin -install mobz/elasticsearch-head
/mnt/elasticsearch-1.1.1/bin/plugin -install elasticsearch/elasticsearch-cloud-aws/2.1.1
echo " network.bind_host: $(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')" >> /mnt/elasticsearch-1.1.1/config/elasticsearch.yml
screen -dmS new_screen sh
@djamelz
djamelz / FunWithAwsCli.txt
Last active October 6, 2015 07:45
Fun with aws-cli
#pop emr cluster with spark job to execute
aws emr create-cluster --name "<cluster name>"\
--region us-east-1\
--log-uri s3://<bucket>/logs/<cluster name>/\
--enable-debugging\
--release-label emr-4.0.0\
--applications Name=Spark\
--ec2-attributes KeyName=<keyname>\
--use-default-roles\
@djamelz
djamelz / spark-ec2.py
Created November 16, 2014 00:27 — forked from amar-analytx/spark-ec2.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance