Skip to content

Instantly share code, notes, and snippets.

View flolas's full-sized avatar
🧭

Felipe Lolas flolas

🧭
12 followers · 3 following
  • Santiago, Chile
View GitHub Profile
@flolas
flolas / unpack_pandas.py
Last active July 25, 2016 22:27
Unpack Pandas series to multiple Pandas series(unpacking cols)
def unpack_col(col_to_unpack, df_to_append = None, header = 'col', sep=',', na_value=''):
import pandas as pd
unpacked_cols = col_to_unpack.fillna(na_value).apply(lambda x: pd.Series(x.split(','))).fillna(na_value)
#add dynamic columns names based on # of rows and parameter header passed for prefix (header_#)
col_names = []
for i in unpacked_cols.columns:
col_names.append(header + '_' + str(i))
unpacked_cols.columns = col_names
if isinstance(df_to_append, pd.DataFrame):
#return df concatenated with previously unpacked columns
@flolas
flolas / multiproc_trim.py
Created August 24, 2016 03:40
Multiprocessing Trim Rows Python
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# multiproc_trim.py
"""A program that reads integer values from a CSV file and writes out their
sums to another CSV file, using multiple processes if desired.
"""
import csv
import multiprocessing
import optparse
@flolas
flolas / Dockerfile.bcp
Created October 6, 2016 03:21
FROM ubuntu
ENV LANGUAGE en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LC_CTYPE en_US.UTF-8
ENV LC_MESSAGES en_US.UTF-8
ENV LC_ALL en_US.UTF-8
RUN apt update && \
@flolas
flolas / Event.json
Created March 30, 2017 16:41
EventStreaming
{
context : {
session : <int>,
id: <int>
},
timestamp: <timestamp(6)>,
event: <str>,
subevent: <str>,
fields: [
{
@flolas
flolas / teradataop.py
Created October 16, 2017 18:24
from airflow.contrib.operators.teradata_operator import TeradataOperator
task = TeradataOperator(sql='query.sql', teradata_conn_id='teradata-prod', task_id='Ejecuta_query_sql', dag=dag)
@flolas
flolas / james.sh
Created October 22, 2017 20:42
sudo apt-get -y update
sudo apt-get -y install default-jre libc6-i386 libc6-dev-i386 curl unzip vim netcat telnet
sudo useradd james
sudo mkdir /home/james
sudo chown -R james:james /home/james
sudo adduser james sudo
mkdir /opt/james
sudo curl -o /opt/james/james-server-app-3.0.0-app.zip http://www.apache.org/dist/james/server/3.0.0/james-server-app-3.0.0-app.zip
sudo unzip /opt/james/james-server-app-3.0.0-app.zip -d /opt/james
sudo chown -R james:james /opt/james
@flolas
flolas / index.html
Last active January 3, 2018 15:46
PBI
<!DOCTYPE html>
<html>
<head>
<title>Minimal sample using ADAL.JS</title>
<meta charset="utf-8" />
<script src="https://secure.aadcdn.microsoftonline-p.com/lib/1.0.11/js/adal.min.js"></script>
<script src="powerbi.js"></script>
<script src="https://code.jquery.com/jquery-3.2.1.min.js" integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4=" crossorigin="anonymous"></script>
<script>window.jQuery || document.write('<script src="js/vendor/jquery-3.2.1.min.js"><\/script>')</script>
</head>
@flolas
flolas / README.md
Last active August 20, 2018 04:08
Siddhi with Kerberized Kafka Source

Kerberized Kafka with Siddhi

  • krb5-user package with krb.conf
  • crt in JKS
  • valid keytab for renewing token
@flolas
flolas / SlackCallBack.py
Created January 8, 2019 19:43
def get_slack_on_retry_callback(owners_list):
def retry_callback(context):
from airflow.operators import SlackAPIPostOperator
slack_conf = getVarIfExists("slack_conf", json=True)
message = '[{DS}]\n[{DAG}]\n[{TASK}]\n :warning: La tarea se está reintentando :warning: \n Alerta para {OWNERS} \n -----------'\
.format(
DAG = context['dag'].dag_id,
TASK = context['task'].task_id,
DS = context['ds'],
OWNERS = ' '.join(owners_list)
@flolas
flolas / bteq.py
Last active January 12, 2021 02:05
"""
Code that goes along with the Airflow tutorial located at:
https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py
"""
from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.docker_operator import DockerOperator
start = datetime.combine(datetime.today() - timedelta(2), datetime.min.time())