Skip to content

Instantly share code, notes, and snippets.

View Wittline's full-sized avatar
:octocat:
Hi there

Ramses Alexander Coraspe Valdez Wittline

:octocat:
Hi there
104 followers · 123 following
View GitHub Profile
@Wittline
Wittline / all_values.yaml
Created July 9, 2022 00:38
All values found
5
_name
disc_6
values
na
cnt
70755
_type
STRING
14
@Wittline
Wittline / data_types.yaml
Created July 9, 2022 00:37
All possible data types and their percentages
5
name
disc_6
types
STRING
10.061573902903785
INTEGER
89.93842609709621
nullable
True
@Wittline
Wittline / inference_result.yaml
Last active July 9, 2022 00:28
Example of Inference result
0
name
key_1
type
STRING
nullable
False
1
name
date_2
@Wittline
Wittline / data.csv
Created July 9, 2022 00:02
Example of data set in csv format
key_1 date_2 cont_3 cont_4 disc_5 disc_6 cat_7 cat_8 cont_9 cont_10
HC2030 2016-11-16 622.27 -2.36 2 6 frequent happy 0.24 0.25
sP8147 2004-02-18 1056.16 59.93 2 8 never happy 1.94 2.29
Cq3823 2007-03-25 210.73 -93.94 1 1 never happy -0.11 -0.1
Hw9428 2013-12-28 1116.48 80.58 3 10 never surprised 1.27 1.15
xZ0360 2003-08-25 1038.3 12.37 6 17 never happy 1.76 1.76
IK2721 2012-10-19 835.17 16.3 4 11 frequent surprised 2.04 2.3
iK8875 2005-02-04 769.02 75.69 3 2 never happy -1.53 -1.56
qd0312 2014-11-17 273.11 66.2 1 8 frequent surprised 2.67 2.95
IO1104 2020-11-24 1844.21 -54.11 1 11 never surprised -0.42 -0.43
@Wittline
Wittline / test_livy.ipynb
Last active June 13, 2022 09:53
test_livy.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@Wittline
Wittline / docker-compose.yaml
Created June 12, 2022 22:44
docker-compose. with Apache Spark Cluster, PostgreSQL and Apache Livy server
version: '3'
services:
spark-master:
image: docker.io/bitnami/spark:2
environment:
- SPARK_MODE=master
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
@Wittline
Wittline / Dockerfile
Created June 12, 2022 22:37
Dockerfile for an Apache Livy server
FROM docker.io/bitnami/spark:2
USER root
ENV LIVY_HOME /opt/bitnami/livy
WORKDIR /opt/bitnami/
RUN install_packages unzip \
&& curl "https://downloads.apache.org/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip" -O \
&& unzip "apache-livy-0.7.1-incubating-bin" \
&& rm -rf "apache-livy-0.7.1-incubating-bin.zip" \
@Wittline
Wittline / Dockerfile
Created June 5, 2022 16:37
FROM python:3.8
WORKDIR /ingestion
COPY requirements.txt .
COPY wait-for-it.sh /usr/wait-for-it.sh
RUN pip install -r requirements.txt
@Wittline
Wittline / main.py
Created June 5, 2022 16:28
FastAPI Request
from typing import Optional
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import uvicorn
import pandas as pd
import psycopg2
import db_engine as dbe
import os
from pydantic import BaseModel
import configparser
@Wittline
Wittline / db_engine.py
Created June 5, 2022 16:22
Class Responsible of handle the connection with PostgreSQL
import psycopg2
import pandas as pd
from sql_queries import create_table_queries, drop_table_queries, fill_table_queries, create_constraints
def create_connection(params):
"""
create a new connection with the postgreSQL
database and return the cur and conn object
:param params: connection string