Skip to content

Instantly share code, notes, and snippets.

View AllieUbisse's full-sized avatar
🎯
Focusing

Allie .S Ubisse AllieUbisse

🎯
Focusing
View GitHub Profile
@AllieUbisse
AllieUbisse / ec2run.sh
Last active August 23, 2020 02:37
Mini script to help start, stop and check the status/public ip address for your aws ec2 via aws-cli
#!/bin/bash
########################################################################################
# START, STOP or STATUS #
# ---------------------------- #
# This scrip is intended to help you start, stop or get the IP address of #
# Current running EC2. #
# This will require you to 1st configure your AWC-CLI, namualy to ensure safety #
# #
# Please read the code to ensure that It does not cause any security issues #
# #
@AllieUbisse
AllieUbisse / docker_installer.sh
Last active August 12, 2020 05:27
How to Automate Docker installation by executing a Bash script from GitHub Gist 📖
#!/bin/bash
##################################################################################
# ----------------------------------------------------------------
# THIS SCRIPT WILL HELP YOUR AUTOMATE THE DOCKER INSTALATION STEPS
# ----------------------------------------------------------------
# Test was ran on aws ec2 instance.
#
# AUTHOR:
#Import All Functions
from pyspark.sql import SQLContext
from pyspark.sql import functions as F
from pyspark.sql import SparkSession
from pyspark.sql.functions import unix_timestamp, to_date, date_format, month, year, dayofyear, dayofweek, col
from pyspark.sql.types import TimestampType
from pyspark.sql import functions as F
from pyspark.sql import SparkSession
from pyspark.sql.functions import unix_timestamp, to_date, date_format, month, year, dayofyear, dayofweek, col
from pyspark.sql.types import TimestampType
@carlleston
carlleston / 3-ln_model.ipynb
Last active August 23, 2020 12:09
pre-processing and linear model in pyspark
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@liorshk
liorshk / mlflow_gridsearch.py
Created April 22, 2020 15:24
Create MLFlow runs with Sklearn Gridsearch object
def log_run(gridsearch: sklearn.GridSearchCV, experiment_name: str, model_name: str, run_index: int, conda_env, tags={}):
"""Logging of cross validation results to mlflow tracking server
Args:
experiment_name (str): experiment name
model_name (str): Name of the model
run_index (int): Index of the run (in Gridsearch)
conda_env (str): A dictionary that describes the conda environment (MLFlow Format)
tags (dict): Dictionary of extra data and tags (usually features)
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score, explained_variance_score
import mlflow
import mlflow.sklearn
import numpy as np
# Launch the experiment on mlflow
experiment_name = "electricityconsumption-forecast"
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
'''
spark/bin/spark-submit \
--master local --driver-memory 4g \
--num-executors 2 --executor-memory 4g \
--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0 \
sstreaming-spark-final.py
'''
from pyspark.sql import SparkSession
from pyspark.sql.types import *
from pyspark.sql.functions import expr