Skip to content

Instantly share code, notes, and snippets.

Avatar
🏄

Berk Orbay berkorbay

🏄
View GitHub Profile
@berkorbay
berkorbay / read_xlsx_to_pandas_fast.py
Created Aug 3, 2021
Blazing fast of xlsx files into pandas
View read_xlsx_to_pandas_fast.py
from datatable import fread
import pandas as pd
excel_path = "my_excel_file.xlsx"
df = fread(excel_path+"/sheet_name").to_pandas()
@berkorbay
berkorbay / ortools_docker_run.sh
Created Jul 30, 2021
OR-Tools Docker OOM issue (Exit code 137)
View ortools_docker_run.sh
# This is an example docker run command to run streamlit which will include an ortools model
# -m 12g will keep the memory at 12GB
# --oom-kill-disable will prevent the container to go under because of out of memory sigkill
docker run -dp 80:8501 --oom-kill-disable -m 12g <image_name>
@berkorbay
berkorbay / Dockerfile
Last active Jul 29, 2021
Minimal streamlit (v0.85) in Python 3.9.6 Dockerfile
View Dockerfile
FROM python:3.9.6
RUN pip install --no-cache-dir streamlit
EXPOSE 8501
ENTRYPOINT [ "streamlit" ]
CMD ["hello"]
## FOR ANY APPLICATION
ENTRYPOINT [ "streamlit", "run"]
CMD ["my_streamlit_file.py"]
@berkorbay
berkorbay / annoying_problems_python
Last active Apr 24, 2021
Unordered list of solutions to annoying problems with Python
View annoying_problems_python
## Skip failing installations on requirements.txt
## https://stackoverflow.com/a/54053100/3608936
cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install
### Jupyter Notebook not working
## https://stackoverflow.com/a/59571314/3608936
python3 -m notebook
@berkorbay
berkorbay / cognito_r_direct_query.R
Created Feb 17, 2021
AWS Cognito Simple Authentication with R
View cognito_r_direct_query.R
#######
#### This gist is a very quick way to implement just authentication function. This is a minimal example.
#### You need an AWS IAM account and your AWS Cognito User Pool set up.
#### Check the steps in the description of https://github.com/chi2labs/cognitoR
#### After creating your user pool on the left sidebar under General Settings, click on App clients
### Make sure "Enable username password based authentication (ALLOW_USER_PASSWORD_AUTH)" is checked
### Create a user from the interface for testing purposes. Let username be "usertest" and password be "passtest".
#######
#### Variables
@berkorbay
berkorbay / parse_ys_gmail_threads.R
Last active May 24, 2020
Parses YS order emails and brings them under a single data frame
View parse_ys_gmail_threads.R
options(stringsAsFactors=FALSE)
library(tidyverse)
library(rvest)
library(gmailr)
## FOLLOW AUTH INSTRUCTIONS FROM HERE https://gmailr.r-lib.org/articles/gmailr.html
parse_order_table <-function(my_msg,full_info_list=FALSE){
my_info <- read_html(gm_body(my_msg)) %>% html_nodes(xpath="/html/body/table/tr/td/center/table[1]/tr[2]/td") %>% html_children() %>% `[[`(2)
@berkorbay
berkorbay / qnb_finansbank_api_v0.r
Created Apr 6, 2020
QNB Finansbank Rest API Methods
View qnb_finansbank_api_v0.r
# Source: https://developer.qnbfinansbank.com
# Currently in beta phase so we DO NOT use id and secret instead we use a "code" from one of the accounts.
# Go to: https://developer.qnbfinansbank.com/discovery/GettingStarted/UseTestPlatform
## Follow steps until step 5. The code you get is your token
url = "https://sandbox-api.qnbfinansbank.com/v0/accounts"
my_token = "MY_VERY_LONG_TOKEN"
resval <- httr::GET(url, httr::add_headers(Authorization = paste("Bearer", the_token, sep = " ")))
@berkorbay
berkorbay / github_desktop_ubuntu.sh
Last active Sep 23, 2021
To install Github Desktop for Ubuntu
View github_desktop_ubuntu.sh
#sudo wget https://github.com/shiftkey/desktop/releases/download/release-2.1.0-linux1/GitHubDesktop-linux-2.1.0-linux1.deb
#sudo gdebi GitHubDesktop-linux-2.1.0-linux1.deb
# UPDATE (2021-03-05): Thanks to PaoloRanzi81's comment, the updated code is as follows https://gist.github.com/PaoloRanzi81
sudo wget https://github.com/shiftkey/desktop/releases/download/release-2.6.3-linux1/GitHubDesktop-linux-2.6.3-linux1.deb
### Uncomment below line if you have not installed gdebi-core before
# sudo apt-get install gdebi-core
sudo gdebi GitHubDesktop-linux-2.6.3-linux1.deb
@berkorbay
berkorbay / ibb_solar_panel_analysis.R
Created Jan 19, 2020
İBB Açık Veri Portalı İkitelli Güneş Enerjisi Santrali üretim verisi inceleme kodu
View ibb_solar_panel_analysis.R
library(tidyverse)
library(lubridate)
library(jsonlite)
library(devtools)
#https://data.ibb.gov.tr/dataset/ikitelli-gunes-enerjisi-santrali-elektrik-uretim-miktarlari/resource/52afa9a3-2ea1-420b-a783-505cfe635ece
devtools::source_gist("https://gist.github.com/berkorbay/493e82b3d48245317c57a5d0c334492e")
raw_df <- ibb_data_pull("52afa9a3-2ea1-420b-a783-505cfe635ece")
wip_df <- raw_df %>% transmute(dt = as_datetime(Tarih),production=as.numeric(Uretim__kWh_))
@berkorbay
berkorbay / ibb_open_data_query_function.R
Created Jan 19, 2020
İBB Açık Veri Portalı'ndan veri çekme R fonksiyonu
View ibb_open_data_query_function.R
## İlgili veri setinde önizleme'ye tıklayın çıkan url adresinde "/resource/"tan sonraki kodu kopyalayın (data_id)
ibb_data_pull <- function(data_id="e1e2b771-b4de-49d3-b779-17b4bba10fb7"){
ibb_query_url <- paste0("https://data.ibb.gov.tr/api/3/action/datastore_search_sql?sql=SELECT%20*%20from%20%22",data_id,"%22")
json_query_response <- jsonlite::fromJSON(ibb_query_url)
return_df <- json_query_response$result$records %>% rename_all(~gsub("^_","",.)) %>% rename_all(~gsub(" |\\(|\\)|/","_",.)) %>% tbl_df()
return(return_df)