Skip to content

Instantly share code, notes, and snippets.

View eidosam's full-sized avatar

Samir Aleido eidosam

11 followers · 0 following
View GitHub Profile
@eidosam
eidosam / create-transfer-job.sh
Created March 7, 2019 14:03
Start a Google Cloud Storage Transfer job from command-line
#!/usr/bin/env bash
### ------ Preparation ------ ###
# Acquire new user credentials to use for Application Default Credentials
# Run: gcloud auth application-default login
### ------ -------------- --- ###
function printUsage() {
echo -e "
\rUsage: bash ./create-transfer-job.sh [options] <s3-source-data> <gcs-bucket-name>
@eidosam
eidosam / parquet-s3
Last active July 9, 2020 15:25
Inspect parquet files stored in S3 using `parquet-tools`
#!/usr/bin/env bash
set -o errexit
### ------ Preparation ------ ###
# brew tap adoptopenjdk/openjdk && brew cask install adoptopenjdk8
# command -v mvn > /dev/null || brew install maven
# command -v parquet-tools > /dev/null || brew install parquet-tools
@eidosam
eidosam / parquet-head
Created February 7, 2019 15:39
Read the first couple of rows in Parquet file stored in S3
#!/usr/bin/env bash
profile=default
limit=100
while [ -n "$1" ]; do
case $1 in
--profile )
shift
profile=$1
@eidosam
eidosam / sftp-only.sh
Created February 9, 2018 09:48
Add SFTP user can access only SFTP service
adduser sftping
mkdir -p /var/sftp/
chmod 755 /var/sftp
chown sftping: /var/sftp/
cat <<'EOF' >> /etc/ssh/sshd_config
Subsystem sftp internal-sftp
Match User sftping
X11Forwarding no
@eidosam
eidosam / install-orc-tools.sh
Created January 3, 2018 10:27
Install Apache ORC tools
apt-get -y install build-essential cmake
curl -sSLO http://www.mirrorservice.org/sites/ftp.apache.org/orc/orc-1.4.1/orc-1.4.1.tar.gz
tar -zxf orc-1.4.1.tar.gz
cd orc-1.4.1
mkdir build
cd build
cmake .. -DBUILD_JAVA=OFF
make install
@eidosam
eidosam / crc16-xmodem.awk
Last active February 20, 2022 23:14
CRC16 XMODEM ZMODEM implementation in AWK, it is the model used by Redis Cluster for key distribution
BEGIN {
cmpl = 0xffff;
initial = 0x0000;
TABLE[ 0] = 0x0000;
TABLE[ 1] = 0x1021;
TABLE[ 2] = 0x2042;
TABLE[ 3] = 0x3063;
TABLE[ 4] = 0x4084;
@eidosam
eidosam / install-r-rstudio-ubuntu.sh
Created September 8, 2017 11:59
#!/bin/bash
#install R
yum install -y R
#install RStudio-Server
wget https://download2.rstudio.org/rstudio-server-rhel-0.99.465-x86_64.rpm
yum install -y --nogpgcheck rstudio-server-rhel-0.99.465-x86_64.rpm
#install shiny and shiny-server
R -e "install.packages('shiny', repos='http://cran.rstudio.com/')"
wget https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.4.0.718-rh5-x86_64.rpm
yum install -y --nogpgcheck shiny-server-1.4.0.718-rh5-x86_64.rpm
@eidosam
eidosam / install-r-rstudio.sh
Last active June 9, 2018 18:11
apt-get update && apt-get -y install r-base curl
LIBSSL=libssl1.0.0_1.0.1t-1+deb8u6_amd64.deb
curl -sSO http://security.debian.org/debian-security/pool/updates/main/o/openssl/$LIBSSL
dpkg -i $LIBSSL
RSTUDIO=rstudio-server-1.0.153-amd64.deb
curl -sSO https://download2.rstudio.org/$RSTUDIO
gdebi $RSTUDIO