Superset: Airbnb’s data exploration platform
Getting Started WithThese instructions are for Amazon Linux Version 2
Update Python and PIP versions on EC2 (Amazon AMI)
sudo yum update -y
sudo yum install python3 -y
Install VMWare Workstation PRO 17 (Read it right. PRO!) | |
Sub to me on youtube pls - PurpleVibe32 | |
if you want more keys - call my bot on telegram. @purector_bot (THE BOT WONT REPLY ANYMORE) - Or: https://cdn.discordapp.com/attachments/1040615179894935645/1074016373228978277/keys.zip - the password in the zip is 102me. | |
--- | |
This gist can get off at any time. | |
PLEASE, DONT COPY THIS. IF YOU FORK IT, DONT EDIT IT. | |
*If you have a problem comment and people will try to help you! | |
*No virus | |
*No spam just license key |
#!/usr/bin/env python3 | |
from argparse import ArgumentParser | |
from os import listdir, stat | |
from os.path import sep as pathsep | |
from glob import glob | |
from shutil import SameFileError, copy, get_terminal_size | |
from typing import Union | |
from sys import exit, stdout | |
from alive_progress import alive_bar | |
from concurrent.futures import ProcessPoolExecutor, as_completed |
#!/usr/bin/env python | |
import os | |
import boto3 | |
import argparse | |
import json | |
# `pip install -U PTable` will get you the right fork of PrettyTable | |
from prettytable import PrettyTable | |
from botocore.exceptions import ClientError |
import pkg_resources | |
installed_packages = pkg_resources.working_set | |
for package in sorted(installed_packages, lambda x: x.key): | |
print "{}=={}".format(package.key, package.version) |
awscli==1.14.45 | |
boto3==1.5.35 | |
botocore==1.8.49 | |
colorama==0.3.7 | |
docutils==0.14 | |
futures==3.2.0 | |
jmespath==0.9.3 | |
numpy==1.14.0 | |
pandas==0.22.0 | |
pip==9.0.1 |
These instructions are for Amazon Linux Version 2
sudo yum update -y
sudo yum install python3 -y
AWSTemplateFormatVersion: "2010-09-09" | |
Description: Put a SecureString parameter in SSM Parameter Store | |
Parameters: | |
KmsKeyId: | |
Type: String | |
Description: The KMS Key you want to use to encrypt the string. | |
SSMParameterKey: | |
Type: String | |
Description: The Parameter Store Key | |
SSMParameterValue: |
def s3_to_pandas(client, bucket, key, header=None): | |
# get key using boto3 client | |
obj = client.get_object(Bucket=bucket, Key=key) | |
gz = gzip.GzipFile(fileobj=obj['Body']) | |
# load stream directly to DF | |
return pd.read_csv(gz, header=header, dtype=str) | |
def s3_to_pandas_with_processing(client, bucket, key, header=None): |
At the time of writing, Python v3.5 and PIP v9.0.1 were available on AWS EC2.
sudo yum update -y
sudo yum install python35 -y