sany2k8

API workthough

1. Open a browser

   # start an instance of firefox with selenium-webdriver
   driver = Selenium::WebDriver.for :firefox
   # :chrome -> chrome
   # :ie     -> iexplore
   

• Go to a specified URL

sany2k8

# 10_basic.py
# 15_make_soup.py
# 20_search.py
# 25_navigation.py
# 30_edit.py
# 40_encoding.py
# 50_parse_only_part.py

sany2k8

Scrapy Cheatsheet

For test

scrapy shell https://example.com/

Run

scrapy crawl spider_name

Output

scrapy crawl dapps -o data/07-07-dapps.csv
scrapy crawl dapps -t csv -o - >"data/dapp/$DATE-dapp.csv"

sany2k8

def create_link(file_name, file_id):
    return f'http://localhost:port/{file_name}/{file_id}'
  
def make_clickable(val):
    # target _blank to open new window
    return '<a target="_blank" href="{}">{}</a>'.format(val, val)

data['link'] = data.apply(lambda x: create_link(str(x['file_path']), x['file_id']), axis=1)
clickable_table = data.style.format({'link': make_clickable})

sany2k8

1) see re: increasing shmmax http://stackoverflow.com/a/10629164/1283020

2) add to postgresql.conf:
shared_preload_libraries = 'pg_stat_statements'   # (change requires restart)
136 pg_stat_statements.max = 1000
137 pg_stat_statements.track = all

3) restart postgres

4) check it out in psql

sany2k8

Presto Specific

• Don’t SELECT *, Specify explicit column names (columnar store)
• Avoid large JOINs (filter each table first)
  • In PRESTO tables are joined in the order they are listed!!
  • Join small tables earlier in the plan and leave larger fact tables to the end
  • Avoid cross joins or 1 to many joins as these can degrade performance
• Order by and group by take time
  • only use order by in subqueries if it is really necessary
• When using GROUP BY, order the columns by the highest cardinality (that is, most number of unique values) to the lowest.

sany2k8

MySQL Cheat Sheet

Help with SQL commands to interact with a MySQL database

MySQL Locations

• Mac /usr/local/mysql/bin
• Windows /Program Files/MySQL/MySQL version/bin
• Xampp /xampp/mysql/bin

Add mysql to your PATH

sany2k8

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                       7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * *  command_to_execute

sany2k8

Tar Usage / Cheat Sheet

Compress a file or directory

e.g: tar -czvf name-of-archive.tar.gz /path/to/directory-or-file

• -c: Create an archive.
• -z: Compress the archive with gzip.
• -v: makes tar talk a lot. Verbose output shows you all the files being archived and much.
• -f: Allows you to specify the filename of the archive.

sany2k8

How to install virtualenv:

Install pip first

sudo apt-get install python3-pip

Then install virtualenv using pip3

sudo pip3 install virtualenv