lxneng

install PostgreSQL 9 in Mac OSX via Homebrew


Mac OS X Snow Leopard
System Version: Mac OS X 10.6.5
Kernel Version: Darwin 10.5.0


Install notes for PostgreSQL 9.0.1 install using Homebrew:
	sh-3.2# brew install postgresql

lxneng

(env)~/env pip install pycurl
Downloading/unpacking pycurl
  Downloading pycurl-7.19.0.tar.gz (71Kb): 71Kb downloaded
  Running setup.py egg_info for package pycurl
    sh: curl-config: not found
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/home/eric/env/build/pycurl/setup.py", line 90, in <module>
        raise Exception, ("`%s' not found -- please install the libcurl development files" % CURL_CONFIG)
    Exception: `curl-config' not found -- please install the libcurl development files

lxneng

Latency Comparison Numbers

Note: "Forked" from Latency Numbers Every Programmer Should Know

Event	Nanoseconds	Microseconds	Milliseconds	Comparison
L1 cache reference	0.5	-	-	-
Branch mispredict	5.0	-	-	-
L2 cache reference	7.0	-	-	14x L1 cache
Mutex lock/unlock	25.0	-	-	-

lxneng

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

lxneng

import streamlit as st
import os
import sys
import importlib.util

# Parse command-line arguments.
if len(sys.argv) > 1:
    folder = os.path.abspath(sys.argv[1])
else:
    folder = os.path.abspath(os.getcwd())

lxneng

Presto Specific

• Don’t SELECT *, Specify explicit column names (columnar store)
• Avoid large JOINs (filter each table first)
  • In PRESTO tables are joined in the order they are listed!!
  • Join small tables earlier in the plan and leave larger fact tables to the end
  • Avoid cross joins or 1 to many joins as these can degrade performance
• Order by and group by take time
  • only use order by in subqueries if it is really necessary
• When using GROUP BY, order the columns by the highest cardinality (that is, most number of unique values) to the lowest.

lxneng

class Translator(object):
    translate_url = "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=%(message)s&langpair=%(from)s%%7C%(to)s"

    def __init__(self):
        self.browser = Browser()

    def translate(self, message, lang_to='en', lang_from=''):
        """
        Given a 'message' translate it from 'lang_from' to 'lang_to'.
        If 'lang_from' is empty, auto-detects the language.

lxneng

curl ifconfig.me/ip -> IP Adress

curl ifconfig.me/host -> Remote Host

curl ifconfig.me/ua ->User Agent

curl ifconfig.me/port -> Port

thonks to http://ifconfig.me/

lxneng

Chapter 2 Indexing


	Adding indexes to tables
			ALTER TABLE books ADD INDEX IDX_author(author), ADD INDEX IDX_title(title);
			ALTER TABLE books ADD INDEX IDX_title(title(20)); -- tell MySQL to only copy the first 20 characters of the title to the index:

	Adding a fulltext index
			ALTER TABLE posts ADD FULLTEXT INDEX IDX_content(content);
			

lxneng

PySpark on YARN in self-contained environments

Author: https://github.com/seanorama

Reference:

• https://community.cloudera.com/t5/Community-Articles/Using-VirtualEnv-with-PySpark/ta-p/245905
• https://community.cloudera.com/t5/Community-Articles/Running-PySpark-with-Conda-Env/ta-p/247551
• https://stackoverflow.com/questions/48770263/bundling-python3-packages-for-pyspark-results-in-missing-imports
• https://stackoverflow.com/questions/53524696/livy-no-yarn-application-is-found-with-tag-livy-batch-10-hg3po7kp-in-120-seconds