root-11

If 2fa is enabled on github switch to ssh instead of https on linux

1. generate an ssh keypair on your linux box
ssh-keygen -t {rsa|dsa} 

2. add the public key to github: profile - settings - ssh keys

3. switch from https to ssh

Check your repo remote:

root-11

import time
import requests
import zlib
# pip install lz4 pylzma
import lz4
import pylzma as lzma


def analysis_ratios(source):
    time_before_zip = time.time()

root-11

from tqdm import tqdm
import requests

# file url
url = 'https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-stub-meta-history.xml.gz' #file size is '70.5GB'
# stream true is required
response = requests.get(url, stream=True)
# total file size
t = int(response.headers.get('content-length', 0))
block_size = 1024**2 #1 Mbit

root-11

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD