xpzouying

# uncomments the following line if use notebook
# %matplotlib inline

import torchvision
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import ImageFolder
import matplotlib.pyplot as plt

data_dir = '/tmp/images2/'

xpzouying

package main

import (
    "fmt"
    "log"

    "flag"

    "time"

xpzouying

This is a gist with a set of helpful performance tricks and best practices that I've found on the internet.

xpzouying

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

xpzouying

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns             
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs

xpzouying

1. Clone your fork:

git clone git@github.com:YOUR-USERNAME/YOUR-FORKED-REPO.git

2. Add remote from original repository in your forked repository:

cd into/cloned/fork-repo
git remote add upstream git://github.com/ORIGINAL-DEV-USERNAME/REPO-YOU-FORKED-FROM.git
git fetch upstream

xpzouying

Crawling Anonymously with Tor in Python

adapted from the article "Crawling anonymously with Tor in Python" by S. Acharya, Nov 2, 2013.

The most common use-case is to be able to hide one's identity using TOR or being able to change identities programmatically, for example when you are crawling a website like Google and you don’t want to be rate-limited or blocked via IP address.

Tor

Install Tor.

xpzouying

# Should work on all Debian based distros with systemd; tested on Ubuntu 16.04+.
# This will by default install all plugins; you can customize this behavior on line 6. Selecting too many plugins can cause issues when downloading. 
# Run as root (or sudo before every line) please. Note this is not designed to be run automatically; I recommend executing this line by line.

apt install curl
curl https://getcaddy.com | bash -s personal dns,docker,dyndns,hook.service,http.authz,http.awses,http.awslambda,http.cache,http.cgi,http.cors,http.datadog,http.expires,http.filemanager,http.filter,http.forwardproxy,http.geoip,http.git,http.gopkg,http.grpc,http.hugo,http.ipfilter,http.jekyll,http.jwt,http.locale,http.login,http.mailout,http.minify,http.nobots,http.prometheus,http.proxyprotocol,http.ratelimit,http.realip,http.reauth,http.restic,http.upload,http.webdav,net,tls.dns.auroradns,tls.dns.azure,tls.dns.cloudflare,tls.dns.cloudxns,tls.dns.digitalocean,tls.dns.dnsimple,tls.dns.dnsmadeeasy,tls.dns.dnspod,tls.dns.dyn,tls.

xpzouying

# Should work on all Debian based distros with systemd; tested on Ubuntu 16.04+.
# This will by default install all plugins; you can customize this behavior on line 6. Selecting too many plugins can cause issues when downloading. 
# Run as root (or sudo before every line) please. Note this is not designed to be run automatically; I recommend executing this line by line.

apt install curl
curl https://getcaddy.com | bash -s personal dns,docker,dyndns,hook.service,http.authz,http.awses,http.awslambda,http.cache,http.cgi,http.cors,http.datadog,http.expires,http.filemanager,http.filter,http.forwardproxy,http.geoip,http.git,http.gopkg,http.grpc,http.hugo,http.ipfilter,http.jekyll,http.jwt,http.locale,http.login,http.mailout,http.minify,http.nobots,http.prometheus,http.proxyprotocol,http.ratelimit,http.realip,http.reauth,http.restic,http.upload,http.webdav,net,tls.dns.auroradns,tls.dns.azure,tls.dns.cloudflare,tls.dns.cloudxns,tls.dns.digitalocean,tls.dns.dnsimple,tls.dns.dnsmadeeasy,tls.dns.dnspod,tls.dns.dyn,tls.

xpzouying

package main

import (
	"log"
	"unsafe"
)

func main() {
	x := make([]byte, 8)
	x[0] = 3