Skip to content

Instantly share code, notes, and snippets.

View andrefsp's full-sized avatar

andre da palma andrefsp

View GitHub Profile
@andrefsp
andrefsp / pylibmc_concurrency_test.py
Last active August 29, 2015 14:02
Race condition on pylibmc
import argparse
import multiprocessing
import threading
from pylibmc.test import make_test_client
import random
NUM_THREADS = 2
NUM_PROCS = 4
NUM_SETS = 10000

A general rule of thumb we work with is:

  • Use the Oracle JVM, not OpenJDK. OpenJDK has a lot of issues with large amounts of memory
  • You'll want twice as much memory dedicated to the JVM as the index size is on disk. i.e. if your committed/optimized index is 20GB in size, you'll want 40GB of RAM + a little spare.
  • Use a good garbage collector, e.g.::

    JAVA_OPTS="${JAVA_OPTS} -XX:+UseConcMarkSweepGC -XX:+UseParNewGC"

  • Logging helps a LOT when shit is not working as expected::

    JAVA_OPTS="${JAVA_OPTS} -Xloggc:/var/log/tomcat6/log_GarbageCollection -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"JAVA_OPTS="${JAVA_OPTS} -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/tomcat6/oom_log"

REMEMBER, CACHES NEED RAM. ALLOCATED MORE RAM TO THE JVM AS IS REQUIRED BASED ON YOUR CACHES.

## At the http level
map $http_cookie $is_secure {
default 0;
~SESS 1; # there's a session cookie (use SSL - authenticated user)
}
map $is_secure $not_secure {
1 0;
0 1;
}
@andrefsp
andrefsp / postgres_db_stats
Created October 22, 2013 13:20
Shell script to get per database stats
#!/bin/bash
case $1 in
pg_stat_activity)
sudo -u postgres psql -t -c "select $3 from pg_stat_activity where datname='$2' ;" | head -1 | sed s/\ \\+//g
;;
pg_stat_database)
sudo -u postgres psql -t -c "select $3 from pg_stat_database where datname='$2' ;" | head -1 | sed s/\ \\+//g
;;
pg_database_size)
@andrefsp
andrefsp / gevent_non_blocking.py
Last active February 9, 2016 23:36
Gevent non blocking socket
import gevent
from gevent import monkey
monkey.patch_all()
import requests
def fetch_request(url):
print("fetching %s" % url)
requests.get(url)
print("Finish fetching %s" % url)
@andrefsp
andrefsp / parallel_url_fetch_aiohttp.py
Created February 10, 2016 18:26
This gist shows how to make a non-blocking parallel url fetcher using aiohttp and asyncio
from datetime import datetime
import logging
import asyncio
from aiohttp import web
import requests
logger = logging.getLogger()
@asyncio.coroutine
@andrefsp
andrefsp / parallel_url_fetch.go
Created February 12, 2016 15:12
Parallel url fetcher written in go
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
func fetch_url(url string, responses *chan string) {
@andrefsp
andrefsp / Squid Allow All configuration
Created January 9, 2018 00:18
Squid configuration file
#Recommended minimum configuration:
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 0.0.0.0/0 192.168.100.0/24 192.168.101.0/24
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
@andrefsp
andrefsp / squid.inboud.conf
Created July 15, 2019 09:06
Squid Double Legged proxy
# Recommended minimum configuration:
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
# Specify incoming networks of incoming traffic
acl localnet src 0.0.0.0/0 192.168.100.0/24 192.168.101.0/24 10.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80 # http
package main
import (
"context"
"crypto/tls"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
)