Skip to content

Instantly share code, notes, and snippets.

@jedi4ever
jedi4ever / gist:903751
Created April 5, 2011 14:51
Tuning stuff for Ubuntu hosts
# /etc/security/limits.conf
* soft nofile 999999
* hard nofile 999999
root soft nofile 999999
root hard nofile 999999
===========================================================
# /etc/sysctl.conf
# sysctl for maximum tuning
@cuimuxi
cuimuxi / gist:3719516
Created September 14, 2012 02:50
gevent crawler
import gevent
from gevent import monkey, queue
monkey.patch_all()
import urllib2
from time import sleep
import traceback
import logging
@feilaoda
feilaoda / gist:3984714
Created October 31, 2012 03:59 — forked from observerss/gist:3798922
Google Keyword Tool Scraper(casperjs version)
// requires
var utils = require('utils');
var casper = require('casper').create()
var casper = require('casper').create({
verbose: true,
logLevel: "debug"
});
// setup globals
var email = casper.cli.options['email'] || 'REPLACE THIS EMAIL';
@feilaoda
feilaoda / gist:4043549
Created November 9, 2012 03:35 — forked from cuimuxi/gist:3719535
gevent crawler test
from gcrawler import GCrawler, Downloader
import unittest
import urllib2
import logging
import traceback
from datetime import datetime
import re
logging.basicConfig(level=logging.DEBUG)
@feilaoda
feilaoda / marktodo规范
Created January 17, 2013 02:52
marktodo规范
Marktodo: A syntax for TODO lists
#any.do#
=project name
-todo list
[-] todo item A
[=] doing item B
[+] done item C
+done list
[+] done item a
@ashrithr
ashrithr / kafka.md
Last active March 14, 2024 21:16
kafka introduction

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

  • Producers send messages to brokers
  • Consumers read messages from brokers
  • Messages are sent to a topic
@jkreps
jkreps / benchmark-commands.txt
Last active June 17, 2024 03:54
Kafka Benchmark Commands
Producer
Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3
Single thread, no replication
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
@KexinFeng
KexinFeng / BertTranslator.java
Last active July 14, 2024 00:22
Deploying HuggingFace QA model in Java
import ai.djl.modality.nlp.DefaultVocabulary;
import ai.djl.modality.nlp.Vocabulary;
import ai.djl.modality.nlp.bert.BertToken;
import ai.djl.modality.nlp.bert.BertTokenizer;
import ai.djl.modality.nlp.qa.QAInput;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.NDManager;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;