Skip to content

Instantly share code, notes, and snippets.

vipmax / kafka concurrent batch commit
Last active Sep 20, 2018
kafka concurrent batch commit
View kafka concurrent batch commit
val kafkaProps = new Properties()
kafkaProps.put("bootstrap.servers", endpoint)
kafkaProps.put("key.serializer", classOf[ByteArraySerializer])
kafkaProps.put("key.deserializer", classOf[ByteArrayDeserializer])
kafkaProps.put("value.serializer", classOf[ByteArraySerializer])
kafkaProps.put("value.deserializer", classOf[ByteArrayDeserializer])
kafkaProps.put("", "CrawlerTasksStorage")
kafkaProps.put("max.poll.records", "1000")
View kafka installation and systemd
cd /opt
tar xvzf kafka_2.11-
ln -s kafka_2.11- kafka
vi /etc/systemd/system/kafka-zookeeper.service
Description=Apache Zookeeper server (Kafka)
View systemd java service
cat social/scripts/social-site.service
cat social/scripts/social-crawler.service
View python mongo
GeekPicnic = pymongo.MongoClient(host="")['GeekPicnic']
for doc in GeekPicnic['GeekPicnic_spb_photos'].find():
GeekPicnic['GeekPicnic_all_photos'].update({'_id': doc['_id']},{'$set': doc}, True, False)
vipmax / VkExamples.scala
Last active Jul 11, 2016
just vk api examples for scala and python
View VkExamples.scala
import com.mongodb.{BasicDBList, BasicDBObject, DBObject}
import com.mongodb.util.JSON
import scalaj.http.Http
object VkGetLastPostsTest {
def main(args: Array[String]) {
val ownerId = "1"
val posts = JSON.parse(
View extend polyglot sentiment dictionary
import sys
reload(sys) # Reload does the trick!
import tarfile
# extract dictionary package
tar ="polyglot_data/sentiment2/ru/ru.sent.pkl.tar.bz2")
You can’t perform that action at this time.