Skip to content

Instantly share code, notes, and snippets.

@vipmax
vipmax / kafka concurrent batch commit
Last active Sep 20, 2018
kafka concurrent batch commit
View kafka concurrent batch commit
val kafkaProps = new Properties()
kafkaProps.put("bootstrap.servers", endpoint)
kafkaProps.put("key.serializer", classOf[ByteArraySerializer])
kafkaProps.put("key.deserializer", classOf[ByteArrayDeserializer])
kafkaProps.put("value.serializer", classOf[ByteArraySerializer])
kafkaProps.put("value.deserializer", classOf[ByteArrayDeserializer])
kafkaProps.put("group.id", "CrawlerTasksStorage")
kafkaProps.put("max.poll.records", "1000")
kafkaProps.put("enable.auto.commit","false")
View kafka installation and systemd
cd /opt
wget http://apache-mirror.rbc.ru/pub/apache/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
tar xvzf kafka_2.11-0.10.1.0.tgz
ln -s kafka_2.11-0.10.1.0/ kafka
vi /etc/systemd/system/kafka-zookeeper.service
[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
View systemd java service
cat social/scripts/social-site.service
cat social/scripts/social-crawler.service
View python mongo
GeekPicnic = pymongo.MongoClient(host="192.168.13.133")['GeekPicnic']
for doc in GeekPicnic['GeekPicnic_spb_photos'].find():
GeekPicnic['GeekPicnic_all_photos'].update({'_id': doc['_id']},{'$set': doc}, True, False)
@vipmax
vipmax / VkExamples.scala
Last active Jul 11, 2016
just vk api examples for scala and python
View VkExamples.scala
import com.mongodb.{BasicDBList, BasicDBObject, DBObject}
import com.mongodb.util.JSON
import scalaj.http.Http
object VkGetLastPostsTest {
def main(args: Array[String]) {
val ownerId = "1"
val posts = JSON.parse(
View extend polyglot sentiment dictionary
import sys
reload(sys) # Reload does the trick!
sys.setdefaultencoding('UTF8')
import tarfile
# extract dictionary package
tar = tarfile.open("polyglot_data/sentiment2/ru/ru.sent.pkl.tar.bz2")
tar.extractall()
tar.close()
You can’t perform that action at this time.