Skip to content

Instantly share code, notes, and snippets.

Max Petrov vipmax

Block or report user

Report or block vipmax

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@vipmax
vipmax / kafka concurrent batch commit
Last active Sep 20, 2018
kafka concurrent batch commit
View kafka concurrent batch commit
val kafkaProps = new Properties()
kafkaProps.put("bootstrap.servers", endpoint)
kafkaProps.put("key.serializer", classOf[ByteArraySerializer])
kafkaProps.put("key.deserializer", classOf[ByteArrayDeserializer])
kafkaProps.put("value.serializer", classOf[ByteArraySerializer])
kafkaProps.put("value.deserializer", classOf[ByteArrayDeserializer])
kafkaProps.put("group.id", "CrawlerTasksStorage")
kafkaProps.put("max.poll.records", "1000")
kafkaProps.put("enable.auto.commit","false")
View kafka installation and systemd
cd /opt
wget http://apache-mirror.rbc.ru/pub/apache/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
tar xvzf kafka_2.11-0.10.1.0.tgz
ln -s kafka_2.11-0.10.1.0/ kafka
vi /etc/systemd/system/kafka-zookeeper.service
[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
View systemd java service
cat social/scripts/social-site.service
cat social/scripts/social-crawler.service
View python mongo
GeekPicnic = pymongo.MongoClient(host="192.168.13.133")['GeekPicnic']
for doc in GeekPicnic['GeekPicnic_spb_photos'].find():
GeekPicnic['GeekPicnic_all_photos'].update({'_id': doc['_id']},{'$set': doc}, True, False)
@vipmax
vipmax / VkExamples.scala
Last active Jul 11, 2016
just vk api examples for scala and python
View VkExamples.scala
import com.mongodb.{BasicDBList, BasicDBObject, DBObject}
import com.mongodb.util.JSON
import scalaj.http.Http
object VkGetLastPostsTest {
def main(args: Array[String]) {
val ownerId = "1"
val posts = JSON.parse(
View extend polyglot sentiment dictionary
import sys
reload(sys) # Reload does the trick!
sys.setdefaultencoding('UTF8')
import tarfile
# extract dictionary package
tar = tarfile.open("polyglot_data/sentiment2/ru/ru.sent.pkl.tar.bz2")
tar.extractall()
tar.close()
You can’t perform that action at this time.