Skip to content

Instantly share code, notes, and snippets.

@hivefans
hivefans / sync_offsets.rb
Last active March 17, 2020 02:03 — forked from killme2008/sync_offsets.rb
|-|{"files":{"sync_offsets.rb":{"env":"plain"}},"tag":"bigdata"}
####
# Description:a ruby script to sync consumers offsets with brokers offsets.
# Requirements: zookeeper
# sudo gem install zookeeper
#
#####
require 'rubygems'
require 'zookeeper'
require 'socket'
@hivefans
hivefans / README.markdown
Last active March 17, 2020 02:03 — forked from karmi/.gitignore
Example Nginx configurations for Elasticsearch|-|{"files":{"passwords":{"env":"plain"},"nginx_load_balancer.conf":{"env":"plain"},"nginx_keep_alive.conf":{"env":"plain"},"admins":{"env":"plain"},"nginx_basic_proxy.conf":{"env":"plain"},"nginx_http_auth_deny_path.conf":{"env":"plain"},"nginx_http_auth_roles.conf":{"env":"plain"},"nginx_http_auth_…

Example Nginx Configurations for Elasticsearch

This repository contains couple of example configurations for using Nginx as a proxy for Elasticsearch.

These examples can be run standalone from this repository -- the general pattern is:

$ nginx -p $PWD/nginx/ -c $PWD/<CONFIG FILE>

When you change the configuration, simply reload the Nginx process to pick up the changes:

@hivefans
hivefans / tornadocrud.py
Last active March 17, 2020 02:03 — forked from cjgiridhar/tornadocrud.py
|-|{"files":{"tornadocrud.py":{"env":"plain"}},"tag":"bigdata"}
import tornado.ioloop
import tornado.web
from datetime import datetime
import urlparse
from bson.json_util import dumps
import pymongo
from pymongo import Connection
class Home(tornado.web.RequestHandler):
@hivefans
hivefans / out_reloadable_copy.rb
Last active March 17, 2020 02:03 — forked from bash0C7/out_reloadable_copy.rb
|-|{"files":{"out_reloadable_copy.rb":{"env":"plain"}},"tag":"bigdata"}
#
# Fluent
#
# Copyright (C) 2011 FURUHASHI Sadayuki
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
@hivefans
hivefans / slideshare-dl.py
Last active March 17, 2020 02:03 — forked from julionc/slideshare-dl.py
|-|{"files":{"slideshare-dl.py":{"env":"plain"}},"tag":"bigdata"}
#!/usr/bin/env python2.7
#-*- coding: utf-8 -*-
import os,sys
import urllib2
from BeautifulSoup import BeautifulSoup
from reportlab.pdfgen import canvas
from reportlab.lib.pagesizes import portrait,A4,landscape
reload(sys)
sys.setdefaultencoding('utf-8')
@hivefans
hivefans / 115.py
Last active March 17, 2020 02:03 — forked from wusuopu/115.py
a script auto to login 115|-|{"files":{"115.py":{"env":"plain"}},"tag":"bigdata"}
#!/usr/bin/env python
#-*- coding:utf-8 -*-
##
##
# Copyright (C)
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation.
# 本程序是免费软件,基于GPL许可发布。
@hivefans
hivefans / HBaseNewAPI.scala
Last active March 17, 2020 02:03 — forked from wuchong/HBaseNewAPI.scala
Spark 下 操作 HBase 1.0.0 新版API|-|{"files":{"SparkOnHBase.scala":{"env":"plain"},"build.sbt":{"env":"plain"},"HBaseNewAPI.scala":{"env":"plain"}},"tag":"Uncategorized"}
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HColumnDescriptor, HTableDescriptor, TableName, HBaseConfiguration}
import org.apache.hadoop.hbase.client._
import org.apache.spark.SparkContext
import scala.collection.JavaConversions._
/**
* HBase 1.0.0 新版API, CRUD 的基本操作代码示例
**/
object HBaseNewAPI {
@hivefans
hivefans / flume-ng-agent.sh
Last active March 17, 2020 02:03 — forked from ashrithr/flume-ng-agent.sh
Custom Flume NG Agent INIT script for centos for runnig multiple agents on same machine|-|{"files":{"flume-ng-agent.sh":{"env":"plain"},"usage.md":{"env":"plain"}},"tag":"Uncategorized"}
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
@hivefans
hivefans / squid-elasticsearch.json
Last active March 17, 2020 02:03
squid日志导入elasticsearch的模板设置|-|{"files":{"squid-elasticsearch.json":{"env":"plain"}},"tag":"bigdata"}
{
"order": 0,
"template": "hangout-*",
"settings": {
"index": {
"number_of_shards": "40",
"codec": "best_compression",
"refresh_interval": "30s"
}
},
@hivefans
hivefans / EsPerformanceTest.java
Last active March 17, 2020 02:03
Elasticsearch 批量index性能测试|-|{"files":{"build.gradle":{"env":"plain"},"EsPerformanceTest.java":{"env":"plain"}},"tag":"Uncategorized"}
package kingsoft.com;
import org.apache.log4j.BasicConfigurator;
import org.elasticsearch.action.admin.indices.flush.FlushRequest;
import org.elasticsearch.action.bulk.BackoffPolicy;
import org.elasticsearch.action.bulk.BulkProcessor;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.transport.TransportClient;