mingderwang

mkdir /etc/chef

cat << EOF > /etc/chef/solo.rb
file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
node_name "app_server6"
EOF

cat << EOF > /etc/chef/chef.json
{

mingderwang

Installing Logstash 1.4.0, ElasticSearch 1.1.0, and Kibana 3 on Ubuntu 12.04

• indexer and shipper setups
• TODO: add auth layer between kibana and elasticsearch... https://github.com/fangli/kibana-authentication-proxy ?

References

• http://logstash.net/docs/1.4.0/tutorials/getting-started-with-logstash
• https://github.com/elasticsearch/kibana
• http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

mingderwang

Logstash Performance Testing

Server Details

HP BL460

• 48 Gb Memory
• 2 x X5675 @ 3.07GHz
• 2 x 10 gbps NIC
• 2tb NetApp NFS volume for ES data

mingderwang

FROM ubuntu:12.04
MAINTAINER Andrew Hromis "ahromis@gmail.com"

RUN apt-get update -q
RUN DEBIAN_FRONTEND=noninteractive apt-get install -qy build-essential curl git
RUN curl -s https://go.googlecode.com/files/go1.2.src.tar.gz | tar -v -C /usr/local -xz
RUN cd /usr/local/go/src && ./make.bash --no-clean 2>&1
ENV PATH /usr/local/go/bin:$PATH

RUN git clone git://github.com/elasticsearch/logstash-forwarder.git /opt/logstash-forwarder

mingderwang

1. General Background and Overview

• Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
• Models and Issues in Data Stream Systems
• Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
• Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
• [Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep

mingderwang

root@redmine:~/stress_test/gke# ../vegeta attack -rate=10000 -targets=targets.txt > results_23_101_1000_1.bin
root@redmine:~/stress_test/gke# ../vegeta report -inputs="results_23_101_1000_1.bin"
Requests	[total, rate]			100000, 9954.37
Duration	[total, attack, wait]		10.659396317s, 10.045843051s, 613.553266ms
Latencies	[mean, 50, 95, 99, max]		131.628089ms, 22.243637ms, 677.705875ms, 6.066950532s, 6.066950532s
Bytes In	[total, mean]			17340748, 173.41
Bytes Out	[total, mean]			0, 0.00
Success		[ratio]				15.10%
Status Codes	[code:count]			200:15096  0:84904
Error Set:

mingderwang

[
  {
    "code": "1001",
    "category": "HOME",
    "type": "SUBSCRIPTION",
    "name": "Home emergency assistance and insurance",
    "description": "We cover the typical mishaps you may encounter: If you lost your keys, a window or door gets broken and many more. Our assistance service is able to dispatch every type of craftsmen at no additional cost & at high quality and guaranteed low cost. 12 month subscription, coverage max. 1,500 per incident.",
    "price": 119
  },

mingderwang

input {
  
}

filter {
  
}

output {

mingderwang

Installing ElasticSearch with Monit & nginx

This repository contains files tailored for bootstrapping, installing and configuring ElasticSearch with Chef Solo on the Ubuntu operating system, inspired by this article.

Launch and ssh into Ubuntu 12.04 instance (if on ec2, use ami from alestic.com and with a security group as detailed here)

Now, copy the files to the machine and execute the bootstrap script:

mingderwang

eth allows to create custom/private blockchain by using several options:

--private defines the name of the custom chain (optional).

--config <filename> 
<filename> contains a JSON description of the network:
	- sealEngine (engine use to mine block)
	- params (general network information like minGasLimit, minimumDifficulty, blockReward, networkID)
	- genesis (genesis block description)
	- accounts (setup an original state that contains accounts/contracts)