Skip to content

Instantly share code, notes, and snippets.

View qiaohaijun's full-sized avatar
🎯
Focusing

qiao hai-jun qiaohaijun

🎯
Focusing
  • Beijing,China
View GitHub Profile
@qiaohaijun
qiaohaijun / protobuf3.spec
Created January 4, 2018 03:35 — forked from apeyser/protobuf3.spec
RPM spec file for building protobuf version 3.0.0-beta-1 into /opt in parallel to old, old versions of protobuf
#
%global srcvers 3.0.0-beta-1
%global srcname protobuf-%{srcvers}
%global _prefix /opt/%{srcname}
#
Summary: Protocol Buffers Google''s data interchange format
Name: protobuf3
git config --global https.proxy http://127.0.0.1:1080
git config --global https.proxy https://127.0.0.1:1080
git config --global --unset http.proxy
git config --global --unset https.proxy
npm config delete proxy
@qiaohaijun
qiaohaijun / es.sh
Created August 26, 2014 07:35 — forked from rajraj/es.sh
cd ~
sudo yum update
sudo yum install java-1.7.0-openjdk.i686 -y
wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.9.tar.gz -O elasticsearch.tar.gz
tar -xf elasticsearch.tar.gz
rm elasticsearch.tar.gz
mv elasticsearch-* elasticsearch
sudo mv elasticsearch /usr/local/share
Resolving Dependencies
--> Running transaction check
---> Package salt-minion.noarch 0:0.17.4-1.el5 set to be updated
--> Processing Dependency: salt = 0.17.4-1.el5 for package: salt-minion
--> Running transaction check
---> Package salt.noarch 0:0.17.4-1.el5 set to be updated
--> Processing Dependency: yum-utils for package: salt
--> Processing Dependency: sshpass for package: salt
--> Processing Dependency: python26-zmq for package: salt
--> Processing Dependency: python26-msgpack for package: salt
If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:
- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so
{
"template": "logstash-*",
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0,
"index" : {
"query" : { "default_field" : "@message" },
"store" : { "compress" : { "stored" : true, "tv": true } }
}
},
cluster:
name: CMTPRODLDN
network:
host: *********
node:
data: true
path:
# %W% %E%
#
# Copyright (c) 2009, 2011, Oracle and/or its affiliates. All rights reserved.
# URL: http://willerce.com/post/intellij-ide-fontconfig
# Version
version=1
# Component Font Mappings
@qiaohaijun
qiaohaijun / mg.csv
Created January 10, 2013 06:17 — forked from nanxstats/mg.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 10 should actually have 7 columns, instead of 6. in line 9.
1.1335086e+00, 1.0653587e+00, 8.4250325e-01, 7.0782984e-01, 1.0178784e+00, 1.2476827e+00, 9.0000000e-01
1.1645714e+00, 1.1355961e+00, 5.5885374e-01, 7.9370192e-01, 1.0346794e+00, 1.0067861e+00, 9.0000000e-01
1.0315895e+00, 1.1533842e+00, 4.2572016e-01, 1.0558440e+00, 1.1640420e+00, 7.1297054e-01, 9.0000000e-01
7.9058261e-01, 1.0678134e+00, 7.5043060e-01, 1.2000510e+00, 1.0670009e+00, 5.0349420e-01, 9.4136145e-01
6.3878936e-01, 8.1349649e-01, 9.7291951e-01, 1.2998549e+00, 1.0026658e+00, 5.8392542e-01, 1.1188041e+00
6.4385427e-01, 6.6246660e-01, 9.3785503e-01, 1.1857500e+00, 7.7000347e-01, 9.4327347e-01, 1.2161867e+00
9.7290237e-01, 6.2210481e-01, 1.0653587e+00, 8.4250325e-01, 7.0782984e-01, 1.0178784e+00, 1.2476827e+00
1.1330941e+00, 9.3890430e-01, 1.1355961e+00, 5.5885374e-01, 7.9370192e-01, 1.0346794e+00, 1.0067861e+00
1.1738960e+00, 1.1287058e+00, 1.1533842e+00, 4.2572016e-01, 1.0558440e+00, 1.1640420e+00, 7.1297054e-01
1.2210669e+00, 1.1699535e+00, 1.0678134e+00, 7.5043060e-01, 1.2000510e+00, 1.0670009e+00