Skip to content

Instantly share code, notes, and snippets.

View qiaohaijun's full-sized avatar
🎯
Focusing

qiao hai-jun qiaohaijun

🎯
Focusing
  • Beijing,China
View GitHub Profile

first gitst

  • sogou
  • facebook
  • twitter

cron学习笔记

  • cron.d
  • cron.monthly
  • cron.hourly
cluster:
name: CMTPRODLDN
network:
host: *********
node:
data: true
path:
{
"template": "logstash-*",
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0,
"index" : {
"query" : { "default_field" : "@message" },
"store" : { "compress" : { "stored" : true, "tv": true } }
}
},
If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:
- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so
Resolving Dependencies
--> Running transaction check
---> Package salt-minion.noarch 0:0.17.4-1.el5 set to be updated
--> Processing Dependency: salt = 0.17.4-1.el5 for package: salt-minion
--> Running transaction check
---> Package salt.noarch 0:0.17.4-1.el5 set to be updated
--> Processing Dependency: yum-utils for package: salt
--> Processing Dependency: sshpass for package: salt
--> Processing Dependency: python26-zmq for package: salt
--> Processing Dependency: python26-msgpack for package: salt
stderr logs
8E%E6%98%9F_%E5%A8%B1%E4%B9%90_%E7%83%9F%E5%8F%B0%E5%A4%A7...&- HTTP/1.0\" \"http://www.sogou.com/sogou?pid=AQxRG-5001&query=%D2%A6%B5%D1&p=99350103&oq=&ri=-2\" \"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0\" 786D4A4417E55C0A3D48CA0683309204 1375547964854602458054882","logdate":"2014040101"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:544)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:64)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:396)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
at org.apache.hadoop.mapred.Child$4.run(Child.java:275)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
stderr logs
5&sst0=1396287725531&lkt=0%2C0%2C0"},"useragent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; doyo 2.6.1)","yyid":"285ADD761F28D4F3D6509AECBB3947CA","suv":"1396287426443862413197370","line":"111.85.242.45 [01/Apr/2014:01:42:04 +0800:1396287724.319] \"GET /pv.gif?uigs_productid=web&uigs_t=1396287728562559&uigs_cookie=SUID%3D2DF2556F7E23900A000000005339931C%26sct%3D3&uigs_uuid=1396287728562529&scrnwi=1024&scrnhi=768&uigs_pbtag=A&abtestid=2&query=%25C8%25FD%25C8%25E9%25C3%25C0%25C5%25AE&rn=149988&stype=0&htn=9&qcn=0&hbn=0&uuid=bdffdbbe-1d7d-4c7d-a237-f5b49496c907&vr=null_0-10000201_1-null_2-null_3-null_4-null_5-30000901_6-null_7-null_8-null_9&vrdetail=null_0-10000201_1-null_2-null_3-null_4-null_5-30000901_6-null_7-null_8-null_9&sm=d0_0-d1_1-d0_2-d0_3-d0_4-d0_5-d0_6-d0_7-d0_8-d0_9&msrc=sm&loc=CN5201&adn=0&adltbn=0&adltan=0&radn=1&bkcolor=0&expand=0&qflag=0&qtype=0&warnLevel=127&eg=1&cost=167&bl=-1_127_0_0&pid=sogou&qjf=sogou&servuri=%252Fwebsearch%252Fsogou.jsp&rw=&idc=cnc&pn=9&jhh
stderr logs
i=1920&scrnhi=1080&uigs_pbtag=A&abtestid=7&query=%25D0%25A1%25C5%25DA%25CC%25EC%25B8%25B3&rn=5796&stype=3&htn=9&qcn=0&hbn=0&uuid=e335bf4a-3329-4454-aab5-3e17766d266a&vr=null_0-null_1-null_2-null_3-null_4-null_5-null_6-null_7-null_8-30000201_9&vrdetail=null_0-null_1-null_2-null_3-null_4-null_5-null_6-null_7-null_8-30000201_9&sm=d0_0-d0_1-d0_2-d0_3-d0_4-d0_5-d0_6-d0_7-d0_8-d0_9&msrc=sm&loc=CN5000&adn=1&adltbn=1&adltan=0&radn=0&bkcolor=0&expand=1&qflag=0&qtype=0&warnLevel=127&eg=1&cost=194&bl=-1_127_0_0&pid=sogou-netb-33267e5dc58fad34&qjf=sogou-netb-33267e5dc58fad34&servuri=%252Fwebsearch%252Fbdsearch.jsp&rw=&idc=zw&pn=10&jhhint=1&jhshuxing=0&intcat=web&inttab=61-0_28-1_15-2_40-3_41-4_39-5_43-6_9-7_29-8_30-9_45-10_62-11_&jhly=top&jhlysite=all-0_tieba.baidu.com-1_duowan.com-2_yzz.cn-3_&legalad=1&kmap=qa_0-right_21_1_0&qatype=0&googlead=null&uigs_version=v1.1&uigs_refer=http%3A%2F%2Fwww.sogou.com%2Findex.htm%3Fpid%3Dsogou-netb-33267e5dc58fad34&- HTTP/1.0\" \"http://www.sogou.com/sogou?query=%D0%A1%C5%D
./bin/spark-class org.apache.spark.deploy.yarn.Client --jar examples/lib/spark-examples_2.10-0.9.0-cdh5.0.0.jar --class org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers 3 --master-memory 4g --worker-memory 2g --worker-cores 1