Skip to content

Instantly share code, notes, and snippets.

View qiaohaijun's full-sized avatar
🎯
Focusing

qiao hai-jun qiaohaijun

🎯
Focusing
  • Beijing,China
View GitHub Profile
@qiaohaijun
qiaohaijun / mg.csv
Created January 10, 2013 06:17 — forked from nanxstats/mg.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 10 should actually have 7 columns, instead of 6. in line 9.
1.1335086e+00, 1.0653587e+00, 8.4250325e-01, 7.0782984e-01, 1.0178784e+00, 1.2476827e+00, 9.0000000e-01
1.1645714e+00, 1.1355961e+00, 5.5885374e-01, 7.9370192e-01, 1.0346794e+00, 1.0067861e+00, 9.0000000e-01
1.0315895e+00, 1.1533842e+00, 4.2572016e-01, 1.0558440e+00, 1.1640420e+00, 7.1297054e-01, 9.0000000e-01
7.9058261e-01, 1.0678134e+00, 7.5043060e-01, 1.2000510e+00, 1.0670009e+00, 5.0349420e-01, 9.4136145e-01
6.3878936e-01, 8.1349649e-01, 9.7291951e-01, 1.2998549e+00, 1.0026658e+00, 5.8392542e-01, 1.1188041e+00
6.4385427e-01, 6.6246660e-01, 9.3785503e-01, 1.1857500e+00, 7.7000347e-01, 9.4327347e-01, 1.2161867e+00
9.7290237e-01, 6.2210481e-01, 1.0653587e+00, 8.4250325e-01, 7.0782984e-01, 1.0178784e+00, 1.2476827e+00
1.1330941e+00, 9.3890430e-01, 1.1355961e+00, 5.5885374e-01, 7.9370192e-01, 1.0346794e+00, 1.0067861e+00
1.1738960e+00, 1.1287058e+00, 1.1533842e+00, 4.2572016e-01, 1.0558440e+00, 1.1640420e+00, 7.1297054e-01
1.2210669e+00, 1.1699535e+00, 1.0678134e+00, 7.5043060e-01, 1.2000510e+00, 1.0670009e+00
# %W% %E%
#
# Copyright (c) 2009, 2011, Oracle and/or its affiliates. All rights reserved.
# URL: http://willerce.com/post/intellij-ide-fontconfig
# Version
version=1
# Component Font Mappings

first gitst

  • sogou
  • facebook
  • twitter

cron学习笔记

  • cron.d
  • cron.monthly
  • cron.hourly
cluster:
name: CMTPRODLDN
network:
host: *********
node:
data: true
path:
{
"template": "logstash-*",
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0,
"index" : {
"query" : { "default_field" : "@message" },
"store" : { "compress" : { "stored" : true, "tv": true } }
}
},
If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:
- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so
Resolving Dependencies
--> Running transaction check
---> Package salt-minion.noarch 0:0.17.4-1.el5 set to be updated
--> Processing Dependency: salt = 0.17.4-1.el5 for package: salt-minion
--> Running transaction check
---> Package salt.noarch 0:0.17.4-1.el5 set to be updated
--> Processing Dependency: yum-utils for package: salt
--> Processing Dependency: sshpass for package: salt
--> Processing Dependency: python26-zmq for package: salt
--> Processing Dependency: python26-msgpack for package: salt
stderr logs
8E%E6%98%9F_%E5%A8%B1%E4%B9%90_%E7%83%9F%E5%8F%B0%E5%A4%A7...&- HTTP/1.0\" \"http://www.sogou.com/sogou?pid=AQxRG-5001&query=%D2%A6%B5%D1&p=99350103&oq=&ri=-2\" \"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0\" 786D4A4417E55C0A3D48CA0683309204 1375547964854602458054882","logdate":"2014040101"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:544)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:64)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:396)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
at org.apache.hadoop.mapred.Child$4.run(Child.java:275)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
stderr logs
5&sst0=1396287725531&lkt=0%2C0%2C0"},"useragent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; doyo 2.6.1)","yyid":"285ADD761F28D4F3D6509AECBB3947CA","suv":"1396287426443862413197370","line":"111.85.242.45 [01/Apr/2014:01:42:04 +0800:1396287724.319] \"GET /pv.gif?uigs_productid=web&uigs_t=1396287728562559&uigs_cookie=SUID%3D2DF2556F7E23900A000000005339931C%26sct%3D3&uigs_uuid=1396287728562529&scrnwi=1024&scrnhi=768&uigs_pbtag=A&abtestid=2&query=%25C8%25FD%25C8%25E9%25C3%25C0%25C5%25AE&rn=149988&stype=0&htn=9&qcn=0&hbn=0&uuid=bdffdbbe-1d7d-4c7d-a237-f5b49496c907&vr=null_0-10000201_1-null_2-null_3-null_4-null_5-30000901_6-null_7-null_8-null_9&vrdetail=null_0-10000201_1-null_2-null_3-null_4-null_5-30000901_6-null_7-null_8-null_9&sm=d0_0-d1_1-d0_2-d0_3-d0_4-d0_5-d0_6-d0_7-d0_8-d0_9&msrc=sm&loc=CN5201&adn=0&adltbn=0&adltan=0&radn=1&bkcolor=0&expand=0&qflag=0&qtype=0&warnLevel=127&eg=1&cost=167&bl=-1_127_0_0&pid=sogou&qjf=sogou&servuri=%252Fwebsearch%252Fsogou.jsp&rw=&idc=cnc&pn=9&jhh