Skip to content

Instantly share code, notes, and snippets.

Lars George larsgeorge

Block or report user

Report or block larsgeorge

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View presto-histogramm-20161125.log
2016-11-25T12:39:20.420+0100: 177.558: [SoftReference, 0 refs, 0.0000547 secs]2016-11-25T12:39:20.420+0100: 177.558: [WeakReference, 2766 refs, 0.0004900 secs]2016-11-25T12:39:20.421+0100: 177.558: [FinalReference, 589 refs, 0.0007352 secs]2016-11-25T12:39:20.422+0100: 177.559: [PhantomReference, 0 refs, 182 refs, 0.0000369 secs]2016-11-25T12:39:20.422+0100: 177.559: [JNI Weak Reference, 0.0000666 secs] 177.562: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 8925478912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC]
177.562: [G1Ergonomics (Mixed GCs) do not continue mixed GCs, reason: reclaimable percentage not over threshold, candidate old regions: 94 regions, reclaimable: 850530200 bytes (4.95 %), threshold: 5.00 %]
, 0.0895976 secs]
[Parallel Time: 82.9 ms, GC Workers: 33]
[GC Worker Start (ms): Min: 177473.8, Avg: 177474.2, Max: 177474.6, Diff: 0.8]
[Ext Root Scann
View gist:e5797433a32edd6487886069f150e9c7
# See: https://www.websequencediagrams.com/
title HBase Metrics Call Sequence Diagramm
MetricsSystem->MetricsSystem: onTimerEvent()
activate MetricsSystem
MetricsSystem->MetricsSystem: publishMetrics()
MetricsSystem->MetricsSystem: sampleMetrics()
MetricsSystem->+MetricsBufferBuilder: constructor
MetricsBufferBuilder->-MetricsSystem: MetricsBufferBuilder
@larsgeorge
larsgeorge / gist:dbfbb09f914eb5079be1
Last active Mar 2, 2017
Hidden HBase Properties
View gist:dbfbb09f914eb5079be1
IMPORTANT DISCLAIMER: Many of the found "properties" aren't really useful or mistakenly listed,
like those starting with "org.apache.hadoop.*". The tool simply greps a pattern in the matching
files, and some of them are not used as properties. On the other hand, all named properties from
the given configuration file are removed, so quite a few are valid. Again, some of those are still
hidden on purpose, since it is not useful to tinker with them unless in extreme situations, or
during test runs. Be VERY CAREFUL of what you do. I warned ya!
$ sh target/bin/run-finder -d /projects/opensource/hbase/hbase-trunk-rw-git/ -u -s \
--exclude **/generated --exclude **/generated/** --exclude **/target \
--exclude **/test/** -c hbase-common/src/main/resources/hbase-default.xml -o /dev/null
@larsgeorge
larsgeorge / gist:3f7daee9f96200d2f229
Last active Jan 10, 2017
HBase Configuration Differences (based on the hbase-default.xml file in each release)
View gist:3f7daee9f96200d2f229
/projects/private/hadoop-config-differ [git:master]$ sh target/bin/run-differ /projects/opensource/hbase/hbase-r1130916/src/main/resources/hbase-default.xml r1130916 /projects/opensource/hbase/hbase-0.92-rw/src/main/resources/hbase-default.xml 0.92 /projects/opensource/hbase/hbase-0.94-rw/src/main/resources/hbase-default.xml 0.94 /projects/opensource/hbase/hbase-0.96-rw/hbase-common/src/main/resources/hbase-default.xml 0.96 /projects/opensource/hbase/hbase-0.98-rw/hbase-common/src/main/resources/hbase-default.xml 0.98 /projects/opensource/hbase/hbase-trunk-rw-git/hbase-common/src/main/resources/hbase-default.xml 0.99
=========================================================
Start
=========================================================
Checking differences across versions...
Added or renamed keys in 0.92:
added: Property{key='dfs.support.append', value='true', description='Does HDFS allow appends to files? This is an hdfs config. set in here so the hdfs client will do append support. You must ensure tha
@larsgeorge
larsgeorge / hdfs-timing.txt
Created Jan 8, 2014
Measure time it takes to create and read a file in HDFS
View hdfs-timing.txt
$ time dd if=/dev/zero bs=1G count=1 | hadoop fs -put - 1g.bin
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 64.4632 s, 16.7 MB/s
real 1m5.239s
user 0m9.560s
sys 0m3.394s
$ time hadoop fs -get 1g.bin - > /dev/null
@larsgeorge
larsgeorge / Hadoop log4j.properties
Created Aug 9, 2011
Use syslog (on MacOS) with Hadoop's userlog/syslog
View Hadoop log4j.properties
#...
#Default values
hadoop.tasklog.taskid=null
hadoop.tasklog.iscleanup=false
hadoop.tasklog.noKeepSplits=4
hadoop.tasklog.totalLogFileSize=100
hadoop.tasklog.purgeLogSplits=true
hadoop.tasklog.logsRetainHours=12
log4j.appender.TLA=org.apache.log4j.net.SyslogAppender
@larsgeorge
larsgeorge / gist:825646
Created Feb 14, 2011
HBase Replication Notes
View gist:825646
HBase 1:
506 wget http://apache.easy-webs.de//hbase/hbase-0.90.0/hbase-0.90.0.tar.gz
507 tar -zxvf hbase-0.90.0.tar.gz
508 cd hbase-0.90.0
522 cp -pR conf conf.2
523 vim conf/hbase-site.xml
524 cp conf/hbase-site.xml conf.2/
525 vim conf.2/hbase-site.xml
View gist:822865
2011-02-11 19:49:11,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:22,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:33,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:44,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:54,964 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.54 MB, max=199.18 MB, blocks=4, accesses=55, hits=51, hitRatio=92.72%%, cachingAccesses=55, cachingHits=51, cachingHitsRatio=92.72%%, evictions=0, evicted=0, evictedPerRun=NaN
2011-02-11 19:49:55,748 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
==> logs/hbase-larsgeorge-master-d
View gist:822679
hbase(main):014:0> add_peer '1', 'localhost:2181:/hbase-2'
0 row(s) in 0.0580 seconds
hbase(main):015:0> start_replication
2011-02-11 18:04:58,347 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Replication is now started
0 row(s) in 0.0500 seconds
hbase(main):016:0> put 'test', '2011-02-11 18:05:22,003 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [10.0.0.57,60020,1297437319991]
2011-02-11 18:05:22,016 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs for 10.0.0.57,60020,1297437319991
2011-02-11 18:05:22,016 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=10.0.0.57,60020,1297437319991 to dead servers, submitted shutdown handler to be executed, root=true, meta=true
You can’t perform that action at this time.