Skip to content

Instantly share code, notes, and snippets.

View larsgeorge's full-sized avatar

Lars George larsgeorge

286 followers · 7 following
View GitHub Profile
@larsgeorge
larsgeorge / gist:822679
Created February 11, 2011 17:09
hbase(main):014:0> add_peer '1', 'localhost:2181:/hbase-2'
0 row(s) in 0.0580 seconds
hbase(main):015:0> start_replication
2011-02-11 18:04:58,347 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Replication is now started
0 row(s) in 0.0500 seconds
hbase(main):016:0> put 'test', '2011-02-11 18:05:22,003 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [10.0.0.57,60020,1297437319991]
2011-02-11 18:05:22,016 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs for 10.0.0.57,60020,1297437319991
2011-02-11 18:05:22,016 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=10.0.0.57,60020,1297437319991 to dead servers, submitted shutdown handler to be executed, root=true, meta=true
@larsgeorge
larsgeorge / gist:822865
Created February 11, 2011 19:35
2011-02-11 19:49:11,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:22,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:33,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:44,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
2011-02-11 19:49:54,964 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.54 MB, max=199.18 MB, blocks=4, accesses=55, hits=51, hitRatio=92.72%%, cachingAccesses=55, cachingHits=51, cachingHitsRatio=92.72%%, evictions=0, evicted=0, evictedPerRun=NaN
2011-02-11 19:49:55,748 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10
==> logs/hbase-larsgeorge-master-d
@larsgeorge
larsgeorge / gist:825646
Created February 14, 2011 09:14
HBase Replication Notes
HBase 1:
506 wget http://apache.easy-webs.de//hbase/hbase-0.90.0/hbase-0.90.0.tar.gz
507 tar -zxvf hbase-0.90.0.tar.gz
508 cd hbase-0.90.0
522 cp -pR conf conf.2
523 vim conf/hbase-site.xml
524 cp conf/hbase-site.xml conf.2/
525 vim conf.2/hbase-site.xml
@larsgeorge
larsgeorge / Hadoop log4j.properties
Created August 9, 2011 12:58
Use syslog (on MacOS) with Hadoop's userlog/syslog
#...
#Default values
hadoop.tasklog.taskid=null
hadoop.tasklog.iscleanup=false
hadoop.tasklog.noKeepSplits=4
hadoop.tasklog.totalLogFileSize=100
hadoop.tasklog.purgeLogSplits=true
hadoop.tasklog.logsRetainHours=12
log4j.appender.TLA=org.apache.log4j.net.SyslogAppender
@larsgeorge
larsgeorge / hdfs-timing.txt
Created January 8, 2014 09:22
Measure time it takes to create and read a file in HDFS
$ time dd if=/dev/zero bs=1G count=1 | hadoop fs -put - 1g.bin
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 64.4632 s, 16.7 MB/s
real 1m5.239s
user 0m9.560s
sys 0m3.394s
$ time hadoop fs -get 1g.bin - > /dev/null
@larsgeorge
larsgeorge / gist:3f7daee9f96200d2f229
Last active January 10, 2017 11:11
HBase Configuration Differences (based on the hbase-default.xml file in each release)
/projects/private/hadoop-config-differ [git:master]$ sh target/bin/run-differ /projects/opensource/hbase/hbase-r1130916/src/main/resources/hbase-default.xml r1130916 /projects/opensource/hbase/hbase-0.92-rw/src/main/resources/hbase-default.xml 0.92 /projects/opensource/hbase/hbase-0.94-rw/src/main/resources/hbase-default.xml 0.94 /projects/opensource/hbase/hbase-0.96-rw/hbase-common/src/main/resources/hbase-default.xml 0.96 /projects/opensource/hbase/hbase-0.98-rw/hbase-common/src/main/resources/hbase-default.xml 0.98 /projects/opensource/hbase/hbase-trunk-rw-git/hbase-common/src/main/resources/hbase-default.xml 0.99
=========================================================
Start
=========================================================
Checking differences across versions...
Added or renamed keys in 0.92:
added: Property{key='dfs.support.append', value='true', description='Does HDFS allow appends to files? This is an hdfs config. set in here so the hdfs client will do append support. You must ensure tha
@larsgeorge
larsgeorge / gist:dbfbb09f914eb5079be1
Last active April 20, 2021 12:20
Hidden HBase Properties
IMPORTANT DISCLAIMER: Many of the found "properties" aren't really useful or mistakenly listed,
like those starting with "org.apache.hadoop.*". The tool simply greps a pattern in the matching
files, and some of them are not used as properties. On the other hand, all named properties from
the given configuration file are removed, so quite a few are valid. Again, some of those are still
hidden on purpose, since it is not useful to tinker with them unless in extreme situations, or
during test runs. Be VERY CAREFUL of what you do. I warned ya!
$ sh target/bin/run-finder -d /projects/opensource/hbase/hbase-trunk-rw-git/ -u -s \
--exclude **/generated --exclude **/generated/** --exclude **/target \
--exclude **/test/** -c hbase-common/src/main/resources/hbase-default.xml -o /dev/null
@larsgeorge
larsgeorge / gist:e5797433a32edd6487886069f150e9c7
Created June 10, 2016 12:46
Metric Sequence Diagram
# See: https://www.websequencediagrams.com/
title HBase Metrics Call Sequence Diagramm
MetricsSystem->MetricsSystem: onTimerEvent()
activate MetricsSystem
MetricsSystem->MetricsSystem: publishMetrics()
MetricsSystem->MetricsSystem: sampleMetrics()
MetricsSystem->+MetricsBufferBuilder: constructor
MetricsBufferBuilder->-MetricsSystem: MetricsBufferBuilder
@larsgeorge
larsgeorge / presto-histogramm-20161125.log
Created November 25, 2016 13:08
2016-11-25T12:39:20.420+0100: 177.558: [SoftReference, 0 refs, 0.0000547 secs]2016-11-25T12:39:20.420+0100: 177.558: [WeakReference, 2766 refs, 0.0004900 secs]2016-11-25T12:39:20.421+0100: 177.558: [FinalReference, 589 refs, 0.0007352 secs]2016-11-25T12:39:20.422+0100: 177.559: [PhantomReference, 0 refs, 182 refs, 0.0000369 secs]2016-11-25T12:39:20.422+0100: 177.559: [JNI Weak Reference, 0.0000666 secs] 177.562: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 8925478912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC]
177.562: [G1Ergonomics (Mixed GCs) do not continue mixed GCs, reason: reclaimable percentage not over threshold, candidate old regions: 94 regions, reclaimable: 850530200 bytes (4.95 %), threshold: 5.00 %]
, 0.0895976 secs]
[Parallel Time: 82.9 ms, GC Workers: 33]
[GC Worker Start (ms): Min: 177473.8, Avg: 177474.2, Max: 177474.6, Diff: 0.8]
[Ext Root Scann
@larsgeorge
larsgeorge / gist:9577d2cef0c8ff39c7c8c6ca2e7b570b
Last active June 12, 2020 17:13
-- cleanup previous demo
-- drop database and permissions
drop database if exists marketing cascade INCLUDING PERMISSIONS;
drop database if exists demo cascade INCLUDING PERMISSIONS;
-- ensure roles are all created cleanly
drop role if exists analyst_role;
create role if not exists analyst_role;
grant role analyst_role to group analyst_group;
-- create US role for old view row filter demo.
-- TODO remove once we are only using 2.1 to demo