Skip to content

Instantly share code, notes, and snippets.

View zcox's full-sized avatar

Zach Cox zcox

52 followers · 7 following
View GitHub Profile
@zcox
zcox / gist:44e35eb369c26491fe8f
Created April 11, 2015 13:56
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/druid-0.7.1.1/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-04-11T13:40:15,661 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2015-04-11T13:40:15,747 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2015-04-11T13:40:16,469 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:postgresql-metadata-storage, io.druid.extensions:druid-hdfs-storage], defaultVersion='0.7.1.1', localRepository='/opt/druid/repository', remot
@zcox
zcox / gist:fca6a58e3736ed97d3c2
Created April 11, 2015 15:17
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/druid-0.7.1.1/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-04-11T15:08:28,550 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2015-04-11T15:08:28,595 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2015-04-11T15:08:29,325 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:postgresql-metadata-storage, io.druid.extensions:druid-hdfs-storage], defaultVersion='0.7.1.1', localRepository='/opt/druid/repository', remot
@zcox
zcox / gist:7b70f447a99bd7e9aded
Created April 11, 2015 16:13
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/druid-0.7.1.1/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-04-11T16:01:28,229 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2015-04-11T16:01:28,299 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2015-04-11T16:01:29,011 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:postgresql-metadata-storage, io.druid.extensions:druid-hdfs-storage], defaultVersion='0.7.1.1', localRepository='/opt/druid/repository', remot
@zcox
zcox / gist:0d72c2dd45d23842b2a1
Created April 11, 2015 16:13
2015-04-11 12:04:51,486 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1428768036084_0001_000001
2015-04-11 12:04:51,779 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-04-11 12:04:51,793 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-04-11 12:04:51,864 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-04-11 12:04:51,887 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-04-11 12:04:51,935 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@3628d98d)
2015-04-11 12:04:51,957 INFO [
@zcox
zcox / gist:56c77860fb75bb844f80
Created April 11, 2015 17:21
bash-4.1# bin/hadoop fs -ls /druid-working/classpath | grep hadoop
-rw-r--r-- 3 root supergroup 126192 2015-04-11 13:10 /druid-working/classpath/druid-indexing-hadoop-0.7.1.1.jar
-rw-r--r-- 3 root supergroup 16778 2015-04-11 13:10 /druid-working/classpath/hadoop-annotations-2.3.0.jar
-rw-r--r-- 3 root supergroup 17036 2015-04-11 13:10 /druid-working/classpath/hadoop-annotations-2.4.1.jar
-rw-r--r-- 3 root supergroup 49920 2015-04-11 13:10 /druid-working/classpath/hadoop-auth-2.3.0.jar
-rw-r--r-- 3 root supergroup 50524 2015-04-11 13:10 /druid-working/classpath/hadoop-auth-2.4.1.jar
-rw-r--r-- 3 root supergroup 2559 2015-04-11 13:10 /druid-working/classpath/hadoop-client-2.3.0.jar
-rw-r--r-- 3 root supergroup 2831223 2015-04-11 13:10 /druid-working/classpath/hadoop-common-2.3.0.jar
-rw-r--r-- 3 root supergroup 1510793 2015-04-11 13:10 /druid-working/classpath/hadoop-common-2.4.1-tests.jar
-rw-r--r-- 3 root supergroup 2850191 2015-04-11 13:10 /druid-worki
@zcox
zcox / gist:b6eddaea76343baebad9
Created April 12, 2015 19:02
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/druid-0.7.1.1/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-04-12T18:49:10,051 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2015-04-12T18:49:10,123 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2015-04-12T18:49:10,810 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:postgresql-metadata-storage], defaultVersion='0.7.1.1', localRepository='/opt/druid/repository', remoteRepositories=[https://repo1.maven.org/m
@zcox
zcox / gist:0fc76242702fb803c0c1
Created April 12, 2015 19:02
2015-04-12 14:51:13,231 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1428864489761_0001_000001
2015-04-12 14:51:13,487 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-04-12 14:51:13,501 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-04-12 14:51:13,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-04-12 14:51:13,616 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@787582d3)
2015-04-12 14:51:13,627 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max attempts: 2 for application: 1. Attempt num: 1 is last retry: false
2015-04-12 14:51:13,634 INFO [main] org.
@zcox
zcox / wordcounts.rb
Created January 4, 2010 15:24
Revised Ruby version from this blog post: http://blogs.sourceallies.com/2009/12/word-counts-example-in-ruby-and-scala
rootDir = "/home/zcox/dev/20_newsgroups"
raise rootDir + " does not exist" unless File.directory? rootDir
counts = Hash.new(0) #0 will be the default value for non-existent keys
Dir["#{rootDir}/**/*"].reject{|file| File.directory? file}.each do |file|
IO.read(file).scan(/\w+/) { |word| counts[word.downcase] += 1 }
end
open("counts-descreasing-ruby", "w") do |out|
counts.sort { |a, b| b[1] <=> a[1] }.each { |pair| out << "#{pair[0]}\t#{pair[1]}\n" }
@zcox
zcox / wordcounts.scala
Created January 4, 2010 15:25
Revised Scala version from this blog post: http://blogs.sourceallies.com/2009/12/word-counts-example-in-ruby-and-scala
import java.io._
val rootDir = new File("/home/zcox/dev/20_newsgroups")
if (!rootDir.exists) throw new IllegalArgumentException(rootDir + " does not exist")
var counts = Map.empty[String, Int].withDefaultValue(0)
files(rootDir) { _.split("""\W+""").foreach { word => counts = counts(word.toLowerCase) += 1 }}
write(counts, "counts-descreasing-scala") {_._2 > _._2}
write(counts, "counts-alphabetical-scala") {_._1 < _._1}
/** Writes the specified map to the specified file in tab-delimited format, sorted accordingly. */
@zcox
zcox / using.scala
Created February 3, 2010 15:07
def using[A <: {def close(): Unit}, B](param: A)(f: A => B): B =
try {
f(param)
} finally {
param.close()
}