Skip to content

Instantly share code, notes, and snippets.

@costin
Created September 24, 2012 14:08
Show Gist options
  • Save costin/3776128 to your computer and use it in GitHub Desktop.
Save costin/3776128 to your computer and use it in GitHub Desktop.
Mahout SHDP Tool Config
17:33:32,579 INFO main context.TestContextManager - @TestExecutionListeners is not present for class [class org.springframework.data.hadoop.MahoutTests]: using defaults.
17:33:32,710 INFO main xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/MahoutTests-context.xml]
17:33:32,917 INFO main xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
17:33:33,074 INFO main support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@59556d12: startup date [Mon Sep 24 17:33:33 EEST 2012]; root of context hierarchy
17:33:33,196 INFO main config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
17:33:33,234 INFO main support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7a2431b9: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,hadoopFs,org.springframework.data.hadoop.mapreduce.MapReducePropertyEditorRegistrar.ns.registration,hadoopResourceLoader,hadoopConfiguration,org.springframework.data.hadoop.scripting.HdfsScriptFactoryBean#0,mahout-tool,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
17:33:33,279 WARN main conf.Configuration - DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
17:33:33,688 WARN main util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17:33:33,688 WARN main snappy.LoadSnappy - Snappy native library not loaded
Resource type class org.springframework.core.io.UrlResource file:/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar 30037387
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/costin/.gradle/caches/artifacts-14/filestore/org.slf4j/slf4j-log4j12/1.6.6/jar/5cd9b4fbc3ff6a97beaade3206137d76f65df805/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a "org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [ParentLastURLCL
URLs: [file:/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar]
Parent CL: sun.misc.Launcher$AppClassLoader@63c78e57
System CL: sun.misc.Launcher$ExtClassLoader@425224ee
] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by [sun.misc.Launcher$AppClassLoader@63c78e57].
log4j:ERROR Could not instantiate appender named "stdout".
17:33:35,091 INFO main mapreduce.ToolRunner - Invoking [org.apache.mahout.cf.taste.hadoop.item.RecommenderJob@252a78ee] from jar [file:/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar] with args [[-s SIMILARITY_LOGLIKELIHOOD, -i/tmp/mah/intro.csv, -o recommendations]]
log4j:WARN No appenders could be found for logger (org.apache.mahout.common.AbstractJob).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
17:33:35,634 INFO main input.FileInputFormat - Total input paths to process : 1
17:33:36,079 INFO main mapred.JobClient - Running job: job_local_0001
17:33:36,147 INFO Thread-10 mapred.Task - Using ResourceCalculatorPlugin : null
17:33:36,176 INFO Thread-10 mapred.MapTask - io.sort.mb = 100
17:33:36,221 INFO Thread-10 mapred.MapTask - data buffer = 79691776/99614720
17:33:36,221 INFO Thread-10 mapred.MapTask - record buffer = 262144/327680
17:33:36,254 WARN Thread-10 mapred.LocalJobRunner - job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:47)
at org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
17:33:37,083 INFO main mapred.JobClient - map 0% reduce 0%
17:33:37,086 INFO main mapred.JobClient - Job complete: job_local_0001
17:33:37,086 INFO main mapred.JobClient - Counters: 0
17:33:37,189 WARN main mapreduce.ExecutionUtils - Trying to patch leaked cl [ParentLastURLCL
URLs: [file:/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar]
Parent CL: sun.misc.Launcher$AppClassLoader@63c78e57
System CL: sun.misc.Launcher$ExtClassLoader@425224ee
] in thread [Thread[communication thread,5,main]]
17:33:37,189 WARN main mapreduce.ExecutionUtils - Trying to patch leaked cl [ParentLastURLCL
URLs: [file:/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar]
Parent CL: sun.misc.Launcher$AppClassLoader@63c78e57
System CL: sun.misc.Launcher$ExtClassLoader@425224ee
] in thread [Thread[SpillThread,5,main]]
17:33:37,200 INFO Thread-3 support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@59556d12: startup date [Mon Sep 24 17:33:33 EEST 2012]; root of context hierarchy
17:33:37,201 INFO Thread-3 support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7a2431b9: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,hadoopFs,org.springframework.data.hadoop.mapreduce.MapReducePropertyEditorRegistrar.ns.registration,hadoopResourceLoader,hadoopConfiguration,org.springframework.data.hadoop.scripting.HdfsScriptFactoryBean#0,mahout-tool,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:hdp="http://www.springframework.org/schema/hadoop"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<import resource="./hadoop-ctx.xml"/>
<hdp:script run-at-startup="true" language="javascript">
if (fsh.test("temp"))
fsh.rmr("temp")
if (fsh.test("tmp/mah/intro.csv"))
fsh.put("/libs/hadoop/mahout/mahout-distribution-0.7/intro.csv", "/tmp/mah/intro.csv"
</hdp:script>
<hdp:tool-runner id="mahout-tool"
tool-class="org.apache.mahout.cf.taste.hadoop.item.RecommenderJob"
jar="/libs/hadoop/mahout/mahout-distribution-0.7/mahout-examples-0.7-job.jar">
<hdp:arg value="-s SIMILARITY_LOGLIKELIHOOD" />
<hdp:arg value="-i/tmp/mah/intro.csv" />
<hdp:arg value="-o recommendations" />
</hdp:tool-runner>
</beans>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment