OZAWA Tsuyoshi oza
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| private DAGProtos.DAGPlan getDAGPlan() throws IOException { | |
| String stagingDir = conf.get(TezConfiguration.TEZ_AM_STAGING_DIR, | |
| TezConfiguration.TEZ_AM_STAGING_DIR_DEFAULT); | |
| Path binaryPlan = new Path( | |
| stagingDir, TezConfiguration.TEZ_PB_PLAN_BINARY_NAME); | |
| FileInputStream dagPBBinaryStream = null; | |
| DAGProtos.DAGPlan dagPlan = null; | |
| try { | |
| // Read the protobuf DAG | |
| dagPBBinaryStream = new FileInputStream(binaryPlan.getName()); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| private DAGProtos.DAGPlan getDAGPlan() throws IOException { | |
| String stagingDir = conf.get(TezConfiguration.TEZ_AM_STAGING_DIR, | |
| TezConfiguration.TEZ_AM_STAGING_DIR_DEFAULT); | |
| Path binaryPlan = new Path( | |
| stagingDir, TezConfiguration.TEZ_PB_PLAN_BINARY_NAME); | |
| FileInputStream dagPBBinaryStream = null; | |
| DAGProtos.DAGPlan dagPlan = null; | |
| try { | |
| // Read the protobuf DAG | |
| dagPBBinaryStream = new FileInputStream(binaryPlan.getName()); |
oza
/ hadoop-yarn-project-test.log
Last active
August 29, 2015 13:58
A result of "mvn test" with hadoop-2.4.0-rc0 source code.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Running org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell | |
| Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 219.414 sec <<< FAILURE! - in org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell | |
| testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell) Time elapsed: 30.716 sec <<< FAILURE! | |
| junit.framework.AssertionFailedError: null | |
| at junit.framework.Assert.fail(Assert.java:48) | |
| at junit.framework.Assert.assertTrue(Ass |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [INFO] Scanning for projects... | |
| [WARNING] | |
| [WARNING] Some problems were encountered while building the effective model for org.apache.tez:tez-api:jar:0.5.0-incubating-SNAPSHOT | |
| [WARNING] 'build.pluginManagement.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.codehaus.mojo:findbugs-maven-plugin @ org.apache.tez:tez:0.5.0-incubating-SNAPSHOT, /Users/ozawa/workspace/tez/pom.xml, line 496, column 17 | |
| [WARNING] | |
| [WARNING] Some problems were encountered while building the effective model for org.apache.tez:tez-common:jar:0.5.0-incubating-SNAPSHOT | |
| [WARNING] 'build.pluginManagement.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.codehaus.mojo:findbugs-maven-plugin @ org.apache.tez:tez:0.5.0-incubating-SNAPSHOT, /Users/ozawa/workspace/tez/pom.xml, line 496, column 17 | |
| [WARNING] | |
| [WARNING] Some problems were encountered while building the effective model for org.apache.tez:tez-runtime-library:jar:0.5.0-incubating-SNAPSHOT | |
| [ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @Test (timeout = 20000) | |
| public void testRetriedFinishApplicationMasterRequest() | |
| throws Exception { | |
| conf.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, 1); | |
| MemoryRMStateStore memStore = new MemoryRMStateStore(); | |
| memStore.init(conf); | |
| // start RM | |
| rm1 = new MockRM(conf, memStore); | |
| rm1.start(); |
oza
/ SortBench.scala
Last active
August 29, 2015 14:08
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // scala> conf.get("spark.shuffle.manager") | |
| // res3: String = org.apache.spark.shuffle.sort.SortShuffleManager | |
| new scala.testing.Benchmark { | |
| def run() { | |
| sc.textFile("hdfs:///user/ozawa/wordCountInput20GB") | |
| .flatMap(line => line.split(" ")) | |
| .map(word => (word, 1)) | |
| .sortByKey(true, 37) | |
| .saveAsTextFile("hdfs:///user/ozawa/outs/sort/3"); | |
| } |
oza
/ core-site.xml
Last active
August 29, 2015 14:09
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <?xml version="1.0" encoding="UTF-8"?> | |
| <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> | |
| <!-- | |
| Licensed under the Apache License, Version 2.0 (the "License"); | |
| you may not use this file except in compliance with the License. | |
| You may obtain a copy of the License at | |
| http://www.apache.org/licenses/LICENSE-2.0 | |
| Unless required by applicable law or agreed to in writing, software |
oza
/ spark-native-yarn-am.log
Created
November 27, 2014 02:19
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"entity":"task_1417036912823_0003_1_02_000027","entitytype":"TEZ_TASK_ID","events":[{"ts":1417051621649,"eventtype":"TASK_FINISHED"}],"otherinfo":{"startTime":1417051619126,"endTime":1417051621649,"timeTaken":2523,"status":"KILLED","diagnostics":"Task is terminated due to:OTHER_TASK_FAILURE","counters":{}}} | |
| {"entity":"tez_container_1417036912823_0003_01_000007","entitytype":"TEZ_CONTAINER_ID","relatedEntities":[{"entity":"appattempt_1417036912823_0003_000001","entitytype":"TEZ_APPLICATION_ATTEMPT"},{"entity":"container_1417036912823_0003_01_000007","entitytype":"containerId"}],"events":[{"ts":1417051621649,"eventtype":"CONTAINER_STOPPED"}],"otherinfo":{"exitStatus":0}} | |
| {"entity":"tez_container_1417036912823_0003_01_000006","entitytype":"TEZ_CONTAINER_ID","relatedEntities":[{"entity":"appattempt_1417036912823_0003_000001","entitytype":"TEZ_APPLICATION_ATTEMPT"},{"entity":"container_1417036912823_0003_01_000006","entitytype":"containerId"}],"events":[{"ts":1417051621650,"eventtype":"CONTAINER_STOPPED"}],"other |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 14/11/27 15:50:26 INFO tez.DAGBuilder: DAG execution complete | |
| 14/11/27 15:50:26 ERROR tez.DAGBuilder: DAG diagnostics: [Vertex failed, vertexName=1, vertexId=vertex_1417036912823_0031_1_00, diagnostics=[Task failed, taskId=task_1417036912823_0031_1_00_000007, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure | |
| while running task:java.lang.IllegalArgumentException | |
| at com.google.common.base.Preconditions.checkArgument(Preconditions.java:76) | |
oza
/ join-on-sparkshell-info.log
Last active
August 29, 2015 14:10
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Vertex failed, vertexName=0, vertexId=vertex_1417036912823_0059_1_02, diagnostics=[Task failed, taskId=task_1417036912823_0059_1_02_000014, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.IllegalStateException: Failed to execute processor for Vertex 2 | |
| at org.apache.spark.tez.SparkTaskProcessor.run(SparkTaskProcessor.scala:55) | |
| at org.apache.tez.runtime.library.processor.SimpleProcessor.run(SimpleProcessor.java:53) | |
| at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) | |
| at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) | |
| at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:415) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) | |
| at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRu |
OlderNewer