Skip to content

Instantly share code, notes, and snippets.

View aajisaka's full-sized avatar

Akira Ajisaka aajisaka

View GitHub Profile
@aajisaka
aajisaka / test.log
Created May 3, 2018 14:34
Test log with HADOOP-14178.014.patch (mvn test -fn)
This file has been truncated, but you can view the full file.
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project:pom:3.2.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ org.apache.hadoop:hadoop-main:3.2.0-SNAPSHOT, /Users/ajisaka/git/hadoop/pom.xml, line 431, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project-dist:pom:3.2.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ org.apache.hadoop:hadoop-main:3.2.0-SNAPSHOT, /Users/ajisaka/git/hadoop/pom.xml, line 431, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-assemblies:jar:3.2.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(
@aajisaka
aajisaka / jdk9.patch
Created December 2, 2017 14:44
Apply the patch and compile Apache Hadoop with Java 9.0.1
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
index be85497209b..eb7b34f8bd4 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
@@ -27,6 +27,7 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Seekable;
+import org.apache.hadoop.util.CleanerUtil;
@aajisaka
aajisaka / log
Created July 19, 2017 05:02
$ mvn clean install -DskipTests -DskipShade -Dzookeeper.version=3.5.3-beta 2>&1 > log
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-dist:jar:3.0.0-beta1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
@aajisaka
aajisaka / qbt.log
Created December 5, 2016 07:14
qbt log
/tmp/yetus-4429.25592 has been created
Modes: UnitTests
============================================================================
============================================================================
Confirming git environment
============================================================================
============================================================================
@aajisaka
aajisaka / Test.java
Created December 1, 2016 13:41
JDK-9045797 reproduce
import java.io.File;
import java.io.FileWriter;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.BasicFileAttributeView;
import java.nio.file.attribute.FileTime;
public class Test {
public static void main(String args[]) throws Exception {
// create a non-empty file.
1 3.5
1 4.0
1 5.9
1 8.7
2 9.3
2 9.6
2 10.3
1 10.2
2 3.8
3 11.5
@aajisaka
aajisaka / hadoop-bundled-jars.txt
Last active November 4, 2016 19:36
find . -name "*.jar" | xargs -n1 basename | sort | uniq | grep -v 3.0.0-alpha1
ST4-4.0.7.jar
aliyun-sdk-oss-2.2.1.jar
annotations-api.jar
antlr-2.7.7.jar
antlr-3.5.jar
antlr-runtime-3.5.jar
aopalliance-1.0.jar
asm-all-5.0.2.jar
avro-1.7.4.jar
aws-java-sdk-core-1.10.6.jar
@aajisaka
aajisaka / Benchmark.md
Created April 26, 2016 13:40
Benchmark MAPREDUCE-2841

Native off: 506sec

$ hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar wordcount -Dmapreduce.job.reduces=10 wikimedia out1
(snip)
16/04/26 12:43:26 INFO mapreduce.Job: Counters: 50
	File System Counters
		FILE: Number of bytes read=32688875608
		FILE: Number of bytes written=46793855889
		FILE: Number of read operations=0
@aajisaka
aajisaka / THIRD-PARTY.txt.branch-2.6
Created April 25, 2016 14:43
(branch-2.6) $ mvn license:aggregate-add-third-party
Lists of 212 third-party dependencies.
(BSD License) AntLR Parser Generator (antlr:antlr:2.7.7 - http://www.antlr.org/)
(Public Domain) AOP alliance (aopalliance:aopalliance:1.0 - http://aopalliance.sourceforge.net)
(BSD) ASM Core (asm:asm:3.2 - http://asm.objectweb.org/asm/)
(Unknown license) bcprov-jdk15 (bouncycastle:bcprov-jdk15:140 - no url defined)
(ASF 2.0) Code Generation Library (cglib:cglib:2.2 - http://cglib.sourceforge.net/)
(Apache License, Version 2.0) AWS SDK for Java (com.amazonaws:aws-java-sdk:1.7.4 - https://aws.amazon.com/sdkforjava)
(Apache License 2.0) Metrics Core (com.codahale.metrics:metrics-core:3.0.1 - http://metrics.codahale.com/metrics-core/)
(GNU Lesser General Public License, Version 2.1) (The Apache Software License, Version 2.0) Jackson-annotations (com.fasterxml.jackson.core:jackson-annotations:2.2.3 - http://wiki.fasterxml.com/JacksonHome)
@aajisaka
aajisaka / THIRD-PARTY.txt.branch-2.7
Created April 25, 2016 14:22
(branch-2.7) $ mvn license:aggregate-add-third-party
Lists of 214 third-party dependencies.
(BSD License) AntLR Parser Generator (antlr:antlr:2.7.7 - http://www.antlr.org/)
(Public Domain) AOP alliance (aopalliance:aopalliance:1.0 - http://aopalliance.sourceforge.net)
(BSD) ASM Core (asm:asm:3.2 - http://asm.objectweb.org/asm/)
(Unknown license) bcprov-jdk15 (bouncycastle:bcprov-jdk15:140 - no url defined)
(ASF 2.0) Code Generation Library (cglib:cglib:2.2 - http://cglib.sourceforge.net/)
(Apache License, Version 2.0) AWS SDK for Java (com.amazonaws:aws-java-sdk:1.7.4 - https://aws.amazon.com/sdkforjava)
(Apache License 2.0) Metrics Core (com.codahale.metrics:metrics-core:3.0.1 - http://metrics.codahale.com/metrics-core/)
(GNU Lesser General Public License, Version 2.1) (The Apache Software License, Version 2.0) Jackson-annotations (com.fasterxml.jackson.core:jackson-annotations:2.2.3 - http://wiki.fasterxml.com/JacksonHome)