Skip to content

Instantly share code, notes, and snippets.

View aajisaka's full-sized avatar

Akira Ajisaka aajisaka

246 followers · 48 following
View GitHub Profile
@aajisaka
aajisaka / hadoop-bundled-jars-20181203-trunk.txt
Created December 3, 2018 06:42
hadoop-bundled-jars-20181203-trunk.txt
accessors-smart-1.2.jar
aliyun-java-sdk-core-3.4.0.jar
aliyun-java-sdk-ecs-4.2.0.jar
aliyun-java-sdk-ram-3.0.0.jar
aliyun-java-sdk-sts-3.0.0.jar
aliyun-sdk-oss-3.0.0.jar
aopalliance-1.0.jar
asm-5.0.4.jar
asm-commons-6.0.jar
asm-tree-6.0.jar
@aajisaka
aajisaka / test.log
Created May 3, 2018 14:34
Test log with HADOOP-14178.014.patch (mvn test -fn)
This file has been truncated, but you can view the full file.
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project:pom:3.2.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ org.apache.hadoop:hadoop-main:3.2.0-SNAPSHOT, /Users/ajisaka/git/hadoop/pom.xml, line 431, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project-dist:pom:3.2.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ org.apache.hadoop:hadoop-main:3.2.0-SNAPSHOT, /Users/ajisaka/git/hadoop/pom.xml, line 431, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-assemblies:jar:3.2.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(
@aajisaka
aajisaka / jdk9.patch
Created December 2, 2017 14:44
Apply the patch and compile Apache Hadoop with Java 9.0.1
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
index be85497209b..eb7b34f8bd4 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
@@ -27,6 +27,7 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Seekable;
+import org.apache.hadoop.util.CleanerUtil;
@aajisaka
aajisaka / log
Created July 19, 2017 05:02
$ mvn clean install -DskipTests -DskipShade -Dzookeeper.version=3.5.3-beta 2>&1 > log
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-dist:jar:3.0.0-beta1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
@aajisaka
aajisaka / qbt.log
Created December 5, 2016 07:14
qbt log
/tmp/yetus-4429.25592 has been created
Modes: UnitTests
============================================================================
============================================================================
Confirming git environment
============================================================================
============================================================================
@aajisaka
aajisaka / Test.java
Created December 1, 2016 13:41
JDK-9045797 reproduce
import java.io.File;
import java.io.FileWriter;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.BasicFileAttributeView;
import java.nio.file.attribute.FileTime;
public class Test {
public static void main(String args[]) throws Exception {
// create a non-empty file.
@aajisaka
aajisaka / hadoop-bundled-jars.txt
Last active November 4, 2016 19:36
find . -name "*.jar" | xargs -n1 basename | sort | uniq | grep -v 3.0.0-alpha1
ST4-4.0.7.jar
aliyun-sdk-oss-2.2.1.jar
annotations-api.jar
antlr-2.7.7.jar
antlr-3.5.jar
antlr-runtime-3.5.jar
aopalliance-1.0.jar
asm-all-5.0.2.jar
avro-1.7.4.jar
aws-java-sdk-core-1.10.6.jar
@aajisaka
aajisaka / ratings.csv
Created September 29, 2016 07:54
1 3.5
1 4.0
1 5.9
1 8.7
2 9.3
2 9.6
2 10.3
1 10.2
2 3.8
3 11.5
@aajisaka
aajisaka / updateHadoop.sh
Last active May 23, 2016 21:14
Shell script used for updating hadoop (trunk) in my development VM
#!/bin/sh
if [ $# -ne 1 ]; then
echo "Usage ./updateHadoop.sh <NAME>"
echo "New hadoop package will be installed into /usr/hadoop/NAME"
echo "We recommend to use <YYYYMMDD-ISSUEID> for NAME."
exit 1
fi
# New hadoop package is installed to /usr/local/NAME
NAME=$1
@aajisaka
aajisaka / Benchmark.md
Created April 26, 2016 13:40
Benchmark MAPREDUCE-2841

Native off: 506sec

$ hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar wordcount -Dmapreduce.job.reduces=10 wikimedia out1
(snip)
16/04/26 12:43:26 INFO mapreduce.Job: Counters: 50
	File System Counters
		FILE: Number of bytes read=32688875608
		FILE: Number of bytes written=46793855889
		FILE: Number of read operations=0