Skip to content

Instantly share code, notes, and snippets.

View busbey's full-sized avatar

Sean Busbey busbey

  • Champaign, IL USA
View GitHub Profile
#! /usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
git fetch apache
remote: Counting objects: 387, done.
remote: Compressing objects: 100% (113/113), done.
remote: Total 264 (delta 111), reused 179 (delta 72)
Receiving objects: 100% (264/264), 28.46 KiB, done.
Resolving deltas: 100% (111/111), completed with 55 local objects.
From http://git-wip-us.apache.org/repos/asf/accumulo
8669b80..65d5fba 1.4.5-SNAPSHOT -> apache/1.4.5-SNAPSHOT
* [new branch] 1.5.0 -> apache/1.5.0
c23126a..65aaddd 1.5.1-SNAPSHOT -> apache/1.5.1-SNAPSHOT
Failed tests: listscans(org.apache.accumulo.test.ShellServerTest)
maxrow(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<1>
importDirectory(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<2>
constraint(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<2>
renametable(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<2>
listcompactions(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<3>
classpath(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<3>
du(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<4>
grep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<5>
help(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<5>
-bash-4.1$ $ACCUMULO_HOME/bin/accumulo classpath
Level 1: Java System Classloader (loads Java system resources) URL classpath items are:
file:/usr/java/jdk1.6.0_31/jre/lib/ext/sunjce_provider.jar
file:/usr/java/jdk1.6.0_31/jre/lib/ext/localedata.jar
file:/usr/java/jdk1.6.0_31/jre/lib/ext/sunpkcs11.jar
file:/usr/java/jdk1.6.0_31/jre/lib/ext/dnsns.jar
Level 2: Java Classloader (loads everything defined by java classpath) URL classpath items are:
file:/etc/accumulo/continuous/
file:/usr/lib/accumulo-1.5.1-rc1/lib/accumulo-start.jar
[accumulo@localhost accumulo]$ bin/accumulo version
1.4.3-cdh4.3.0
[accumulo@localhost accumulo]$ bin/accumulo classpath
Accumulo List of classpath items are:
file:/usr/lib/accumulo/lib/commons-jci-fam-1.0.jar
file:/usr/lib/accumulo/lib/commons-jci-core-1.0.jar
file:/usr/lib/accumulo/lib/accumulo-core-1.4.3-cdh4.3.0.jar
file:/usr/lib/accumulo/lib/examples-simple-1.4.3-cdh4.3.0.jar
file:/usr/lib/accumulo/lib/commons-collections-3.2.jar
file:/usr/lib/accumulo/lib/accumulo-start-1.4.3-cdh4.3.0.jar
[alias]
hist = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset' --abbrev-commit --date=relative
oldest-ancestor = !zsh -c 'diff --old-line-format='' --new-line-format='' <(git rev-list --first-parent "${1:-master}") <(git rev-list --first-parent "${2:-HEAD}") | head -1' -
[core]
editor = /usr/bin/vim

In order to increase the likelihood that our planned WAL interfaces make sense, I'll try to step through how we get from the current APIs to the new ones and how we build on them to make the other WAL improvements we have planned. At the moment, I'd like this to be branch-1 / master.

Step 1, clean up WAL interface (HBASE-10378 / (HBASE-8610)

Current interface/class relationship in 0.98 / branch-1 / master

HLog <-- is_a -- FSHLog

Problem Statement

Currently, Hadoop exposes downstream clients to a variety of third party libraries. As our code base grows and matures we increase the set of libraries we rely on. At the same time, as our user base grows we increase the likelihood that some downstream project will run into a conflict while attempting to use a different version of some library we depend on. While there are hot-button third party libraries that drive most of the development and support issues (e.g. Guava, Apache Commons, and Jackson), a coherent general practice will ensure that we avoid future complications. Simply attempting to coordinate library versions among Hadoop and various downstream projects is untenable, because each project has its own release schedule and often attempts to support multiple versions of other ecosystem projects. Furthermore, our current approach of taking a conservative approach to dependency updates leads to reliance on stale versions of everything. Those stale versions include

@busbey
busbey / Version Numbers.md
Last active August 29, 2015 14:21
YCSB #236

The YCSB project uses the version numbers on our releases to set expectations about the impact of upgrading for downstream users. Each release will have a version number that looks like x.y.z. Starting with version 1.0.0, these numbers map directly to:

  • Major version x: incremented when pre-existing custom Workloads or Database bindings would break JVM Binary Compatibility. incremented when a change in how statistics are gathered or presented would impact performance analysis.
  • Minor version y: incremented when new Workloads or Database bindings are added. incremented when we add to the statistics gathered or the means to present them.
  • Patch version z: incremented when backwards and forwards compatible bug fixes are made.

Note that prior to version 1.0.0, minor version increments may introduce changes normally reserved for major versions.

Even when using the same major version, remember that YCSB is best suited for comparing sy

@busbey
busbey / Table.javap
Created July 2, 2015 17:45
HBase Table class output for javap
Classfile /Users/busbey/projects/hbase-priority/hbase-shaded/hbase-shaded-client/target/org/apache/hadoop/hbase/client/Table.class
Last modified Jul 2, 2015; size 6712 bytes
MD5 checksum 67a1591fb1a821d095f89d5e62f2cb3d
Compiled from "Table.java"
public interface org.apache.hadoop.hbase.client.Table extends java.io.Closeable
SourceFile: "Table.java"
InnerClasses:
public static #14= #11 of #13; //Callback=class org/apache/hadoop/hbase/client/coprocessor/Batch$Callback of class org/apache/hadoop/hbase/client/coprocessor/Batch
public static final #19= #16 of #18; //CompareOp=class org/apache/hadoop/hbase/filter/CompareFilter$CompareOp of class org/apache/hadoop/hbase/filter/CompareFilter
public static #22= #21 of #13; //Call=class org/apache/hadoop/hbase/client/coprocessor/Batch$Call of class org/apache/hadoop/hbase/client/coprocessor/Batch