Skip to content

Instantly share code, notes, and snippets.

View bbeaudreault's full-sized avatar

Bryan Beaudreault bbeaudreault

View GitHub Profile
@bbeaudreault
bbeaudreault / HBaseMapReduceUtils.java
Last active November 8, 2017 06:28
Control # of mappers per RegionServer
package org.apache.hadoop.hbase.mapreduce;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.mapreduce.Job;
@bbeaudreault
bbeaudreault / HMerge.java
Last active December 28, 2015 21:49
Modifications to 0.94.x HMerge tool, to allow running against an online cluster with target table disabled. Important: See README.
/**
* Copyright 2009 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
@bbeaudreault
bbeaudreault / gist:5834357
Created June 21, 2013 21:13
Cleanup hadoop jobcache files
#!/bin/bash
for DIR in `find /mnt/mapred/local/taskTracker/*/jobcache/* -maxdepth 0 -type d -mmin +60`; do
if ! find $DIR | grep attempt > /dev/null; then
rm -rf $DIR;
fi;
done;
# There is also another bug that results in jobcache directories being duplicated
# within the attempt_ directories we filter out above. These directories never go away and so jobs with
@bbeaudreault
bbeaudreault / gist:5710269
Last active December 18, 2015 02:19
Alternate implementation of Bookie#getBookieAddress(ServerConfiguration conf) which is configurable to look at a specific interface.
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.InterfaceAddress;
import java.net.NetworkInterface;
import java.net.SocketException;
import java.net.UnknownHostException;
import org.apache.bookkeeper.conf.ServerConfiguration;
public class Bookie {