Skip to content

Instantly share code, notes, and snippets.

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
index 10b0012016b..6f53ee436ba 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
@@ -210,6 +210,11 @@ private Integer listCorruptFileBlocks(String dir, String baseUrl)
allDone = true;
break;
}
+ if (line.startsWith("Access denied for user")) {
+ numCorrupt = 0;
@phstudy
phstudy / HyperLogLogStoreUDAF.scala
Created June 6, 2017 17:12 — forked from MLnick/HyperLogLogStoreUDAF.scala
Experimenting with Spark SQL UDAF - HyperLogLog UDAF for distinct counts, that stores the actual HLL for each row to allow further aggregation
class HyperLogLogStoreUDAF extends UserDefinedAggregateFunction {
override def inputSchema = new StructType()
.add("stringInput", BinaryType)
override def update(buffer: MutableAggregationBuffer, input: Row) = {
// This input Row only has a single column storing the input value in String (or other Binary data).
// We only update the buffer when the input value is not null.
if (!input.isNullAt(0)) {
if (buffer.isNullAt(0)) {
@phstudy
phstudy / gist:fdf40c04feb1b52c99e7bc2fe843bf56
Created June 1, 2017 15:11 — forked from saetia/gist:1623487
Clean Install – OS X 10.11 El Capitan

OS X Preferences


most of these require logout/restart to take effect

# Enable character repeat on keydown
defaults write -g ApplePressAndHoldEnabled -bool false

# Set a shorter Delay until key repeat
@phstudy
phstudy / gist:c180afa138325a4bd8fe7c566d1ed839
Created June 1, 2017 15:11 — forked from saetia/gist:1623487
Clean Install – OS X 10.11 El Capitan

OS X Preferences


most of these require logout/restart to take effect

# Enable character repeat on keydown
defaults write -g ApplePressAndHoldEnabled -bool false

# Set a shorter Delay until key repeat
@phstudy
phstudy / install.md
Created June 1, 2017 15:10 — forked from hlb/Brewfile
clean install

System Preferences

# Enable character repeat on keydown
defaults write -g ApplePressAndHoldEnabled -bool false

# Set a shorter Delay until key repeat
defaults write NSGlobalDomain InitialKeyRepeat -int 12

# Set a blazingly fast keyboard repeat rate
@phstudy
phstudy / ncdc.sh
Last active July 22, 2019 17:34 — forked from aligusnet/ncdc.sh
Download a weather dataset from the National Climatic Data Center (NCDC, http://www .ncdc.noaa.gov/). Prepare it for examples of "Hadoop: The Definitive Guide" book by Tom White. http://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/1449311520Usage:./ncdc.sh 1901 1930 # download wheather datasets for period from 1901 to 1930.
#!/usr/bin/env bash
# global parameters
g_tmp_folder="ncdc_tmp";
g_output_folder="ncdc_data";
g_remote_host="ftp.ncdc.noaa.gov";
g_remote_path="pub/data/noaa";
@phstudy
phstudy / core-site.xml.template
Last active April 5, 2017 08:11
azure blob storage core-site.xml.template
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
<!--
<property>
<name>fs.azure.account.key.{WASB_ACCOUNT_NAME}.blob.core.windows.net</name>
<value>{WASB_ACCOUNT_KEY}</value>
</property>
@phstudy
phstudy / Iterables.java
Last active June 4, 2016 16:59 — forked from popcornylu/Iterables.java
Iterable transformation
import org.apache.commons.collections4.IterableUtils;
import java.util.function.Function;
public class Iterables {
public static <F, T> Iterable<T> map(
Iterable<F> iterable,
Function<F, T> mapper) {
return IterableUtils.transformedIterable(iterable, (x) -> mapper.apply(x));
}
@phstudy
phstudy / crash.log
Created February 5, 2016 09:30
2016-02-05 azure sg vm crash
Feb 5 09:01:01 localhost systemd: Started Session 81 of user root.
Feb 5 09:01:01 localhost systemd: Starting Session 81 of user root.
Feb 5 09:01:01 localhost systemd: Removed slice user-0.slice.
Feb 5 09:01:01 localhost systemd: Stopping user-0.slice.
Feb 5 10:01:01 localhost systemd: Created slice user-0.slice.
Feb 5 10:01:01 localhost systemd: Starting user-0.slice.
Feb 5 10:01:01 localhost systemd: Started Session 82 of user root.
Feb 5 10:01:01 localhost systemd: Starting Session 82 of user root.
Feb 5 10:01:01 localhost systemd: Removed slice user-0.slice.
@phstudy
phstudy / create_docker_vm.ps1
Created April 25, 2015 04:13
create docker vm
azure vm docker create -e 22 -l 'West US' VM_NAME "b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140618.1-en-us-30GB" ACCOUNT PASSWORD