oza

# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.

# Example:
# spark.master                     spark://master:7077
spark.eventLog.enabled           true
spark.eventLog.dir               file:///home/ozawa/sparkeventlogs
# spark.serializer                 org.apache.spark.serializer.KryoSerializer
# spark.driver.memory              5g
# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

oza

YARN

1. General resource management layer on HDFS
2. A part of Hadoop

Spark

1. In memory processing framework

Spark on YARN

oza

Tez source code reading

LICENSE

• This document is distributed under Apache 2 Licenses.
• All source code described in this document is from Apache Tez project.

Links

• Apache Tez : Accelerating Hadoop Query Processing by Bikas Saha

oza

Presto source code reading #1

• This document is created for Presto Source Code Reading #1.
  • http://atnd.org/events/47149
• Target: trunk code, Feb 10th, 2014
  • Main Topic: Coordinator and Executor ./presto-main/src/main/java/com/facebook/presto/executions
  • http://www.slideshare.net/frsyuki/hadoop-source-code-reading-15-in-japan-presto/10
• NOTE
  1. All source code noted here is quoted from https://github.com/facebook/presto. License bases on https://github.com/facebook/presto/blob/master/LICENSE.
  2. This document can include something wrong. Pull requests to fix them are always welcome :-)

oza

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

oza

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>

oza

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,

oza

Keybase proof

I hereby claim:

• I am oza on github.
• I am ozw (https://keybase.io/ozw) on keybase.
• I have a public key ASBEgZ5dBs8y7vz5MdexvhUgBH9vijrAse0P2YodzXZxxQo

To claim this, I am signing this object:

oza

Fault Tolerance in MRv2 over Hadoop/YARN

What is/Why YARN?

• Generic Resource Manager
• http://www.slideshare.net/ozax86/ntt-meets-hadoop-and-cloudera-world-tokyo#17

Architecture Overview

• ResourceManager/NodeManager

oza

Kudu

What's Kudu?

• From http://getkudu.io/
  • Kudu completes Hadoop's storage layer to enable fast analytics on fast data.
  • Distributed Insertable/Updatable columnar store.
• Schema on write.