Skip to content

Instantly share code, notes, and snippets.

View jeesim2's full-sized avatar

Jihun No jeesim2

  • Naver
  • Seoul, Korea
View GitHub Profile
```
mkdir -p ~/Library/KeyBindings
vi ~/Library/KeyBindings/DefaultkeyBinding.dict
```
```
{
"\UF729" = "moveToBeginningOfLine:"; /* Home */
"\UF72B" = "moveToEndOfLine:"; /* End */
```
{
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"_all": {
"primaries": {
"docs": {
@jeesim2
jeesim2 / environment.html
Created September 7, 2017 08:07
zeppelin0.7.3-spark2.2.0.environment.html
<html><head>
<meta http-equiv="Content-type" content="text/html; charset=utf-8"><link rel="stylesheet" href="/proxy/application_1504093826660_6034/static/bootstrap.min.css" type="text/css"><link rel="stylesheet" href="/proxy/application_1504093826660_6034/static/vis.min.css" type="text/css"><link rel="stylesheet" href="/proxy/application_1504093826660_6034/static/webui.css" type="text/css"><link rel="stylesheet" href="/proxy/application_1504093826660_6034/static/timeline-view.css" type="text/css"><script src="/proxy/application_1504093826660_6034/static/sorttable.js"></script><script src="/proxy/application_1504093826660_6034/static/jquery-1.11.1.min.js"></script><script src="/proxy/application_1504093826660_6034/static/vis.min.js"></script><script src="/proxy/application_1504093826660_6034/static/bootstrap-tooltip.js"></script><script src="/proxy/application_1504093826660_6034/static/initialize-tooltips.js"></script><script src="/proxy/application_1504093826660_6034/static/table.js"></script><script
%spark
import java.io.File
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.functions._
import org.apache.spark.sql._
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.Vector
import scala.collection.mutable.WrappedArray
import org.apache.spark.sql.expressions.Window
@jeesim2
jeesim2 / spark-submit-streaming-yarn.sh
Created August 23, 2017 00:43 — forked from bernhardschaefer/spark-submit-streaming-yarn.sh
spark-submit template for running Spark Streaming on YARN
#!/bin/bash
# Minimum TODOs on a per job basis:
# 1. define name, application jar path, main class, queue and log4j-yarn.properties path
# 2. remove properties not applicable to your Spark version (Spark 1.x vs. Spark 2.x)
# 3. tweak num_executors, executor_memory (+ overhead), and backpressure settings
# the two most important settings:
num_executors=6
executor_memory=3g
// Helper: root(), and rootDir() are defined at the bottom
var path = require('path');
var webpack = require('webpack');
// Webpack Plugins
var CommonsChunkPlugin = webpack.optimize.CommonsChunkPlugin;
var autoprefixer = require('autoprefixer');
var HtmlWebpackPlugin = require('html-webpack-plugin');
var ExtractTextPlugin = require('extract-text-webpack-plugin');
var CopyWebpackPlugin = require('copy-webpack-plugin');
Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1467878636123_1323_8_00, diagnostics=[Task failed, taskId=task_1467878636123_1323_8_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable null at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172
at java.security.AccessController.doPrivileged(Native Method
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.77-b03 mixed mode):
"Attach Listener" #58 daemon prio=9 os_prio=0 tid=0x00007fd6c400d000 nid=0x78c4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"elasticsearch[dev-jpl-elastic004.ncl][management][T#1]" #57 daemon prio=5 os_prio=0 tid=0x00007fd69c081800 nid=0x1ea5 waiting on condition [0x00007fd6a92d7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b1f12678> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><meta http-equiv="Content-type" content="text/html;charset=x-windows-949"><title>Problem Suspect 1</title><link rel="stylesheet" type="text/css" href="../styles.css"><script src="../code.js" type="text/javascript"></script></head><body onload="preparepage();"><input type="hidden" id="imageBase" value="../img/"><div id="header"><ul><li><a href="../index.html#17">Leak Suspects</a></li><li>&raquo; <a href="../index.html#17">Leaks</a></li><li>&raquo; <a href="#">Problem Suspect 1</a></li></ul></div>
<h5><a href="#" onclick="hide(this, 'exp18'); return false;" title="hide / unhide"><img src="../img/opened.gif" alt=""></a> <a name="18">Description</a></h5><div id="exp18"><div class="important"><div><p>One instance of <b>&quot;net.sf.ehcache.Cache&quot;</b> loaded by <b>&quot;sun.misc.Launcher$AppClassLoader @ 0x701619998&quot;</b> occupies <b>337,472,304 (92.18%)</b> bytes. The memory is accumulated in