Skip to content

Instantly share code, notes, and snippets.

View coderplay's full-sized avatar

Min Zhou coderplay

192 followers · 2 following
View GitHub Profile
@coderplay
coderplay / scala.md
Last active February 3, 2017 22:07
scala学习

变量

// 变量
var x = 6
// 常量
val x = 6
// 显式类型
var x:Double = 6
@coderplay
coderplay / mmap-cn.md
Last active August 15, 2016 03:49

mmap是lazy加载数据进内存的. 在映射内存的时候, 不会引起任何I/O. 加载的时候I/O会有readahead, 这个可以由madvise调整; 内存的分配是按page为单位添加内存, 可以由下面代码得到证明.

import sun.misc.Unsafe;
import sun.nio.ch.DirectBuffer;

import java.io.File;
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
@coderplay
coderplay / netty3.md
Last active August 3, 2016 18:41
Netty 3 & 4 workflow vs Nio Workflow

Netty 3 workflow vs Nio Workflow

Server Side

each thread in the threadpool create a new selector

1. selector = Selector.open()

main@1, prio=5, in group 'main', status: 'RUNNING'
	  at java.nio.channels.Selector.open(Selector.java:227)
@coderplay
coderplay / push-shuffle.md
Last active August 29, 2015 13:59
Push mode data shuffling desgin notes

MapReduce Pull mode

Push mode

Map启动时, 需要知道下游所有Reduce的地址

Reduce启动时， 需要知道上游所有Map的地址

Fault tolerance

@coderplay
coderplay / Yarn.md
Last active August 29, 2015 14:01
Hadoop Yarn 研究

日志

NodeManager日志

在页面左边有一个Nav, Tools里面的Local logs可以看到NodeManager本身的日志. 这个日志是由HttpServer提供的标准servlets. 对应的还有 /static, /stacks, /logLevel, /metrics, /jmx, /conf等众多servlet.由于这个HttpServer是hadoop-common提供的，所以所有hadoop页面都会有这几个servlet.

Application日志

Application日志由AppBlock渲染. 没看到列出Application所有的Containers的页面. RMAppAttempt接口里也没有提供这种方法. 事实上RMAppAttempt有一个Set getRanNodes()的接口, 如果页面渲染Application申请过的节点, 那也可以通过ApplicationId从该节点NodeManager页面找到对应的Container日志。但目前没有提供，运维起来不好查错. Set getRanNodes()接口的值是通过RMAppAttemptImpl.ContainerAcquiredTransition得到, 不知道Yarn为什么不直接把事件的Container存下来，而是存NodeId.

@coderplay
coderplay / netstat.s
Last active August 29, 2015 14:07
tcp retrans
$ netstat -s
Ip:
7222283184 total packets received
0 forwarded
0 incoming packets discarded
7222283182 incoming packets delivered
7226682964 requests sent out
1 outgoing packets dropped
Icmp:
21 ICMP messages received
@coderplay
coderplay / perf.txt
Last active August 29, 2015 14:07
sys high
java [kernel.kallsyms] [k] pvclock_clocksource_read
|
--- pvclock_clocksource_read
|
|--98.10%-- xen_clocksource_get_cycles
| ktime_get_ts
| posix_ktime_get_ts
| SyS_clock_gettime
| system_call_fastpath
| 0x7fff910847d2
@coderplay
coderplay / jobs.md
Last active August 29, 2015 14:13

综合

  • 税务
  • 退货：amazon尿盆、TBS礼盒六套、3B被子
  • 宽带
  • 垃圾
  • 找园丁打理
  • 院子除落叶
  • 寄出新房水电费的check
  • 预约DMV, 取得最新的注册卡
  • DMV 换地址
@coderplay
coderplay / cassandra.md
Last active August 29, 2015 14:20 
  java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
   at org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(WaitQueue.java:307)
   at org.apache.cassandra.utils.concurrent.SimpleCondition.await(SimpleCondition.java:63)
   at org.apache.cassandra.service.ReadCallback.await(ReadCallback.java:90)
   at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:100)
   at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
   at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1338)
@coderplay
coderplay / BytesToBytesMapBenchmark.java
Created June 22, 2015 23:02
BytesToBytesMapBenchmark
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0