coderplay

TCP/IP传输层协议

此次花了三天时间读了 PART II-8 TCP/IP transport layer protocols 从第42章至第49章, 收获很大. 明白了以下一些问题:

• 为什么需要tcp? 同时又需要udp?
• tcp/udp端口的意义
• tcp 和 udp端口的区别, 是否可以在一个终端上同时使用相同的tcp端口和udp端口
• tcp端口范围,为什么要保留一些端口
• udp格式及为什么要这么设计udp格式
• tcp的主要功能
• MSS和MTU的区别及联系

coderplay

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0

coderplay

  java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
   at org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(WaitQueue.java:307)
   at org.apache.cassandra.utils.concurrent.SimpleCondition.await(SimpleCondition.java:63)
   at org.apache.cassandra.service.ReadCallback.await(ReadCallback.java:90)
   at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:100)
   at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
   at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1338)

coderplay

综合

• 税务
• 退货：amazon尿盆、TBS礼盒六套、3B被子
• 宽带
• 垃圾
• 找园丁打理
• 院子除落叶
• 寄出新房水电费的check
• 预约DMV, 取得最新的注册卡
• DMV 换地址

coderplay

 java  [kernel.kallsyms]  [k] pvclock_clocksource_read
                  |
                  --- pvclock_clocksource_read
                     |
                     |--98.10%-- xen_clocksource_get_cycles
                     |          ktime_get_ts
                     |          posix_ktime_get_ts
                     |          SyS_clock_gettime
                     |          system_call_fastpath
                     |          0x7fff910847d2

coderplay

$ netstat -s
Ip:
    7222283184 total packets received
    0 forwarded
    0 incoming packets discarded
    7222283182 incoming packets delivered
    7226682964 requests sent out
    1 outgoing packets dropped
Icmp:
    21 ICMP messages received

coderplay

日志

NodeManager日志

在页面左边有一个Nav, Tools里面的Local logs可以看到NodeManager本身的日志. 这个日志是由HttpServer提供的标准servlets. 对应的还有 /static, /stacks, /logLevel, /metrics, /jmx, /conf等众多servlet.由于这个HttpServer是hadoop-common提供的，所以所有hadoop页面都会有这几个servlet.

Application日志

Application日志由AppBlock渲染. 没看到列出Application所有的Containers的页面. RMAppAttempt接口里也没有提供这种方法. 事实上RMAppAttempt有一个Set getRanNodes()的接口, 如果页面渲染Application申请过的节点, 那也可以通过ApplicationId从该节点NodeManager页面找到对应的Container日志。但目前没有提供，运维起来不好查错. Set getRanNodes()接口的值是通过RMAppAttemptImpl.ContainerAcquiredTransition得到, 不知道Yarn为什么不直接把事件的Container存下来，而是存NodeId.

coderplay

MapReduce Pull mode

Push mode

Map启动时, 需要知道下游所有Reduce的地址

Reduce启动时， 需要知道上游所有Map的地址

Fault tolerance

coderplay

Netty 3 workflow vs Nio Workflow

Server Side

each thread in the threadpool create a new selector

1. selector = Selector.open()

main@1, prio=5, in group 'main', status: 'RUNNING'
	  at java.nio.channels.Selector.open(Selector.java:227)

coderplay

mmap是lazy加载数据进内存的. 在映射内存的时候, 不会引起任何I/O. 加载的时候I/O会有readahead, 这个可以由madvise调整; 内存的分配是按page为单位添加内存, 可以由下面代码得到证明.

import sun.misc.Unsafe;
import sun.nio.ch.DirectBuffer;

import java.io.File;
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;