gangliao

### Keybase proof

I hereby claim:

  * I am gangliao on github.
  * I am gangliao (https://keybase.io/gangliao) on keybase.
  * I have a public key ASBMZAZ036MtjOrnnl1dpRYibO_ZNf5pxSS1EM8iPhogJQo

To claim this, I am signing this object:

gangliao

import tensorflow as tf

filename_queue = tf.train.string_input_producer(["hdfs://10.152.104.73:8020/sogou/train-data/1_final.feature_transform"])

reader = tf.WholeFileReader()
key, value = reader.read(filename_queue)


with tf.Session() as sess:
  # Start populating the filename queue.

gangliao

import tensorflow as tf

filenames = ["hdfs://10.152.104.73:8020/sogou/train_data/1_final.feature_transform"]

dataset = tf.data.TextLineDataset(filenames)

iterator = dataset.make_one_shot_iterator()

next_batch = iterator.get_next()

gangliao

import matplotlib.pyplot as plt
#for plotting
import numpy as np 

# create plot
fig, ax = plt.subplots()
bar_width = 0.15
opacity = 0.8

xlabel= np.array([8, 16, 32, 64])

gangliao

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  ms/call  ms/call  name    
 22.44      0.35     0.35     3000     0.12     0.12  rgb_ycc_convert
 20.51      0.67     0.32    52828     0.01     0.01  encode_mcu_huff
 18.59      0.96     0.29   211031     0.00     0.00  forward_DCT
 17.95      1.24     0.28   316406     0.00     0.00  jpeg_fdct_islow
 12.18      1.43     0.19                             preload_image

gangliao

-- Logs begin at Thu 2018-04-12 23:33:59 CST, end at Thu 2018-05-03 16:28:49 CST. --
May 03 15:52:34 nmyjs_170_36 kubelet[10616]: W0503 15:52:34.980720   10616 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
May 03 15:52:36 nmyjs_170_36 kubelet[10616]: I0503 15:52:36.358267   10616 kubelet_node_status.go:609] Update capacity for nvidia.com/gpu to 8
May 03 15:52:45 nmyjs_170_36 kubelet[10616]: W0503 15:52:45.002416   10616 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
May 03 15:52:46 nmyjs_170_36 kubelet[10616]: I0503 15:52:46.393167   10616 kubelet_node_status.go:609] Update capacity for nvidia.com/gpu to 8
May 03 15:52:55 nmyjs_170_36 kubelet[10616]: W0503 15:52:55.023856   10616 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
May 03 15:52:56 nmyjs_170_36 kubelet[10616]: I0503 15:52:56.428377   10616 kubelet_node_status.go:609] Update capacity 

gangliao

Freeflow

TCP Physical Bandwidth

ethtool eth0 | grep Speed
# 	Speed: 1000Mb/s

gangliao

Freeflow

TCP Physical Bandwidth

ethtool eth0 | grep Speed
# 	Speed: 1000Mb/s

gangliao

# check os kernel version
echo "dump os kernel info:"
uname -a

# check IP
echo "dump IP info:"
ip -4 addr show scope global dev eth0 | grep inet | awk '{print $2}' | cut -d / -f 1

# check infiniband
echo "dump IB info:"

gangliao

#include <arpa/inet.h>
#include <stdio.h>
#include <unistd.h>

/*
IP Checksum

Benefits: fast, easy to compute and check which is motivated by earliest
software implementation.