liyuqi/python_q.md

## python_q.md

      
    Raw
  

              python_q.md
            
          
    Q1.pykafka
給定kafka-producer-perf-test條件，模仿輸出下列結果。
條件說明:100000筆，每筆大小100字，滿200字送出，不壓縮，對perf11類別，同步1 thread
# kafka-producer-perf-test.sh
# --broker-list 172.28.128.22:9092,172.28.128.23:9092,172.28.128.24:9092
# --messages 100000
# --threads 1
# --message-size 100
# --batch-size 200
# -compression-codec 0
# --topic ktest

# produce
#       開始|      結束|     訊息size|                總size| 每秒?MB|                 總筆數|     每秒?筆
#  start.time| end.time| message.size| total.data.sent.in.MB| MB.sec| total.data.sent.in.nMsg|   nMsg.sec
#    02:10:30| 02:19:20|          100|               4768.37| 8.9941|                50000000| 94309.9061
#
# consume
#        開始|     結束|   訊息size|                總size|  每秒?MB|                總筆數|       每秒?筆
#  start.time| end.time| fetch.size|   data.consumed.in.MB|   MB.sec| data.consumed.in.nMsg|     nMsg.sec
#    02:22:22| 02:22:29|    1048576|              277.9675| 137.8123|               2914700| 1445066.9311
Q2.storm petrel 串流寫法
給定輸入源 file ，輸出下列結果。
流程 Spout,Bolt,
概念 emit, process
  .,.,       .,              .,          TIMESTAMP,               .,      MSISDN,           .,            .
'85,0,40187421,466977200122266,2015-11-05 01:59:55,3587160516678701,886912345678,103.2.218.49,103.2.216.227'
'85,0,40187421,466977200122266,2015-11-05 01:59:56,3587160516678701,886912345678,103.2.218.49,103.2.216.227'
'85,0,40187421,466977200122266,2015-11-05 01:59:57,3587160516678701,886912345678,103.2.218.49,103.2.216.227'

MISIDN: 886912345678,
TIMESTAMP: [2015-11-05 01:59:55,
2015-11-05 01:59:56,
2015-11-05 01:59:57]
Q3.pykafka agg寫法
給定輸入源 db.lte_pgw，輸出下列結果。
概念 List, Dict, Tuple
{TIMESTAMP:ISO(2015-11-05 01:59:55),MSISDN:'886912345678'}
{TIMESTAMP:ISO(2015-11-05 01:59:56),MSISDN:'886912345678'}
{TIMESTAMP:ISO(2015-11-05 01:59:57),MSISDN:'886912345678'}

{MISIDN: 886912345678,
TIMESTAMP: [2015-11-05 01:59:55,
2015-11-05 01:59:56,
2015-11-05 01:59:57]}