Create a topic with 48 partitions
[root@y136 ~]# kafka-topics.sh --create --topic pcap48-1hour --partitions 48 --replication-factor 1 --zookeeper y113:2181
Created topic "pcap48-1hour".
Grant access
[root@y136 ~]# kafka-acls.sh \
> --authorizer kafka.security.auth.SimpleAclAuthorizer \
> -authorizer-properties zookeeper.connect=y113:2181 \
> --add \
> --allow-principal User:fastcapa \
> --topic pcap48-1hour \
> --group fastcapa
Adding ACLs for resource `Topic:pcap48-1hour`:
User:fastcapa has Allow permission for operations: All from hosts: *
Adding ACLs for resource `Group:fastcapa`:
User:fastcapa has Allow permission for operations: All from hosts: *
Current ACLs for resource `Topic:pcap48-1hour`:
User:fastcapa has Allow permission for operations: All from hosts: *
Current ACLs for resource `Group:fastcapa`:
User:fastcapa has Allow permission for operations: All from hosts: *
[root@y136 ~]# kafka-acls.sh \
> --authorizer kafka.security.auth.SimpleAclAuthorizer \
> -authorizer-properties zookeeper.connect=y113:2181 \
> --add \
> --allow-principal User:metron \
> --topic pcap48-1hour \
> --group metron
Adding ACLs for resource `Topic:pcap48-1hour`:
User:metron has Allow permission for operations: All from hosts: *
Adding ACLs for resource `Group:metron`:
User:metron has Allow permission for operations: All from hosts: *
Current ACLs for resource `Topic:pcap48-1hour`:
User:fastcapa has Allow permission for operations: All from hosts: *
User:metron has Allow permission for operations: All from hosts: *
Current ACLs for resource `Group:metron`:
User:metron has Allow permission for operations: All from hosts: *
[root@y136 ~]# kafka-acls.sh \
> --authorizer kafka.security.auth.SimpleAclAuthorizer \
> -authorizer-properties zookeeper.connect=y113:2181 \
> --add \
> --allow-principal User:pycapa\
> --topic pcap48-1hour \
> --group pycapa
Adding ACLs for resource `Topic:pcap48-1hour`:
User:pycapa has Allow permission for operations: All from hosts: *
Adding ACLs for resource `Group:pycapa`:
User:pycapa has Allow permission for operations: All from hosts: *
Current ACLs for resource `Topic:pcap48-1hour`:
User:fastcapa has Allow permission for operations: All from hosts: *
User:pycapa has Allow permission for operations: All from hosts: *
User:metron has Allow permission for operations: All from hosts: *
Current ACLs for resource `Group:pycapa`:
User:pycapa has Allow permission for operations: All from hosts: *
Start Fastcapa
[root@y138 fastcapa]# fastcapa -l 8-15 --huge-dir /mnt/huge_1GB -- -t pcap48-1hour -c /etc/fastcapa.ycluster -b 128 -x 262144 -s fastcapa-kafka.log
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL: probe driver: 1137:43 net_enic
EAL: PCI device 0000:0a:00.0 on NUMA socket 0
EAL: probe driver: 1137:43 net_enic
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:81:00.1 on NUMA socket 1
EAL: probe driver: 8086:10fb net_ixgbe
[ -t KAFKA_TOPIC ] defined as pcap48-1hour
[ -c KAFKA_CONFIG ] defined as /etc/fastcapa.ycluster
[ -b BURST_SIZE ] defined as 128
[ -x TX_RING_SIZE ] defined as 262144
[ -s KAFKA_STATS ] defined as fastcapa-kafka.log
[ -p PORT_MASK ] undefined; defaulting to 0x01
[ -r NB_RX_DESC ] undefined; defaulting to 1024
[ -q NB_RX_QUEUE ] undefined; defaulting to 2
USER1: Initializing port 0
USER1: Device setup successfully; port=0, mac=90 e2 ba d9 3c f9
USER1: Appending Kafka client stats to 'fastcapa-kafka.log'
USER1: Launching receive worker; worker=0, core=9, queue=0
USER1: Receive worker started; core=9, socket=1, queue=0 attempts=0
USER1: Launching receive worker; worker=1, core=10, queue=1
USER1: Launching transmit worker; worker=0, core=11 ring=0
USER1: Transmit worker started; core=11, socket=1
USER1: Receive worker started; core=10, socket=1, queue=1 attempts=0
USER1: Launching transmit worker; worker=1, core=12 ring=1
USER1: Transmit worker started; core=12, socket=1
USER1: Launching transmit worker; worker=2, core=13 ring=0
USER1: Transmit worker started; core=13, socket=1
USER1: Launching transmit worker; worker=3, core=14 ring=1
USER1: Transmit worker started; core=14, socket=1
USER1: Launching transmit worker; worker=4, core=15 ring=0
USER1: Transmit worker started; core=15, socket=1
USER1: Starting to monitor workers; core=8, socket=1
----- in ----- --- queued --- ----- out ----- ---- drops ----
[nic] 0 - - -
[rx] 0 - 0 0
[tx] 0 - 0 0
[kaf] 0 0 0 0
...
Check the Kafka offsets for the new topic. Expect them to be 0.
[root@y137 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list y135:6667 --topic pcap48-1hour --security-protocol PLAINTEXTSASL --time -1 | grep pcap | awk -F: '{p+=$3} END {print p}'
[2017-05-10 21:06:47,335] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
0
Start packet replay.
[root@y137 ~]# time tcpreplay -i enp129s0f1 --loop=0 --stats=15 --preload-pcap --mbps 1100 example.pcap
File Cache is enabled
Actual: 3166583 packets (2062501418 bytes) sent in 15.00 seconds.
Rated: 137499900.0 Bps, 1099.99 Mbps, 211105.37 pps
Wait for 1 hour.
Stop Tcpreplay.
Actual: 759979287 packets (495000215740 bytes) sent in 3600.00 seconds.
Rated: 137499900.0 Bps, 1099.99 Mbps, 211105.25 pps
^C User interrupt...
sendpacket_abort
Actual: 762763664 packets (496813838161 bytes) sent in 3613.01 seconds.
Rated: 137499900.0 Bps, 1099.99 Mbps, 211105.23 pps
Flows: 68 flows, 0.01 fps, 762763998 flow packets, 0 non-flow
Statistics for network device: enp129s0f1
Successful packets: 762763664
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
real 60m14.717s
user 40m33.978s
sys 19m37.613s
Stop Fastcapa
----- in ----- --- queued --- ----- out ----- ---- drops ----
[nic] 762763664 - - -
[rx] 762763664 - 762763664 0
[tx] 762763664 - 762763664 0
[kaf] 762763664 0 762763664 0
^CUSER1: Exiting on signal '2'
USER1: Finished monitoring workers; core=8, socket=1
USER1: Receive worker finished; core=10, socket=1, queue=1
USER1: Transmit worker finished; core=13, socket=1
USER1: Transmit worker finished; core=11, socket=1
USER1: Receive worker finished; core=9, socket=1, queue=0
USER1: Transmit worker finished; core=12, socket=1
USER1: Transmit worker finished; core=15, socket=1
USER1: Transmit worker finished; core=14, socket=1
USER1: Closing all Kafka connections
USER1: '1' message(s) queued on fastcapa-y138-enp129s0f1#producer-1
USER1: '0' message(s) queued on fastcapa-y138-enp129s0f1#producer-2
USER1: '1' message(s) queued on fastcapa-y138-enp129s0f1#producer-3
USER1: '0' message(s) queued on fastcapa-y138-enp129s0f1#producer-4
USER1: '1' message(s) queued on fastcapa-y138-enp129s0f1#producer-5
USER1: Waiting for '1' message(s) on fastcapa-y138-enp129s0f1#producer-1
USER1: All messages cleared on fastcapa-y138-enp129s0f1#producer-1
USER1: All messages cleared on fastcapa-y138-enp129s0f1#producer-2
USER1: Waiting for '1' message(s) on fastcapa-y138-enp129s0f1#producer-3
USER1: All messages cleared on fastcapa-y138-enp129s0f1#producer-3
USER1: All messages cleared on fastcapa-y138-enp129s0f1#producer-4
USER1: Waiting for '1' message(s) on fastcapa-y138-enp129s0f1#producer-5
USER1: All messages cleared on fastcapa-y138-enp129s0f1#producer-5
Grab the latest offset from Kafka.
[2017-05-10 22:10:58,153] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
762763664
[root@y137 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list y135:6667 --topic pcap48-1hour --security-protocol PLAINTEXTSASL --time -1 | grep pcap | awk -F: '{p+=$3} END {print p}'
Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file:
13936
Try using the -Djava.io.tmpdir= option to select an alternate temp location.
[2017-05-10 22:11:05,064] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
762763664
| Pkt Count | |
|---|---|
| Tcpreplay Says.. | 762,763,664 |
| Fastcapa Says... | 762,763,664 |
| Kafka Says... | 762,763,664 |
✅