Skip to content

Instantly share code, notes, and snippets.

Avatar
🦉
Dry eyes ...

Daein Park bysnupy

🦉
Dry eyes ...
View GitHub Profile
View ocp4__trace_network9.md
$ grep -E "06:34:1[45]" ./AWS_ACCOUNT_ID_elasticloadbalancing_ap-northeast-1_a17a43204638c41818a7d130503aaaaa_20210405T0635Z_52.199.57.199_ysy02mf0.log
2021-04-05T06:34:14.307137Z a17a43204638c41818a7d130503aaaaa 217.178.150.68:35088 10.0.134.187:31790 0.000499 0.000009 0.000013 - - 567 346 "- - - " "-" - -
2021-04-05T06:34:15.426174Z a17a43204638c41818a7d130503aaaaa 217.178.150.68:35111 10.0.134.187:31790 0.00048 0.000009 0.000013 - - 567 346 "- - - " "-" - -
View ocp4__trace_network9.md
$ grep 31566 ./AWS_ACCOUNT_ID_vpcflowlogs_ap-northeast-1_fl-0e34a6dfefc0db8a9_20210405T0635Z_6427407c.log
// Columns of the VPC Flow logs
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

// ELB ENI
2 AWS_ACCOUNT_ID eni-047359708552d4999 10.0.134.187 10.0.14.231 31790 31566 6 4 562 1617604407 1617604466 ACCEPT OK
2 AWS_ACCOUNT_ID eni-047359708552d4999 10.0.14.231 10.0.134.187 31566 31790 6 7 988 1617604407 1617604466 ACCEPT OK

// EC2 ENI
View ocp4__trace_network8.md
// "tcp.stream eq" can filter a certain tcp session as follows.
$ TZ=UTC tshark -n -t ud  -r tcpdump_20210405_063341.pcap \
  -Y "tcp.stream eq 794"

23152 2021-04-05 06:34:14.307291  10.0.14.231 → 10.0.134.187 TCP 76 31566 → 31790 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=2042992871 TSecr=0 WS=256
23157 2021-04-05 06:34:14.307408 10.0.134.187 → 10.0.14.231  TCP 76 31790 → 31566 [SYN, ACK] Seq=0 Ack=1 Win=26697 Len=0 MSS=8911 SACK_PERM=1 TSval=2647712296 TSecr=2042992871 WS=128
23158 2021-04-05 06:34:14.307733  10.0.14.231 → 10.0.134.187 TCP 68 31566 → 31790 [ACK] Seq=1 Ack=1 Win=27136 Len=0 TSval=2042992871 TSecr=2647712296
23161 2021-04-05 06:34:14.307828  10.0.14.231 → 10.0.134.187 PROXYv1 117 31566 → 31790 [PSH, ACK] Seq=1 Ack=1 Win=27136 Len=49 TSval=2042992871 TSecr=2647712296
// HTTP Request
View ocp4__trace_network7.md
// I've requested "/dddddddddd" URL with "request-id: abcdefghijk" in request header.
// Check it's matched as expected.
$ TZ=UTC tshark -n -t ud  -r tcpdump_20210405_063341.pcap  \
  -Y 'http.request and http contains "request-id: abcdefghijk"'
23783 2021-04-05 06:34:15.428013   10.128.2.7 → 10.131.0.23  HTTP 885 GET /dddddddddd HTTP/1.1 

// Then, we can filter the "Stream index" from the packet details for specify a specific tcp session.
$ TZ=UTC tshark -n -t ud  -r tcpdump_20210405_063341.pcap  \
  -Y 'http.request and http contains "request-id: abcdefghijk"' -V | \
View ocp4__trace_network6.md
$ oc get pod -A | grep openshift-debug-node
openshift-debug-node-2dnn5zgqp2                    ip-10-0-198-72ap-northeast-1computeinternal-debug                          1/1     Running        0          176m
openshift-debug-node-8rxtlcxcbq                    ip-10-0-134-187ap-northeast-1computeinternal-debug                         1/1     Running        0          3h15m

// We can select a specific pcap using the file name which show us the generated timestamp.
$ oc cp -n openshift-debug-node-2dnn5zgqp2 \
  ip-10-0-198-72ap-northeast-1computeinternal-debug:tcpdump_20210405_054132.pcap \
  ./tcpdump_20210405_054132.pcap
  
View ocp4__trace_network5.md
$ oc annotate -n openshift-ingress service router-default --overwrite         \
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled="true"    \
    service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval="5" \
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name="test-trace-logs"
    
service/router-default annotated
View ocp4__trace_network4.md
// Replace AWS_ACCOUNT_ID and ELB_ACCOUNT_ID with your real ones before running.
$ cat <<EOF > s3-bucket-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ELB_ACCOUNT_ID:root"
View ocp4__trace_network3.md
$ aws s3api create-bucket  \
  --bucket test-trace-logs \
  --create-bucket-configuration "LocationConstraint=ap-northeast-1"
{
    "Location": "http://test-trace-logs.s3.amazonaws.com/"
}

$ aws s3 ls | grep test-trace-logs
2021-04-05 12:16:27 test-trace-logs
View ocp4__trace_network2.md
$ oc debug node/ip-10-0-134-187.ap-northeast-1.compute.internal
Creating debug namespace/openshift-debug-node-8rxtl ...
Starting pod/ip-10-0-134-187ap-northeast-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.134.187
If you don't see a command prompt, try pressing enter.

// Capture the traffic against all interfaces for suppressing unexpected lost traffic.
// But many noises and duplicated traffic would be in the pcap files.
View ocp4__trace_network1.md
$ oc get pod -n openshift-ingress -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE                                           
router-default-56589566bc-nb52j   1/1     Running   0          4d22h   10.128.2.7    ip-10-0-134-187.ap-northeast-1.compute.internal
router-default-56589566bc-tghb9   1/1     Running   1          4d22h   10.129.2.10   ip-10-0-198-72.ap-northeast-1.compute.internal