Networkdb events log for moby issue

swarm.networkdb.events.log

## Common events related to online endpoint from bulk sync
2021-01-19T06:38:10.886180117Z {Type:CREATE,LTime:93557892,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:38:40.905904777Z {Type:CREATE,LTime:93557892,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:39:10.885634958Z {Type:CREATE,LTime:93557892,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:39:41.193478062Z {Type:CREATE,LTime:93557892,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}

## Deploy process started, node 1b906f7d2be0 set ServiceDisabled=true and update LTime of the endpoint record
## We got CREATE event from bulk sync earlier than UPDATE event from gossip
2021-01-19T06:39:50.899209788Z {Type:CREATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
## CREATE event ignored ServiceDisabled=true flag and didn't update IPVS table

## Next events are ignored because LTime is the same as before
## during these events endpoint has weight 1 in IPVS table
2021-01-19T06:39:51.372175542Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:39:51.520065112Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:39:52.240751438Z {Type:CREATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:39:53.174523912Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:01.415950175Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:01.920180779Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:02.098081468Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:05.270438843Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:05.953138848Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:06.920765495Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:07.142009496Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:08.304224480Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:10.999052920Z {Type:CREATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:11.372201674Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:13.022040640Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:14.097906052Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:16.670842605Z {Type:UPDATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}
2021-01-19T06:40:42.339699998Z {Type:CREATE,LTime:93561596,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:0}

## We started to have timeouts and connection problems, because container is already DOWN, but has weight 1 on some network peers
## LTime changed, container deleted, record disappeared from the ipvs
2021-01-19T06:40:51.504647244Z {Type:DELETE,LTime:93561883,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:1800}
2021-01-19T06:40:51.520386920Z {Type:DELETE,LTime:93561883,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:1800}
2021-01-19T06:40:51.710730260Z {Type:DELETE,LTime:93561883,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:1800}
2021-01-19T06:40:51.791670427Z {Type:DELETE,LTime:93561883,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:1800}
2021-01-19T06:40:51.992225524Z {Type:DELETE,LTime:93561883,NodeName:1b906f7d2be0,NetworkID:f312pn3rfujkpupqzeyumtvpm,TableName:endpoint_table,Key:96309316d9ca369cfae8bd02f9ecfa4cc468e4fd60cd0e33863ec33ef4a0bdd6,ResidualReapTime:1800}
...