tail -F /var/log/mesos/*
==> /var/log/mesos/mesos-master.INFO <==
I1029 21:01:33.413650 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (185)@172.31.8.161:5050
I1029 21:01:33.604188 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (25)@172.31.32.172:5050
I1029 21:01:33.604387 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:33.613934 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:34.297235 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (187)@172.31.8.161:5050
I1029 21:01:34.404985 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (27)@172.31.32.172:5050
I1029 21:01:34.405491 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:34.414019 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:35.067514 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (189)@172.31.8.161:5050
I1029 21:01:35.198773 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (29)@172.31.32.172:5050
I1029 21:01:35.198968 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:35.202798 23711 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:35.691251 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (191)@172.31.8.161:5050
I1029 21:01:35.862488 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (31)@172.31.32.172:5050
I1029 21:01:35.862676 23711 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:35.865612 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:36.251734 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (193)@172.31.8.161:5050
I1029 21:01:36.541522 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (33)@172.31.32.172:5050
I1029 21:01:36.545967 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:36.551113 23711 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:37.166829 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (195)@172.31.8.161:5050
I1029 21:01:37.526115 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (35)@172.31.32.172:5050
I1029 21:01:37.526305 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:37.535156 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:37.903717 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (197)@172.31.8.161:5050
I1029 21:01:38.210244 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (37)@172.31.32.172:5050
I1029 21:01:38.210446 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:38.217936 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:38.656182 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (199)@172.31.8.161:5050
I1029 21:01:38.811022 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (39)@172.31.32.172:5050
I1029 21:01:38.811249 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:38.817950 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:39.498637 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (41)@172.31.32.172:5050
I1029 21:01:39.498885 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:39.505938 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:39.625705 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (201)@172.31.8.161:5050
I1029 21:01:40.302103 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (203)@172.31.8.161:5050
I1029 21:01:40.504750 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (43)@172.31.32.172:5050
I1029 21:01:40.504946 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:40.513991 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:41.173871 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (205)@172.31.8.161:5050
I1029 21:01:41.203955 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (45)@172.31.32.172:5050
I1029 21:01:41.204502 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:41.209949 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:42.108628 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (207)@172.31.8.161:5050
I1029 21:01:42.142719 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (47)@172.31.32.172:5050
I1029 21:01:42.143245 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:42.148748 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:42.692729 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (49)@172.31.32.172:5050
I1029 21:01:42.693552 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:42.698707 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:42.789019 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (209)@172.31.8.161:5050
I1029 21:01:43.328192 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (211)@172.31.8.161:5050
I1029 21:01:43.633188 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (51)@172.31.32.172:5050
I1029 21:01:43.637933 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:43.645963 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:44.075378 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (213)@172.31.8.161:5050
I1029 21:01:44.212348 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (53)@172.31.32.172:5050
I1029 21:01:44.212860 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:44.219771 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:44.801172 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (215)@172.31.8.161:5050
I1029 21:01:44.980567 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (55)@172.31.32.172:5050
I1029 21:01:44.982524 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:44.989936 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:45.473219 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (217)@172.31.8.161:5050
I1029 21:01:45.805968 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (57)@172.31.32.172:5050
I1029 21:01:45.806330 23711 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:45.810176 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:46.141086 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (219)@172.31.8.161:5050
I1029 21:01:46.469466 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (59)@172.31.32.172:5050
I1029 21:01:46.470470 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:46.477938 23710 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:47.138464 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (221)@172.31.8.161:5050
I1029 21:01:47.153671 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (61)@172.31.32.172:5050
I1029 21:01:47.154216 23711 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:47.161934 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:47.723335 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (63)@172.31.32.172:5050
I1029 21:01:47.723915 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:47.733957 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:48.019807 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (223)@172.31.8.161:5050
I1029 21:01:48.656723 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (225)@172.31.8.161:5050
I1029 21:01:48.664783 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (65)@172.31.32.172:5050
I1029 21:01:48.665173 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:48.671875 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:49.527300 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (227)@172.31.8.161:5050
I1029 21:01:49.576516 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (67)@172.31.32.172:5050
I1029 21:01:49.582736 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:49.585146 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:50.139086 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (69)@172.31.32.172:5050
I1029 21:01:50.139619 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:50.145957 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:50.407510 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (229)@172.31.8.161:5050
I1029 21:01:51.097223 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (71)@172.31.32.172:5050
I1029 21:01:51.102023 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:51.103528 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:51.165827 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (231)@172.31.8.161:5050
I1029 21:01:51.663630 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (73)@172.31.32.172:5050
I1029 21:01:51.663935 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:51.667739 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:52.166867 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (233)@172.31.8.161:5050
I1029 21:01:52.561971 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (75)@172.31.32.172:5050
I1029 21:01:52.562168 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:52.569963 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:53.088008 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (235)@172.31.8.161:5050
I1029 21:01:53.566822 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (77)@172.31.32.172:5050
I1029 21:01:53.567049 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:53.573951 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:53.608208 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (237)@172.31.8.161:5050
I1029 21:01:53.631526 23705 http.cpp:338] HTTP GET for /master/state.json from 172.31.32.172:53746 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:41.0) Gecko/20100101 Firefox/41.0'
I1029 21:01:54.129922 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (80)@172.31.32.172:5050
I1029 21:01:54.130142 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:54.137969 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:54.292804 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (239)@172.31.8.161:5050
I1029 21:01:54.662554 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (82)@172.31.32.172:5050
I1029 21:01:54.663100 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:54.669941 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:54.879179 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (241)@172.31.8.161:5050
I1029 21:01:55.462734 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (84)@172.31.32.172:5050
I1029 21:01:55.462930 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:55.469950 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:55.523864 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (243)@172.31.8.161:5050
I1029 21:01:56.131597 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (86)@172.31.32.172:5050
I1029 21:01:56.131801 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:56.137974 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:56.248246 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (245)@172.31.8.161:5050
I1029 21:01:56.738811 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (88)@172.31.32.172:5050
I1029 21:01:56.739403 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:56.743214 23711 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:57.224309 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (247)@172.31.8.161:5050
I1029 21:01:57.324568 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (90)@172.31.32.172:5050
I1029 21:01:57.326550 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:57.332455 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:57.741603 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (249)@172.31.8.161:5050
I1029 21:01:58.275244 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (92)@172.31.32.172:5050
I1029 21:01:58.275586 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:58.281939 23710 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:58.738765 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (251)@172.31.8.161:5050
I1029 21:01:59.036819 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (94)@172.31.32.172:5050
I1029 21:01:59.037017 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:59.045948 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:59.335553 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (253)@172.31.8.161:5050
I1029 21:01:59.802465 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (96)@172.31.32.172:5050
I1029 21:01:59.802666 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:59.806767 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:01:59.900445 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (255)@172.31.8.161:5050
I1029 21:02:00.720649 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (98)@172.31.32.172:5050
I1029 21:02:00.720856 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:00.730060 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:00.805922 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (257)@172.31.8.161:5050
I1029 21:02:01.639317 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (260)@172.31.8.161:5050
I1029 21:02:01.654454 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (100)@172.31.32.172:5050
I1029 21:02:01.654646 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:01.661950 23710 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:02.450860 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (262)@172.31.8.161:5050
I1029 21:02:02.509847 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (102)@172.31.32.172:5050
I1029 21:02:02.510409 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:02.517946 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:03.110954 23709 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (104)@172.31.32.172:5050
I1029 21:02:03.111207 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:03.114717 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:03.313666 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (264)@172.31.8.161:5050
I1029 21:02:04.035717 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (106)@172.31.32.172:5050
I1029 21:02:04.035928 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:04.041950 23709 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:04.303705 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (266)@172.31.8.161:5050
I1029 21:02:04.577500 23705 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (108)@172.31.32.172:5050
I1029 21:02:04.578018 23705 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:04.582152 23704 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:04.988745 23704 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (268)@172.31.8.161:5050
I1029 21:02:05.108814 23707 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (110)@172.31.32.172:5050
I1029 21:02:05.109359 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:05.116274 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:05.785549 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (270)@172.31.8.161:5050
I1029 21:02:06.076949 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (112)@172.31.32.172:5050
I1029 21:02:06.077147 23708 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:06.081825 23707 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:06.452611 23711 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (272)@172.31.8.161:5050
I1029 21:02:07.050070 23706 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (114)@172.31.32.172:5050
I1029 21:02:07.050266 23706 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:07.057988 23710 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1029 21:02:07.159643 23708 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (274)@172.31.8.161:5050
I1029 21:02:07.644961 23710 replica.cpp:642] Replica in EMPTY status received a broadcasted recover request from (116)@172.31.32.172:5050
I1029 21:02:07.645397 23710 recover.cpp:195] Received a recover response from a replica in EMPTY status
How to fix:
-
Ensure mesos master count == (quorum_size*2)-1.
-
Ensure all mesos masters are running; for example: the cluster won't start if only 2/3 masters are running.