manishrajkarnikar/gist:6a48979f4cfaa2d445e7

## gistfile1.txt
[14:54] == manishrajkarnika [32f97f1b@gateway/web/freenode/ip.50.249.127.27] has joined #mesos
[14:54] -ChanServ- [#mesos] This channel is publicly logged.
[14:54] <manishrajkarnika> hi
[14:54] == lambert [~lambert@4.53.133.222] has joined #mesos
[14:55] <manishrajkarnika> i am having issue with docker 1.7/1.8 container deployment with mesos 0.24.1 and marathon 0.11.0
[14:55] <manishrajkarnika> I have documented the issue here http://stackoverflow.com/questions/33050112/marathon-not-loading-docker-container-failed-to-get-resource-statistics-for-exe
[14:56] <manishrajkarnika> wondering if somebody can help me out
[15:00] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
[15:00] <dtrainor> can you docker inspect that image and see what it gives you?
[15:01] == jieyu [~jieyu@8.25.197.25] has quit [Remote host closed the connection]
[15:02] == Yasumoto [~yasumoto@unaffiliated/yasumoto] has joined #mesos
[15:05] == jieyu [~jieyu@8.25.197.25] has joined #mesos
[15:06] == lambert [~lambert@4.53.133.222] has quit [Quit: lambert]
[15:07] <dtrainor> BluePeppers, do you have any command-line tools for parsing individual values in a json tree?
[15:07] == jieyu [~jieyu@8.25.197.25] has quit [Remote host closed the connection]
[15:07] <dtrainor> otherwise I'd do something dumb like sed/awk/grep'ing through it
[15:09] == hellertime1 [~hellertim@a23-79-238-10.deploy.static.akamaitechnologies.com] has quit [Ping timeout: 256 seconds]
[15:13] <manishrajkarnika> yes i can do the docker inspect and provide a link here
[15:14] == sobersocialist [Instantbir@cet-nat.comcastcntr.pa.bo.comcast.net] has quit [Quit: Instantbird 1.5 -- http://www.instantbird.com]
[15:16] <manishrajkarnika> docker inspect response : https://gist.github.com/manishrajkarnikar/d9e0b7f233bd75e68e9c
[15:17] == dtrainor [~dtrainor@wsip-98-172-67-54.ph.ph.cox.net]
[15:17] ==  realname : Dan Trainor
[15:17] ==  channels : #mesos
[15:17] ==  server   : card.freenode.net [Washington, DC, USA]
[15:17] ==           : is using a secure connection
[15:17] ==  account  : dan__t
[15:17] == End of WHOIS
[15:17] == dtrainor [~dtrainor@wsip-98-172-67-54.ph.ph.cox.net]
[15:17] ==  realname : Dan Trainor
[15:17] ==  channels : #mesos
[15:17] ==  server   : card.freenode.net [Washington, DC, USA]
[15:17] ==           : is using a secure connection
[15:17] ==  account  : dan__t
[15:17] == End of WHOIS
[15:18] <manishrajkarnika> @Dan do you think if there is anything wrong with the way i am spinning up my docker container for master/slave ?
[15:26] == zhitao [~zhitao@4.53.133.222] has joined #mesos
[15:28] == zhitao [~zhitao@4.53.133.222] has quit [Remote host closed the connection]
[15:28] == zhitao [~zhitao@4.53.133.222] has joined #mesos
[15:28] <dtrainor> sorry doing a few things at once, give me a bit
[15:29] == Punkrex [~punkrex@50.253.9.243] has quit [Remote host closed the connection]
[15:33] == eghobo [~eghobo@161.170.193.135] has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
[15:34] <BluePeppers> dreewoo: jq
[15:34] <BluePeppers> dtrainor: jq
[15:35] <BluePeppers> sorry dreewoo :)
[15:35] <BluePeppers> that said, with docker inspect you can use the -f param to get a particular value
[15:36] == Punkrex [~punkrex@50.253.9.243] has joined #mesos
[15:38] <BluePeppers> manishrajkarnika: sounds like the status api call is failing. can you do things like `docker stats $(docker ps -q)`?
[15:40] == lambert [~lambert@4.53.133.222] has joined #mesos
[15:40] == adam-mesos [~adam-meso@c-107-3-190-75.hsd1.ca.comcast.net] has joined #mesos
[15:41] <dtrainor> oh that's right.
[15:41] <dtrainor> thank you.
[15:41] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has joined #mesos
[15:41] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
[15:42] <manishrajkarnika> for the failed container f6a4ca265d30        0.00%               2.609 MB/100.7 MB   2.59%               0 B/0 B
[15:42] <manishrajkarnika> thats the output bluePeppers
[15:43] <BluePeppers> not sure, the error message seems to be saying that it couldn't get the information that that command (docker stats) is returning
[15:44] <BluePeppers> could maybe be a race condition, but that seems unlikely
[15:44] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has joined #mesos
[15:44] <manishrajkarnika> which error message are you referring to
[15:44] == jieyu [~jieyu@8.25.197.25] has joined #mesos
[15:44] <BluePeppers> 'Failed to get resource statistics for executor ...'
[15:46] <manishrajkarnika> well from what I can see looks like its trying to do 'docker inspect mesos-20151014-203940-1684252864-5050-7-S5.45284402-b0dd-4e34-86b3-e555e6b9ecf0'
[15:46] <manishrajkarnika> but the failed container name is mesos-20151014-203940-1684252864-5050-7-S5.45284402-b0dd-4e34-86b3-e555e6b9ecf0.executor
[15:46] <manishrajkarnika> it has executor at the end
[15:47] <manishrajkarnika> so docker inspect is failing
[15:47] <BluePeppers> afaict, an executor container is started, but in the inspect output you can see it mounts the docker socket
[15:47] <BluePeppers> that means that it can then start the actual container
[15:47] == Punkrex [~punkrex@50.253.9.243] has quit [Remote host closed the connection]
[15:48] <BluePeppers> what does docker logs mesos-20151014-203940-1684252864-5050-7-S5.45284402-b0dd-4e34-86b3-e555e6b9ecf0.executor give?
[15:48] <BluePeppers> just the one line about failing to get resource statistics?
[15:49] <manishrajkarnika> https://gist.github.com/manishrajkarnikar/a363f54e281478bceb2a
[15:50] <manishrajkarnika> btw i microbot=helloworld
[15:52] <BluePeppers> hmm not sure. i think this issue might be best escalated to the mailing list
[15:52] <BluePeppers> sorry, my knowledge of mesos internals is lacking
[15:52] <manishrajkarnika> do you see anything wrong with the way i have docker image created and started?
[15:54] <BluePeppers> only thing i can think of is that 'privileged' should be under the 'docker' key
[15:54] <BluePeppers> but can't see why that'd cause the issue your having
[15:55] <manishrajkarnika> oh for the master?
[15:56] <BluePeppers> for the marathon task definition
[15:56] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
[15:58] == eghobo [~eghobo@c-73-15-80-159.hsd1.ca.comcast.net] has joined #mesos
[15:58] == faddat [~textual@118.70.67.125] has quit [Ping timeout: 268 seconds]
[15:58] <manishrajkarnika> so something like this:  docker run --name master -d \  --privileged      --net=host \     -e MESOS_CLUSTER=localmesos \     -e MESOS_HOSTNAME=192.168.99.100 \     -e MESOS_IP=192.168.99.100 \     -e MESOS_LOG_DIR=/var/log/mesos/master \     -e MESOS_PORT=5050 \     -e MESOS_QUORUM=1 \     -e MESOS_WORK_DIR=/var/lib/mesos/master \     -e MESOS_ZK=zk://192.168.99.100:2181/mesos \     -e MARATHON_HOSTNAME=192.168.99.100 \     -e MARATHON_HTTPS_ADDRESS=192.168.99.100 \
[16:00] <BluePeppers> oh god, docker in docker
[16:00] == lambert [~lambert@4.53.133.222] has quit [Quit: lambert]
[16:02] == lambert [~lambert@4.53.133.222] has joined #mesos
[16:04] == adamt [~adamt@unaffiliated/adamt] has quit [Ping timeout: 260 seconds]
[16:06] <manishrajkarnika> is that bad
[16:09] == jrb [~jrb@2601:648:8102:3c08:ea2a:eaff:fe60:d416] has joined #mesos
[16:10] <BluePeppers> manishrajkarnika: it can cause problems. in genreal, i try and avoid it
[16:11] == vinodkone [~vinodkone@8.25.197.24] has joined #mesos
[16:12] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has joined #mesos
[16:16] == adamt [~adamt@GW.dbc.dk] has joined #mesos
[16:16] == adamt [~adamt@GW.dbc.dk] has quit [Changing host]
[16:16] == adamt [~adamt@unaffiliated/adamt] has joined #mesos
[16:21] <manishrajkarnika> I had used https://spof.io/blog/2015/06/23/mesos-sandbox-using-docker-compose/ before
[16:21] <manishrajkarnika> and it had worked
[16:22] <manishrajkarnika> thought it would work with latest version also
[16:23] == rhoml[away] [~rhoml@14-202-215-171.tpgi.com.au] has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
[16:32] == t3hSteve [~textual@8.25.197.24] has joined #mesos
[16:36] == clehene [~clehene@192.150.10.209] has joined #mesos
[16:37] == zhitao [~zhitao@4.53.133.222] has quit [Remote host closed the connection]
[16:37] == bshi1 [~Adium@credible.static.monkeybrains.net] has joined #mesos
[16:38] == zhitao [~zhitao@4.53.133.222] has joined #mesos
[16:38] == luch0 [~luch0@OL119-36.fibertel.com.ar] has quit [Remote host closed the connection]
[16:38] == dnorris [~dnorris@unaffiliated/protochron] has quit [Ping timeout: 260 seconds]
[16:39] == bshi1 [~Adium@credible.static.monkeybrains.net] has quit [Client Quit]
[16:39] == bshi1 [~Adium@credible.static.monkeybrains.net] has joined #mesos
[16:40] == bshi [~Adium@credible.static.monkeybrains.net] has quit [Ping timeout: 250 seconds]
[16:40] <dtrainor> BluePeppers, btw, there's a "Suspend" button in the Marathon UI that then pops up a dialogue asking you to confirm a scale to 0 instances.
[16:42] <dtrainor> eh we knew that heh
	[14:54] == manishrajkarnika [32f97f1b@gateway/web/freenode/ip.50.249.127.27] has joined #mesos
	[14:54] -ChanServ- [#mesos] This channel is publicly logged.
	[14:54] <manishrajkarnika> hi
	[14:54] == lambert [~lambert@4.53.133.222] has joined #mesos
	[14:55] <manishrajkarnika> i am having issue with docker 1.7/1.8 container deployment with mesos 0.24.1 and marathon 0.11.0
	[14:55] <manishrajkarnika> I have documented the issue here http://stackoverflow.com/questions/33050112/marathon-not-loading-docker-container-failed-to-get-resource-statistics-for-exe
	[14:56] <manishrajkarnika> wondering if somebody can help me out
	[15:00] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
	[15:00] <dtrainor> can you docker inspect that image and see what it gives you?
	[15:01] == jieyu [~jieyu@8.25.197.25] has quit [Remote host closed the connection]
	[15:02] == Yasumoto [~yasumoto@unaffiliated/yasumoto] has joined #mesos
	[15:05] == jieyu [~jieyu@8.25.197.25] has joined #mesos
	[15:06] == lambert [~lambert@4.53.133.222] has quit [Quit: lambert]
	[15:07] <dtrainor> BluePeppers, do you have any command-line tools for parsing individual values in a json tree?
	[15:07] == jieyu [~jieyu@8.25.197.25] has quit [Remote host closed the connection]
	[15:07] <dtrainor> otherwise I'd do something dumb like sed/awk/grep'ing through it
	[15:09] == hellertime1 [~hellertim@a23-79-238-10.deploy.static.akamaitechnologies.com] has quit [Ping timeout: 256 seconds]
	[15:13] <manishrajkarnika> yes i can do the docker inspect and provide a link here
	[15:14] == sobersocialist [Instantbir@cet-nat.comcastcntr.pa.bo.comcast.net] has quit [Quit: Instantbird 1.5 -- http://www.instantbird.com]
	[15:16] <manishrajkarnika> docker inspect response : https://gist.github.com/manishrajkarnikar/d9e0b7f233bd75e68e9c
	[15:17] == dtrainor [~dtrainor@wsip-98-172-67-54.ph.ph.cox.net]
	[15:17] == realname : Dan Trainor
	[15:17] == channels : #mesos
	[15:17] == server : card.freenode.net [Washington, DC, USA]
	[15:17] == : is using a secure connection
	[15:17] == account : dan__t
	[15:17] == End of WHOIS
	[15:17] == dtrainor [~dtrainor@wsip-98-172-67-54.ph.ph.cox.net]
	[15:17] == realname : Dan Trainor
	[15:17] == channels : #mesos
	[15:17] == server : card.freenode.net [Washington, DC, USA]
	[15:17] == : is using a secure connection
	[15:17] == account : dan__t
	[15:17] == End of WHOIS
	[15:18] <manishrajkarnika> @Dan do you think if there is anything wrong with the way i am spinning up my docker container for master/slave ?
	[15:26] == zhitao [~zhitao@4.53.133.222] has joined #mesos
	[15:28] == zhitao [~zhitao@4.53.133.222] has quit [Remote host closed the connection]
	[15:28] == zhitao [~zhitao@4.53.133.222] has joined #mesos
	[15:28] <dtrainor> sorry doing a few things at once, give me a bit
	[15:29] == Punkrex [~punkrex@50.253.9.243] has quit [Remote host closed the connection]
	[15:33] == eghobo [~eghobo@161.170.193.135] has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
	[15:34] <BluePeppers> dreewoo: jq
	[15:34] <BluePeppers> dtrainor: jq
	[15:35] <BluePeppers> sorry dreewoo :)
	[15:35] <BluePeppers> that said, with docker inspect you can use the -f param to get a particular value
	[15:36] == Punkrex [~punkrex@50.253.9.243] has joined #mesos
	[15:38] <BluePeppers> manishrajkarnika: sounds like the status api call is failing. can you do things like `docker stats $(docker ps -q)`?
	[15:40] == lambert [~lambert@4.53.133.222] has joined #mesos
	[15:40] == adam-mesos [~adam-meso@c-107-3-190-75.hsd1.ca.comcast.net] has joined #mesos
	[15:41] <dtrainor> oh that's right.
	[15:41] <dtrainor> thank you.
	[15:41] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has joined #mesos
	[15:41] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
	[15:42] <manishrajkarnika> for the failed container f6a4ca265d30 0.00% 2.609 MB/100.7 MB 2.59% 0 B/0 B
	[15:42] <manishrajkarnika> thats the output bluePeppers
	[15:43] <BluePeppers> not sure, the error message seems to be saying that it couldn't get the information that that command (docker stats) is returning
	[15:44] <BluePeppers> could maybe be a race condition, but that seems unlikely
	[15:44] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has joined #mesos
	[15:44] <manishrajkarnika> which error message are you referring to
	[15:44] == jieyu [~jieyu@8.25.197.25] has joined #mesos
	[15:44] <BluePeppers> 'Failed to get resource statistics for executor ...'
	[15:46] <manishrajkarnika> well from what I can see looks like its trying to do 'docker inspect mesos-20151014-203940-1684252864-5050-7-S5.45284402-b0dd-4e34-86b3-e555e6b9ecf0'
	[15:46] <manishrajkarnika> but the failed container name is mesos-20151014-203940-1684252864-5050-7-S5.45284402-b0dd-4e34-86b3-e555e6b9ecf0.executor
	[15:46] <manishrajkarnika> it has executor at the end
	[15:47] <manishrajkarnika> so docker inspect is failing
	[15:47] <BluePeppers> afaict, an executor container is started, but in the inspect output you can see it mounts the docker socket
	[15:47] <BluePeppers> that means that it can then start the actual container
	[15:47] == Punkrex [~punkrex@50.253.9.243] has quit [Remote host closed the connection]
	[15:48] <BluePeppers> what does docker logs mesos-20151014-203940-1684252864-5050-7-S5.45284402-b0dd-4e34-86b3-e555e6b9ecf0.executor give?
	[15:48] <BluePeppers> just the one line about failing to get resource statistics?
	[15:49] <manishrajkarnika> https://gist.github.com/manishrajkarnikar/a363f54e281478bceb2a
	[15:50] <manishrajkarnika> btw i microbot=helloworld
	[15:52] <BluePeppers> hmm not sure. i think this issue might be best escalated to the mailing list
	[15:52] <BluePeppers> sorry, my knowledge of mesos internals is lacking
	[15:52] <manishrajkarnika> do you see anything wrong with the way i have docker image created and started?
	[15:54] <BluePeppers> only thing i can think of is that 'privileged' should be under the 'docker' key
	[15:54] <BluePeppers> but can't see why that'd cause the issue your having
	[15:55] <manishrajkarnika> oh for the master?
	[15:56] <BluePeppers> for the marathon task definition
	[15:56] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
	[15:58] == eghobo [~eghobo@c-73-15-80-159.hsd1.ca.comcast.net] has joined #mesos
	[15:58] == faddat [~textual@118.70.67.125] has quit [Ping timeout: 268 seconds]
	[15:58] <manishrajkarnika> so something like this: docker run --name master -d \ --privileged --net=host \ -e MESOS_CLUSTER=localmesos \ -e MESOS_HOSTNAME=192.168.99.100 \ -e MESOS_IP=192.168.99.100 \ -e MESOS_LOG_DIR=/var/log/mesos/master \ -e MESOS_PORT=5050 \ -e MESOS_QUORUM=1 \ -e MESOS_WORK_DIR=/var/lib/mesos/master \ -e MESOS_ZK=zk://192.168.99.100:2181/mesos \ -e MARATHON_HOSTNAME=192.168.99.100 \ -e MARATHON_HTTPS_ADDRESS=192.168.99.100 \
	[16:00] <BluePeppers> oh god, docker in docker
	[16:00] == lambert [~lambert@4.53.133.222] has quit [Quit: lambert]
	[16:02] == lambert [~lambert@4.53.133.222] has joined #mesos
	[16:04] == adamt [~adamt@unaffiliated/adamt] has quit [Ping timeout: 260 seconds]
	[16:06] <manishrajkarnika> is that bad
	[16:09] == jrb [~jrb@2601:648:8102:3c08:ea2a:eaff:fe60:d416] has joined #mesos
	[16:10] <BluePeppers> manishrajkarnika: it can cause problems. in genreal, i try and avoid it
	[16:11] == vinodkone [~vinodkone@8.25.197.24] has joined #mesos
	[16:12] == achanda [~abhishek@c-98-234-181-87.hsd1.ca.comcast.net] has joined #mesos
	[16:16] == adamt [~adamt@GW.dbc.dk] has joined #mesos
	[16:16] == adamt [~adamt@GW.dbc.dk] has quit [Changing host]
	[16:16] == adamt [~adamt@unaffiliated/adamt] has joined #mesos
	[16:21] <manishrajkarnika> I had used https://spof.io/blog/2015/06/23/mesos-sandbox-using-docker-compose/ before
	[16:21] <manishrajkarnika> and it had worked
	[16:22] <manishrajkarnika> thought it would work with latest version also
	[16:23] == rhoml[away] [~rhoml@14-202-215-171.tpgi.com.au] has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
	[16:32] == t3hSteve [~textual@8.25.197.24] has joined #mesos
	[16:36] == clehene [~clehene@192.150.10.209] has joined #mesos
	[16:37] == zhitao [~zhitao@4.53.133.222] has quit [Remote host closed the connection]
	[16:37] == bshi1 [~Adium@credible.static.monkeybrains.net] has joined #mesos
	[16:38] == zhitao [~zhitao@4.53.133.222] has joined #mesos
	[16:38] == luch0 [~luch0@OL119-36.fibertel.com.ar] has quit [Remote host closed the connection]
	[16:38] == dnorris [~dnorris@unaffiliated/protochron] has quit [Ping timeout: 260 seconds]
	[16:39] == bshi1 [~Adium@credible.static.monkeybrains.net] has quit [Client Quit]
	[16:39] == bshi1 [~Adium@credible.static.monkeybrains.net] has joined #mesos
	[16:40] == bshi [~Adium@credible.static.monkeybrains.net] has quit [Ping timeout: 250 seconds]
	[16:40] <dtrainor> BluePeppers, btw, there's a "Suspend" button in the Marathon UI that then pops up a dialogue asking you to confirm a scale to 0 instances.
	[16:42] <dtrainor> eh we knew that heh