Skip to content

Instantly share code, notes, and snippets.

@haosdent

haosdent/mesos_health_check_manual_test_cases.md Secret

Last active August 26, 2016 16:24
Show Gist options
  • Save haosdent/577e4a527e2a80e2aa1b1eb783870f8d to your computer and use it in GitHub Desktop.
Save haosdent/577e4a527e2a80e2aa1b1eb783870f8d to your computer and use it in GitHub Desktop.
Download ZIP
Mesos Health Check Manual Test Cases
Raw
mesos_health_check_manual_test_cases.md
  • Apply this patch r51431 to mesos-execute before test:

unified-containerizer + without image + command health check

healthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"COMMAND","command":{"value":"exit 0"}}'

I0825 22:05:46.194346 141040 scheduler.cpp:172] Version: 1.1.0
I0825 22:05:46.230329 141058 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0000
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"COMMAND","command":{"value":"exit 1"}}'

I0825 22:06:11.639521 141390 scheduler.cpp:172] Version: 1.1.0
I0825 22:06:11.675516 141410 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0001
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

unified-containerizer + without image + HTTP health check

healthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="python -m SimpleHTTPServer 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":8000}}'

I0825 22:09:21.542626 141600 scheduler.cpp:172] Version: 1.1.0
I0825 22:09:21.578807 141583 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0002
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="python -m SimpleHTTPServer 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":8001}}'

I0825 22:10:25.277760 142265 scheduler.cpp:172] Version: 1.1.0
I0825 22:10:25.313411 142258 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0003
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

unified-containerizer + without image + HTTPS health check

Because a bit complex to set up an image provide HTTPS service, I start a Nginx with HTTPS locally first.

healthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":443, "scheme": "https"}}'

I0825 22:38:09.221585 143310 scheduler.cpp:172] Version: 1.1.0
I0825 22:38:09.257640 143326 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0008
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":80, "scheme": "https"}}'

I0825 22:38:59.007895 143599 scheduler.cpp:172] Version: 1.1.0
I0825 22:38:59.042330 143595 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0009
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

stderr

W0825 22:38:59.694854 143654 health_checker.cpp:196] Health check failed 3 times consecutively: HTTP health check failed: curl returned exited with status 35: curl: (35) SSL received a record that exceeded the maximum permissible length.

unified-containerizer + without image + TCP health check

healthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="python -m SimpleHTTPServer 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"TCP","tcp":{"port":8000}}'

I0825 22:11:28.953541 142399 scheduler.cpp:172] Version: 1.1.0
I0825 22:11:28.990389 142393 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0004
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--master=127.0.0.1:5050 \
	--command="python -m SimpleHTTPServer 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"TCP","tcp":{"port":8001}}'

I0825 22:11:53.439486 142737 scheduler.cpp:172] Version: 1.1.0
I0825 22:11:53.475617 142754 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0005
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

unified-containerizer + with image + command health check

healthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"COMMAND","command":{"value":"python3 --version"}}'

I0825 22:44:07.756103 143860 scheduler.cpp:172] Version: 1.1.0
I0825 22:44:07.792418 143863 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0011
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

stderr

Python 3.5.2

unhealthy case

$ ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"COMMAND","command":{"value":"python4 --version"}}'

I0825 22:46:10.337110 145088 scheduler.cpp:172] Version: 1.1.0
I0825 22:46:10.372409 145102 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0012
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

unified-containerizer + with image + HTTP health check

healthy case

$ sudo ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="python3 -m http.server 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"HTTP","http":{"port":8000}}'

I0825 22:50:44.228790 145913 scheduler.cpp:172] Version: 1.1.0
I0825 22:50:44.265033 145915 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0016
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ sudo ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="python3 -m http.server 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"HTTP","http":{"port":8081}}'

I0825 22:51:52.293392 146396 scheduler.cpp:172] Version: 1.1.0
I0825 22:51:52.329563 146383 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0017
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

unified-containerizer + with image + HTTPS health check

Because a bit complex to set up an image provide HTTPS service, I start a Nginx with HTTPS locally first.

healthy case

$ sudo ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":443, "scheme": "https"}}'

I0825 22:53:37.153532 146597 scheduler.cpp:172] Version: 1.1.0
I0825 22:53:38.960068 146598 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0018
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ sudo ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="sleep 200" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":80, "scheme": "https"}}'

I0825 22:54:32.813844 146932 scheduler.cpp:172] Version: 1.1.0
I0825 22:54:32.850049 146931 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0019
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

stderr

W0825 22:54:34.984119 147053 health_checker.cpp:196] Health check failed 3 times consecutively: HTTP health check failed: curl returned exited with status 35: curl: (35) SSL received a record that exceeded the maximum permissible length.

unified-containerizer + with image + TCP health check

healthy case

$ sudo ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="python3 -m http.server 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8000}}'

I0825 23:06:53.210116  2608 scheduler.cpp:172] Version: 1.1.0
I0825 23:06:53.244458  2616 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0030
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'

Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ sudo ./src/mesos-execute \
	--name=test_health_check \
	--containerizer=mesos \
	--docker_image=python:alpine \
	--master=127.0.0.1:5050 \
	--command="python3 -m http.server 8000" \
	--health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8001}}'

I0825 23:08:41.282997  3957 scheduler.cpp:172] Version: 1.1.0
I0825 23:08:41.318316  3973 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 00a49c9a-e2a6-430c-b377-38f0957a789c-0032
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Command terminated with signal Terminated'
  source: SOURCE_EXECUTOR
  healthy?: 0

stderr

W0825 23:08:53.364130  4089 health_checker.cpp:196] Health check failed 3 times consecutively: TCP health check failed: bash returned exited with status 1: bash: connect: Connection refused
bash: /dev/tcp/127.0.0.1/8001: Connection refused

Prepare works for docker bridge network

For bridge network, I patched this to make mesos-execute launch container in bridged network mode.

diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
index 6189529..a48e94c 100644
--- a/src/cli/execute.cpp
+++ b/src/cli/execute.cpp
@@ -662,6 +662,9 @@ private:
       containerInfo.set_type(ContainerInfo::DOCKER);
       containerInfo.mutable_docker()->set_image(dockerImage.get());

+      containerInfo.mutable_docker()->set_network(
+          ContainerInfo::DockerInfo::BRIDGE);
+
       if (networks.isSome() && !networks->empty()) {
         vector<string> tokens = strings::tokenize(networks.get(), ",");
         if (tokens.size() > 1) {

docker-containerizer + HTTP health check + bridge network

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":8000}}'

# Make sure the container didn't listen in host. 
# the output of `netstat -antp|grep 8000` is empty.

I0826 22:20:27.770289 111029 scheduler.cpp:172] Version: 1.1.0
I0826 22:20:29.579452 111042 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0000
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":8001}}'

I0826 22:22:28.373203 112451 scheduler.cpp:172] Version: 1.1.0
I0826 22:22:28.408643 112429 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0001
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0

docker-containerizer + TCP health check + bridge network

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8000}}'

# Make sure the container didn't listen in host. 
# the output of `netstat -antp|grep 8000` is empty.

I0826 22:23:31.599761 112763 scheduler.cpp:172] Version: 1.1.0
I0826 22:23:31.634266 112766 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0002
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8001}}'

I0826 22:24:47.260185 113387 scheduler.cpp:172] Version: 1.1.0
I0826 22:24:47.296603 113394 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0003
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0

Prepare works for docker overlay network

  • Create a consul cluster and restart docker engine with --cluster-store, --cluster-advertise options.
  • Create a overlay network
$ docker network create --driver overlay --subnet 10.0.9.0/24 test-health-check-overlay-network
$ docker network ls
NETWORK ID          NAME                                DRIVER              SCOPE
652616cf30a8        bridge                              bridge              local
9ff9b3405981        host                                host                local
6cf658676fec        none                                null                local
eddab822e499        test-health-check-overlay-network   overlay             global

docker-containerizer + HTTP health check + overlay network

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --networks=test-health-check-overlay-network \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"HTTP","http":{"port":8000}}'

I0826 23:10:57.589120 127465 scheduler.cpp:172] Version: 1.1.0
I0826 23:10:57.735872 127482 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0009
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --networks=test-health-check-overlay-network \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"HTTP","http":{"port":8001}}'

I0826 23:11:54.786170 128296 scheduler.cpp:172] Version: 1.1.0
I0826 23:11:54.821864 128269 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0010
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0

docker-containerizer + TCP health check + overlay network

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --networks=test-health-check-overlay-network \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8000}}'

I0826 23:12:44.081990 128765 scheduler.cpp:172] Version: 1.1.0
I0826 23:12:44.118736 128764 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0011
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --networks=test-health-check-overlay-network \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8001}}'

I0826 23:13:22.172219 129211 scheduler.cpp:172] Version: 1.1.0
I0826 23:13:22.208231 129239 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID 2684ae3c-cce8-445c-b821-2e4e466f4130-0012
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0

docker-containerizer + command health check

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="sleep 200" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"COMMAND","command":{"value":"exit 0"}}'

I0826 22:06:39.329999 67313 scheduler.cpp:172] Version: 1.1.0
I0826 22:06:39.366371 67302 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0000
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'

Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

docker-containerizer + HTTP health check + host network

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":8000}}'

I0826 22:10:08.432780 106246 scheduler.cpp:172] Version: 1.1.0
I0826 22:10:08.470535 106259 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0001
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":8001}}'

I0826 22:11:03.058123 106857 scheduler.cpp:172] Version: 1.1.0
I0826 22:11:03.124912 106863 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0002
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0

docker-containerizer + HTTPS health check + host network

Because a bit complex to set up an image provide HTTPS service, I start a Nginx with HTTPS locally first.

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="sleep 200" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":443, "scheme": "https"}}'

I0826 22:12:59.134943 107125 scheduler.cpp:172] Version: 1.1.0
I0826 22:12:59.170428 107130 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0003
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="sleep 200" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":0.0,"type":"HTTP","http":{"port":80, "scheme": "https"}}'

I0826 22:13:37.906347 107548 scheduler.cpp:172] Version: 1.1.0
I0826 22:13:37.942514 107520 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0004
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0

docker-containerizer + TCP health check + host network

healthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8000}}'

I0826 22:15:00.613901 107749 scheduler.cpp:172] Version: 1.1.0
I0826 22:15:00.683385 107743 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0005
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 1

unhealthy case

$ ./src/mesos-execute \
    --name=test_health_check \
    --containerizer=docker \
    --docker_image=python:alpine \
    --master=127.0.0.1:5050 \
    --command="python3 -m http.server 8000" \
    --health_check='{"delay_seconds":0.0,"interval_seconds":0.0,"timeout_seconds":20.0,"consecutive_failures":3,"grace_period_seconds":10.0,"type":"TCP","tcp":{"port":8001}}'

I0826 22:15:51.035229 108083 scheduler.cpp:172] Version: 1.1.0
I0826 22:15:51.178683 108060 scheduler.cpp:461] New master detected at master@127.0.0.1:5050
Subscribed with ID af58d209-4b28-457a-aed4-e63376a1a480-0006
Submitted task 'test_health_check' to agent 'bd02a3c8-d447-4b8b-a9fc-9f32d455009f-S0'
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_RUNNING for task 'test_health_check'
  source: SOURCE_EXECUTOR
  healthy?: 0
Received status update TASK_KILLING for task 'test_health_check'
  source: SOURCE_EXECUTOR
Received status update TASK_KILLED for task 'test_health_check'
  message: 'Container exited with status 137'
  source: SOURCE_EXECUTOR
  healthy?: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment