hectorj2f/density-test-error

## density-test-error
E1017 14:18:26.198091   57527 clusterloader.go:177] Test Finished
E1017 14:18:26.198096   57527 clusterloader.go:178]   Test: /Users/hector/myRepo/mesos_repo/gopath/src/github.com/mesosphere/konvoy-mwt/mwt-perf-tests/density/config.yaml
E1017 14:18:26.198103   57527 clusterloader.go:179]   Status: Fail
E1017 14:18:26.198108   57527 clusterloader.go:181]   Errors: [measurement call APIResponsiveness - APIResponsiveness error: top latency metric: there should be no high-latency requests, but: [got: {Resource:pods Subresource: Verb:LIST Scope:cluster Latency:{Perc50:186.177ms Perc90:519.097ms Perc99:23.293676s} Count:705}; expected perc99 <= 1s got: {Resource:ipamblocks Subresource: Verb:LIST Scope:cluster Latency:{Perc50:64.781ms Perc90:6.074028s Perc99:21.777461s} Count:269}; expected perc99 <= 1s got: {Resource:blockaffinities Subresource: Verb:LIST Scope:cluster Latency:{Perc50:943.254ms Perc90:5.511028s Perc99:20.536341s} Count:2817}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:DELETE Scope:resource Latency:{Perc50:32.383ms Perc90:833.785ms Perc99:11.503623s} Count:3280}; expected perc99 <= 1s got: {Resource:blockaffinities Subresource: Verb:GET Scope:resource Latency:{Perc50:773.845ms Perc90:3.114154s Perc99:6.798896s} Count:2866}; expected perc99 <= 1s got: {Resource:ipamblocks Subresource: Verb:GET Scope:resource Latency:{Perc50:74.492ms Perc90:1.95523s Perc99:5.997523s} Count:5972}; expected perc99 <= 1s got: {Resource:ipamblocks Subresource: Verb:UPDATE Scope:resource Latency:{Perc50:76.913ms Perc90:1.595613s Perc99:5.948409s} Count:5896}; expected perc99 <= 1s got: {Resource:kubefedconfigs Subresource: Verb:GET Scope:namespace Latency:{Perc50:1.783ms Perc90:3.18ms Perc99:5.947272s} Count:30}; expected perc99 <= 1s got: {Resource:ipamconfigs Subresource: Verb:GET Scope:resource Latency:{Perc50:539.648ms Perc90:2.346781s Perc99:5.620288s} Count:2678}; expected perc99 <= 1s got: {Resource:pods Subresource:status Verb:PATCH Scope:namespace Latency:{Perc50:56.663ms Perc90:1.6424s Perc99:5.27123s} Count:22116}; expected perc99 <= 1s got: {Resource:ippools Subresource: Verb:LIST Scope:cluster Latency:{Perc50:233.904ms Perc90:2.010937s Perc99:5.064156s} Count:11407}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:CREATE Scope:cluster Latency:{Perc50:466.044ms Perc90:2.17751s Perc99:4.953352s} Count:2930}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:UPDATE Scope:resource Latency:{Perc50:16.106ms Perc90:264.759ms Perc99:4.780576s} Count:3280}; expected perc99 <= 1s got: {Resource:pods Subresource: Verb:GET Scope:namespace Latency:{Perc50:23.394ms Perc90:1.285418s Perc99:4.303172s} Count:43811}; expected perc99 <= 1s got: {Resource:namespaces Subresource: Verb:GET Scope:cluster Latency:{Perc50:14.711ms Perc90:678.128ms Perc99:4.135945s} Count:13353}; expected perc99 <= 1s got: {Resource:clusterinformations Subresource: Verb:GET Scope:resource Latency:{Perc50:19.679ms Perc90:397.541ms Perc99:3.56387s} Count:14647}; expected perc99 <= 1s got: {Resource:authrequests Subresource: Verb:LIST Scope:namespace Latency:{Perc50:2.047ms Perc90:18.549ms Perc99:3.131796s} Count:92}; expected perc99 <= 1s got: {Resource:minioinstances Subresource: Verb:UPDATE Scope:namespace Latency:{Perc50:3.13ms Perc90:13.096ms Perc99:2.979028s} Count:930}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:GET Scope:resource Latency:{Perc50:14.305ms Perc90:187.707ms Perc99:2.962293s} Count:32688}; expected perc99 <= 1s got: {Resource:authrequests Subresource: Verb:DELETE Scope:namespace Latency:{Perc50:6.162ms Perc90:25.684ms Perc99:2.878391s} Count:1830}; expected perc99 <= 1s got: {Resource:nodes Subresource:status Verb:PATCH Scope:cluster Latency:{Perc50:5.236ms Perc90:25.087ms Perc99:2.129877s} Count:32465}; expected perc99 <= 1s got: {Resource:persistentvolumes Subresource: Verb:GET Scope:cluster Latency:{Perc50:1.642ms Perc90:8.889ms Perc99:1.910016s} Count:2593}; expected perc99 <= 1s got: {Resource:kommanderclusters Subresource: Verb:LIST Scope:namespace Latency:{Perc50:1.891ms Perc90:27.117ms Perc99:1.784962s} Count:64}; expected perc99 <= 1s got: {Resource:persistentvolumeclaims Subresource: Verb:GET Scope:namespace Latency:{Perc50:1.878ms Perc90:13.145ms Perc99:1.772436s} Count:2593}; expected perc99 <= 1s got: {Resource:apiservices Subresource:status Verb:PUT Scope:cluster Latency:{Perc50:9.508ms Perc90:103.606ms Perc99:1.605307s} Count:144}; expected perc99 <= 1s got: {Resource:endpoints Subresource: Verb:PUT Scope:namespace Latency:{Perc50:1.462913s Perc90:1.501068s Perc99:1.501068s} Count:6}; expected perc99 <= 1s got: {Resource:authcodes Subresource: Verb:LIST Scope:namespace Latency:{Perc50:1.756ms Perc90:14.04ms Perc99:1.415822s} Count:92}; expected perc99 <= 1s got: {Resource:services Subresource: Verb:LIST Scope:cluster Latency:{Perc50:2.837ms Perc90:5.097ms Perc99:1.321886s} Count:459}; expected perc99 <= 1s got: {Resource:authrequests Subresource: Verb:CREATE Scope:namespace Latency:{Perc50:5.605ms Perc90:23.03ms Perc99:1.241129s} Count:1830}; expected perc99 <= 1s got: {Resource:leases Subresource: Verb:GET Scope:namespace Latency:{Perc50:1.913ms Perc90:12.238ms Perc99:1.141908s} Count:188837}; expected perc99 <= 1s got: {Resource:nodes Subresource: Verb:LIST Scope:cluster Latency:{Perc50:38.061ms Perc90:71.693ms Perc99:1.007216s} Count:856}; expected perc99 <= 1s]
measurement call TestMetrics - TestMetrics error: [resource constraints: 12 constraints violated: [container coredns-5c98db65d4-h8jlz/coredns is using 0.075232815/0.05 CPU container coredns-5c98db65d4-jlxk8/coredns is using 0.08440147/0.05 CPU container kube-apiserver-ip-10-0-192-9.us-west-2.compute.internal/kube-apiserver is using 3.580078747/2.2 CPU container kube-apiserver-ip-10-0-192-9.us-west-2.compute.internal/kube-apiserver is using 3909.34765625/1200 MB of memory container kube-apiserver-ip-10-0-195-15.us-west-2.compute.internal/kube-apiserver is using 3.235164221/2.2 CPU container kube-apiserver-ip-10-0-195-15.us-west-2.compute.internal/kube-apiserver is using 2905.59765625/1200 MB of memory container kube-apiserver-ip-10-0-195-173.us-west-2.compute.internal/kube-apiserver is using 3.353332096/2.2 CPU container kube-apiserver-ip-10-0-195-173.us-west-2.compute.internal/kube-apiserver is using 2883.87890625/1200 MB of memory container kube-controller-manager-ip-10-0-195-15.us-west-2.compute.internal/kube-controller-manager is using 540.94921875/300 MB of memory container kube-scheduler-ip-10-0-192-9.us-west-2.compute.internal/kube-scheduler is using 145.6953125/120 MB of memory container kube-scheduler-ip-10-0-195-15.us-west-2.compute.internal/kube-scheduler is using 145.44921875/120 MB of memory container kube-scheduler-ip-10-0-195-173.us-west-2.compute.internal/kube-scheduler is using 205.40234375/120 MB of memory]]]
	E1017 14:18:26.198091 57527 clusterloader.go:177] Test Finished
	E1017 14:18:26.198096 57527 clusterloader.go:178] Test: /Users/hector/myRepo/mesos_repo/gopath/src/github.com/mesosphere/konvoy-mwt/mwt-perf-tests/density/config.yaml
	E1017 14:18:26.198103 57527 clusterloader.go:179] Status: Fail
	E1017 14:18:26.198108 57527 clusterloader.go:181] Errors: [measurement call APIResponsiveness - APIResponsiveness error: top latency metric: there should be no high-latency requests, but: [got: {Resource:pods Subresource: Verb:LIST Scope:cluster Latency:{Perc50:186.177ms Perc90:519.097ms Perc99:23.293676s} Count:705}; expected perc99 <= 1s got: {Resource:ipamblocks Subresource: Verb:LIST Scope:cluster Latency:{Perc50:64.781ms Perc90:6.074028s Perc99:21.777461s} Count:269}; expected perc99 <= 1s got: {Resource:blockaffinities Subresource: Verb:LIST Scope:cluster Latency:{Perc50:943.254ms Perc90:5.511028s Perc99:20.536341s} Count:2817}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:DELETE Scope:resource Latency:{Perc50:32.383ms Perc90:833.785ms Perc99:11.503623s} Count:3280}; expected perc99 <= 1s got: {Resource:blockaffinities Subresource: Verb:GET Scope:resource Latency:{Perc50:773.845ms Perc90:3.114154s Perc99:6.798896s} Count:2866}; expected perc99 <= 1s got: {Resource:ipamblocks Subresource: Verb:GET Scope:resource Latency:{Perc50:74.492ms Perc90:1.95523s Perc99:5.997523s} Count:5972}; expected perc99 <= 1s got: {Resource:ipamblocks Subresource: Verb:UPDATE Scope:resource Latency:{Perc50:76.913ms Perc90:1.595613s Perc99:5.948409s} Count:5896}; expected perc99 <= 1s got: {Resource:kubefedconfigs Subresource: Verb:GET Scope:namespace Latency:{Perc50:1.783ms Perc90:3.18ms Perc99:5.947272s} Count:30}; expected perc99 <= 1s got: {Resource:ipamconfigs Subresource: Verb:GET Scope:resource Latency:{Perc50:539.648ms Perc90:2.346781s Perc99:5.620288s} Count:2678}; expected perc99 <= 1s got: {Resource:pods Subresource:status Verb:PATCH Scope:namespace Latency:{Perc50:56.663ms Perc90:1.6424s Perc99:5.27123s} Count:22116}; expected perc99 <= 1s got: {Resource:ippools Subresource: Verb:LIST Scope:cluster Latency:{Perc50:233.904ms Perc90:2.010937s Perc99:5.064156s} Count:11407}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:CREATE Scope:cluster Latency:{Perc50:466.044ms Perc90:2.17751s Perc99:4.953352s} Count:2930}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:UPDATE Scope:resource Latency:{Perc50:16.106ms Perc90:264.759ms Perc99:4.780576s} Count:3280}; expected perc99 <= 1s got: {Resource:pods Subresource: Verb:GET Scope:namespace Latency:{Perc50:23.394ms Perc90:1.285418s Perc99:4.303172s} Count:43811}; expected perc99 <= 1s got: {Resource:namespaces Subresource: Verb:GET Scope:cluster Latency:{Perc50:14.711ms Perc90:678.128ms Perc99:4.135945s} Count:13353}; expected perc99 <= 1s got: {Resource:clusterinformations Subresource: Verb:GET Scope:resource Latency:{Perc50:19.679ms Perc90:397.541ms Perc99:3.56387s} Count:14647}; expected perc99 <= 1s got: {Resource:authrequests Subresource: Verb:LIST Scope:namespace Latency:{Perc50:2.047ms Perc90:18.549ms Perc99:3.131796s} Count:92}; expected perc99 <= 1s got: {Resource:minioinstances Subresource: Verb:UPDATE Scope:namespace Latency:{Perc50:3.13ms Perc90:13.096ms Perc99:2.979028s} Count:930}; expected perc99 <= 1s got: {Resource:ipamhandles Subresource: Verb:GET Scope:resource Latency:{Perc50:14.305ms Perc90:187.707ms Perc99:2.962293s} Count:32688}; expected perc99 <= 1s got: {Resource:authrequests Subresource: Verb:DELETE Scope:namespace Latency:{Perc50:6.162ms Perc90:25.684ms Perc99:2.878391s} Count:1830}; expected perc99 <= 1s got: {Resource:nodes Subresource:status Verb:PATCH Scope:cluster Latency:{Perc50:5.236ms Perc90:25.087ms Perc99:2.129877s} Count:32465}; expected perc99 <= 1s got: {Resource:persistentvolumes Subresource: Verb:GET Scope:cluster Latency:{Perc50:1.642ms Perc90:8.889ms Perc99:1.910016s} Count:2593}; expected perc99 <= 1s got: {Resource:kommanderclusters Subresource: Verb:LIST Scope:namespace Latency:{Perc50:1.891ms Perc90:27.117ms Perc99:1.784962s} Count:64}; expected perc99 <= 1s got: {Resource:persistentvolumeclaims Subresource: Verb:GET Scope:namespace Latency:{Perc50:1.878ms Perc90:13.145ms Perc99:1.772436s} Count:2593}; expected perc99 <= 1s got: {Resource:apiservices Subresource:status Verb:PUT Scope:cluster Latency:{Perc50:9.508ms Perc90:103.606ms Perc99:1.605307s} Count:144}; expected perc99 <= 1s got: {Resource:endpoints Subresource: Verb:PUT Scope:namespace Latency:{Perc50:1.462913s Perc90:1.501068s Perc99:1.501068s} Count:6}; expected perc99 <= 1s got: {Resource:authcodes Subresource: Verb:LIST Scope:namespace Latency:{Perc50:1.756ms Perc90:14.04ms Perc99:1.415822s} Count:92}; expected perc99 <= 1s got: {Resource:services Subresource: Verb:LIST Scope:cluster Latency:{Perc50:2.837ms Perc90:5.097ms Perc99:1.321886s} Count:459}; expected perc99 <= 1s got: {Resource:authrequests Subresource: Verb:CREATE Scope:namespace Latency:{Perc50:5.605ms Perc90:23.03ms Perc99:1.241129s} Count:1830}; expected perc99 <= 1s got: {Resource:leases Subresource: Verb:GET Scope:namespace Latency:{Perc50:1.913ms Perc90:12.238ms Perc99:1.141908s} Count:188837}; expected perc99 <= 1s got: {Resource:nodes Subresource: Verb:LIST Scope:cluster Latency:{Perc50:38.061ms Perc90:71.693ms Perc99:1.007216s} Count:856}; expected perc99 <= 1s]
	measurement call TestMetrics - TestMetrics error: [resource constraints: 12 constraints violated: [container coredns-5c98db65d4-h8jlz/coredns is using 0.075232815/0.05 CPU container coredns-5c98db65d4-jlxk8/coredns is using 0.08440147/0.05 CPU container kube-apiserver-ip-10-0-192-9.us-west-2.compute.internal/kube-apiserver is using 3.580078747/2.2 CPU container kube-apiserver-ip-10-0-192-9.us-west-2.compute.internal/kube-apiserver is using 3909.34765625/1200 MB of memory container kube-apiserver-ip-10-0-195-15.us-west-2.compute.internal/kube-apiserver is using 3.235164221/2.2 CPU container kube-apiserver-ip-10-0-195-15.us-west-2.compute.internal/kube-apiserver is using 2905.59765625/1200 MB of memory container kube-apiserver-ip-10-0-195-173.us-west-2.compute.internal/kube-apiserver is using 3.353332096/2.2 CPU container kube-apiserver-ip-10-0-195-173.us-west-2.compute.internal/kube-apiserver is using 2883.87890625/1200 MB of memory container kube-controller-manager-ip-10-0-195-15.us-west-2.compute.internal/kube-controller-manager is using 540.94921875/300 MB of memory container kube-scheduler-ip-10-0-192-9.us-west-2.compute.internal/kube-scheduler is using 145.6953125/120 MB of memory container kube-scheduler-ip-10-0-195-15.us-west-2.compute.internal/kube-scheduler is using 145.44921875/120 MB of memory container kube-scheduler-ip-10-0-195-173.us-west-2.compute.internal/kube-scheduler is using 205.40234375/120 MB of memory]]]