nomad-autoscaler-debug-log.txt

2024-03-13T20:40:32.389Z [DEBUG] policy_eval.broker: dequeue eval: queue=cluster
2024-03-13T20:40:32.389Z [DEBUG] policy_eval.broker: waiting for eval: queue=cluster
2024-03-13T20:40:32.555Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=5a183123-8682-efb8-212c-3e7dbfe1ca41 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:40:32.555Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=5a183123-8682-efb8-212c-3e7dbfe1ca41 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:40:32.555Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=5a183123-8682-efb8-212c-3e7dbfe1ca41 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=f46a1bd9-38b5-caeb-4440-465f62f2b1fa
2024-03-13T20:40:32.555Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:40:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:40:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:40:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:39:32.556042645 +0000 UTC m=+7801.467185859 2024-03-13 20:40:32.556042645 +0000 UTC m=+7861.467185859}"
2024-03-13T20:40:32.557Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:40:32.557Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:40:32.557Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:40:32.557Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:40:32.557Z [DEBUG] policy_eval.broker: ack eval: eval_id=5a183123-8682-efb8-212c-3e7dbfe1ca41 token=f46a1bd9-38b5-caeb-4440-465f62f2b1fa eval_id=5a183123-8682-efb8-212c-3e7dbfe1ca41 token=f46a1bd9-38b5-caeb-4440-465f62f2b1fa
2024-03-13T20:40:32.557Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:40:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:40:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:40:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=08863560-96ec-6925-31b2-2ebe77e35ce8 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:40:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=08863560-96ec-6925-31b2-2ebe77e35ce8 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:40:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=08863560-96ec-6925-31b2-2ebe77e35ce8 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=b522807f-daf7-4c6b-36c4-f933de0e2119
2024-03-13T20:40:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:40:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:40:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:40:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:39:32.858238044 +0000 UTC m=+7801.769381251 2024-03-13 20:40:32.858238044 +0000 UTC m=+7861.769381251}"
2024-03-13T20:40:32.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:40:32.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:40:32.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:40:32.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:40:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=08863560-96ec-6925-31b2-2ebe77e35ce8 token=b522807f-daf7-4c6b-36c4-f933de0e2119 eval_id=08863560-96ec-6925-31b2-2ebe77e35ce8 token=b522807f-daf7-4c6b-36c4-f933de0e2119
2024-03-13T20:40:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:40:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:40:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:40:33.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=6e826293-eef6-79c5-b4bc-315f899a24a8 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:40:33.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=6e826293-eef6-79c5-b4bc-315f899a24a8 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:40:33.857Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=6e826293-eef6-79c5-b4bc-315f899a24a8 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=4f699f5c-d1aa-3171-1e3e-8b86cd1678a4
2024-03-13T20:40:33.857Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:40:33.857Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:40:33.857Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:40:33.857Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:39:33.857593758 +0000 UTC m=+7802.768736969 2024-03-13 20:40:33.857593758 +0000 UTC m=+7862.768736969}"
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=6e826293-eef6-79c5-b4bc-315f899a24a8 token=4f699f5c-d1aa-3171-1e3e-8b86cd1678a4 eval_id=6e826293-eef6-79c5-b4bc-315f899a24a8 token=4f699f5c-d1aa-3171-1e3e-8b86cd1678a4
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:40:33.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:41:31.355Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=11a64112-9287-cec7-8ab1-6f3a54cef2df policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=11a64112-9287-cec7-8ab1-6f3a54cef2df policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=11a64112-9287-cec7-8ab1-6f3a54cef2df policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=153c6c2c-572c-532d-1997-54c4230ba53d
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:31.356086965 +0000 UTC m=+7860.267230175 2024-03-13 20:41:31.356086965 +0000 UTC m=+7920.267230175}"
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=d9d75c90-7fbf-1fdf-edf1-cba157eb29db policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=d9d75c90-7fbf-1fdf-edf1-cba157eb29db policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=d9d75c90-7fbf-1fdf-edf1-cba157eb29db policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=a927c800-d392-ad67-98ae-bf6c64e889c3
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:31.356484241 +0000 UTC m=+7860.267627456 2024-03-13 20:41:31.356484241 +0000 UTC m=+7920.267627456}"
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=d9d75c90-7fbf-1fdf-edf1-cba157eb29db token=a927c800-d392-ad67-98ae-bf6c64e889c3 eval_id=d9d75c90-7fbf-1fdf-edf1-cba157eb29db token=a927c800-d392-ad67-98ae-bf6c64e889c3
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=11a64112-9287-cec7-8ab1-6f3a54cef2df token=153c6c2c-572c-532d-1997-54c4230ba53d eval_id=11a64112-9287-cec7-8ab1-6f3a54cef2df token=153c6c2c-572c-532d-1997-54c4230ba53d
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e0b0ed37-fd71-b902-3992-0085601d038a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:41:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=e0b0ed37-fd71-b902-3992-0085601d038a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:41:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=e0b0ed37-fd71-b902-3992-0085601d038a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=b409128e-7283-44a3-92fa-b8fa1bfa01bd
2024-03-13T20:41:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:41:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:31.541535656 +0000 UTC m=+7860.452678863 2024-03-13 20:41:31.541535656 +0000 UTC m=+7920.452678863}"
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=e0b0ed37-fd71-b902-3992-0085601d038a token=b409128e-7283-44a3-92fa-b8fa1bfa01bd eval_id=e0b0ed37-fd71-b902-3992-0085601d038a token=b409128e-7283-44a3-92fa-b8fa1bfa01bd
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=b23f685a-ebe9-4eb1-316b-1012a84bcf83 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:41:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=b23f685a-ebe9-4eb1-316b-1012a84bcf83 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:41:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=b23f685a-ebe9-4eb1-316b-1012a84bcf83 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=d7f7d911-c926-7761-3a44-1dd7d1653870
2024-03-13T20:41:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:41:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:31.557509089 +0000 UTC m=+7860.468652296 2024-03-13 20:41:31.557509089 +0000 UTC m=+7920.468652296}"
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=b23f685a-ebe9-4eb1-316b-1012a84bcf83 token=d7f7d911-c926-7761-3a44-1dd7d1653870 eval_id=b23f685a-ebe9-4eb1-316b-1012a84bcf83 token=d7f7d911-c926-7761-3a44-1dd7d1653870
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:32.075Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=c16d2739-b10e-9aa1-e915-15f055fae3aa policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:41:32.075Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=c16d2739-b10e-9aa1-e915-15f055fae3aa policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:41:32.075Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=c16d2739-b10e-9aa1-e915-15f055fae3aa policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 token=e00a71db-0592-62dd-1bc2-8db38fb98e4a
2024-03-13T20:41:32.075Z [DEBUG] policy_eval.worker: received policy for evaluation: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg
2024-03-13T20:41:32.162Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=8e15491e-8488-1578-26da-e66524e39ece policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:41:32.162Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=8e15491e-8488-1578-26da-e66524e39ece policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:41:32.255Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=04d70c7b-ccdc-dccb-7c37-d863c6dee741 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=04d70c7b-ccdc-dccb-7c37-d863c6dee741 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=8f39b2d8-d30d-47c3-1880-55a18bb71d3e policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=8f39b2d8-d30d-47c3-1880-55a18bb71d3e policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=04d70c7b-ccdc-dccb-7c37-d863c6dee741 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=da6b919b-1a45-f6a1-3514-c1b5833ce855
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:32.256100337 +0000 UTC m=+7861.167243548 2024-03-13 20:41:32.256100337 +0000 UTC m=+7921.167243548}"
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=8f39b2d8-d30d-47c3-1880-55a18bb71d3e policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=e9c3aa96-6176-4af6-e5ac-14cae77bf8d9
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:32.256335113 +0000 UTC m=+7861.167478327 2024-03-13 20:41:32.256335113 +0000 UTC m=+7921.167478327}"
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=8f39b2d8-d30d-47c3-1880-55a18bb71d3e token=e9c3aa96-6176-4af6-e5ac-14cae77bf8d9 eval_id=8f39b2d8-d30d-47c3-1880-55a18bb71d3e token=e9c3aa96-6176-4af6-e5ac-14cae77bf8d9
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=04d70c7b-ccdc-dccb-7c37-d863c6dee741 token=da6b919b-1a45-f6a1-3514-c1b5833ce855 eval_id=04d70c7b-ccdc-dccb-7c37-d863c6dee741 token=da6b919b-1a45-f6a1-3514-c1b5833ce855
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:32.281Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:41:32.281Z [DEBUG] policy_eval.worker.check_handler: querying source: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg query="sum(nomad_client_allocated_cpu{node_class=\"linux\"}*100/(nomad_client_unallocated_cpu{node_class=\"linux\"}+nomad_client_allocated_cpu{node_class=\"linux\"}))/count(nomad_client_allocated_cpu{node_class=\"linux\"})" source=prometheus
2024-03-13T20:41:32.281Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_cpu{node_class=\"linux\"}*100/(nomad_client_unallocated_cpu{node_class=\"linux\"}+nomad_client_allocated_cpu{node_class=\"linux\"}))/count(nomad_client_allocated_cpu{node_class=\"linux\"})" range="{2024-03-13 20:40:32.281716205 +0000 UTC m=+7861.192859415 2024-03-13 20:41:32.281716205 +0000 UTC m=+7921.192859415}"
2024-03-13T20:41:32.283Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg count=3
2024-03-13T20:41:32.283Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:41:32.283Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:41:32.283Z [DEBUG] policy_eval.worker.check_handler: querying source: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg query="sum(nomad_client_allocated_memory{node_class=\"linux\"}*100/(nomad_client_unallocated_memory{node_class=\"linux\"}+nomad_client_allocated_memory{node_class=\"linux\"}))/count(nomad_client_allocated_memory{node_class=\"linux\"})" source=prometheus
2024-03-13T20:41:32.283Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_memory{node_class=\"linux\"}*100/(nomad_client_unallocated_memory{node_class=\"linux\"}+nomad_client_allocated_memory{node_class=\"linux\"}))/count(nomad_client_allocated_memory{node_class=\"linux\"})" range="{2024-03-13 20:40:32.283964812 +0000 UTC m=+7861.195108020 2024-03-13 20:41:32.283964812 +0000 UTC m=+7921.195108020}"
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg count=3
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg from=3 to=3
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.worker: check linux_cpu_allocated_percentage selected in group : id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg direction=none count=0
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.worker: no checks need to be executed: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.broker: ack eval: eval_id=c16d2739-b10e-9aa1-e915-15f055fae3aa token=e00a71db-0592-62dd-1bc2-8db38fb98e4a eval_id=c16d2739-b10e-9aa1-e915-15f055fae3aa token=e00a71db-0592-62dd-1bc2-8db38fb98e4a
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.broker: dequeue eval: queue=cluster
2024-03-13T20:41:32.285Z [DEBUG] policy_eval.broker: waiting for eval: queue=cluster
2024-03-13T20:41:32.555Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=6c1c73e4-ca12-21a7-4a35-e6ae2c3a4f83 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:41:32.555Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=6c1c73e4-ca12-21a7-4a35-e6ae2c3a4f83 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:41:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=6c1c73e4-ca12-21a7-4a35-e6ae2c3a4f83 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=979c565d-0834-5b49-69a4-e6f323ff6354
2024-03-13T20:41:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:41:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:32.55621029 +0000 UTC m=+7861.467353498 2024-03-13 20:41:32.55621029 +0000 UTC m=+7921.467353498}"
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=6c1c73e4-ca12-21a7-4a35-e6ae2c3a4f83 token=979c565d-0834-5b49-69a4-e6f323ff6354 eval_id=6c1c73e4-ca12-21a7-4a35-e6ae2c3a4f83 token=979c565d-0834-5b49-69a4-e6f323ff6354
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:32.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=cc96bbb8-1154-28e2-e84d-247a96904582 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:41:32.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=cc96bbb8-1154-28e2-e84d-247a96904582 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:41:32.857Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=cc96bbb8-1154-28e2-e84d-247a96904582 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=1547ea4c-9cee-0fa8-ba08-b10a38c716a9
2024-03-13T20:41:32.857Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:41:32.857Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:32.857Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:32.857Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:32.857520313 +0000 UTC m=+7861.768663522 2024-03-13 20:41:32.857520313 +0000 UTC m=+7921.768663522}"
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=cc96bbb8-1154-28e2-e84d-247a96904582 token=1547ea4c-9cee-0fa8-ba08-b10a38c716a9 eval_id=cc96bbb8-1154-28e2-e84d-247a96904582 token=1547ea4c-9cee-0fa8-ba08-b10a38c716a9
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:32.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:41:33.856Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=539926bb-6596-2e47-c02f-11d48bb07b64 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:41:33.856Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=539926bb-6596-2e47-c02f-11d48bb07b64 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:41:33.856Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=539926bb-6596-2e47-c02f-11d48bb07b64 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=3f777aff-87f7-bc84-c666-cdc81d45a721
2024-03-13T20:41:33.856Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:41:33.856Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:41:33.856Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:41:33.856Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:40:33.856752749 +0000 UTC m=+7862.767895957 2024-03-13 20:41:33.856752749 +0000 UTC m=+7922.767895957}"
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.broker: ack eval: eval_id=539926bb-6596-2e47-c02f-11d48bb07b64 token=3f777aff-87f7-bc84-c666-cdc81d45a721 eval_id=539926bb-6596-2e47-c02f-11d48bb07b64 token=3f777aff-87f7-bc84-c666-cdc81d45a721
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:41:33.858Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:42:31.355Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=8813b461-f6dd-e992-d4a3-59ed3b496495 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:42:31.355Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=8813b461-f6dd-e992-d4a3-59ed3b496495 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:42:31.355Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=8813b461-f6dd-e992-d4a3-59ed3b496495 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=76915646-ada3-e9fc-f96a-1fc780518a47
2024-03-13T20:42:31.355Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:42:31.355Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:31.355Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:31.355Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:31.355277508 +0000 UTC m=+7920.266420729 2024-03-13 20:42:31.355277508 +0000 UTC m=+7980.266420729}"
2024-03-13T20:42:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=0a10256e-e065-8d26-cfa6-ac3522fb2227 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:42:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=0a10256e-e065-8d26-cfa6-ac3522fb2227 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:42:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=0a10256e-e065-8d26-cfa6-ac3522fb2227 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=e2740a0f-011f-86d5-b810-15ce19e594e8
2024-03-13T20:42:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:31.357Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:31.357039398 +0000 UTC m=+7920.268182613 2024-03-13 20:42:31.357039398 +0000 UTC m=+7980.268182613}"
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.broker: ack eval: eval_id=8813b461-f6dd-e992-d4a3-59ed3b496495 token=76915646-ada3-e9fc-f96a-1fc780518a47 eval_id=8813b461-f6dd-e992-d4a3-59ed3b496495 token=76915646-ada3-e9fc-f96a-1fc780518a47
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:31.357Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=0a10256e-e065-8d26-cfa6-ac3522fb2227 token=e2740a0f-011f-86d5-b810-15ce19e594e8 eval_id=0a10256e-e065-8d26-cfa6-ac3522fb2227 token=e2740a0f-011f-86d5-b810-15ce19e594e8
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=854b0d4e-b394-7fca-56a2-1b5caabd3a32 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:42:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=854b0d4e-b394-7fca-56a2-1b5caabd3a32 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:42:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=854b0d4e-b394-7fca-56a2-1b5caabd3a32 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=b9a735cf-74eb-6b0a-ca2e-2b7f194406a6
2024-03-13T20:42:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:42:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:31.541746822 +0000 UTC m=+7920.452890029 2024-03-13 20:42:31.541746822 +0000 UTC m=+7980.452890029}"
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=854b0d4e-b394-7fca-56a2-1b5caabd3a32 token=b9a735cf-74eb-6b0a-ca2e-2b7f194406a6 eval_id=854b0d4e-b394-7fca-56a2-1b5caabd3a32 token=b9a735cf-74eb-6b0a-ca2e-2b7f194406a6
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=8dd949dc-ffaf-3f6f-d6a7-f08f76361e2b policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:42:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=8dd949dc-ffaf-3f6f-d6a7-f08f76361e2b policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:42:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=8dd949dc-ffaf-3f6f-d6a7-f08f76361e2b policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=314739e2-6198-2a87-908c-93f620f6fdfe
2024-03-13T20:42:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:42:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:31.557626507 +0000 UTC m=+7920.468769725 2024-03-13 20:42:31.557626507 +0000 UTC m=+7980.468769725}"
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=8dd949dc-ffaf-3f6f-d6a7-f08f76361e2b token=314739e2-6198-2a87-908c-93f620f6fdfe eval_id=8dd949dc-ffaf-3f6f-d6a7-f08f76361e2b token=314739e2-6198-2a87-908c-93f620f6fdfe
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:32.087Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=7012a88c-365c-3cc2-d1d7-2c34df61ec4b policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:42:32.087Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=7012a88c-365c-3cc2-d1d7-2c34df61ec4b policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:42:32.087Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=7012a88c-365c-3cc2-d1d7-2c34df61ec4b policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 token=7b5b067d-aec7-4203-a043-d1abad2934ae
2024-03-13T20:42:32.087Z [DEBUG] policy_eval.worker: received policy for evaluation: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg
2024-03-13T20:42:32.124Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=9c124054-5e4e-e706-c3dd-cca0959298cb policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:42:32.124Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=9c124054-5e4e-e706-c3dd-cca0959298cb policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:42:32.130Z [DEBUG] policy_eval.broker: nack eval: eval_id=1e55334d-03ea-f5fc-eb7d-680b5453587e token=499ac0f5-2987-7cec-86c1-a4124a47d247
2024-03-13T20:42:32.130Z [WARN]  policy_eval.broker: eval delivery limit reached: eval_id=1e55334d-03ea-f5fc-eb7d-680b5453587e policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf token=499ac0f5-2987-7cec-86c1-a4124a47d247 count=1 limit=1
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=4dbbb042-ced2-0dc4-1fd8-96cf2806a87b policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=4dbbb042-ced2-0dc4-1fd8-96cf2806a87b policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=07894b05-7c4d-bec7-ed83-a41158deb59a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=07894b05-7c4d-bec7-ed83-a41158deb59a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=4dbbb042-ced2-0dc4-1fd8-96cf2806a87b policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=daaa2d33-6e5b-5a26-7d1e-f7523a51d657
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:32.256754374 +0000 UTC m=+7921.167897598 2024-03-13 20:42:32.256754374 +0000 UTC m=+7981.167897598}"
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=07894b05-7c4d-bec7-ed83-a41158deb59a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=2746942c-1cc2-37c3-c9b2-c2847b92cff0
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:32.257Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:32.257073637 +0000 UTC m=+7921.168216844 2024-03-13 20:42:32.257073637 +0000 UTC m=+7981.168216844}"
2024-03-13T20:42:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: ack eval: eval_id=4dbbb042-ced2-0dc4-1fd8-96cf2806a87b token=daaa2d33-6e5b-5a26-7d1e-f7523a51d657 eval_id=4dbbb042-ced2-0dc4-1fd8-96cf2806a87b token=daaa2d33-6e5b-5a26-7d1e-f7523a51d657
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: ack eval: eval_id=07894b05-7c4d-bec7-ed83-a41158deb59a token=2746942c-1cc2-37c3-c9b2-c2847b92cff0 eval_id=07894b05-7c4d-bec7-ed83-a41158deb59a token=2746942c-1cc2-37c3-c9b2-c2847b92cff0
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:32.259Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:32.290Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:42:32.290Z [DEBUG] policy_eval.worker.check_handler: querying source: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg query="sum(nomad_client_allocated_cpu{node_class=\"linux\"}*100/(nomad_client_unallocated_cpu{node_class=\"linux\"}+nomad_client_allocated_cpu{node_class=\"linux\"}))/count(nomad_client_allocated_cpu{node_class=\"linux\"})" source=prometheus
2024-03-13T20:42:32.290Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_cpu{node_class=\"linux\"}*100/(nomad_client_unallocated_cpu{node_class=\"linux\"}+nomad_client_allocated_cpu{node_class=\"linux\"}))/count(nomad_client_allocated_cpu{node_class=\"linux\"})" range="{2024-03-13 20:41:32.290399306 +0000 UTC m=+7921.201542514 2024-03-13 20:42:32.290399306 +0000 UTC m=+7981.201542514}"
2024-03-13T20:42:32.292Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg count=3
2024-03-13T20:42:32.292Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:42:32.292Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:42:32.292Z [DEBUG] policy_eval.worker.check_handler: querying source: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg query="sum(nomad_client_allocated_memory{node_class=\"linux\"}*100/(nomad_client_unallocated_memory{node_class=\"linux\"}+nomad_client_allocated_memory{node_class=\"linux\"}))/count(nomad_client_allocated_memory{node_class=\"linux\"})" source=prometheus
2024-03-13T20:42:32.292Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_memory{node_class=\"linux\"}*100/(nomad_client_unallocated_memory{node_class=\"linux\"}+nomad_client_allocated_memory{node_class=\"linux\"}))/count(nomad_client_allocated_memory{node_class=\"linux\"})" range="{2024-03-13 20:41:32.29270264 +0000 UTC m=+7921.203845864 2024-03-13 20:42:32.29270264 +0000 UTC m=+7981.203845864}"
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg count=3
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg from=3 to=3
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.worker: check linux_cpu_allocated_percentage selected in group : id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg direction=none count=0
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.worker: no checks need to be executed: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.broker: ack eval: eval_id=7012a88c-365c-3cc2-d1d7-2c34df61ec4b token=7b5b067d-aec7-4203-a043-d1abad2934ae eval_id=7012a88c-365c-3cc2-d1d7-2c34df61ec4b token=7b5b067d-aec7-4203-a043-d1abad2934ae
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.broker: dequeue eval: queue=cluster
2024-03-13T20:42:32.294Z [DEBUG] policy_eval.broker: waiting for eval: queue=cluster
2024-03-13T20:42:32.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=ceb005e3-10e5-6945-20d7-e15f9bca3913 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:42:32.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=ceb005e3-10e5-6945-20d7-e15f9bca3913 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:42:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=ceb005e3-10e5-6945-20d7-e15f9bca3913 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=71db714c-2dce-f0b5-a3c5-0d4044c108d0
2024-03-13T20:42:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:42:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:32.5568186 +0000 UTC m=+7921.467961815 2024-03-13 20:42:32.5568186 +0000 UTC m=+7981.467961815}"
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=ceb005e3-10e5-6945-20d7-e15f9bca3913 token=71db714c-2dce-f0b5-a3c5-0d4044c108d0 eval_id=ceb005e3-10e5-6945-20d7-e15f9bca3913 token=71db714c-2dce-f0b5-a3c5-0d4044c108d0
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=622df581-df7e-68cb-b871-5e6acce7b777 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:42:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=622df581-df7e-68cb-b871-5e6acce7b777 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:42:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=622df581-df7e-68cb-b871-5e6acce7b777 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=5f48e5bd-157c-7597-4e17-e2ff3030b589
2024-03-13T20:42:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:42:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:32.8582586 +0000 UTC m=+7921.769401808 2024-03-13 20:42:32.8582586 +0000 UTC m=+7981.769401808}"
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=622df581-df7e-68cb-b871-5e6acce7b777 token=5f48e5bd-157c-7597-4e17-e2ff3030b589 eval_id=622df581-df7e-68cb-b871-5e6acce7b777 token=5f48e5bd-157c-7597-4e17-e2ff3030b589
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:42:33.856Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=deb73a9c-36f5-b097-9b6a-1de83b9b60fd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:42:33.856Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=deb73a9c-36f5-b097-9b6a-1de83b9b60fd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:42:33.856Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=deb73a9c-36f5-b097-9b6a-1de83b9b60fd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=598d1482-9440-b59a-7e91-12c833294373
2024-03-13T20:42:33.856Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:42:33.856Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:42:33.856Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:42:33.856Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:41:33.856734771 +0000 UTC m=+7922.767877979 2024-03-13 20:42:33.856734771 +0000 UTC m=+7982.767877979}"
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.broker: ack eval: eval_id=deb73a9c-36f5-b097-9b6a-1de83b9b60fd token=598d1482-9440-b59a-7e91-12c833294373 eval_id=deb73a9c-36f5-b097-9b6a-1de83b9b60fd token=598d1482-9440-b59a-7e91-12c833294373
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:42:33.858Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:43:31.355Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=d80767e2-c4c0-3a29-9d94-689564d534f9 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:43:31.355Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=d80767e2-c4c0-3a29-9d94-689564d534f9 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:43:31.355Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=d80767e2-c4c0-3a29-9d94-689564d534f9 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=b992ab61-5ddd-741b-0f41-8c6792526d72
2024-03-13T20:43:31.355Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:43:31.355Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:31.355Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:31.355Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:31.355386156 +0000 UTC m=+7980.266529362 2024-03-13 20:43:31.355386156 +0000 UTC m=+8040.266529362}"
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=a45b8f3d-717d-d88a-5375-1734cbf48d09 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=a45b8f3d-717d-d88a-5375-1734cbf48d09 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=a45b8f3d-717d-d88a-5375-1734cbf48d09 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=34b9fd27-4f8c-f4c3-2868-45a5ffd90ea5
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:31.357Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:31.357094048 +0000 UTC m=+7980.268237262 2024-03-13 20:43:31.357094048 +0000 UTC m=+8040.268237262}"
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: ack eval: eval_id=d80767e2-c4c0-3a29-9d94-689564d534f9 token=b992ab61-5ddd-741b-0f41-8c6792526d72 eval_id=d80767e2-c4c0-3a29-9d94-689564d534f9 token=b992ab61-5ddd-741b-0f41-8c6792526d72
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:31.357Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=a45b8f3d-717d-d88a-5375-1734cbf48d09 token=34b9fd27-4f8c-f4c3-2868-45a5ffd90ea5 eval_id=a45b8f3d-717d-d88a-5375-1734cbf48d09 token=34b9fd27-4f8c-f4c3-2868-45a5ffd90ea5
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=903e4731-01a9-0b3c-8e9e-32523557fb06 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:43:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=903e4731-01a9-0b3c-8e9e-32523557fb06 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:43:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=903e4731-01a9-0b3c-8e9e-32523557fb06 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=1bf69cd2-f0af-2af4-bc8a-01ffa3b20b84
2024-03-13T20:43:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:43:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:31.541760102 +0000 UTC m=+7980.452903309 2024-03-13 20:43:31.541760102 +0000 UTC m=+8040.452903309}"
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=903e4731-01a9-0b3c-8e9e-32523557fb06 token=1bf69cd2-f0af-2af4-bc8a-01ffa3b20b84 eval_id=903e4731-01a9-0b3c-8e9e-32523557fb06 token=1bf69cd2-f0af-2af4-bc8a-01ffa3b20b84
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=805bdd67-c45b-e426-8408-c4ad2fa1e352 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:43:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=805bdd67-c45b-e426-8408-c4ad2fa1e352 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:43:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=805bdd67-c45b-e426-8408-c4ad2fa1e352 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=5ec53ebc-c449-b0fd-a3c6-54115ac03223
2024-03-13T20:43:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:43:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:31.557531027 +0000 UTC m=+7980.468674234 2024-03-13 20:43:31.557531027 +0000 UTC m=+8040.468674234}"
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=805bdd67-c45b-e426-8408-c4ad2fa1e352 token=5ec53ebc-c449-b0fd-a3c6-54115ac03223 eval_id=805bdd67-c45b-e426-8408-c4ad2fa1e352 token=5ec53ebc-c449-b0fd-a3c6-54115ac03223
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:32.055Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=a3be04fc-d019-b2c9-d738-3dee87162ca2 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:43:32.055Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=a3be04fc-d019-b2c9-d738-3dee87162ca2 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:43:32.055Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=a3be04fc-d019-b2c9-d738-3dee87162ca2 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 token=65132494-c3e9-592d-599e-08f134db5633
2024-03-13T20:43:32.055Z [DEBUG] policy_eval.worker: received policy for evaluation: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg
2024-03-13T20:43:32.093Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=466a774d-b3e6-f7f8-8390-5d7c843dac8d policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:43:32.093Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=466a774d-b3e6-f7f8-8390-5d7c843dac8d policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=ed8d6fe1-928b-a72f-8337-f3eb887ee0e5 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=ed8d6fe1-928b-a72f-8337-f3eb887ee0e5 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=5e1a3d8a-5c01-cc94-4dea-96547544285d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=5e1a3d8a-5c01-cc94-4dea-96547544285d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=5e1a3d8a-5c01-cc94-4dea-96547544285d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=e100506b-ce2f-9ffd-c75e-52c00377cc60
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=ed8d6fe1-928b-a72f-8337-f3eb887ee0e5 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=51d17695-c680-9a54-e37c-a071ed2c0947
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:32.256298114 +0000 UTC m=+7981.167441329 2024-03-13 20:43:32.256298114 +0000 UTC m=+8041.167441329}"
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:32.256489778 +0000 UTC m=+7981.167632992 2024-03-13 20:43:32.256489778 +0000 UTC m=+8041.167632992}"
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=5e1a3d8a-5c01-cc94-4dea-96547544285d token=e100506b-ce2f-9ffd-c75e-52c00377cc60 eval_id=5e1a3d8a-5c01-cc94-4dea-96547544285d token=e100506b-ce2f-9ffd-c75e-52c00377cc60
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=ed8d6fe1-928b-a72f-8337-f3eb887ee0e5 token=51d17695-c680-9a54-e37c-a071ed2c0947 eval_id=ed8d6fe1-928b-a72f-8337-f3eb887ee0e5 token=51d17695-c680-9a54-e37c-a071ed2c0947
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:32.282Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:43:32.282Z [DEBUG] policy_eval.worker.check_handler: querying source: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg query="sum(nomad_client_allocated_cpu{node_class=\"linux\"}*100/(nomad_client_unallocated_cpu{node_class=\"linux\"}+nomad_client_allocated_cpu{node_class=\"linux\"}))/count(nomad_client_allocated_cpu{node_class=\"linux\"})" source=prometheus
2024-03-13T20:43:32.282Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_cpu{node_class=\"linux\"}*100/(nomad_client_unallocated_cpu{node_class=\"linux\"}+nomad_client_allocated_cpu{node_class=\"linux\"}))/count(nomad_client_allocated_cpu{node_class=\"linux\"})" range="{2024-03-13 20:42:32.28225826 +0000 UTC m=+7981.193401472 2024-03-13 20:43:32.28225826 +0000 UTC m=+8041.193401472}"
2024-03-13T20:43:32.284Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg count=3
2024-03-13T20:43:32.284Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=linux_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:43:32.284Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg
2024-03-13T20:43:32.284Z [DEBUG] policy_eval.worker.check_handler: querying source: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg query="sum(nomad_client_allocated_memory{node_class=\"linux\"}*100/(nomad_client_unallocated_memory{node_class=\"linux\"}+nomad_client_allocated_memory{node_class=\"linux\"}))/count(nomad_client_allocated_memory{node_class=\"linux\"})" source=prometheus
2024-03-13T20:43:32.284Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_memory{node_class=\"linux\"}*100/(nomad_client_unallocated_memory{node_class=\"linux\"}+nomad_client_allocated_memory{node_class=\"linux\"}))/count(nomad_client_allocated_memory{node_class=\"linux\"})" range="{2024-03-13 20:42:32.284379978 +0000 UTC m=+7981.195523194 2024-03-13 20:43:32.284379978 +0000 UTC m=+8041.195523194}"
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg count=3
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=linux_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster source=prometheus strategy=target-value target=linux-aws-asg from=3 to=3
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.worker: check linux_cpu_allocated_percentage selected in group : id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg direction=none count=0
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.worker: no checks need to be executed: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster target=linux-aws-asg
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.broker: ack eval: eval_id=a3be04fc-d019-b2c9-d738-3dee87162ca2 token=65132494-c3e9-592d-599e-08f134db5633 eval_id=a3be04fc-d019-b2c9-d738-3dee87162ca2 token=65132494-c3e9-592d-599e-08f134db5633
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.broker: dequeue eval: queue=cluster
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=466a774d-b3e6-f7f8-8390-5d7c843dac8d policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf token=3d615dd4-d3e7-7510-44dc-bb438726e0e7
2024-03-13T20:43:32.286Z [DEBUG] policy_eval.worker: received policy for evaluation: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster target=windows-aws-asg
2024-03-13T20:43:32.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=windows_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster source=prometheus strategy=target-value target=windows-aws-asg
2024-03-13T20:43:32.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=windows_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster source=prometheus strategy=target-value target=windows-aws-asg query="sum(nomad_client_allocated_cpu{node_class=\"windows\"}*100/(nomad_client_unallocated_cpu{node_class=\"windows\"}+nomad_client_allocated_cpu{node_class=\"windows\"}))/count(nomad_client_allocated_cpu{node_class=\"windows\"})" source=prometheus
2024-03-13T20:43:32.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_cpu{node_class=\"windows\"}*100/(nomad_client_unallocated_cpu{node_class=\"windows\"}+nomad_client_allocated_cpu{node_class=\"windows\"}))/count(nomad_client_allocated_cpu{node_class=\"windows\"})" range="{2024-03-13 20:42:32.541994239 +0000 UTC m=+7981.453137448 2024-03-13 20:43:32.541994239 +0000 UTC m=+8041.453137448}"
2024-03-13T20:43:32.547Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=windows_cpu_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster source=prometheus strategy=target-value target=windows-aws-asg count=32
2024-03-13T20:43:32.547Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=windows_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster source=prometheus strategy=target-value target=windows-aws-asg
2024-03-13T20:43:32.547Z [DEBUG] policy_eval.worker.check_handler: querying source: check=windows_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster source=prometheus strategy=target-value target=windows-aws-asg query="sum(nomad_client_allocated_memory{node_class=\"windows\"}*100/(nomad_client_unallocated_memory{node_class=\"windows\"}+nomad_client_allocated_memory{node_class=\"windows\"}))/count(nomad_client_allocated_memory{node_class=\"windows\"})" source=prometheus
2024-03-13T20:43:32.547Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="sum(nomad_client_allocated_memory{node_class=\"windows\"}*100/(nomad_client_unallocated_memory{node_class=\"windows\"}+nomad_client_allocated_memory{node_class=\"windows\"}))/count(nomad_client_allocated_memory{node_class=\"windows\"})" range="{2024-03-13 20:42:32.547407602 +0000 UTC m=+7981.458550810 2024-03-13 20:43:32.547407602 +0000 UTC m=+8041.458550810}"
2024-03-13T20:43:32.552Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=windows_mem_allocated_percentage id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster source=prometheus strategy=target-value target=windows-aws-asg count=32
2024-03-13T20:43:32.552Z [DEBUG] policy_eval.worker: check windows_cpu_allocated_percentage selected in group : id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster target=windows-aws-asg direction=down count=26
2024-03-13T20:43:32.552Z [DEBUG] policy_eval.worker: check windows_cpu_allocated_percentage selected: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster target=windows-aws-asg direction=down count=26
2024-03-13T20:43:32.552Z [INFO]  policy_eval.worker: scaling target: id=797e3c65-e480-a3cb-e8c4-2fe03ada2168 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster target=windows-aws-asg from=32 to=26 reason="scaling down because factor is 0.787877" meta=map[nomad_policy_id:580ba51a-5277-71a7-0bc9-e8319b7110bf]
2024-03-13T20:43:32.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=9e039c13-9235-915f-e5f4-9e321679e488 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:43:32.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=9e039c13-9235-915f-e5f4-9e321679e488 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:43:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=9e039c13-9235-915f-e5f4-9e321679e488 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=84853344-e653-6328-bc6e-4703bbc37972
2024-03-13T20:43:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:43:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:32.556576557 +0000 UTC m=+7981.467719764 2024-03-13 20:43:32.556576557 +0000 UTC m=+8041.467719764}"
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=9e039c13-9235-915f-e5f4-9e321679e488 token=84853344-e653-6328-bc6e-4703bbc37972 eval_id=9e039c13-9235-915f-e5f4-9e321679e488 token=84853344-e653-6328-bc6e-4703bbc37972
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-00761f514c21aaf48
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-00aca9cf65be2da43
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0249fd37afd49e104
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-024b0fe8233862054
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-02f84b272e7cd0a9c
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-04bf922f3301aa6f7
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-05d9388c5d49ee018
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0634a7792de03e3e3
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-06458c3709d4efa7c
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0674b06ba1a75db9e
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-06a6ef7c0a75c5f99
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-06e26dd836d852263
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0770bbdece5bcc25a
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-077c93ff09c084b09
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-07ddd8bb1439cca8c
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-07e60de66c6403caa
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-085268f7e317abcb5
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0a19bf5084074d0db
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0a88d17cbfeda198e
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0abdb2a55865097b3
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0afd6c5efddf11b3a
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0b75309ff5d92d8e4
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0c2f7af9a7c9bd1a1
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0c360df2c83923f78
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0d2eb45b5987769be
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0d4e79bea942ba2f2
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0dca40a8c9c393667
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0e2d6dbfee1fb862e
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0e4383abf0ec59800
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0e609a812459607df
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0ea1a446a0cf48ed1
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: found eligible instance: action=scale_in asg_name=spot-default-windows scale_in_protection=false instance_id=i-0eb42e36085e22943
2024-03-13T20:43:32.734Z [DEBUG] internal_plugin.windows-aws-asg: performing node pool filtering: node_class=windows
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=f74b348b-891d-a42f-2b5d-06926b260954 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=8c8a2b4c-e880-4a1a-8175-dcf848d6fe47 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=81e22068-a34c-a078-29d7-469baf8d3f6f datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=3e2610c1-7ccf-6e3b-d4af-442338261731 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=ineligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=371c4479-6c4c-1b9d-ee4b-c0e0258d28ac datacenter=dc1 node_class=windows node_pool=default status=down eligibility=ineligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=e4b6461c-ec81-99f8-5ce2-036962a1a250 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=60a7979c-398b-555f-f9aa-07619757a61c datacenter=dc1 node_class=windows node_pool=default status=down eligibility=ineligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=15f0a5e7-6ee8-8020-baa5-5320317a722d datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=fa010a9c-15fe-7dd0-0608-37ad8d20454e datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=0381ef2f-787b-29e7-c20d-f8cb4accc4bf datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=99dac997-acdf-8ac8-5395-3d7caaf6ca43 datacenter=dc1 node_class=linux node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=1a50065b-5065-c2ce-7750-bcbcfed1ba58 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=de71fd67-bb97-9679-da2e-9576728b105e datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=ineligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=cdc8c759-7cc4-f57d-aea6-2310c139b87a datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=7e2f2c9e-e60a-1ec2-b603-cdb4667d2d4e datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=ineligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=2810ff6a-c459-29b9-77b8-4bcdb5c41ffe datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=56bd9b52-552e-6c52-b0a5-bfc0eedbd15a datacenter=dc1 node_class=linux node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=2868adee-794c-fe46-2e2c-9a5337818b33 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=7781af5c-eeb0-82de-ee37-433f073881af datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.752Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=fcb814eb-e102-3c9b-d2b6-6fc7ae6f6a49 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=ineligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=ineligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=ineligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=123bd2c2-b6af-d025-98a2-6199583d1469 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=bb4285a6-3664-0b96-fed1-33c0e9043871 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=16032d7d-5019-4b4c-ce7a-68a4925d27e0 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=c31d1fc6-0039-83fe-0e35-2b23e8a7d937 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=ineligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=6b10ba96-04c5-cf5c-339c-1cabc4fc88cb datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=6d6aa92c-b43f-c035-91fa-434fdf1a8c5c datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=ab50dc24-326f-eadb-e653-34b658927ca8 datacenter=dc1 node_class=linux node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=d85341d0-02ca-bca7-2b90-47c858434f4d datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=9d388642-f36e-95c7-5b83-ee37ebbd6fa5 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=6447b6c3-9fa1-5fe6-c807-a4ebc7035221 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=ineligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=82e27108-63f3-a89a-ce82-38bd7addd073 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=6d039ab1-f516-3cb6-c26e-08fdfce87c80 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=40fb1e95-42ba-2260-0c0d-4c1ff43e7f4b datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=8db6fda9-366d-b566-2237-aa7f9485f3e2 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=40fc0b84-aa75-189c-51a8-3882898b71c4 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=85ad306b-4615-46de-c17e-a011f676fbd4 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=aed6bdfa-b7bb-4dc6-4d5e-6571e052dbb0 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=122e8d89-e52c-c7dd-3488-d75a9cdd3b55 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=49f6f0dc-098a-5f02-d908-39a11a7d37f4 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=d84719c3-b2d8-c3af-5e00-7678d29ce339 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=ee79ae51-c58e-3976-8b92-84df289f736d datacenter=dc1 node_class=windows node_pool=default status=down eligibility=ineligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=894c861d-a8e9-2fea-fc1c-38a4a8be1e25 datacenter=dc1 node_class=windows node_pool=default status=ready eligibility=eligible draining=false
2024-03-13T20:43:32.753Z [DEBUG] internal_plugin.windows-aws-asg: found node: node_id=62c4cd79-d4d0-2874-cab1-cb1d5ef259b8 datacenter=dc1 node_class=windows node_pool=default status=down eligibility=ineligible draining=false
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=f74b348b-891d-a42f-2b5d-06926b260954
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=8c8a2b4c-e880-4a1a-8175-dcf848d6fe47
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=81e22068-a34c-a078-29d7-469baf8d3f6f
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=0381ef2f-787b-29e7-c20d-f8cb4accc4bf
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=de71fd67-bb97-9679-da2e-9576728b105e
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=cdc8c759-7cc4-f57d-aea6-2310c139b87a
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=2810ff6a-c459-29b9-77b8-4bcdb5c41ffe
2024-03-13T20:43:32.754Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=2868adee-794c-fe46-2e2c-9a5337818b33
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=7781af5c-eeb0-82de-ee37-433f073881af
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=fcb814eb-e102-3c9b-d2b6-6fc7ae6f6a49
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=123bd2c2-b6af-d025-98a2-6199583d1469
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=bb4285a6-3664-0b96-fed1-33c0e9043871
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=16032d7d-5019-4b4c-ce7a-68a4925d27e0
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=c31d1fc6-0039-83fe-0e35-2b23e8a7d937
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=6b10ba96-04c5-cf5c-339c-1cabc4fc88cb
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=6d6aa92c-b43f-c035-91fa-434fdf1a8c5c
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=d85341d0-02ca-bca7-2b90-47c858434f4d
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=9d388642-f36e-95c7-5b83-ee37ebbd6fa5
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=82e27108-63f3-a89a-ce82-38bd7addd073
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=6d039ab1-f516-3cb6-c26e-08fdfce87c80
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=40fb1e95-42ba-2260-0c0d-4c1ff43e7f4b
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=8db6fda9-366d-b566-2237-aa7f9485f3e2
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=40fc0b84-aa75-189c-51a8-3882898b71c4
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=85ad306b-4615-46de-c17e-a011f676fbd4
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=aed6bdfa-b7bb-4dc6-4d5e-6571e052dbb0
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=122e8d89-e52c-c7dd-3488-d75a9cdd3b55
2024-03-13T20:43:32.755Z [DEBUG] internal_plugin.windows-aws-asg: node passed filter criteria: node_id=894c861d-a8e9-2fea-fc1c-38a4a8be1e25
2024-03-13T20:43:32.759Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=f74b348b-891d-a42f-2b5d-06926b260954 remote_id=i-0a88d17cbfeda198e
2024-03-13T20:43:32.761Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=8c8a2b4c-e880-4a1a-8175-dcf848d6fe47 remote_id=i-06e26dd836d852263
2024-03-13T20:43:32.764Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=81e22068-a34c-a078-29d7-469baf8d3f6f remote_id=i-085268f7e317abcb5
2024-03-13T20:43:32.766Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=0381ef2f-787b-29e7-c20d-f8cb4accc4bf remote_id=i-0b75309ff5d92d8e4
2024-03-13T20:43:32.769Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=de71fd67-bb97-9679-da2e-9576728b105e remote_id=i-00761f514c21aaf48
2024-03-13T20:43:32.772Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=cdc8c759-7cc4-f57d-aea6-2310c139b87a remote_id=i-0afd6c5efddf11b3a
2024-03-13T20:43:32.774Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa remote_id=i-07e60de66c6403caa
2024-03-13T20:43:32.777Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=2810ff6a-c459-29b9-77b8-4bcdb5c41ffe remote_id=i-0634a7792de03e3e3
2024-03-13T20:43:32.779Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=2868adee-794c-fe46-2e2c-9a5337818b33 remote_id=i-0eb42e36085e22943
2024-03-13T20:43:32.782Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=7781af5c-eeb0-82de-ee37-433f073881af remote_id=i-02f84b272e7cd0a9c
2024-03-13T20:43:32.784Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=fcb814eb-e102-3c9b-d2b6-6fc7ae6f6a49 remote_id=i-05d9388c5d49ee018
2024-03-13T20:43:32.787Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 remote_id=i-0abdb2a55865097b3
2024-03-13T20:43:32.790Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 remote_id=i-0e2d6dbfee1fb862e
2024-03-13T20:43:32.798Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e remote_id=i-06458c3709d4efa7c
2024-03-13T20:43:32.801Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=123bd2c2-b6af-d025-98a2-6199583d1469 remote_id=i-0770bbdece5bcc25a
2024-03-13T20:43:32.804Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=bb4285a6-3664-0b96-fed1-33c0e9043871 remote_id=i-0ea1a446a0cf48ed1
2024-03-13T20:43:32.806Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=16032d7d-5019-4b4c-ce7a-68a4925d27e0 remote_id=i-0a19bf5084074d0db
2024-03-13T20:43:32.809Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=c31d1fc6-0039-83fe-0e35-2b23e8a7d937 remote_id=i-0dca40a8c9c393667
2024-03-13T20:43:32.812Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df remote_id=i-0c2f7af9a7c9bd1a1
2024-03-13T20:43:32.814Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=6b10ba96-04c5-cf5c-339c-1cabc4fc88cb remote_id=i-0d4e79bea942ba2f2
2024-03-13T20:43:32.817Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=6d6aa92c-b43f-c035-91fa-434fdf1a8c5c remote_id=i-0e609a812459607df
2024-03-13T20:43:32.820Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=d85341d0-02ca-bca7-2b90-47c858434f4d remote_id=i-0249fd37afd49e104
2024-03-13T20:43:32.822Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=9d388642-f36e-95c7-5b83-ee37ebbd6fa5 remote_id=i-024b0fe8233862054
2024-03-13T20:43:32.825Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=82e27108-63f3-a89a-ce82-38bd7addd073 remote_id=i-0d2eb45b5987769be
2024-03-13T20:43:32.827Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=6d039ab1-f516-3cb6-c26e-08fdfce87c80 remote_id=i-0674b06ba1a75db9e
2024-03-13T20:43:32.830Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=40fb1e95-42ba-2260-0c0d-4c1ff43e7f4b remote_id=i-00aca9cf65be2da43
2024-03-13T20:43:32.836Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=8db6fda9-366d-b566-2237-aa7f9485f3e2 remote_id=i-07ddd8bb1439cca8c
2024-03-13T20:43:32.839Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=40fc0b84-aa75-189c-51a8-3882898b71c4 remote_id=i-0e4383abf0ec59800
2024-03-13T20:43:32.841Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=85ad306b-4615-46de-c17e-a011f676fbd4 remote_id=i-0c360df2c83923f78
2024-03-13T20:43:32.844Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=aed6bdfa-b7bb-4dc6-4d5e-6571e052dbb0 remote_id=i-06a6ef7c0a75c5f99
2024-03-13T20:43:32.847Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=122e8d89-e52c-c7dd-3488-d75a9cdd3b55 remote_id=i-04bf922f3301aa6f7
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: identified remote provider ID for node: node_id=894c861d-a8e9-2fea-fc1c-38a4a8be1e25 remote_id=i-077c93ff09c084b09
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=f74b348b-891d-a42f-2b5d-06926b260954 remote_id=i-0a88d17cbfeda198e
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=8c8a2b4c-e880-4a1a-8175-dcf848d6fe47 remote_id=i-06e26dd836d852263
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=81e22068-a34c-a078-29d7-469baf8d3f6f remote_id=i-085268f7e317abcb5
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=0381ef2f-787b-29e7-c20d-f8cb4accc4bf remote_id=i-0b75309ff5d92d8e4
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=de71fd67-bb97-9679-da2e-9576728b105e remote_id=i-00761f514c21aaf48
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=cdc8c759-7cc4-f57d-aea6-2310c139b87a remote_id=i-0afd6c5efddf11b3a
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa remote_id=i-07e60de66c6403caa
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=2810ff6a-c459-29b9-77b8-4bcdb5c41ffe remote_id=i-0634a7792de03e3e3
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=2868adee-794c-fe46-2e2c-9a5337818b33 remote_id=i-0eb42e36085e22943
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=7781af5c-eeb0-82de-ee37-433f073881af remote_id=i-02f84b272e7cd0a9c
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=fcb814eb-e102-3c9b-d2b6-6fc7ae6f6a49 remote_id=i-05d9388c5d49ee018
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 remote_id=i-0abdb2a55865097b3
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 remote_id=i-0e2d6dbfee1fb862e
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e remote_id=i-06458c3709d4efa7c
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=123bd2c2-b6af-d025-98a2-6199583d1469 remote_id=i-0770bbdece5bcc25a
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=bb4285a6-3664-0b96-fed1-33c0e9043871 remote_id=i-0ea1a446a0cf48ed1
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=16032d7d-5019-4b4c-ce7a-68a4925d27e0 remote_id=i-0a19bf5084074d0db
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=c31d1fc6-0039-83fe-0e35-2b23e8a7d937 remote_id=i-0dca40a8c9c393667
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df remote_id=i-0c2f7af9a7c9bd1a1
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=6b10ba96-04c5-cf5c-339c-1cabc4fc88cb remote_id=i-0d4e79bea942ba2f2
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=6d6aa92c-b43f-c035-91fa-434fdf1a8c5c remote_id=i-0e609a812459607df
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=d85341d0-02ca-bca7-2b90-47c858434f4d remote_id=i-0249fd37afd49e104
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=9d388642-f36e-95c7-5b83-ee37ebbd6fa5 remote_id=i-024b0fe8233862054
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=82e27108-63f3-a89a-ce82-38bd7addd073 remote_id=i-0d2eb45b5987769be
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=6d039ab1-f516-3cb6-c26e-08fdfce87c80 remote_id=i-0674b06ba1a75db9e
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=40fb1e95-42ba-2260-0c0d-4c1ff43e7f4b remote_id=i-00aca9cf65be2da43
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=8db6fda9-366d-b566-2237-aa7f9485f3e2 remote_id=i-07ddd8bb1439cca8c
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=40fc0b84-aa75-189c-51a8-3882898b71c4 remote_id=i-0e4383abf0ec59800
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=85ad306b-4615-46de-c17e-a011f676fbd4 remote_id=i-0c360df2c83923f78
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=aed6bdfa-b7bb-4dc6-4d5e-6571e052dbb0 remote_id=i-06a6ef7c0a75c5f99
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=122e8d89-e52c-c7dd-3488-d75a9cdd3b55 remote_id=i-04bf922f3301aa6f7
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: node is part of the policy target: node_id=894c861d-a8e9-2fea-fc1c-38a4a8be1e25 remote_id=i-077c93ff09c084b09
2024-03-13T20:43:32.849Z [DEBUG] internal_plugin.windows-aws-asg: performing node selection: selector_strategy=least_busy
2024-03-13T20:43:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=6a51ffba-b95f-03c6-eefe-eac6c23e35c9 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:43:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=6a51ffba-b95f-03c6-eefe-eac6c23e35c9 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:43:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=6a51ffba-b95f-03c6-eefe-eac6c23e35c9 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=9ae7a5fa-025a-04a3-c09c-879caaffe523
2024-03-13T20:43:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:43:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:32.858540305 +0000 UTC m=+7981.769683519 2024-03-13 20:43:32.858540305 +0000 UTC m=+8041.769683519}"
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=6a51ffba-b95f-03c6-eefe-eac6c23e35c9 token=9ae7a5fa-025a-04a3-c09c-879caaffe523 eval_id=6a51ffba-b95f-03c6-eefe-eac6c23e35c9 token=9ae7a5fa-025a-04a3-c09c-879caaffe523
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:43:33.232Z [DEBUG] internal_plugin.windows-aws-asg: node selected for removal: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e remote_id=i-06458c3709d4efa7c
2024-03-13T20:43:33.232Z [DEBUG] internal_plugin.windows-aws-asg: node selected for removal: node_id=de71fd67-bb97-9679-da2e-9576728b105e remote_id=i-00761f514c21aaf48
2024-03-13T20:43:33.233Z [DEBUG] internal_plugin.windows-aws-asg: node selected for removal: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 remote_id=i-0e2d6dbfee1fb862e
2024-03-13T20:43:33.233Z [DEBUG] internal_plugin.windows-aws-asg: node selected for removal: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa remote_id=i-07e60de66c6403caa
2024-03-13T20:43:33.233Z [DEBUG] internal_plugin.windows-aws-asg: node selected for removal: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 remote_id=i-0abdb2a55865097b3
2024-03-13T20:43:33.233Z [DEBUG] internal_plugin.windows-aws-asg: node selected for removal: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df remote_id=i-0c2f7af9a7c9bd1a1
2024-03-13T20:43:33.233Z [INFO]  internal_plugin.windows-aws-asg: triggering drain on node: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df deadline=5m0s
2024-03-13T20:43:33.233Z [INFO]  internal_plugin.windows-aws-asg: triggering drain on node: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 deadline=5m0s
2024-03-13T20:43:33.233Z [INFO]  internal_plugin.windows-aws-asg: triggering drain on node: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e deadline=5m0s
2024-03-13T20:43:33.234Z [INFO]  internal_plugin.windows-aws-asg: triggering drain on node: node_id=de71fd67-bb97-9679-da2e-9576728b105e deadline=5m0s
2024-03-13T20:43:33.235Z [INFO]  internal_plugin.windows-aws-asg: triggering drain on node: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa deadline=5m0s
2024-03-13T20:43:33.237Z [INFO]  internal_plugin.windows-aws-asg: triggering drain on node: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 deadline=5m0s
2024-03-13T20:43:33.256Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 msg="Drain complete for node bf9b3997-c595-b3ab-0547-6e1e8be5f556"
2024-03-13T20:43:33.289Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=de71fd67-bb97-9679-da2e-9576728b105e msg="Drain complete for node de71fd67-bb97-9679-da2e-9576728b105e"
2024-03-13T20:43:33.290Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 msg="Drain complete for node c35c9225-3629-6cfd-80b6-e4b7d152fda2"
2024-03-13T20:43:33.295Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa msg="Drain complete for node e1252bcf-2a01-04d0-1a35-ed36e95185aa"
2024-03-13T20:43:33.298Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556 msg="All allocations on node \"bf9b3997-c595-b3ab-0547-6e1e8be5f556\" have stopped"
2024-03-13T20:43:33.298Z [DEBUG] internal_plugin.windows-aws-asg: node drain complete: node_id=bf9b3997-c595-b3ab-0547-6e1e8be5f556
2024-03-13T20:43:33.302Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2 msg="All allocations on node \"c35c9225-3629-6cfd-80b6-e4b7d152fda2\" have stopped"
2024-03-13T20:43:33.302Z [DEBUG] internal_plugin.windows-aws-asg: node drain complete: node_id=c35c9225-3629-6cfd-80b6-e4b7d152fda2
2024-03-13T20:43:33.305Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa msg="All allocations on node \"e1252bcf-2a01-04d0-1a35-ed36e95185aa\" have stopped"
2024-03-13T20:43:33.305Z [DEBUG] internal_plugin.windows-aws-asg: node drain complete: node_id=e1252bcf-2a01-04d0-1a35-ed36e95185aa
2024-03-13T20:43:33.307Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=de71fd67-bb97-9679-da2e-9576728b105e msg="All allocations on node \"de71fd67-bb97-9679-da2e-9576728b105e\" have stopped"
2024-03-13T20:43:33.307Z [DEBUG] internal_plugin.windows-aws-asg: node drain complete: node_id=de71fd67-bb97-9679-da2e-9576728b105e
2024-03-13T20:43:33.308Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e msg="Drain complete for node 8e541bc4-c1d3-c2e6-9ee6-7426f94d818e"
2024-03-13T20:43:33.310Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e msg="All allocations on node \"8e541bc4-c1d3-c2e6-9ee6-7426f94d818e\" have stopped"
2024-03-13T20:43:33.310Z [DEBUG] internal_plugin.windows-aws-asg: node drain complete: node_id=8e541bc4-c1d3-c2e6-9ee6-7426f94d818e
2024-03-13T20:43:33.319Z [INFO]  internal_plugin.windows-aws-asg: received node drain message: node_id=9fef61b9-3cc3-7782-3bb3-189a56e616df msg="Drain complete for node 9fef61b9-3cc3-7782-3bb3-189a56e616df"
2024-03-13T20:43:33.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=cb9f4c6a-a257-4028-b2d1-09ca9e985e17 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:43:33.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=cb9f4c6a-a257-4028-b2d1-09ca9e985e17 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:43:33.857Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=cb9f4c6a-a257-4028-b2d1-09ca9e985e17 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=e5498cbd-52d4-2129-69b7-b0bf1d282d64
2024-03-13T20:43:33.857Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:43:33.857Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:43:33.857Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:43:33.857Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:42:33.857619359 +0000 UTC m=+7982.768762574 2024-03-13 20:43:33.857619359 +0000 UTC m=+8042.768762574}"
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=cb9f4c6a-a257-4028-b2d1-09ca9e985e17 token=e5498cbd-52d4-2129-69b7-b0bf1d282d64 eval_id=cb9f4c6a-a257-4028-b2d1-09ca9e985e17 token=e5498cbd-52d4-2129-69b7-b0bf1d282d64
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:43:33.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:44:31.355Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=cae10971-6557-66a1-5314-32e553a9ed72 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:44:31.355Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=cae10971-6557-66a1-5314-32e553a9ed72 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:44:31.355Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=cae10971-6557-66a1-5314-32e553a9ed72 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=311f2edd-2930-bc74-c8af-355b817d456c
2024-03-13T20:44:31.355Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:44:31.355Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:31.355Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:31.355Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:31.355923676 +0000 UTC m=+8040.267066885 2024-03-13 20:44:31.355923676 +0000 UTC m=+8100.267066885}"
2024-03-13T20:44:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=1760eea6-0d35-392f-15be-0cd431c00c76 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:44:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=1760eea6-0d35-392f-15be-0cd431c00c76 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:44:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=1760eea6-0d35-392f-15be-0cd431c00c76 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=34875bc1-1171-f23d-c1fe-a3c15476d54c
2024-03-13T20:44:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:44:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:31.356905169 +0000 UTC m=+8040.268048382 2024-03-13 20:44:31.356905169 +0000 UTC m=+8100.268048382}"
2024-03-13T20:44:31.357Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:44:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=cae10971-6557-66a1-5314-32e553a9ed72 token=311f2edd-2930-bc74-c8af-355b817d456c eval_id=cae10971-6557-66a1-5314-32e553a9ed72 token=311f2edd-2930-bc74-c8af-355b817d456c
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: ack eval: eval_id=1760eea6-0d35-392f-15be-0cd431c00c76 token=34875bc1-1171-f23d-c1fe-a3c15476d54c eval_id=1760eea6-0d35-392f-15be-0cd431c00c76 token=34875bc1-1171-f23d-c1fe-a3c15476d54c
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:31.359Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=b1a74dc7-e8a6-0979-97be-e898a4a527aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:44:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=b1a74dc7-e8a6-0979-97be-e898a4a527aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:44:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=b1a74dc7-e8a6-0979-97be-e898a4a527aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=51736ffa-d99e-52ae-167a-87122a0b9f3b
2024-03-13T20:44:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:44:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:31.541143831 +0000 UTC m=+8040.452287039 2024-03-13 20:44:31.541143831 +0000 UTC m=+8100.452287039}"
2024-03-13T20:44:31.542Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:31.542Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:31.542Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:31.542Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:44:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=b1a74dc7-e8a6-0979-97be-e898a4a527aa token=51736ffa-d99e-52ae-167a-87122a0b9f3b eval_id=b1a74dc7-e8a6-0979-97be-e898a4a527aa token=51736ffa-d99e-52ae-167a-87122a0b9f3b
2024-03-13T20:44:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:44:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:31.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=6a2b608b-e0ed-defd-6e84-88825000e1ec policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:44:31.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=6a2b608b-e0ed-defd-6e84-88825000e1ec policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:44:31.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=6a2b608b-e0ed-defd-6e84-88825000e1ec policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=0cb1854b-bdcb-ff00-71fa-ff33c381e7c1
2024-03-13T20:44:31.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:44:31.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:31.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:31.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:31.556995205 +0000 UTC m=+8040.468138423 2024-03-13 20:44:31.556995205 +0000 UTC m=+8100.468138423}"
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=6a2b608b-e0ed-defd-6e84-88825000e1ec token=0cb1854b-bdcb-ff00-71fa-ff33c381e7c1 eval_id=6a2b608b-e0ed-defd-6e84-88825000e1ec token=0cb1854b-bdcb-ff00-71fa-ff33c381e7c1
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:31.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:32.017Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=a35da20c-5f6a-7bce-5a58-904a384f7c90 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:44:32.017Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=a35da20c-5f6a-7bce-5a58-904a384f7c90 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:44:32.110Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=7ab77eab-fbb5-9c28-9b40-e8ddd89ec7cc policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:44:32.111Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=7ab77eab-fbb5-9c28-9b40-e8ddd89ec7cc policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=bc6f5873-2092-735c-27cb-f4a26d27be72 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=bc6f5873-2092-735c-27cb-f4a26d27be72 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=97acaa92-c002-2562-3757-cb41cd9d41b9 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=97acaa92-c002-2562-3757-cb41cd9d41b9 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=bc6f5873-2092-735c-27cb-f4a26d27be72 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=9f110af6-e900-f7b2-167a-e3b5240b5dd0
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:32.256276551 +0000 UTC m=+8041.167419762 2024-03-13 20:44:32.256276551 +0000 UTC m=+8101.167419762}"
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=97acaa92-c002-2562-3757-cb41cd9d41b9 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=541725d2-3d8f-4c86-6761-2952097b2550
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:32.256514183 +0000 UTC m=+8041.167657393 2024-03-13 20:44:32.256514183 +0000 UTC m=+8101.167657393}"
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=97acaa92-c002-2562-3757-cb41cd9d41b9 token=541725d2-3d8f-4c86-6761-2952097b2550 eval_id=97acaa92-c002-2562-3757-cb41cd9d41b9 token=541725d2-3d8f-4c86-6761-2952097b2550
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=bc6f5873-2092-735c-27cb-f4a26d27be72 token=9f110af6-e900-f7b2-167a-e3b5240b5dd0 eval_id=bc6f5873-2092-735c-27cb-f4a26d27be72 token=9f110af6-e900-f7b2-167a-e3b5240b5dd0
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:32.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=3196c2dd-e161-9a2e-94d2-d95de39a740f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:44:32.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=3196c2dd-e161-9a2e-94d2-d95de39a740f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:44:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=3196c2dd-e161-9a2e-94d2-d95de39a740f policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=85720e3f-2088-1ecb-4efe-cb67d89b1a8a
2024-03-13T20:44:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:44:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:32.556927812 +0000 UTC m=+8041.468071023 2024-03-13 20:44:32.556927812 +0000 UTC m=+8101.468071023}"
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=3196c2dd-e161-9a2e-94d2-d95de39a740f token=85720e3f-2088-1ecb-4efe-cb67d89b1a8a eval_id=3196c2dd-e161-9a2e-94d2-d95de39a740f token=85720e3f-2088-1ecb-4efe-cb67d89b1a8a
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=664ffcc1-65dd-69e8-edfb-507e816ccffd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:44:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=664ffcc1-65dd-69e8-edfb-507e816ccffd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:44:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=664ffcc1-65dd-69e8-edfb-507e816ccffd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=f14e53ae-09a9-81d9-c795-3332c83739f4
2024-03-13T20:44:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:44:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:32.858153393 +0000 UTC m=+8041.769296601 2024-03-13 20:44:32.858153393 +0000 UTC m=+8101.769296601}"
2024-03-13T20:44:32.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:32.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:32.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:32.860Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:44:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=664ffcc1-65dd-69e8-edfb-507e816ccffd token=f14e53ae-09a9-81d9-c795-3332c83739f4 eval_id=664ffcc1-65dd-69e8-edfb-507e816ccffd token=f14e53ae-09a9-81d9-c795-3332c83739f4
2024-03-13T20:44:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:44:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:44:33.856Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=bd398959-58d8-f0d7-9998-2f36f5f2a460 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:44:33.856Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=bd398959-58d8-f0d7-9998-2f36f5f2a460 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:44:33.856Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=bd398959-58d8-f0d7-9998-2f36f5f2a460 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=9a034c0f-e117-42e9-47a6-34ac020b7b9c
2024-03-13T20:44:33.856Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:44:33.856Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:44:33.856Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:44:33.856Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:43:33.856706056 +0000 UTC m=+8042.767849268 2024-03-13 20:44:33.856706056 +0000 UTC m=+8102.767849268}"
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.broker: ack eval: eval_id=bd398959-58d8-f0d7-9998-2f36f5f2a460 token=9a034c0f-e117-42e9-47a6-34ac020b7b9c eval_id=bd398959-58d8-f0d7-9998-2f36f5f2a460 token=9a034c0f-e117-42e9-47a6-34ac020b7b9c
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:44:33.858Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:45:31.355Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=95b919c3-bce8-696c-3581-cce67434aaf0 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:45:31.355Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=95b919c3-bce8-696c-3581-cce67434aaf0 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:45:31.355Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=95b919c3-bce8-696c-3581-cce67434aaf0 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=f69d4ce1-2828-4855-33e9-36cdea11d7d8
2024-03-13T20:45:31.355Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:45:31.355Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:31.355Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:31.356002968 +0000 UTC m=+8100.267146175 2024-03-13 20:45:31.356002968 +0000 UTC m=+8160.267146175}"
2024-03-13T20:45:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=cf37705a-a283-8e21-2c8f-a10c844e8df6 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:45:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=cf37705a-a283-8e21-2c8f-a10c844e8df6 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:45:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=cf37705a-a283-8e21-2c8f-a10c844e8df6 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=8b38d077-4063-022b-1033-5decd6f38be8
2024-03-13T20:45:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:45:31.357Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:31.357Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:31.357Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:31.357033649 +0000 UTC m=+8100.268176855 2024-03-13 20:45:31.357033649 +0000 UTC m=+8160.268176855}"
2024-03-13T20:45:31.357Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:31.357Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:31.357Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:31.357Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=95b919c3-bce8-696c-3581-cce67434aaf0 token=f69d4ce1-2828-4855-33e9-36cdea11d7d8 eval_id=95b919c3-bce8-696c-3581-cce67434aaf0 token=f69d4ce1-2828-4855-33e9-36cdea11d7d8
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=cf37705a-a283-8e21-2c8f-a10c844e8df6 token=8b38d077-4063-022b-1033-5decd6f38be8 eval_id=cf37705a-a283-8e21-2c8f-a10c844e8df6 token=8b38d077-4063-022b-1033-5decd6f38be8
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:31.540Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=70584077-ac77-43b0-827f-5a70f77b33bb policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:45:31.540Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=70584077-ac77-43b0-827f-5a70f77b33bb policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:45:31.540Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=70584077-ac77-43b0-827f-5a70f77b33bb policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=dc0c83d7-c067-b621-bdd1-16b8c7135412
2024-03-13T20:45:31.540Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:45:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:31.541064249 +0000 UTC m=+8100.452207457 2024-03-13 20:45:31.541064249 +0000 UTC m=+8160.452207457}"
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.broker: ack eval: eval_id=70584077-ac77-43b0-827f-5a70f77b33bb token=dc0c83d7-c067-b621-bdd1-16b8c7135412 eval_id=70584077-ac77-43b0-827f-5a70f77b33bb token=dc0c83d7-c067-b621-bdd1-16b8c7135412
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:31.542Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=77a9cd98-f9ec-0e8f-072d-360388d39b94 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:45:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=77a9cd98-f9ec-0e8f-072d-360388d39b94 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:45:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=77a9cd98-f9ec-0e8f-072d-360388d39b94 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=aff15bfc-d326-2a9e-58e4-d54651315173
2024-03-13T20:45:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:45:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:31.557964713 +0000 UTC m=+8100.469107924 2024-03-13 20:45:31.557964713 +0000 UTC m=+8160.469107924}"
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=77a9cd98-f9ec-0e8f-072d-360388d39b94 token=aff15bfc-d326-2a9e-58e4-d54651315173 eval_id=77a9cd98-f9ec-0e8f-072d-360388d39b94 token=aff15bfc-d326-2a9e-58e4-d54651315173
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:32.093Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e1218807-780b-a852-4514-de4d93d8ad98 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:45:32.093Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=e1218807-780b-a852-4514-de4d93d8ad98 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:45:32.093Z [DEBUG] policy_eval.broker: new eval is newer, policy updated: eval_id=e1218807-780b-a852-4514-de4d93d8ad98 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:45:32.130Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=41def4d9-996e-2bee-5472-02c84304f6b1 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:45:32.130Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=41def4d9-996e-2bee-5472-02c84304f6b1 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=32dffa51-d453-ee5f-745f-c86401bcc45e policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=32dffa51-d453-ee5f-745f-c86401bcc45e policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=eaa288f3-f68b-82c1-88a0-4b6cf40c7e1e policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=eaa288f3-f68b-82c1-88a0-4b6cf40c7e1e policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=32dffa51-d453-ee5f-745f-c86401bcc45e policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=0f4d9873-2b98-52cb-58c5-05476ed63f57
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=eaa288f3-f68b-82c1-88a0-4b6cf40c7e1e policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=e3c071fa-d534-be61-1af3-f94a578fa956
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:32.256453753 +0000 UTC m=+8101.167596993 2024-03-13 20:45:32.256453753 +0000 UTC m=+8161.167596993}"
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:32.256627719 +0000 UTC m=+8101.167770929 2024-03-13 20:45:32.256627719 +0000 UTC m=+8161.167770929}"
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=32dffa51-d453-ee5f-745f-c86401bcc45e token=0f4d9873-2b98-52cb-58c5-05476ed63f57 eval_id=32dffa51-d453-ee5f-745f-c86401bcc45e token=0f4d9873-2b98-52cb-58c5-05476ed63f57
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=eaa288f3-f68b-82c1-88a0-4b6cf40c7e1e token=e3c071fa-d534-be61-1af3-f94a578fa956 eval_id=eaa288f3-f68b-82c1-88a0-4b6cf40c7e1e token=e3c071fa-d534-be61-1af3-f94a578fa956
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:32.555Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=2b5de648-9e72-4cfc-5d15-485725d042dc policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:45:32.555Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=2b5de648-9e72-4cfc-5d15-485725d042dc policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:45:32.555Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=2b5de648-9e72-4cfc-5d15-485725d042dc policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=88ca7060-a3e9-1c42-f0e1-2eec05dd222a
2024-03-13T20:45:32.555Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:45:32.555Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:32.556019757 +0000 UTC m=+8101.467162969 2024-03-13 20:45:32.556019757 +0000 UTC m=+8161.467162969}"
2024-03-13T20:45:32.557Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:32.557Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:32.557Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:32.557Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:45:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=2b5de648-9e72-4cfc-5d15-485725d042dc token=88ca7060-a3e9-1c42-f0e1-2eec05dd222a eval_id=2b5de648-9e72-4cfc-5d15-485725d042dc token=88ca7060-a3e9-1c42-f0e1-2eec05dd222a
2024-03-13T20:45:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:45:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=39c83790-3007-9c85-6593-4e0bdce06cf4 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:45:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=39c83790-3007-9c85-6593-4e0bdce06cf4 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:45:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=39c83790-3007-9c85-6593-4e0bdce06cf4 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=a915ae5b-1314-1075-1bbf-fcff50a5b73f
2024-03-13T20:45:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:45:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:32.858324679 +0000 UTC m=+8101.769467890 2024-03-13 20:45:32.858324679 +0000 UTC m=+8161.769467890}"
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=39c83790-3007-9c85-6593-4e0bdce06cf4 token=a915ae5b-1314-1075-1bbf-fcff50a5b73f eval_id=39c83790-3007-9c85-6593-4e0bdce06cf4 token=a915ae5b-1314-1075-1bbf-fcff50a5b73f
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:45:33.856Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=4b5dfa3f-a360-9ba4-fcdc-f4c3808c71ac policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:45:33.856Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=4b5dfa3f-a360-9ba4-fcdc-f4c3808c71ac policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:45:33.856Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=4b5dfa3f-a360-9ba4-fcdc-f4c3808c71ac policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=1726aa84-94f2-2fd9-380d-273e0d14e191
2024-03-13T20:45:33.856Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:45:33.856Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:45:33.856Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:45:33.856Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:44:33.856728457 +0000 UTC m=+8102.767871664 2024-03-13 20:45:33.856728457 +0000 UTC m=+8162.767871664}"
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.broker: ack eval: eval_id=4b5dfa3f-a360-9ba4-fcdc-f4c3808c71ac token=1726aa84-94f2-2fd9-380d-273e0d14e191 eval_id=4b5dfa3f-a360-9ba4-fcdc-f4c3808c71ac token=1726aa84-94f2-2fd9-380d-273e0d14e191
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:45:33.858Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=87d051e7-d7d3-c6cc-f11d-07a343ba9e30 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=87d051e7-d7d3-c6cc-f11d-07a343ba9e30 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=87d051e7-d7d3-c6cc-f11d-07a343ba9e30 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=8060b8b7-bf5f-5cf7-8a13-7202bb0964dd
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:31.356254489 +0000 UTC m=+8160.267397697 2024-03-13 20:46:31.356254489 +0000 UTC m=+8220.267397697}"
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=a56f5652-7f12-4ceb-d303-c0b36cccfffb policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=a56f5652-7f12-4ceb-d303-c0b36cccfffb policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=a56f5652-7f12-4ceb-d303-c0b36cccfffb policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=03f403be-ae22-fcb8-a145-3b9b6675ee07
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:31.356432289 +0000 UTC m=+8160.267575498 2024-03-13 20:46:31.356432289 +0000 UTC m=+8220.267575498}"
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=a56f5652-7f12-4ceb-d303-c0b36cccfffb token=03f403be-ae22-fcb8-a145-3b9b6675ee07 eval_id=a56f5652-7f12-4ceb-d303-c0b36cccfffb token=03f403be-ae22-fcb8-a145-3b9b6675ee07
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=87d051e7-d7d3-c6cc-f11d-07a343ba9e30 token=8060b8b7-bf5f-5cf7-8a13-7202bb0964dd eval_id=87d051e7-d7d3-c6cc-f11d-07a343ba9e30 token=8060b8b7-bf5f-5cf7-8a13-7202bb0964dd
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=724636c6-7ee7-6993-5299-6f5155730059 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:46:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=724636c6-7ee7-6993-5299-6f5155730059 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:46:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=724636c6-7ee7-6993-5299-6f5155730059 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=f9ed8dbb-f438-9684-a3cb-a759a78d148d
2024-03-13T20:46:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:46:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:31.541782993 +0000 UTC m=+8160.452926200 2024-03-13 20:46:31.541782993 +0000 UTC m=+8220.452926200}"
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=724636c6-7ee7-6993-5299-6f5155730059 token=f9ed8dbb-f438-9684-a3cb-a759a78d148d eval_id=724636c6-7ee7-6993-5299-6f5155730059 token=f9ed8dbb-f438-9684-a3cb-a759a78d148d
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=542ce9d5-e516-399d-c0ae-8063f50e0f69 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:46:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=542ce9d5-e516-399d-c0ae-8063f50e0f69 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:46:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=542ce9d5-e516-399d-c0ae-8063f50e0f69 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=dc52a86d-45a7-e35d-ef16-d6630b8e141d
2024-03-13T20:46:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:46:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:31.557996004 +0000 UTC m=+8160.469139217 2024-03-13 20:46:31.557996004 +0000 UTC m=+8220.469139217}"
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=542ce9d5-e516-399d-c0ae-8063f50e0f69 token=dc52a86d-45a7-e35d-ef16-d6630b8e141d eval_id=542ce9d5-e516-399d-c0ae-8063f50e0f69 token=dc52a86d-45a7-e35d-ef16-d6630b8e141d
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:32.063Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=5b9d315d-3ceb-623b-f926-adc1b0d5651f policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:46:32.063Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=5b9d315d-3ceb-623b-f926-adc1b0d5651f policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:46:32.063Z [DEBUG] policy_eval.broker: new eval is newer, policy updated: eval_id=5b9d315d-3ceb-623b-f926-adc1b0d5651f policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:46:32.137Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=34b26bd6-c63f-02d2-e03d-5f4771cb9814 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:46:32.137Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=34b26bd6-c63f-02d2-e03d-5f4771cb9814 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=fd0908dd-5cfa-6714-2a8e-1c1915e839d3 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=fd0908dd-5cfa-6714-2a8e-1c1915e839d3 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=fd0908dd-5cfa-6714-2a8e-1c1915e839d3 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=4d2676bc-c5ad-86ea-66fc-00ae3d358b11
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=a22a5f3d-04de-12c8-85f5-8579d4d0c15a policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=a22a5f3d-04de-12c8-85f5-8579d4d0c15a policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=a22a5f3d-04de-12c8-85f5-8579d4d0c15a policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=2a0aeefa-be4e-26b8-3c47-266f9e760626
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:32.255Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:32.255965314 +0000 UTC m=+8161.167108521 2024-03-13 20:46:32.255965314 +0000 UTC m=+8221.167108521}"
2024-03-13T20:46:32.255Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:32.256145513 +0000 UTC m=+8161.167288725 2024-03-13 20:46:32.256145513 +0000 UTC m=+8221.167288725}"
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.broker: ack eval: eval_id=fd0908dd-5cfa-6714-2a8e-1c1915e839d3 token=4d2676bc-c5ad-86ea-66fc-00ae3d358b11 eval_id=fd0908dd-5cfa-6714-2a8e-1c1915e839d3 token=4d2676bc-c5ad-86ea-66fc-00ae3d358b11
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.broker: ack eval: eval_id=a22a5f3d-04de-12c8-85f5-8579d4d0c15a token=2a0aeefa-be4e-26b8-3c47-266f9e760626 eval_id=a22a5f3d-04de-12c8-85f5-8579d4d0c15a token=2a0aeefa-be4e-26b8-3c47-266f9e760626
2024-03-13T20:46:32.257Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:46:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:32.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=fbd36935-f295-f6a0-e9a2-24ef2c4c6380 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:46:32.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=fbd36935-f295-f6a0-e9a2-24ef2c4c6380 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:46:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=fbd36935-f295-f6a0-e9a2-24ef2c4c6380 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=ac27a09f-ad15-7d30-50e5-00551fa6a149
2024-03-13T20:46:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:46:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:32.556785867 +0000 UTC m=+8161.467929074 2024-03-13 20:46:32.556785867 +0000 UTC m=+8221.467929074}"
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=fbd36935-f295-f6a0-e9a2-24ef2c4c6380 token=ac27a09f-ad15-7d30-50e5-00551fa6a149 eval_id=fbd36935-f295-f6a0-e9a2-24ef2c4c6380 token=ac27a09f-ad15-7d30-50e5-00551fa6a149
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:32.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=538457b3-78b9-cc5e-148a-5e380c8d8bae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:46:32.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=538457b3-78b9-cc5e-148a-5e380c8d8bae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:46:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=538457b3-78b9-cc5e-148a-5e380c8d8bae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=58b00fe4-2da4-354e-294a-4a501311f5de
2024-03-13T20:46:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:46:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:32.858097982 +0000 UTC m=+8161.769241197 2024-03-13 20:46:32.858097982 +0000 UTC m=+8221.769241197}"
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=538457b3-78b9-cc5e-148a-5e380c8d8bae token=58b00fe4-2da4-354e-294a-4a501311f5de eval_id=538457b3-78b9-cc5e-148a-5e380c8d8bae token=58b00fe4-2da4-354e-294a-4a501311f5de
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:32.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:46:33.856Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=15b3fbf3-0912-9183-0571-0992f21b89f9 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:46:33.856Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=15b3fbf3-0912-9183-0571-0992f21b89f9 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:46:33.856Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=15b3fbf3-0912-9183-0571-0992f21b89f9 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=9f0e206d-5c5e-8dd1-9873-f8f181763e74
2024-03-13T20:46:33.856Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:46:33.856Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:46:33.856Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:46:33.856Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:45:33.856710568 +0000 UTC m=+8162.767853783 2024-03-13 20:46:33.856710568 +0000 UTC m=+8222.767853783}"
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.broker: ack eval: eval_id=15b3fbf3-0912-9183-0571-0992f21b89f9 token=9f0e206d-5c5e-8dd1-9873-f8f181763e74 eval_id=15b3fbf3-0912-9183-0571-0992f21b89f9 token=9f0e206d-5c5e-8dd1-9873-f8f181763e74
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:46:33.858Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=0a29acb2-d52a-fb62-9510-3408aac7c489 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=0a29acb2-d52a-fb62-9510-3408aac7c489 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=0a29acb2-d52a-fb62-9510-3408aac7c489 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=3f66ed3a-9890-01a5-bbf2-dc09490d933d
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:31.356332117 +0000 UTC m=+8220.267475331 2024-03-13 20:47:31.356332117 +0000 UTC m=+8280.267475331}"
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=58673ef9-bd52-fd50-3671-2dd3e2f260fe policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=58673ef9-bd52-fd50-3671-2dd3e2f260fe policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=58673ef9-bd52-fd50-3671-2dd3e2f260fe policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=e7768cf8-b08b-3a7c-704a-b9674a41ef39
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:31.356567077 +0000 UTC m=+8220.267710285 2024-03-13 20:47:31.356567077 +0000 UTC m=+8280.267710285}"
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=0a29acb2-d52a-fb62-9510-3408aac7c489 token=3f66ed3a-9890-01a5-bbf2-dc09490d933d eval_id=0a29acb2-d52a-fb62-9510-3408aac7c489 token=3f66ed3a-9890-01a5-bbf2-dc09490d933d
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=58673ef9-bd52-fd50-3671-2dd3e2f260fe token=e7768cf8-b08b-3a7c-704a-b9674a41ef39 eval_id=58673ef9-bd52-fd50-3671-2dd3e2f260fe token=e7768cf8-b08b-3a7c-704a-b9674a41ef39
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:31.540Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=0e75d1b1-d228-6289-bdb5-f2b7ef3b1df9 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:47:31.540Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=0e75d1b1-d228-6289-bdb5-f2b7ef3b1df9 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:47:31.540Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=0e75d1b1-d228-6289-bdb5-f2b7ef3b1df9 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=02ce3b84-08be-b31b-bfbe-b4bd9dbb7c90
2024-03-13T20:47:31.540Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:47:31.540Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:31.540Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:31.540Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:31.540893166 +0000 UTC m=+8220.452036372 2024-03-13 20:47:31.540893166 +0000 UTC m=+8280.452036372}"
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.broker: ack eval: eval_id=0e75d1b1-d228-6289-bdb5-f2b7ef3b1df9 token=02ce3b84-08be-b31b-bfbe-b4bd9dbb7c90 eval_id=0e75d1b1-d228-6289-bdb5-f2b7ef3b1df9 token=02ce3b84-08be-b31b-bfbe-b4bd9dbb7c90
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:31.542Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=5ec76896-fc1b-8570-3a35-d3d90cbc58e9 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:47:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=5ec76896-fc1b-8570-3a35-d3d90cbc58e9 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:47:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=5ec76896-fc1b-8570-3a35-d3d90cbc58e9 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=452cba0e-d8a1-69da-17ce-84b2557faf29
2024-03-13T20:47:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:47:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:31.557807811 +0000 UTC m=+8220.468951030 2024-03-13 20:47:31.557807811 +0000 UTC m=+8280.468951030}"
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=5ec76896-fc1b-8570-3a35-d3d90cbc58e9 token=452cba0e-d8a1-69da-17ce-84b2557faf29 eval_id=5ec76896-fc1b-8570-3a35-d3d90cbc58e9 token=452cba0e-d8a1-69da-17ce-84b2557faf29
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:32.070Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e02aa143-2032-5625-4af4-9bd72650dd69 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:47:32.070Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=e02aa143-2032-5625-4af4-9bd72650dd69 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:47:32.071Z [DEBUG] policy_eval.broker: new eval is newer, policy updated: eval_id=e02aa143-2032-5625-4af4-9bd72650dd69 policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:47:32.086Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=0435d33e-db6d-97f1-4dd6-601e6d3f5c8b policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:47:32.086Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=0435d33e-db6d-97f1-4dd6-601e6d3f5c8b policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=df881434-a6a2-5ecf-83c6-1a200ddc808e policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=df881434-a6a2-5ecf-83c6-1a200ddc808e policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=cbcdef8e-693f-01c5-aa4c-32efb55ed941 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=cbcdef8e-693f-01c5-aa4c-32efb55ed941 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=df881434-a6a2-5ecf-83c6-1a200ddc808e policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=361a8e09-f4ee-eb0d-5b43-e28573bfb123
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=cbcdef8e-693f-01c5-aa4c-32efb55ed941 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=a7411619-5709-bc05-a40d-117f12a1450e
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:32.25661924 +0000 UTC m=+8221.167762453 2024-03-13 20:47:32.25661924 +0000 UTC m=+8281.167762453}"
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:32.256846654 +0000 UTC m=+8221.167989861 2024-03-13 20:47:32.256846654 +0000 UTC m=+8281.167989861}"
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=cbcdef8e-693f-01c5-aa4c-32efb55ed941 token=a7411619-5709-bc05-a40d-117f12a1450e eval_id=cbcdef8e-693f-01c5-aa4c-32efb55ed941 token=a7411619-5709-bc05-a40d-117f12a1450e
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=df881434-a6a2-5ecf-83c6-1a200ddc808e token=361a8e09-f4ee-eb0d-5b43-e28573bfb123 eval_id=df881434-a6a2-5ecf-83c6-1a200ddc808e token=361a8e09-f4ee-eb0d-5b43-e28573bfb123
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:32.555Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=09ab6728-397c-72c1-2b0d-8169b0e4abc4 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:47:32.555Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=09ab6728-397c-72c1-2b0d-8169b0e4abc4 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:47:32.555Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=09ab6728-397c-72c1-2b0d-8169b0e4abc4 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=2f6501d3-6ec5-3e5b-93f8-259e9196f88e
2024-03-13T20:47:32.555Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:47:32.555Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:32.556043411 +0000 UTC m=+8221.467186633 2024-03-13 20:47:32.556043411 +0000 UTC m=+8281.467186633}"
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.broker: ack eval: eval_id=09ab6728-397c-72c1-2b0d-8169b0e4abc4 token=2f6501d3-6ec5-3e5b-93f8-259e9196f88e eval_id=09ab6728-397c-72c1-2b0d-8169b0e4abc4 token=2f6501d3-6ec5-3e5b-93f8-259e9196f88e
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:32.557Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=972e93b2-60d2-bd4d-aebc-29806744b405 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:47:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=972e93b2-60d2-bd4d-aebc-29806744b405 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:47:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=972e93b2-60d2-bd4d-aebc-29806744b405 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=b3f445ff-0e93-27aa-02c9-899062e6213d
2024-03-13T20:47:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:47:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:32.858313009 +0000 UTC m=+8221.769456217 2024-03-13 20:47:32.858313009 +0000 UTC m=+8281.769456217}"
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=972e93b2-60d2-bd4d-aebc-29806744b405 token=b3f445ff-0e93-27aa-02c9-899062e6213d eval_id=972e93b2-60d2-bd4d-aebc-29806744b405 token=b3f445ff-0e93-27aa-02c9-899062e6213d
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:47:33.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=dfe2ff54-d444-64a6-6144-081beb7b1b88 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:47:33.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=dfe2ff54-d444-64a6-6144-081beb7b1b88 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:47:33.857Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=dfe2ff54-d444-64a6-6144-081beb7b1b88 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=fd382355-456f-208e-4956-a4fb61dbe3e0
2024-03-13T20:47:33.857Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:47:33.857Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:47:33.857Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:47:33.857Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:46:33.857317122 +0000 UTC m=+8222.768460329 2024-03-13 20:47:33.857317122 +0000 UTC m=+8282.768460329}"
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=dfe2ff54-d444-64a6-6144-081beb7b1b88 token=fd382355-456f-208e-4956-a4fb61dbe3e0 eval_id=dfe2ff54-d444-64a6-6144-081beb7b1b88 token=fd382355-456f-208e-4956-a4fb61dbe3e0
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:47:33.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=b235b5a9-3de5-9609-6b80-9994e8954061 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=b235b5a9-3de5-9609-6b80-9994e8954061 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=b235b5a9-3de5-9609-6b80-9994e8954061 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=9b7e061e-acf1-12d1-7475-085a468aa61b
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:31.356255564 +0000 UTC m=+8280.267398771 2024-03-13 20:48:31.356255564 +0000 UTC m=+8340.267398771}"
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=d8ac1d26-70b1-f328-4118-087d7e644394 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=d8ac1d26-70b1-f328-4118-087d7e644394 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=d8ac1d26-70b1-f328-4118-087d7e644394 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=28ce66ab-14da-d069-3ff0-60e5f6aa1550
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:31.356426711 +0000 UTC m=+8280.267569918 2024-03-13 20:48:31.356426711 +0000 UTC m=+8340.267569918}"
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=b235b5a9-3de5-9609-6b80-9994e8954061 token=9b7e061e-acf1-12d1-7475-085a468aa61b eval_id=b235b5a9-3de5-9609-6b80-9994e8954061 token=9b7e061e-acf1-12d1-7475-085a468aa61b
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=d8ac1d26-70b1-f328-4118-087d7e644394 token=28ce66ab-14da-d069-3ff0-60e5f6aa1550 eval_id=d8ac1d26-70b1-f328-4118-087d7e644394 token=28ce66ab-14da-d069-3ff0-60e5f6aa1550
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:31.540Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=172d374f-e325-2e20-8602-2ec270e0a81b policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:48:31.540Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=172d374f-e325-2e20-8602-2ec270e0a81b policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:48:31.540Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=172d374f-e325-2e20-8602-2ec270e0a81b policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=d9245098-32ff-7d94-5980-2fc413a183d7
2024-03-13T20:48:31.540Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:48:31.540Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:31.540Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:31.540Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:31.540949216 +0000 UTC m=+8280.452092423 2024-03-13 20:48:31.540949216 +0000 UTC m=+8340.452092423}"
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.broker: ack eval: eval_id=172d374f-e325-2e20-8602-2ec270e0a81b token=d9245098-32ff-7d94-5980-2fc413a183d7 eval_id=172d374f-e325-2e20-8602-2ec270e0a81b token=d9245098-32ff-7d94-5980-2fc413a183d7
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:31.542Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=f95434cb-d0d5-1f44-a2e5-b4a69983349e policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:48:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=f95434cb-d0d5-1f44-a2e5-b4a69983349e policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:48:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=f95434cb-d0d5-1f44-a2e5-b4a69983349e policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=60a1fc32-14b1-f916-71d5-8424b6cf05e1
2024-03-13T20:48:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:48:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:31.55786457 +0000 UTC m=+8280.469007777 2024-03-13 20:48:31.55786457 +0000 UTC m=+8340.469007777}"
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=f95434cb-d0d5-1f44-a2e5-b4a69983349e token=60a1fc32-14b1-f916-71d5-8424b6cf05e1 eval_id=f95434cb-d0d5-1f44-a2e5-b4a69983349e token=60a1fc32-14b1-f916-71d5-8424b6cf05e1
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:32.074Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=1f1d307e-aae1-3c29-6ace-12e9828e2e5c policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:48:32.074Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=1f1d307e-aae1-3c29-6ace-12e9828e2e5c policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:48:32.074Z [DEBUG] policy_eval.broker: new eval is newer, policy updated: eval_id=1f1d307e-aae1-3c29-6ace-12e9828e2e5c policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:48:32.153Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e1dbccaf-3432-cda3-55f7-e8fdcb073f2e policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:48:32.153Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=e1dbccaf-3432-cda3-55f7-e8fdcb073f2e policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=8a0bde15-a9f8-6c9a-eeb5-015343d75a49 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=8a0bde15-a9f8-6c9a-eeb5-015343d75a49 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=7b486092-3db0-08dd-cea3-ac63bf9e2f94 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=7b486092-3db0-08dd-cea3-ac63bf9e2f94 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=8a0bde15-a9f8-6c9a-eeb5-015343d75a49 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=b6d9cb35-bcf3-a129-cd10-c47ff7d5e96d
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:32.256291629 +0000 UTC m=+8281.167434837 2024-03-13 20:48:32.256291629 +0000 UTC m=+8341.167434837}"
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=7b486092-3db0-08dd-cea3-ac63bf9e2f94 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=6081c192-84ef-dd5c-b4ca-113fbf8f4bb6
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:32.256525452 +0000 UTC m=+8281.167668659 2024-03-13 20:48:32.256525452 +0000 UTC m=+8341.167668659}"
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=8a0bde15-a9f8-6c9a-eeb5-015343d75a49 token=b6d9cb35-bcf3-a129-cd10-c47ff7d5e96d eval_id=8a0bde15-a9f8-6c9a-eeb5-015343d75a49 token=b6d9cb35-bcf3-a129-cd10-c47ff7d5e96d
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=7b486092-3db0-08dd-cea3-ac63bf9e2f94 token=6081c192-84ef-dd5c-b4ca-113fbf8f4bb6 eval_id=7b486092-3db0-08dd-cea3-ac63bf9e2f94 token=6081c192-84ef-dd5c-b4ca-113fbf8f4bb6
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:32.286Z [DEBUG] policy_eval.broker: nack eval: eval_id=466a774d-b3e6-f7f8-8390-5d7c843dac8d token=3d615dd4-d3e7-7510-44dc-bb438726e0e7
2024-03-13T20:48:32.286Z [WARN]  policy_eval.broker: eval delivery limit reached: eval_id=466a774d-b3e6-f7f8-8390-5d7c843dac8d policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf token=3d615dd4-d3e7-7510-44dc-bb438726e0e7 count=1 limit=1
2024-03-13T20:48:32.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=f9a30091-4ce4-bae7-2f1d-dc1472401e05 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:48:32.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=f9a30091-4ce4-bae7-2f1d-dc1472401e05 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:48:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=f9a30091-4ce4-bae7-2f1d-dc1472401e05 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=de9cc688-87f6-e27d-abaa-01d684bd79c1
2024-03-13T20:48:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:48:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:32.556Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:32.55685086 +0000 UTC m=+8281.467994067 2024-03-13 20:48:32.55685086 +0000 UTC m=+8341.467994067}"
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=f9a30091-4ce4-bae7-2f1d-dc1472401e05 token=de9cc688-87f6-e27d-abaa-01d684bd79c1 eval_id=f9a30091-4ce4-bae7-2f1d-dc1472401e05 token=de9cc688-87f6-e27d-abaa-01d684bd79c1
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:32.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=b420f62b-122e-2ee5-69b8-bd3aa7a404d6 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:48:32.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=b420f62b-122e-2ee5-69b8-bd3aa7a404d6 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:48:32.857Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=b420f62b-122e-2ee5-69b8-bd3aa7a404d6 policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=68d299cf-526a-d039-5c12-22c3bdb5421b
2024-03-13T20:48:32.857Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:48:32.857Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:32.857Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:32.857Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:32.857988555 +0000 UTC m=+8281.769131761 2024-03-13 20:48:32.857988555 +0000 UTC m=+8341.769131761}"
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=b420f62b-122e-2ee5-69b8-bd3aa7a404d6 token=68d299cf-526a-d039-5c12-22c3bdb5421b eval_id=b420f62b-122e-2ee5-69b8-bd3aa7a404d6 token=68d299cf-526a-d039-5c12-22c3bdb5421b
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:32.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:48:33.856Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=9b304228-5040-4bfc-01d0-370de6d5013f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:48:33.856Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=9b304228-5040-4bfc-01d0-370de6d5013f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:48:33.856Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=9b304228-5040-4bfc-01d0-370de6d5013f policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=fc466c0b-ffac-fa1e-8932-d7e40ec0d3de
2024-03-13T20:48:33.856Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:48:33.856Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:48:33.856Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:48:33.856Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:47:33.856723608 +0000 UTC m=+8282.767866822 2024-03-13 20:48:33.856723608 +0000 UTC m=+8342.767866822}"
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.broker: ack eval: eval_id=9b304228-5040-4bfc-01d0-370de6d5013f token=fc466c0b-ffac-fa1e-8932-d7e40ec0d3de eval_id=9b304228-5040-4bfc-01d0-370de6d5013f token=fc466c0b-ffac-fa1e-8932-d7e40ec0d3de
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:48:33.858Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=ce325bac-ffc3-4315-f9d9-878b6ae52c19 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=ce325bac-ffc3-4315-f9d9-878b6ae52c19 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=ce325bac-ffc3-4315-f9d9-878b6ae52c19 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=93609fcd-40e9-f1bc-7cfa-53e1bb291837
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:31.356341551 +0000 UTC m=+8340.267484758 2024-03-13 20:49:31.356341551 +0000 UTC m=+8400.267484758}"
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=72089380-019e-1eae-b582-4e8d24b405c2 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=72089380-019e-1eae-b582-4e8d24b405c2 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=72089380-019e-1eae-b582-4e8d24b405c2 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=eb93a91d-6805-1b1f-3d18-6726fb35bb3d
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:31.356857929 +0000 UTC m=+8340.268001139 2024-03-13 20:49:31.356857929 +0000 UTC m=+8400.268001139}"
2024-03-13T20:49:31.359Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: ack eval: eval_id=72089380-019e-1eae-b582-4e8d24b405c2 token=eb93a91d-6805-1b1f-3d18-6726fb35bb3d eval_id=72089380-019e-1eae-b582-4e8d24b405c2 token=eb93a91d-6805-1b1f-3d18-6726fb35bb3d
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: ack eval: eval_id=ce325bac-ffc3-4315-f9d9-878b6ae52c19 token=93609fcd-40e9-f1bc-7cfa-53e1bb291837 eval_id=ce325bac-ffc3-4315-f9d9-878b6ae52c19 token=93609fcd-40e9-f1bc-7cfa-53e1bb291837
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:31.360Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e2116eb4-8aab-2360-e890-ff2d9759dc99 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:49:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=e2116eb4-8aab-2360-e890-ff2d9759dc99 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:49:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=e2116eb4-8aab-2360-e890-ff2d9759dc99 policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=0d6a35b7-8bc6-9ad7-212a-34746fc2a2ad
2024-03-13T20:49:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:49:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:31.541897926 +0000 UTC m=+8340.453041133 2024-03-13 20:49:31.541897926 +0000 UTC m=+8400.453041133}"
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a2b6c76c-5f5d-c88d-ea01-2cc867bce54a policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=e2116eb4-8aab-2360-e890-ff2d9759dc99 token=0d6a35b7-8bc6-9ad7-212a-34746fc2a2ad eval_id=e2116eb4-8aab-2360-e890-ff2d9759dc99 token=0d6a35b7-8bc6-9ad7-212a-34746fc2a2ad
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:31.557Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=89043a24-0897-e40d-dd4b-0385be0b4237 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:49:31.557Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=89043a24-0897-e40d-dd4b-0385be0b4237 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal token=""
2024-03-13T20:49:31.557Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=89043a24-0897-e40d-dd4b-0385be0b4237 policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 token=55a872a6-5b3d-644b-fbe0-459e91bb7541
2024-03-13T20:49:31.557Z [DEBUG] policy_eval.worker: received policy for evaluation: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:49:31.557Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:31.557Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:31.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:31.557709718 +0000 UTC m=+8340.468852930 2024-03-13 20:49:31.557709718 +0000 UTC m=+8400.468852930}"
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.worker: no checks need to be executed: id=b75aae33-d30b-8d77-a9d2-c84d760cd61f policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5 queue=horizontal target=nomad-target
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.broker: ack eval: eval_id=89043a24-0897-e40d-dd4b-0385be0b4237 token=55a872a6-5b3d-644b-fbe0-459e91bb7541 eval_id=89043a24-0897-e40d-dd4b-0385be0b4237 token=55a872a6-5b3d-644b-fbe0-459e91bb7541
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=00aaf8a1-d731-c099-ce75-9ae3614dcec5
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:31.559Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:32.089Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=62460468-ffdd-b270-3314-d7f50fb5086c policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:49:32.089Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=62460468-ffdd-b270-3314-d7f50fb5086c policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:49:32.089Z [DEBUG] policy_eval.broker: new eval is newer, policy updated: eval_id=62460468-ffdd-b270-3314-d7f50fb5086c policy_id=b12860f1-8e3f-def5-3277-4d4313fd3cf4 queue=cluster token=""
2024-03-13T20:49:32.136Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=565b68c6-a073-ba44-f74b-5e4503b1eea5 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:49:32.136Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=565b68c6-a073-ba44-f74b-5e4503b1eea5 policy_id=580ba51a-5277-71a7-0bc9-e8319b7110bf queue=cluster token=""
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=4a266b2c-fc22-bcf5-57bc-3bd56b666267 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=4a266b2c-fc22-bcf5-57bc-3bd56b666267 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal token=""
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e75c078b-870f-29e1-3ce6-9de5fe2aa737 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=e75c078b-870f-29e1-3ce6-9de5fe2aa737 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal token=""
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=e75c078b-870f-29e1-3ce6-9de5fe2aa737 policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 token=5c5fb418-37df-77ef-1875-ff91b4d43f8d
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=4a266b2c-fc22-bcf5-57bc-3bd56b666267 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 token=4f1b6555-3df3-0fb8-612b-9c5d20e6bc22
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:32.256426221 +0000 UTC m=+8341.167569428 2024-03-13 20:49:32.256426221 +0000 UTC m=+8401.167569428}"
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:32.256Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:32.256Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-a8b1\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:32.256613652 +0000 UTC m=+8341.167756858 2024-03-13 20:49:32.256613652 +0000 UTC m=+8401.167756858}"
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=ad6239ce-aec1-a6dd-9132-096e4de54aa2 policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179 queue=horizontal target=nomad-target
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=4a266b2c-fc22-bcf5-57bc-3bd56b666267 token=4f1b6555-3df3-0fb8-612b-9c5d20e6bc22 eval_id=4a266b2c-fc22-bcf5-57bc-3bd56b666267 token=4f1b6555-3df3-0fb8-612b-9c5d20e6bc22
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c02455d-c262-cb5a-15e9-ea52d6827179
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.worker: no checks need to be executed: id=498f9ae4-2550-3c30-0fa2-cdc42e1b461d policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52 queue=horizontal target=nomad-target
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: ack eval: eval_id=e75c078b-870f-29e1-3ce6-9de5fe2aa737 token=5c5fb418-37df-77ef-1875-ff91b4d43f8d eval_id=e75c078b-870f-29e1-3ce6-9de5fe2aa737 token=5c5fb418-37df-77ef-1875-ff91b4d43f8d
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=4225ef0a-9c2f-9f46-cca6-d40577117a52
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:32.258Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:32.556Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=d13a1a7d-9910-9664-5ab8-420bd80ddd93 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:49:32.556Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=d13a1a7d-9910-9664-5ab8-420bd80ddd93 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal token=""
2024-03-13T20:49:32.556Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=d13a1a7d-9910-9664-5ab8-420bd80ddd93 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e token=8318ed8a-9762-953a-f86c-423da09c34e9
2024-03-13T20:49:32.556Z [DEBUG] policy_eval.worker: received policy for evaluation: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:49:32.556Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:32.556Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:32.557Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:32.557000797 +0000 UTC m=+8341.468144006 2024-03-13 20:49:32.557000797 +0000 UTC m=+8401.468144006}"
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.worker: no checks need to be executed: id=a7c3f88b-763d-6033-e894-24d703791100 policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e queue=horizontal target=nomad-target
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.broker: ack eval: eval_id=d13a1a7d-9910-9664-5ab8-420bd80ddd93 token=8318ed8a-9762-953a-f86c-423da09c34e9 eval_id=d13a1a7d-9910-9664-5ab8-420bd80ddd93 token=8318ed8a-9762-953a-f86c-423da09c34e9
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=0c3bb192-6669-19ea-ef5b-36498c73814e
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:32.558Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:32.858Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=c01e12ee-9b79-9c0b-b192-c4b0dca6045e policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:49:32.858Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=c01e12ee-9b79-9c0b-b192-c4b0dca6045e policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal token=""
2024-03-13T20:49:32.858Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=c01e12ee-9b79-9c0b-b192-c4b0dca6045e policy_id=41431c7f-4405-eaff-31b6-17779270fd57 token=4c736f60-5c39-9a4f-6fd2-07567ad8c4b8
2024-03-13T20:49:32.858Z [DEBUG] policy_eval.worker: received policy for evaluation: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:49:32.858Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:32.858Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:32.858Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:32.858239541 +0000 UTC m=+8341.769382748 2024-03-13 20:49:32.858239541 +0000 UTC m=+8401.769382748}"
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.worker: no checks need to be executed: id=7689c587-94de-7a4b-6815-97e4aaf9d5cd policy_id=41431c7f-4405-eaff-31b6-17779270fd57 queue=horizontal target=nomad-target
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.broker: ack eval: eval_id=c01e12ee-9b79-9c0b-b192-c4b0dca6045e token=4c736f60-5c39-9a4f-6fd2-07567ad8c4b8 eval_id=c01e12ee-9b79-9c0b-b192-c4b0dca6045e token=4c736f60-5c39-9a4f-6fd2-07567ad8c4b8
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=41431c7f-4405-eaff-31b6-17779270fd57
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:32.860Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:49:33.857Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=31452e7f-5cd8-cd66-3074-1f6a13bd23fd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:49:33.857Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=31452e7f-5cd8-cd66-3074-1f6a13bd23fd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal token=""
2024-03-13T20:49:33.857Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=31452e7f-5cd8-cd66-3074-1f6a13bd23fd policy_id=bdd8dabf-945e-6653-af27-079c631ff689 token=5ede03f7-89af-cd79-8a77-78b139ffee83
2024-03-13T20:49:33.857Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:49:33.857Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:49:33.857Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:49:33.857Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:48:33.857481524 +0000 UTC m=+8342.768624740 2024-03-13 20:49:33.857481524 +0000 UTC m=+8402.768624740}"
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5cac5d14-0abc-3ada-2c2e-65d5b17023ae policy_id=bdd8dabf-945e-6653-af27-079c631ff689 queue=horizontal target=nomad-target
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.broker: ack eval: eval_id=31452e7f-5cd8-cd66-3074-1f6a13bd23fd token=5ede03f7-89af-cd79-8a77-78b139ffee83 eval_id=31452e7f-5cd8-cd66-3074-1f6a13bd23fd token=5ede03f7-89af-cd79-8a77-78b139ffee83
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=bdd8dabf-945e-6653-af27-079c631ff689
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:49:33.859Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:50:31.104Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=fe7672ba-f825-0898-e358-70172e92b330 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=fe7672ba-f825-0898-e358-70172e92b330 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal token=""
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=fe7672ba-f825-0898-e358-70172e92b330 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 token=9b133ab9-234f-ae82-cac6-74a3cea0c118
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:50:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-website-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:49:31.356251963 +0000 UTC m=+8400.267395177 2024-03-13 20:50:31.356251963 +0000 UTC m=+8460.267395177}"
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=ac0bf68f-23ea-bfe3-e413-0817dae63717 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=ac0bf68f-23ea-bfe3-e413-0817dae63717 policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal token=""
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=ac0bf68f-23ea-bfe3-e413-0817dae63717 policy_id=c71b428e-71b1-e936-4893-0349309b4083 token=e372865a-7884-8d4a-194f-213dc2ce819d
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.worker: received policy for evaluation: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:50:31.356Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:50:31.356Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webservices-e425\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:49:31.356655617 +0000 UTC m=+8400.267798826 2024-03-13 20:50:31.356655617 +0000 UTC m=+8460.267798826}"
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=8c7ccbef-db69-fc19-c3a6-678da7c6248c policy_id=c71b428e-71b1-e936-4893-0349309b4083 queue=horizontal target=nomad-target
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=ac0bf68f-23ea-bfe3-e413-0817dae63717 token=e372865a-7884-8d4a-194f-213dc2ce819d eval_id=ac0bf68f-23ea-bfe3-e413-0817dae63717 token=e372865a-7884-8d4a-194f-213dc2ce819d
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=c71b428e-71b1-e936-4893-0349309b4083
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e77afde5-f134-b478-8027-fb0ba40db793 policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247 queue=horizontal target=nomad-target
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: ack eval: eval_id=fe7672ba-f825-0898-e358-70172e92b330 token=9b133ab9-234f-ae82-cac6-74a3cea0c118 eval_id=fe7672ba-f825-0898-e358-70172e92b330 token=9b133ab9-234f-ae82-cac6-74a3cea0c118
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=31ee5b4e-ba50-f0c0-baef-0e0b11e76247
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:50:31.358Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal
2024-03-13T20:50:31.541Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=729cfa46-6642-8878-ab0b-c271c50e13cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:50:31.541Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=729cfa46-6642-8878-ab0b-c271c50e13cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal token=""
2024-03-13T20:50:31.541Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=729cfa46-6642-8878-ab0b-c271c50e13cd policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 token=498ac619-ba2f-21ff-34e1-aae418615327
2024-03-13T20:50:31.541Z [DEBUG] policy_eval.worker: received policy for evaluation: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:50:31.541Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target
2024-03-13T20:50:31.541Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" source=prometheus
2024-03-13T20:50:31.541Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="avg(avg_over_time(nomad_client_allocs_cpu_total_percent{exported_job=\"sdi-webapi-17e3\", task=\"win_iis\", namespace=\"sdi-production\"}[5m]))" range="{2024-03-13 20:49:31.54191825 +0000 UTC m=+8400.453061462 2024-03-13 20:50:31.54191825 +0000 UTC m=+8460.453061462}"
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=3
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_usage id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal source=prometheus strategy=target-value target=nomad-target from=3 to=3
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.worker: check cpu_usage selected in group : id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target direction=none count=0
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=6d4921c1-4d84-7b8d-f85a-980cbd9700aa policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2 queue=horizontal target=nomad-target
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.broker: ack eval: eval_id=729cfa46-6642-8878-ab0b-c271c50e13cd token=498ac619-ba2f-21ff-34e1-aae418615327 eval_id=729cfa46-6642-8878-ab0b-c271c50e13cd token=498ac619-ba2f-21ff-34e1-aae418615327
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=6622fa4a-1e20-daed-bec8-d194ecfe4da2
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal
2024-03-13T20:50:31.543Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal