2 kinds of errors for CFME 5820 with OCP 3.7
root@CFME-SCALE-OCP-5820-DB01 log]# grep ERROR evm.log | awk -F'ERROR' '{print $2}' | sort | uniq
-- : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::MetricsCapture#perf_collect_metrics) Hawkular metrics service unavailable: <html>
-- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#fetch_hawk_inv) <html>
Tried revalidating with new tokens but in vain..
[----] I, [2017-09-29T00:47:47.378297 #20730:54b140] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [10.12.70.26] as [] Started Event Monitor Thread
[----] E, [2017-09-29T00:47:47.388398 #20730:69d0534] ERROR -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [10.12.70.26] as [] Event Monitor Thread aborted because [429 Too Many Requests]
[----] E, [2017-09-29T00:47:47.388514 #20730:69d0534] ERROR -- : [KubeException]: 429 Too Many Requests Method:[rescue in block in start_event_monitor]
[----] E, [2017-09-29T00:47:47.388600 #20730:69d0534] ERROR -- : /opt/rh/cfme-gemset/gems/kubeclient-2.4.0/lib/kubeclient/common.rb:117:in `rescue in handle_exception'
/opt/rh/cfme-gemset/gems/kubeclient-2.4.0/lib/kubeclient/common.rb:109:in `handle_exception'
/opt/rh/cfme-gemset/gems/kubeclient-2.4.0/lib/kubeclient/common.rb:461:in `fetch_entities'
/opt/rh/cfme-gemset/gems/kubeclient-2.4.0/lib/kubeclient/common.rb:451:in `load_entities'
/opt/rh/cfme-gemset/gems/kubeclient-2.4.0/lib/kubeclient/common.rb:121:in `discover'
/opt/rh/cfme-gemset/gems/kubeclient-2.4.0/lib/kubeclient/common.rb:88:in `method_missing'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-0fb020cfec87/lib/gems/pending/kubernetes/events/kubernetes_event_monitor.rb:32:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/kubernetes/container_manager/event_catcher_mixin.rb:41:in `monitor_events'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:164:in `block in start_event_monitor'
[----] I, [2017-09-29T00:47:47.521101 #18819:54b140] INFO -- : MIQ(BottleneckEvent.generate_future_events) Generating future bottleneck events for: [MiqEnterprise - Enterprise]... Complete - Added 0 / Deleted 0
[----] I, [2017-09-29T04:46:29.268962 #2633:54b140] INFO -- : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::MetricsCapture#perf_collect_metrics) Collecting metrics for ContainerNode(1634) [realtime] [2017-09-29 07:52:38 UTC] []
[----] E, [2017-09-29T04:46:29.287103 #2633:54b140] ERROR -- : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::MetricsCapture#perf_collect_metrics) Hawkular metrics service unavailable: <html>
.....
.....
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>