Skip to content

Instantly share code, notes, and snippets.

@tyrell
Last active September 29, 2020 19:16
Show Gist options
  • Star 6 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save tyrell/5d0fe1003551dbe051f2a64bd51a9c08 to your computer and use it in GitHub Desktop.
Save tyrell/5d0fe1003551dbe051f2a64bd51a9c08 to your computer and use it in GitHub Desktop.
Running Minishift with Hawkular Metrics Installed

Introduction

A minishift openshift cluster can be created with cluster metrics installed. Although I couldn't find a way to add metrics to an already created minishift VM instance, the steps below helped to get a new instance up and running easily.

Steps

Step 1 - Initialise minishift with metrics.

Create a fresh minishift VM with an openshift cluster using the below command. The --metrics parameter is the only additional thing I had to provide for minishift to start installing cluster metrics and Hawkular.

 $minishift start —metrics

Once minishift startup completes, you will see the following log message on screen.

      -- Server Information ...
               OpenShift server started.
               The server is accessible via web console at:
               https://192.168.64.2:8443

               The metrics service is available at:
               https://metrics-openshift-infra.192.168.64.2.nip.io

               You are logged in as:
                   User:     developer
                   Password: developer

               To login as administrator:
                   oc login -u system:admin

Note the metrics URL listed after the Openshift console URL. This indicates a successful initiation of cluster metrics and Hawkular.

However, depending on the speed of your Internet connection and the power of your local machine, it can be a while before you can actually browse to the above URL and see the Hawkular information page.

Step 2 - Wait for the metrics deployer to initiate other dependent pods.

Hawkular metrics pods will be deployed within the openshift-infra project. Run the following commands to keep track of pod creation progress.

Login as cluster admin

      $oc login -u system:admin

List all pods in the openshift-infra project

      $oc get pods -n openshift-infra

You should see the following metrics pods running.

    NAME                            READY     STATUS    RESTARTS   AGE
    hawkular-cassandra-1-zbwkg      1/1       Running       0      42m
    hawkular-metrics-zf58x          1/1       Running       0      42m
    heapster-x3iqg                  1/1       Running       0      42m
    metrics-deployer-pod            0/1       Error         0      53m    <— This is the deployer pod. Not required to be Running.

As you can see from the table above, it took approximately 42 minutes for the first 3 pods to pull their respective Docker images and eventually reach a running state. The 4th pod is the one orchestrating the installation. It errors out due to a verification script waiting too long. But this did not affect the metrics installation or runtime.

I used the following command to confirm the state of the metrics-deployer-pod

      $oc logs -f metrics-deployer-pod

If you look at the last log output from the command above, you will see something similar to that of below. It looks like the long running script times out while waiting for other pods to report a successful deployment. I don't think this breaks the metrics installation, as the 3 pods that makes up Hawkular metrics eventually reaches a running state.

      --- validate_deployment_artifacts ---
      ======== ERROR =========
      validate_deployment_artifacts:
      Pod hawkular-cassandra-1-zbwkg from ReplicationController hawkular-cassandra-1 is in a Pending state.
      This is most often due to waiting for the container image to pull and should eventually resolve.
        * * * *
      Pod hawkular-metrics-zf58x from ReplicationController hawkular-metrics is in a Pending state.
      This is most often due to waiting for the container image to pull and should eventually resolve.
        * * * *
      Pod heapster-x3iqg from ReplicationController heapster specified an image that cannot be pulled.
      ERROR: This is most often due to the image name being wrong or the docker registry being unavailable.
      Ensure that you used the correct IMAGE_PREFIX and IMAGE_VERSION with the deployment.
      There was an event for this pod with the following message:
      Failed to pull image "openshift/origin-metrics-heapster:v1.4.1": net/http: request canceled
        * * * *
      ========================
      --- validate_deployed_project ---

      VALIDATION FAILED

Step 3 - Verifying your Hawkular metrics installation.

When the first 3 pods are running and ready, you should be able to browse to https://metrics-openshift-infra.192.168.64.2.nip.io (IP will be different) and see a page similar to that of the screenshot hawkular-metrics-installed.png

After confirming above, you can login to your Openshift console and create a sample application. I used my Django example from one of my previous gists at https://gist.github.com/tyrell/8552a8761d98a67f9dbb1d808df35734 for this purpose.

When your application is initialised, you will see a new metrics sections in the pod overview (hawkular-metrics-installed-pod-metrics-overview.png) and also a new tab in the pod details page (hawkular-metrics-installed-pod-metrics-detail.png).

Notes

  1. I noted that minishift has mentioned metrics as an experimental feature. So it may be flaky.
  2. Metrics storage seems to be in-memory and not persisted.
  3. I didn't see any numbers appear in the metrics UI sections mentioned above. This may be due to a number of reasons. But from a proof of concept perspective, I was happy with what I ended up with after Step 3.
  4. I will have to get metrics recording fixed in order to experiment with Grafana http://www.hawkular.org/blog/2016/10/24/hawkular-metrics-openshift-and-grafana.html. But that's for another day.

References

  1. https://minishift.io/docs/minishift_start.html
  2. https://github.com/openshift/origin-metrics
  3. https://docs.openshift.org/latest/install_config/cluster_metrics.html

License

Copyright (c) 2017 Tyrell Perera tyrell.perera@gmail.com Licensed under the MIT license.

@matthewpdias
Copy link

Did you mean $minishift start --metrics in the first snippet as opposed to $minishift start —metrics? (looks like the two dashes got formatted into one m-dash) Otherwise thanks for the great resource!

@phlechev
Copy link

phlechev commented Feb 6, 2018

Hi, thanks for the article!

I am trying to do the same but when I check the pods in openshift-infra, I get this:

oc get pods -n openshift-infra
NAME READY STATUS RESTARTS AGE
openshift-ansible-metrics-job-2991b 0/1 Error 0 3m
openshift-ansible-metrics-job-6rkpw 0/1 Error 0 1m
openshift-ansible-metrics-job-7smng 0/1 Error 0 2m
openshift-ansible-metrics-job-8qxb2 0/1 Error 0 2m
openshift-ansible-metrics-job-fngrm 0/1 Error 0 2m
openshift-ansible-metrics-job-fnqnf 0/1 Error 0 1m
openshift-ansible-metrics-job-fwxxj 0/1 Error 0 3m
openshift-ansible-metrics-job-gzbpp 0/1 Error 0 5m
openshift-ansible-metrics-job-j48fr 0/1 Error 0 1m
openshift-ansible-metrics-job-kh8rz 0/1 Error 0 2m
openshift-ansible-metrics-job-m4x7s 0/1 Error 0 4m
openshift-ansible-metrics-job-r9sb8 0/1 Error 0 3m
openshift-ansible-metrics-job-s402l 0/1 Error 0 3m
openshift-ansible-metrics-job-tln99 0/1 Error 0 3m
openshift-ansible-metrics-job-w7sjh 0/1 Error 0 1m
openshift-ansible-metrics-job-x746z 0/1 Error 0 1m

I suspect the problem is that I am behind a corporate proxy.
This is my setup:
I am starting minishift like this:

minishift start --http-proxy=http://10.0.2.2:3128 --https-proxy=https://10.0.2.2:3128 --vm-driver=virtualbox --metrics
(On my laptop I have Cntlm autehntication proxy configured and running )

Then I can see the following:
-- Checking for IP address ... OK
-- Checking if external host is reachable from the Minishift VM ...
Pinging 8.8.8.8 ... FAIL
VM is unable to ping external host
-- Checking HTTP connectivity from the VM ...
Retrieving http://minishift.io/index.html ... FAIL
VM cannot connect to external URL with HTTP

The Minishift VM can pull docker images without problems. The only thing that I have to do when building docker images is that I have to pass the proxy to the docker build command:

docker build --build-arg http_proxy=http://10.0.2.2:3128 --build-arg https_proxy=https://10.0.2.2:3128 -t <image_name> .

I assume I need to pass somehow the proxy settings to the process installing the mertrics applications but I don't know how.

Any ideas?

@ricardozanini
Copy link

Maybe you're running on this issue:
openshift/openshift-ansible#5756

I'll try to work on the workaround suggested and I'll post my results as soon as I managed to get it resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment