Custom Grafana dashboards for Red Hat OpenShift Container Platform 4.x …

Kamlesh Prajapati
4 min readSep 12, 2021

Overview:
OpenShift Container Platform ships with a pre-configured and self-updating monitoring stack that is based on the Prometheus open source project and its wider eco-system. The Prometheus Operator (PO) creates, configures, and manages Prometheus and Alertmanager instances.

OpenShift administrators often face the same challenges as other system administrators: “I need a tool that will monitor the health of my machine.” Yet, traditional monitoring tools often fall short in their visibility of an OpenShift cluster. Thus, a typical OpenShift monitoring stack includes Prometheus for monitoring both machines and services, and Grafana for analyzing and visualizing metrics.

Administrators are often looking to write custom queries(prom query) and create custom dashboards in Grafana. However, Grafana instances provided with the monitoring stack (and its dashboards) are read-only mode. To solve this problem, we can use the community-powered Grafana operator provided by OperatorHub.

Note: Community Operators are operators which have not been verified by Red Hat. Community Operators should be used with caution because their stability is unknown. Red Hat provides no support for Community Operators.

Deploying Custom Grafana

The community-powered Grafana cannot be deployed to the existing openshift-monitoring namespace, so we will create a new namespace (e.g. my-grafana) to deploy into instead. Navigate to OperatorHub and select the community-powered Grafana Operator. Press Continue to accept the disclaimer, press Install, and press Subscribe to accept the default configuration values and deploy to the my-grafana namespace. Within some time, the Grafana operator will be made available in the my-grafana namespace.

From Installed Operators, select the Grafana Operator. For the Grafana resource, press Create Instance to create a new Grafana instance.

In the Grafana instance YAML, make a note of the default username and password to log in, and press Create.

Connecting Prometheus to Custom Grafana

The next step is to connect the community supported Grafana in the my-grafana namespace to OpenShift monitoring in the openshift-monitoring namespace.

The grafana-serviceaccount service account was created alongside the Grafana instance. We will grant it the cluster-monitoring-view cluster role.

$oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-serviceaccount

The bearer token for this service account is used to authenticate access to Prometheus in the openshift-monitoring namespace. The following command will display this token.

$oc serviceaccounts get-token grafana-serviceaccount -n my-grafana

From the Grafana Data Source resource, press Create Instance, and navigate to the YAML view. In the below YAML, substitute ${BEARER_TOKEN} with the output of the command above, copy the YAML, and press Create.

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
name: prometheus-grafanadatasource
namespace: my-grafana
spec:
datasources:
- access: proxy
editable: true
isDefault: true
jsonData:
httpHeaderName1: 'Authorization'
timeInterval: 10s
tlsSkipVerify: true
name: Prometheus
secureJsonData:
httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
type: prometheus
url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
name: prometheus-grafanadatasource.yaml

Customizing Grafana Dashboard

From the my-grafana namespace, navigate to Networking -> Routes and click on the Grafana <URL> to display the custom Grafana user interface. Click on ‘Sign In’ from the bottom left menu of Grafana, and log in using the default username and password configured earlier. Now, an editable Grafana interface appears and you can view your custom Grafana dashboards or create your own. As a note, administrators should be careful with custom dashboards to query Prometheus as this will have an impact on the performance of the monitoring stack.

To import an existing Grafana dashboard, you can navigate from the Grafana operator menu and create a Grafana Dashboard resource. An alternative is to directly import a custom Grafana dashboard from a JSON file within Grafana. In the screenshot below, I imported a custom Grafana dashboard which displayed the custom metrics I had been looking to view.

Summary..

With the community-powered Grafana, an OpenShift administrator can write their own Prometheus query expression extract metrics and create custom dashboards to visualize the data in grafana.

Happy learning…. :)

--

--

Kamlesh Prajapati

DevOps Practitioner (CKA certified , RHOCP Certified, Azure Certified on az-104,az-400,az-303.)