Upgrade OpenShift from 4.8 to 4.9

Red Hat OpenShift Container Platform is a powerful platform created to provide IT organizations and developers with a hybrid cloud application platform. With this secure and scalable platform, you’re able deploy containerized applications with minimal configuration and management overhead. In this article we look at how you can perform an upgrade from OpenShift 4.8 To OpenShift 4.9.
OpenShift Container Platform 4.9 uses Kubernetes 1.22, which removed a significant number of deprecated v1beta1 APIs.

OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All clusters require this administrator acknowledgment(till ocp 4.8.14) before they can be upgraded to OpenShift Container Platform 4.9.

Removed Kubernetes APIs

Kubernetes 1.22 removed the following deprecated v1beta1 APIs. If your cluster, or any idle workloads or tools use any of these APIs, you must migrate them to the appropriate new version before upgrading to OpenShift Container Platform 4.9.

Evaluate OpenShift 4.8 Cluster for removed APIs

It is a responsibility of OpenShift/K8s Administrator to properly evaluate all Workloads and other integrations to identify where APIs removed in Kubernetes 1.22 are in use.

Ensure you are on OpenShift Cluster 4.8 before trying to perform an upgrade to 4.9. The following commands helps you identify OCP release

$ oc get clusterversions
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.14    True        False         6h45m   Cluster version is 4.8.14 

Reviewing alerts to identify uses of APIs that will be removed

OpenShift Container Platform 4.8 introduced two new alerts that fire when an API is in use that will be removed in the next release:

• APIRemovedInNextReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform release.
• APIRemovedInNextEUSReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform Extended Update Support (EUS) release.

If either of these alerts are firing in your cluster, review the alerts and take action to clear the alerts by migrating manifests and API clients to use the new API version. You can use the APIRequestCount API to get more information about which APIs are in use and which workloads are using APIs that will be removed. Additionally, some APIs might not trigger the above alerts, yet may still be captured by APIRequestCount

Using APIRequestCount to identify uses of APIs that will be removed

You can use the APIRequestCount API to track API requests and review whether any of them are using one of the removed APIs.

$ oc get apirequestcounts

Example output:

NAME                                        REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cloudcredentials.v1.operator.openshift.io                      32                      111
ingresses.v1.networking.k8s.io                                 28                      110
ingresses.v1beta1.extensions                1.22               16                      66
ingresses.v1beta1.networking.k8s.io         1.22               0                       1
installplans.v1alpha1.operators.coreos.com                     93                      167
...

Using APIRequestCount to identify which workloads are using the APIs that will be removed

You can examine the APIRequestCount resource for a given API version to help identify which workloads are using the API.

Run the following command and examine the username and userAgent fields to help identify the workloads that are using the API:

$ oc get apirequestcounts <resource>.<version>.<group> -o yaml

For example:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o yaml

Example output:

VERBS  USERNAME                        USERAGENT
watch  bob                             oc/v4.8.11
watch  system:kube-controller-manager  cluster-policy-controller/v0.0.0

Acknowledge Upgrade to OpenShift Container Platform 4.9

After the evaluation and migration of removed Kubernetes APIs to v1 is complete, as an OpenShift Administrator you can provide the acknowledgment required to proceed.

WARNING: Be aware that all responsibility falls on the administrator to ensure that all uses of removed APIs have been resolved and migrated as necessary before providing this administrator acknowledgment. OpenShift Container Platform can assist with the evaluation, but cannot identify all possible uses of removed APIs, especially idle workloads or external tools.

To acknowledge that you have completed the evaluation and your cluster is ready to upgrade to OpenShift Container Platform 4.9, run the following command:

$ oc -n openshift-config patch cm admin-acks --patch '{"data":{"ack-4.8-kube-1.22-api-removals-in-4.9":"true"}}' --type=merge

Note: You must have access to the cluster as a user with the cluster-admin role in order to run this command.

Begin Upgrade from OpenShift 4.8 To 4.9

Login to OpenShift Web Console and navigate to Administration → Cluster Settings > Details

Fig: 01

Click on “Channel” and update channel to fast-4.9 or stable-4.9.

Select fast-4.9 orstable-4.9 from the list of available channels.

Channel can also be changed from the command line with below command syntax:

oc adm upgrade channel clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "<channel>”}]'

Where <channel> is replaced with either;

• stable-4.9
• fast-4.9
• candidate-4.9

New cluster updates should be visible now. Use the “Update” link to initiate upgrade from OpenShift 4.8 to OpenShift 4.9.

Select the new version of OpenShift 4.9 that you’ll be upgrading to.

The update process to OpenShift Container Platform 4.9 should begin shortly.

You can also check upgrade status from CLI:

$ oc get clusterversions.config.openshift.io
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.2    True        True          1m26s   Working towards 4.9.0:

Output once all the upgrades are completed. You now have OpenShift cluster version 4.9.

List all the available MachineHealthCheck to ensure everything is in healthy state.

$ oc get machinehealthcheck -n openshift-machine-api
NAME                              MAXUNHEALTHY   EXPECTEDMACHINES   CURRENTHEALTHY
machine-api-termination-handler   100%

Check cluster components:

$ oc get cs
W1110 20:39:28.838732 2592723 warnings.go:70] v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-3               Healthy   {"health":"true","reason":""}
etcd-1               Healthy   {"health":"true","reason":""}
etcd-0               Healthy   {"health":"true","reason":""}
etcd-2               Healthy   {"health":"true","reason":""}

List all cluster operators and review current versions.

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.9.5     True        False         False      32h
baremetal                                  4.9.5     True        False         False      84d
cloud-controller-manager                   4.9.5     True        False         False      20d
cloud-credential                           4.9.5     True        False         False      84d
cluster-autoscaler                         4.9.5     True        False         False      84d
config-operator                            4.9.5     True        False         False      84d
console                                    4.9.5     True        False         False      37d
csi-snapshot-controller                    4.9.5     True        False         False      84d
dns                                        4.9.5     True        False         False      84d
etcd                                       4.9.5     True        False         False      84d
image-registry                             4.9.5     True        False         False      84d
ingress                                    4.9.5     True        False         False      84d
insights                                   4.9.5     True        False         False      84d
kube-apiserver                             4.9.5     True        False         False      84d
kube-controller-manager                    4.9.5     True        False         False      84d
kube-scheduler                             4.9.5     True        False         False      84d
kube-storage-version-migrator              4.9.5     True        False         False      8d
machine-api                                4.9.5     True        False         False      84d
machine-approver                           4.9.5     True        False         False      84d
machine-config                             4.9.5     True        False         False      8d
marketplace                                4.9.5     True        False         False      84d
monitoring                                 4.9.5     True        False         False      20d
network                                    4.9.5     True        False         False      84d
node-tuning                                4.9.5     True        False         False      8d
openshift-apiserver                        4.9.5     True        False         False      20d
openshift-controller-manager               4.9.5     True        False         False      7d8h
openshift-samples                          4.9.5     True        False         False      8d
operator-lifecycle-manager                 4.9.5     True        False         False      84d
operator-lifecycle-manager-catalog         4.9.5     True        False         False      84d
operator-lifecycle-manager-packageserver   4.9.5     True        False         False      20d
service-ca                                 4.9.5     True        False         False      84d
storage                                    4.9.5     True        False

Check if all nodes are available and in healthy state.

$ oc get nodes

Summary

In this blog post you will be able to perform an upgrade of OpenShift from version 4.8 to 4.9. Ensure all Operators previously installed through Operator Lifecycle Manager (OLM) are updated to their latest version in their latest channel as selected during upgrade. Also ensure that all machine config pools (MCPs) are running and not paused.

Note: This blog is completely based on my experience on cluster upgrade and the environment which I am working on. Few steps may vary according to your setup.

Note: Upgrade process will be different for disconnected clusters.

Happy reading….