Upgrade OpenShift from 4.8 to 4.9

Red Hat OpenShift Container Platform is a powerful platform created to provide IT organizations and developers with a hybrid cloud application platform. With this secure and scalable platform, you’re able deploy containerized applications with minimal configuration and management overhead. In this article we look at how you can perform an upgrade from OpenShift 4.8 To OpenShift 4.9.
OpenShift Container Platform 4.9 uses Kubernetes 1.22, which removed a significant number of deprecated v1beta1 APIs.

OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All clusters require this administrator acknowledgment(till ocp 4.8.14) before they can be upgraded to OpenShift Container Platform 4.9.

Removed Kubernetes APIs

Kubernetes 1.22 removed the following deprecated v1beta1 APIs. If your cluster, or any idle workloads or tools use any of these APIs, you must migrate them to the appropriate new version before upgrading to OpenShift Container Platform 4.9.

Evaluate OpenShift 4.8 Cluster for removed APIs

It is a responsibility of OpenShift/K8s Administrator to properly evaluate all Workloads and other integrations to identify where APIs removed in Kubernetes 1.22 are in use.

Ensure you are on OpenShift Cluster 4.8 before trying to perform an upgrade to 4.9. The following commands helps you identify OCP release

$ oc get clusterversions
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.14 True False 6h45m Cluster version is 4.8.14

Reviewing alerts to identify uses of APIs that will be removed

OpenShift Container Platform 4.8 introduced two new alerts that fire when an API is in use that will be removed in the next release:

  • APIRemovedInNextReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform release.
  • APIRemovedInNextEUSReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform Extended Update Support (EUS) release.

If either of these alerts are firing in your cluster, review the alerts and take action to clear the alerts by migrating manifests and API clients to use the new API version. You can use the APIRequestCount API to get more information about which APIs are in use and which workloads are using APIs that will be removed. Additionally, some APIs might not trigger the above alerts, yet may still be captured by APIRequestCount

Using APIRequestCount to identify uses of APIs that will be removed

You can use the APIRequestCount API to track API requests and review whether any of them are using one of the removed APIs.

$ oc get apirequestcounts

Example output:

NAME                                        REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cloudcredentials.v1.operator.openshift.io 32 111
ingresses.v1.networking.k8s.io 28 110
ingresses.v1beta1.extensions 1.22 16 66
ingresses.v1beta1.networking.k8s.io 1.22 0 1
installplans.v1alpha1.operators.coreos.com 93 167
...

Using APIRequestCount to identify which workloads are using the APIs that will be removed

You can examine the APIRequestCount resource for a given API version to help identify which workloads are using the API.

Run the following command and examine the username and userAgent fields to help identify the workloads that are using the API:

$ oc get apirequestcounts <resource>.<version>.<group> -o yaml

For example:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o yaml

Example output:

VERBS  USERNAME                        USERAGENT
watch bob oc/v4.8.11
watch system:kube-controller-manager cluster-policy-controller/v0.0.0

Acknowledge Upgrade to OpenShift Container Platform 4.9

After the evaluation and migration of removed Kubernetes APIs to v1 is complete, as an OpenShift Administrator you can provide the acknowledgment required to proceed.

WARNING: Be aware that all responsibility falls on the administrator to ensure that all uses of removed APIs have been resolved and migrated as necessary before providing this administrator acknowledgment. OpenShift Container Platform can assist with the evaluation, but cannot identify all possible uses of removed APIs, especially idle workloads or external tools.

To acknowledge that you have completed the evaluation and your cluster is ready to upgrade to OpenShift Container Platform 4.9, run the following command:

$ oc -n openshift-config patch cm admin-acks --patch '{"data":{"ack-4.8-kube-1.22-api-removals-in-4.9":"true"}}' --type=merge

Note: You must have access to the cluster as a user with the cluster-admin role in order to run this command.

Begin Upgrade from OpenShift 4.8 To 4.9

Login to OpenShift Web Console and navigate to AdministrationCluster Settings > Details

Fig: 01

Click on “Channel” and update channel to fast-4.9 or stable-4.9.

Select fast-4.9 orstable-4.9 from the list of available channels.

Channel can also be changed from the command line with below command syntax:

oc adm upgrade channel clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "<channel>”}]'

Where <channel> is replaced with either;

  • stable-4.9
  • fast-4.9
  • candidate-4.9

New cluster updates should be visible now. Use the “Update” link to initiate upgrade from OpenShift 4.8 to OpenShift 4.9.

Select the new version of OpenShift 4.9 that you’ll be upgrading to.

The update process to OpenShift Container Platform 4.9 should begin shortly.

You can also check upgrade status from CLI:

$ oc get clusterversions.config.openshift.io
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.2 True True 1m26s Working towards 4.9.0:

Output once all the upgrades are completed. You now have OpenShift cluster version 4.9.

List all the available MachineHealthCheck to ensure everything is in healthy state.

$ oc get machinehealthcheck -n openshift-machine-api
NAME MAXUNHEALTHY EXPECTEDMACHINES CURRENTHEALTHY
machine-api-termination-handler 100%

Check cluster components:

$ oc get cs
W1110 20:39:28.838732 2592723 warnings.go:70] v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-3 Healthy {"health":"true","reason":""}
etcd-1 Healthy {"health":"true","reason":""}
etcd-0 Healthy {"health":"true","reason":""}
etcd-2 Healthy {"health":"true","reason":""}

List all cluster operators and review current versions.

$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.9.5 True False False 32h
baremetal 4.9.5 True False False 84d
cloud-controller-manager 4.9.5 True False False 20d
cloud-credential 4.9.5 True False False 84d
cluster-autoscaler 4.9.5 True False False 84d
config-operator 4.9.5 True False False 84d
console 4.9.5 True False False 37d
csi-snapshot-controller 4.9.5 True False False 84d
dns 4.9.5 True False False 84d
etcd 4.9.5 True False False 84d
image-registry 4.9.5 True False False 84d
ingress 4.9.5 True False False 84d
insights 4.9.5 True False False 84d
kube-apiserver 4.9.5 True False False 84d
kube-controller-manager 4.9.5 True False False 84d
kube-scheduler 4.9.5 True False False 84d
kube-storage-version-migrator 4.9.5 True False False 8d
machine-api 4.9.5 True False False 84d
machine-approver 4.9.5 True False False 84d
machine-config 4.9.5 True False False 8d
marketplace 4.9.5 True False False 84d
monitoring 4.9.5 True False False 20d
network 4.9.5 True False False 84d
node-tuning 4.9.5 True False False 8d
openshift-apiserver 4.9.5 True False False 20d
openshift-controller-manager 4.9.5 True False False 7d8h
openshift-samples 4.9.5 True False False 8d
operator-lifecycle-manager 4.9.5 True False False 84d
operator-lifecycle-manager-catalog 4.9.5 True False False 84d
operator-lifecycle-manager-packageserver 4.9.5 True False False 20d
service-ca 4.9.5 True False False 84d
storage 4.9.5 True False

Check if all nodes are available and in healthy state.

$ oc get nodes

Summary

In this blog post you will be able to perform an upgrade of OpenShift from version 4.8 to 4.9. Ensure all Operators previously installed through Operator Lifecycle Manager (OLM) are updated to their latest version in their latest channel as selected during upgrade. Also ensure that all machine config pools (MCPs) are running and not paused.

Note: This blog is completely based on my experience on cluster upgrade and the environment which I am working on. Few steps may vary according to your setup.

Note: Upgrade process will be different for disconnected clusters.

Happy reading….

--

--

DevOps Practitioner (CKA certified , RHOCP Certified, Azure Certified on az-104,az-400,az-303.)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Kamlesh Prajapati

DevOps Practitioner (CKA certified , RHOCP Certified, Azure Certified on az-104,az-400,az-303.)