OpenShift tip: Resolve a terminating state issue

Red Hat OpenShift is a Kubernetes platform designed to empower developers to innovate and ship faster by using the power of hybrid cloud and containers. There is always a significant learning curve involved when you work with any enterprise-level platform, and learning how to resolve issues is an important part of being able to administer a Kubernetes system effectvely. Facing these issues along the way and learning how to resolve them enables you to become a better OpenShift sysadmin and developer.

As an OpenShift sysadmin, you must know how to operate and manage the platform's different states. It is just as important to know how to delete an OpenShift project as it is know how to create a new project, although the benefit is often less obvious.

WARNING: This solution is a work-around but it is not a recommended practice to directly edit the namespace object to remove the finalizer. Please use this option as only a temporarily fix-it-yourself solution. In most cases, a project will be in terminating state if there is another object that has a finalizer set. There needs to be a healthy team discussion on how to find those extraneous objects if you do the above. Otherwise, you then risk having conflicts in ETCD in the future.

Managing terminating states

Your project can get stuck in Terminating state, as you see in the screenshot below.

Image

This problem persists even after you confirm that you deleted all resources found by using the oc get all command in the project. In OpenShift 4, Operators manage the life-cycle of applications. The stuck-in-terminating problem is easily reproducible and one that you may encounter.

Don't worry. In this blog post, I am going to explain how to resolve this error. To give you a sneak preview, this involves modifying an OpenShift Resource called a namespace, which is synonymous with Red Hat project. More specifically, we modify finalizer in OpenShift, which is a mechanism to inform the Kubernetes control plane that an action needs to take place so the standard Kubernetes garbage logic is performed.

To modify any project/namespace, you need the cluster-admin Role-Based Access Control (RBAC) permission. If you do not have the permission, you get an error like this one:

# oc get namespace

Error from server (Forbidden): namespaces is forbidden: User "USERNAME" cannot list namespaces at the cluster scope: no RBAC policy matched

Where USERNAME is your OpenShift username.

Image

Ask your OpenShift administrator to grant you the cluster-admin permission with a command like this:

# oc adm policy add-cluster-role-to-user cluster-admin USERNAME

or

# oc adm policy add-role-to-user admin USERNAME -n PROJECT

If you want the admin privilege for the specific PROJECT that is visible to you, use the following command:

Image

You can still follow the rest of the guide, even if you only have the admin privilege for a specific project.

After you are granted a cluster-admin role, you can then run this command:

# oc get namespace

That command lists all the projects that are available in OpenShift. You will see your project in Terminating state like below:

Image

Run the following command to display the content of the problematic namespace:

# oc get -o yaml namespace/PROJECT

Where PROJECT is your namespace/project name.

Next, modify that value under finalizers. Basically, remove the line - kubernetes.

Image

However, if you try to edit and save the file directly with the oc edit namespace/PROJECT command, your finalizer is not updated.

Image

Instead, back up the namespace to a file with the following command:

# oc get namespace DELETABLE_PROJECT -o yaml > BACKUP_NAMESPACE.yaml

Where DELETABLE_PROJECT is the namespace/project name that you want to delete, and BACKUP_NAMESPACE is the file name where you want to back up your namespace. Open the saved file with a text editor of your choice. In the screen shot below, I am using vim, but you can use any other text editor.

Image

Remove the value under finalizers, and that is all there is to it. In the screenshot below, I deleted the line (line 19), where it says - kubernetes.

Image

After deleting the file, the result looks like this. Save the file and exit.

Image

Next, start a temporary local proxy server to let the local server talk to the remote OpenShift cluster. Running the command below starts the temporary proxy process in the background.

# oc proxy &

Image

The next command is the longest command that you need to enter for this example:

# curl -k -H "Content-Type: application/yaml" -X PUT --data-binary @BACKUP_NAMESPACE.yaml http://127.0.0.1:8001/api/v1/namespaces/DELETABLE_PROJECT/finalize

Where DELETABLE_PROJECT is the namespace/project name you want to delete, and BACKUP_NAMESPACE is the file name for your namespace backup. Note that http://127.0.0.1:8001 is the proxy server that we started above.

Image

The command above gives a successful response, indicating that you updated the namespace by sending the update request with the file. Alternatively, you can try the oc apply -f BACKUP_NAMESPACE.yaml command, but this does not always work.

You should no longer see the namespace/project.

# oc get namespace DELETABLE_PROJECT

Image

That is it! Make sure to kill the proxy server with the following command:

# oc kill -9 %%

Image

Wrap up

In this blog, you learned how to delete a project stuck in Terminating state. Learning how to troubleshoot an issue like this in Red Hat OpenShift prepares you to become an expert with the platform in no time. Thank you for reading.

[ Thinking about a cloud strategy? See why enterprises choose open hybrid cloud. ]