Subscribe to the feed

A Kubernetes namespace isolates specific system resources usually visible to all processes. Each namespace has its own services, pods, and deployments in the cluster. A namespace in Kubernetes is essentially the same as a project in OpenShift.

These days, most products are containerized and can easily be deployed on Kubernetes or OpenShift. This requires continuous application deployment and testing. I recently worked on test automation that involves the creation of namespaces, resources, and data in the namespace, followed by running the test suite, and then removing the data and namespace once the tests are complete.

While frequently creating and deleting namespaces in Kubernetes, I stumbled upon an issue where a namespace got stuck in the Terminating state and just refused to delete.

I use minikube to run Kubernetes locally, but you can use the following steps and commands in any Kubernetes or OpenShift environment.

Note: The example namespace in my screenshots and commands is tackle-operator.

Problem: Deleting a namespace

Sometimes the process to delete namespaces gets stuck, and the command never completes. While the command returns a message showing that the namespace was deleted, querying it indicates that it's actually in a Terminating state:

$ kubectl delete namespace tackle-operator
namespace "tackle-operator" deleted

$ kubectl get namespace
NAME               	   STATUS    	   AGE
default            	                Active    	   148d
ingress-nginx      	    Active    	   148d
kube-node-lease    	    Active    	   148d
kube-public        	    Active    	   148d
kube-system        	    Active    	   148d
kubernetes-dashboard   Active    	   148d
tackle-operator    	   Terminating       10d

The result shows the namespace was deleted, but its status reveals that it's not quite gone yet, and you can still see the namespace in the minikube UI.

Open the minikube dashboard in the Select namespace dropdown on the top, and you can still select the namespace you thought you deleted.

A namespace in the terminating state

You might try to delete it from the user interface (UI). To do so, click on the three dots shown at the right of the namespace and select Delete.

Check after a few minutes (or days, months, years). It still shows up as terminating.

[ Learning path: Getting started with Red Hat OpenShift Service on AWS (ROSA)

Why do some namespaces never delete?

Kubernetes stores each namespace as a YAML or JSON file.

$ kubectl get namespace ${NAMESPACE} -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
	kubectl.kubernetes.io/last-applied-configuration: |
	kubernetes.io/metadata.name: tackle-operator
spec:
  finalizers:
  - kubernetes
status:
  conditions:
 - lastTransitionTime: "2022-01-19T19:05:31Z"
	message: 'Some content in the namespace has finalizers remaining: tackles.tackle.io/finalizer in 1 resource instances'
	reason: SomeFinalizersRemain
	status: "True"
	type: NamespaceFinalizersRemaining
  phase: Terminating

Notice the inclusion of the finalizers field in the above JSON. Some namespaces have a finalizer defined under spec.

A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource.

So when you run a command like kubectl delete namespace abcd, Kubernetes checks for a finalizer in the metadata.finalizers field. If the resource defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.

When an object has been terminating for an excessive time, check its finalizers by inspecting the metadata.finalizers field in its YAML.

[ Deploy an application with Red Hat OpenShift on AWS ]

Deleting a namespace stuck in a terminating state

Once you understand the cause of the problem, you can understand why the solution is to remove the finalizer from the YAML. Only by eliminating the constraint preventing a namespace from being deleted can you remove the namespace successfully.

There are subtleties in this task, though. For example, you can't just edit the namespace YAML from the Kubernetes UI to remove the finalizer because the UI doesn't update the namespace. To see this, edit the namespace to remove the finalizer, and then update. Edit the YAML again, and you'll notice that the finalizer still exists.

Here's the right way to do it.

Step 1: Dump the contents of the namespace in a temporary file called tmp.json:

$ kubectl get namespace ${NAMESPACE} -o json > tmp.json

Step 2: Edit the temporary file in your favorite text editor (mine is Vi):

$ vi tmp.json

Step 3: Remove kubernetes from the finalizer array, and save the file. You can skip to step 4 now unless you're on OpenShift or OKD. For those, you must set up a temporary proxy. Keep this terminal open until the namespace is deleted. To start a proxy server at http://127.0.0.1:8001, use the proxy subcommand:

$ oc proxy

From a separate window, run this command:

$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${PROJECT_NAME}/finalize

Step 4: Call the Kubernetes API application/json against the /finalize endpoint for the namespace to update the JSON. Use the port number appropriate for your instance. I'm using 44315 because that's where my instance of minikube is running:

$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:44315/api/v1/namespaces/${NAMESPACE}/finalize

If you don't know what port your Kubernetes instance is using, look in your browser's URL bar when you navigate to the UI:

$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:44315/api/v1/namespaces/${NAMESPACE}/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
	"name": "tackle-operator",
	"creationTimestamp": "2022-01-10T19:13:58Z",
	"deletionTimestamp": "2022-01-19T19:05:18Z",
  "status": {
	"phase": "Terminating",
	"conditions": [
  	{
    	"type": "NamespaceDeletionDiscoveryFailure",
    	"status": "False",
    	"lastTransitionTime": "2022-01-21T04:51:31Z",
    	"reason": "ResourcesDiscovered",
    	"message": "All resources successfully discovered"
  	},
  	{
    	"type": "NamespaceDeletionContentFailure",
    	"status": "False",
    	"lastTransitionTime": "2022-01-19T19:05:31Z",
    	"reason": "ContentDeleted",
    	"message": "All content successfully deleted, may be waiting on finalization"
  	},
}

And that's it. The minikube dashboard no longer displays the namespace:

$ kubectl get namespace ${NAMESPACE}
Error from server (NotFound): namespaces "${NAMESPACE}" not found

Stop terminating

In my testing, I create and delete namespaces frequently. During this, I discovered an issue where namespaces became stuck in the terminating state. I developed a straightforward process for entirely deleting the namespaces. I hope that this process will help anyone else who encounters the same issue.

[ Learn 16 steps for building production-ready Kubernetes clusters. ]


About the author

Shveta is a senior software engineer at Red Hat, leading a team. She is a subject-matter expert on the Migration Toolkit for Applications (MTA) and Pathfinder that helps customers migrate their applications to containers (Openshift and Kubernetes) and the latest technologies. She continuously develops her technical and domain expertise in MTA, functional testing, Kubernetes, OpenShift, and DevOps and is involved in reporting and validating dozens of issues ensuring high-quality releases to customers. Shveta has independently built the automation framework from scratch for some Red Hat products and has automated the needed test cases in Python and Cypress. Shveta is also contributing to some DevOps tasks and CI/CD tools like building and maintaining Jenkins pipelines to run automation tests on virtual machines for different operating systems, including Linux, Windows, and macOS.

Read full bio
UI_Icon-Red_Hat-Close-A-Black-RGB

Browse by channel

automation icon

Automation

The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon

Security

The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon

Infrastructure

The latest on the world’s leading enterprise Linux platform

application development icon

Applications

Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech