A Kubernetes namespace isolates specific system resources usually visible to all processes. Each namespace has its own services, pods, and deployments in the cluster. A namespace in Kubernetes is essentially the same as a project in OpenShift.
These days, most products are containerized and can easily be deployed on Kubernetes or OpenShift. This requires continuous application deployment and testing. I recently worked on test automation that involves the creation of namespaces, resources, and data in the namespace, followed by running the test suite, and then removing the data and namespace once the tests are complete.
While frequently creating and deleting namespaces in Kubernetes, I stumbled upon an issue where a namespace got stuck in the Terminating state and just refused to delete.
I use minikube to run Kubernetes locally, but you can use the following steps and commands in any Kubernetes or OpenShift environment.
Note: The example namespace in my screenshots and commands is tackle-operator
.
Problem: Deleting a namespace
Sometimes the process to delete namespaces gets stuck, and the command never completes. While the command returns a message showing that the namespace was deleted, querying it indicates that it's actually in a Terminating
state:
$ kubectl delete namespace tackle-operator
namespace "tackle-operator" deleted
$ kubectl get namespace
NAME STATUS AGE
default Active 148d
ingress-nginx Active 148d
kube-node-lease Active 148d
kube-public Active 148d
kube-system Active 148d
kubernetes-dashboard Active 148d
tackle-operator Terminating 10d
The result shows the namespace was deleted, but its status reveals that it's not quite gone yet, and you can still see the namespace in the minikube UI.
Open the minikube dashboard in the Select namespace dropdown on the top, and you can still select the namespace you thought you deleted.

You might try to delete it from the user interface (UI). To do so, click on the three dots shown at the right of the namespace and select Delete.
Check after a few minutes (or days, months, years). It still shows up as terminating.
[ Learning path: Getting started with Red Hat OpenShift Service on AWS (ROSA) ]
Why do some namespaces never delete?
Kubernetes stores each namespace as a YAML or JSON file.
$ kubectl get namespace ${NAMESPACE} -o yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
kubernetes.io/metadata.name: tackle-operator
spec:
finalizers:
- kubernetes
status:
conditions:
- lastTransitionTime: "2022-01-19T19:05:31Z"
message: 'Some content in the namespace has finalizers remaining: tackles.tackle.io/finalizer in 1 resource instances'
reason: SomeFinalizersRemain
status: "True"
type: NamespaceFinalizersRemaining
phase: Terminating
Notice the inclusion of the finalizers field in the above JSON. Some namespaces have a finalizer defined under spec
.
A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource.
So when you run a command like kubectl delete namespace abcd
, Kubernetes checks for a finalizer in the metadata.finalizers
field. If the resource defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.
When an object has been terminating for an excessive time, check its finalizers by inspecting the metadata.finalizers
field in its YAML.
[ Deploy an application with Red Hat OpenShift on AWS ]
Deleting a namespace stuck in a terminating state
Once you understand the cause of the problem, you can understand why the solution is to remove the finalizer from the YAML. Only by eliminating the constraint preventing a namespace from being deleted can you remove the namespace successfully.
There are subtleties in this task, though. For example, you can't just edit the namespace YAML from the Kubernetes UI to remove the finalizer because the UI doesn't update the namespace. To see this, edit the namespace to remove the finalizer, and then update. Edit the YAML again, and you'll notice that the finalizer still exists.
Here's the right way to do it.
Step 1: Dump the contents of the namespace in a temporary file called tmp.json
:
$ kubectl get namespace ${NAMESPACE} -o json > tmp.json
Step 2: Edit the temporary file in your favorite text editor (mine is Vi):
$ vi tmp.json
Step 3: Remove kubernetes from the finalizer array, and save the file. You can skip to step 4 now unless you're on OpenShift or OKD. For those, you must set up a temporary proxy. Keep this terminal open until the namespace is deleted. To start a proxy server at http://127.0.0.1:8001
, use the proxy
subcommand:
$ oc proxy
From a separate window, run this command:
$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${PROJECT_NAME}/finalize
Step 4: Call the Kubernetes API application/json
against the /finalize
endpoint for the namespace to update the JSON. Use the port number appropriate for your instance. I'm using 44315 because that's where my instance of minikube is running:
$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:44315/api/v1/namespaces/${NAMESPACE}/finalize
If you don't know what port your Kubernetes instance is using, look in your browser's URL bar when you navigate to the UI:
$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:44315/api/v1/namespaces/${NAMESPACE}/finalize
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "tackle-operator",
"creationTimestamp": "2022-01-10T19:13:58Z",
"deletionTimestamp": "2022-01-19T19:05:18Z",
"status": {
"phase": "Terminating",
"conditions": [
{
"type": "NamespaceDeletionDiscoveryFailure",
"status": "False",
"lastTransitionTime": "2022-01-21T04:51:31Z",
"reason": "ResourcesDiscovered",
"message": "All resources successfully discovered"
},
{
"type": "NamespaceDeletionContentFailure",
"status": "False",
"lastTransitionTime": "2022-01-19T19:05:31Z",
"reason": "ContentDeleted",
"message": "All content successfully deleted, may be waiting on finalization"
},
}
And that's it. The minikube dashboard no longer displays the namespace:
$ kubectl get namespace ${NAMESPACE}
Error from server (NotFound): namespaces "${NAMESPACE}" not found
Stop terminating
In my testing, I create and delete namespaces frequently. During this, I discovered an issue where namespaces became stuck in the terminating state. I developed a straightforward process for entirely deleting the namespaces. I hope that this process will help anyone else who encounters the same issue.
[ Learn 16 steps for building production-ready Kubernetes clusters. ]
About the author
Shveta is a senior software engineer at Red Hat, leading a team. She is a subject-matter expert on the Migration Toolkit for Applications (MTA) and Pathfinder that helps customers migrate their applications to containers (Openshift and Kubernetes) and the latest technologies. She continuously develops her technical and domain expertise in MTA, functional testing, Kubernetes, OpenShift, and DevOps and is involved in reporting and validating dozens of issues ensuring high-quality releases to customers. Shveta has independently built the automation framework from scratch for some Red Hat products and has automated the needed test cases in Python and Cypress. Shveta is also contributing to some DevOps tasks and CI/CD tools like building and maintaining Jenkins pipelines to run automation tests on virtual machines for different operating systems, including Linux, Windows, and macOS.
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech