Just-in

Kagent Tutorial: Build Cloud-Native AI Agents on Kubernetes

Running Ollama LLM on Kubernetes: A Complete Guide

From Docker to Kubernetes: The Orchestration Journey You Didn’t Know You Needed

What is Kubeflow and how can it be used?

Getting Started with Kubeflow: A Complete Beginner’s Guide

Orchestration

Kubernetes Hierarchical Namespaces: Master Structure

Introduction

Managing resources and permissions in a multi-tenant Kubernetes cluster can quickly become a labyrinthine task. As organizations scale, the flat namespace model of Kubernetes often leads to challenges in governance, policy enforcement, and resource isolation. Teams might accidentally interfere with each other’s deployments, or administrators might struggle to apply consistent policies across related services without resorting to complex, error-prone automation. This is where Kubernetes Hierarchical Namespaces (HNC) steps in, offering a much-needed structural improvement to how we organize and manage our clusters.

HNC, a sub-project of the Kubernetes SIG Multi-cluster, introduces the concept of parent-child relationships between namespaces. This hierarchical structure allows policies, RBAC roles, and resource quotas to be inherited from parent namespaces to their children, simplifying management significantly. Imagine defining a network policy or a resource quota once at a high level, and having it automatically apply to all subordinate teams or applications. This not only reduces operational overhead but also enhances security and compliance by ensuring consistent configurations across related environments.

TL;DR

Hierarchical Namespaces (HNC) brings order to multi-tenant Kubernetes clusters by enabling parent-child relationships between namespaces. This allows policies, RBAC, and resource quotas to be inherited, simplifying management and improving consistency. Install HNC, create a parent namespace, and then define child namespaces linked to it. Policies applied to the parent will automatically propagate to children.

# Install HNC (example using kubectl krew)
kubectl krew install hns

# Create a parent namespace
kubectl create ns parent-ns

# Mark parent-ns as a parent
kubectl hns create parent-ns --parent parent-ns # No, this is incorrect. You mark the child.

# Correct way to make a child namespace
kubectl create ns child-ns
kubectl hns set child-ns --parent parent-ns

# Verify hierarchy
kubectl hns tree parent-ns

# Propagate a network policy from parent to child
# Create policy in parent-ns first
cat <

Prerequisites

Before diving into Hierarchical Namespaces, ensure you have the following:

  • A running Kubernetes cluster (version 1.16+ is recommended for HNC). You can use Minikube, Kind, or a cloud-managed cluster like GKE, EKS, or AKS.
  • kubectl command-line tool installed and configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
  • krew, the kubectl plugin manager, is highly recommended for easily installing the hns plugin. If you don't have it, follow the krew installation guide.
  • Basic understanding of Kubernetes concepts such as Namespaces, RBAC, Network Policies, and Resource Quotas.

Step-by-Step Guide

1. Install the Hierarchical Namespace Controller (HNC)

The first step is to deploy the HNC controller into your Kubernetes cluster. HNC is typically installed in the hnc-system namespace and watches for specific custom resources that define namespace hierarchies and object propagations. Installing it via krew is the easiest method as it also installs the hns kubectl plugin, which simplifies interaction with HNC.

The HNC controller itself is responsible for observing HierarchyConfiguration and SubnamespaceAnchor custom resources, and then performing the necessary actions like creating child namespaces, propagating objects, and enforcing policies. It's a critical component for the entire system to function.

# Install the hns kubectl plugin via krew
kubectl krew install hns

# Install the HNC controller into your cluster
# This command uses the plugin to deploy the controller components.
kubectl hns install

# Verify that HNC is running
kubectl get pods -n hnc-system

Verify: You should see pods running in the hnc-system namespace, indicating that the controller is active.

NAME                                       READY   STATUS    RESTARTS   AGE
hnc-controller-manager-69446f889c-abcde   1/1     Running   0          2m

2. Create a Parent Namespace

With HNC installed, we can now start building our hierarchy. We'll begin by creating a standard Kubernetes namespace that will serve as our parent. This parent namespace will eventually be configured to manage child namespaces and propagate certain resources down its hierarchy.

For this example, let's create a namespace called team-a. This could represent an entire department or a major project within your organization. All sub-teams or applications within team-a will reside in its child namespaces.

# Create the parent namespace
kubectl create namespace team-a

# Verify its creation
kubectl get namespace team-a

Verify: Ensure the namespace is created successfully.

NAME     STATUS   AGE
team-a   Active   10s

3. Create a Child Namespace

Now, let's create a child namespace for team-a. This child namespace will inherit properties from team-a. HNC introduces a custom resource called SubnamespaceAnchor to establish this parent-child relationship. When you create a SubnamespaceAnchor in the parent namespace, HNC automatically creates the corresponding child namespace and links it in the hierarchy.

For instance, team-a-dev could be the development environment for a specific application within team-a. This structure helps in isolating environments while maintaining unified policy management.

# Create a SubnamespaceAnchor in the parent namespace for the child
cat <

Verify: The child namespace team-a-dev should appear as an active namespace.

NAME         STATUS   AGE
team-a-dev   Active   20s

4. View the Namespace Hierarchy

To visualize the relationships we've just created, the hns kubectl plugin provides a convenient tree command. This command helps administrators understand the structure of their namespaces and verify that the hierarchies are set up as intended.

Understanding the hierarchy is crucial for debugging propagation issues or simply for auditing your cluster's multi-tenancy setup. It clearly shows which namespaces are parents and which are children, and how deeply nested they are.

# View the hierarchy starting from the parent namespace
kubectl hns tree team-a

Verify: The output should clearly show team-a as the parent and team-a-dev as its child.

team-a
└── team-a-dev

5. Propagate a ResourceQuota

One of the most powerful features of HNC is resource propagation. You can define a resource in a parent namespace and configure HNC to automatically copy it to all child namespaces. Let's demonstrate this with a ResourceQuota. This is excellent for ensuring that child namespaces don't consume excessive resources, promoting cost optimization and cluster stability.

By applying a ResourceQuota to team-a and marking it for propagation, team-a-dev (and any future children) will automatically inherit these limits, saving administrators the effort of manually applying quotas to every new namespace.

# Define a ResourceQuota in the parent namespace
cat <

Verify: The team-a-quota should now exist in the team-a-dev namespace with the inherited limits.

NAME           AGE   REQUEST                                           LIMIT
team-a-quota   10s   cpu: 0, memory: 0, pods: 0   cpu: 2, memory: 2Gi, pods: 10

6. Propagate a NetworkPolicy

Security policies, such as Kubernetes Network Policies, are another excellent candidate for propagation. By defining a global network policy in a parent namespace, you can ensure that all child namespaces adhere to certain security standards by default. This significantly enhances the security posture of your multi-tenant cluster.

For example, let's create a network policy in team-a that denies all egress traffic by default. This "deny-by-default" approach is a strong security practice. HNC will then ensure this policy is enforced in team-a-dev and any other child namespaces.

# Define a NetworkPolicy in the parent namespace
cat <

Verify: The deny-all-egress network policy should be present in the team-a-dev namespace.

NAME              POD-SELECTOR   AGE
deny-all-egress   <empty>        15s

7. Propagate RBAC Roles and RoleBindings

Managing access control (RBAC) across many namespaces can be tedious. HNC simplifies this by allowing you to propagate Role and RoleBinding objects. This means you can define common roles and assign them to users or service accounts at a higher level, and have those permissions automatically extend to child namespaces.

Let's create a Role and a RoleBinding in team-a that grants read-only access to deployments, and then propagate it to team-a-dev. This ensures that a "viewer" role defined once applies across all environments of team-a.

# Define a Role in the parent namespace
cat <

Verify: Both the deployment-reader Role and team-a-dev-viewer RoleBinding should be present in team-a-dev.

NAME                AGE
deployment-reader   20s

NAME                ROLE                 AGE
team-a-dev-viewer   Role/deployment-reader   20s

Production Considerations

While HNC offers significant benefits, deploying it in a production environment requires careful planning and consideration:

  • Granular Propagation Control: HNC allows fine-grained control over which resource types propagate. Use HierarchyConfiguration to define what propagates and what doesn't. Avoid propagating sensitive resources like Secrets or certain ConfigMaps unless absolutely necessary and with strong justification.
  • RBAC Strategy: Design your RBAC hierarchy thoughtfully. Consider using ClusterRoles and ClusterRoleBindings for cluster-wide permissions, and HNC-propagated Roles/RoleBindings for namespace-specific permissions that need to flow down the hierarchy. For advanced access control, consider integrating with tools like Kyverno for policy enforcement.
  • Monitoring and Alerting: Monitor the HNC controller's health and logs. Set up alerts for any failures in propagation or hierarchy management. eBPF Observability with Hubble could be useful here to understand network interactions between HNC components.
  • Testing and Validation: Thoroughly test your hierarchy and propagation rules in staging environments before deploying to production. Changes to parent namespaces can have widespread effects.
  • Resource Overrides: HNC supports "exceptions" where a child namespace can override a propagated object. Use this feature sparingly and with caution, as it can complicate debugging and policy consistency.
  • Integration with GitOps: HNC works seamlessly with GitOps workflows. Define your SubnamespaceAnchor and annotated resources in Git, and let your GitOps tool (e.g., Argo CD, Flux CD) apply them. This ensures your hierarchy is version-controlled and auditable.
  • Performance Implications: While HNC is designed to be efficient, very deep hierarchies or an extremely large number of propagated objects might introduce some latency during updates. Monitor controller performance in large clusters.
  • Network Isolation: Complement HNC's policy propagation with robust network isolation strategies. Consider advanced CNI features like those offered by Cilium, especially for features like WireGuard Encryption for pod-to-pod traffic, which can further enhance security beyond basic network policies.
  • Service Mesh Integration: If you're using a service mesh like Istio Ambient Mesh, consider how HNC's namespace structure aligns with your mesh's policy enforcement and traffic management.

Troubleshooting

Here are some common issues you might encounter with HNC and their solutions:

1. HNC Controller Pods Not Running

Issue: The hnc-controller-manager pods in the hnc-system namespace are not in a Running state.

Solution: Check the logs of the controller pod and the events in the hnc-system namespace.

kubectl logs -n hnc-system -l control-plane=hnc-controller-manager
kubectl describe pod -n hnc-system -l control-plane=hnc-controller-manager
kubectl get events -n hnc-system

Common causes include insufficient resources (CPU/memory), incorrect RBAC for the controller, or issues during installation. Ensure your cluster meets the basic requirements for HNC.

2. Child Namespace Not Created

Issue: You created a SubnamespaceAnchor, but the corresponding child namespace does not appear.

Solution:

  1. Verify the SubnamespaceAnchor was created successfully in the parent namespace: 
    kubectl get subnamespaceanchor -n parent-ns
  2. Check the status field of the SubnamespaceAnchor for any errors: 
    kubectl get subnamespaceanchor child-ns-name -n parent-ns -o yaml
  3. Inspect HNC controller logs for errors related to namespace creation. 
    kubectl logs -n hnc-system -l control-plane=hnc-controller-manager

A common reason is a misconfiguration in the SubnamespaceAnchor or an issue with HNC's permissions to create namespaces.

3. Object Not Propagating to Child Namespace

Issue: You've applied a resource (e.g., ResourceQuota, NetworkPolicy) to a parent namespace with the hnc.x-k8s.io/propagate: "true" annotation, but it's not appearing in the child namespace.

Solution:

  1. Ensure the hnc.x-k8s.io/propagate: "true" annotation is correctly applied to the resource in the parent namespace. 
    kubectl get <resource-type> <resource-name> -n parent-ns -o yaml | grep propagate
  2. Check if the resource type is configured for propagation in the parent's HierarchyConfiguration. By default, HNC propagates many common types, but custom resources or specific types might need explicit configuration. 
    kubectl get hierarchyconfiguration -n parent-ns -o yaml
  3. Look for any errors in the HNC controller logs. It will log if it attempts to propagate an object but fails due to conflicts or invalid configurations.
  4. Verify there isn't an existing object with the same name in the child namespace that is preventing propagation.

4. Child Namespace Cannot Override Propagated Object

Issue: A child namespace tries to apply its own version of an object that is being propagated from its parent, and HNC prevents it or causes a conflict.

Solution: HNC has a default behavior to prevent children from overriding propagated objects to maintain consistency. If you intend for a child to be able to override, you need to configure the parent's HierarchyConfiguration to allow "exceptions" for that resource type.

# Example HierarchyConfiguration allowing exceptions for NetworkPolicies
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
  name: hierarchy
  namespace: parent-ns
spec:
  allowCascadingDelete: true
  allowPropagating: true
  # Add this section to allow exceptions for NetworkPolicies
  exceptions:
  - group: networking.k8s.io
    resource: networkpolicies

Alternatively, if you don't want propagation for that specific resource in a child, you can mark it as ignored in the child's HierarchyConfiguration using hnc.x-k8s.io/inheritedFrom set to ignore.

5. Deleting a Parent Namespace Fails

Issue: You try to delete a parent namespace, but it gets stuck in a Terminating state.

Solution: HNC prevents accidental deletion of parent namespaces that still have child namespaces. You must delete all child namespaces (or detach them) before deleting the parent.

  1. List all child namespaces using kubectl hns tree parent-ns.
  2. Delete each child namespace: 
    kubectl delete namespace child-ns
  3. Once all children are gone, the parent namespace should terminate successfully. If it still persists, check for finalizers on the parent namespace that HNC might have added, and ensure the HNC controller is healthy.

6. RBAC Issues with HNC

Issue: Users or service accounts cannot perform actions in child namespaces even though a RoleBinding was propagated from a parent.

Solution:

  1. Verify the Role and RoleBinding are correctly propagated to the child namespace.
  2. Ensure the subjects in the RoleBinding correctly identify the user or service account.
  3. Check if there are any conflicting or overriding RoleBindings in the child namespace that might be denying access.
  4. Remember that HNC propagates namespace-scoped RBAC objects (Roles, RoleBindings). Cluster-scoped permissions (ClusterRoles, ClusterRoleBindings) are not propagated by HNC and must be managed separately.
  5. Use kubectl auth can-i <verb> <resource> --as=<user> -n <namespace> to debug specific permissions for a user in a given namespace.

FAQ Section

Q1: What is the main benefit of using Hierarchical Namespaces?

A: The primary benefit is simplified governance and resource management in multi-tenant Kubernetes clusters. HNC allows you to define policies, RBAC, and resource quotas once at a parent level and have them automatically inherited by child namespaces. This reduces operational overhead, ensures consistency, and improves security by enforcing standards across related environments.

Q2: Can I create a hierarchy of more than two levels (e.g., Grandparent > Parent > Child)?

A: Yes, absolutely! HNC supports arbitrary depth for namespace hierarchies. You can create a SubnamespaceAnchor for child-ns within parent-ns, and then create another SubnamespaceAnchor for grandchild-ns within child-ns. Policies and resources will propagate down the entire chain, making it very flexible for complex organizational structures.

Q3: What types of Kubernetes objects can be propagated by HNC?

A: HNC can propagate most namespace-scoped Kubernetes objects. This includes ResourceQuota, NetworkPolicy, Role, RoleBinding, LimitRange, ConfigMap, and Secret (though propagating secrets requires careful consideration). You can configure which types are propagated using the HierarchyConfiguration custom resource in each parent namespace. For more details, refer to the official HNC documentation on propagation.

Q4: How does HNC handle conflicts if a child namespace tries to modify a propagated object?

A: By default, HNC enforces consistency by preventing child namespaces from modifying or deleting propagated objects. If a child tries to create an object with the same name as a propagated one, HNC will typically block it or revert the change. However, you can configure "exceptions" in the parent's HierarchyConfiguration to allow children to override specific propagated objects if that behavior is desired for certain resource types.

Q5: Is HNC an official Kubernetes project?

A: HNC is a sub-project of SIG Multi-cluster, which is part of the official Kubernetes Special Interest Groups. While not a core component of Kubernetes itself, it is developed and maintained by the Kubernetes community under the CNCF umbrella, making it a well-supported and widely adopted solution for multi-tenancy.

Cleanup Commands

To remove the resources created during this tutorial, execute the following commands:

# Delete the parent namespace, which will also trigger the deletion of child namespaces
kubectl delete namespace team-a

# Verify namespaces are gone
kubectl get ns team-a team-a-dev

# Uninstall the HNC controller
kubectl hns uninstall

# If you installed krew specifically for this, you might want to remove it
# (Optional, krew is generally useful)
# kubectl krew uninstall hns

Next Steps / Further Reading

  • Explore the official Hierarchical Namespaces documentation for advanced features like object exceptions, label/annotation propagation, and advanced hierarchy configurations.
  • Deep dive into Kubernetes RBAC to design robust access control strategies in your hierarchical clusters.
  • Learn more about Kubernetes Network Policies and how they can be used with HNC to enforce strong network segmentation.
  • Consider integrating HNC with Flux CD or Argo CD for a GitOps-driven approach to managing your namespace hierarchies and propagated resources.
  • For advanced traffic management and policy enforcement within your hierarchy, explore service mesh solutions like Istio Ambient Mesh or Kubernetes Gateway API.
  • If you're dealing with specific resource types, investigate how HNC interacts with them. For example, if you're running machine learning workloads, consider how HNC might align with LLM GPU Scheduling Best Practices in a multi-tenant setup.

Conclusion

Kubernetes Hierarchical Namespaces provide a powerful and elegant solution to the challenges of multi-tenancy in Kubernetes. By introducing a structured approach to namespace management, HNC simplifies policy enforcement, streamlines RBAC, and ensures consistent resource allocation across diverse teams and applications. This not only reduces the operational burden on cluster administrators but also fosters a more secure and organized cloud-native environment. Embracing HNC can transform your flat, sprawling cluster into a well-governed, easily manageable ecosystem, paving the way for scalable and efficient Kubernetes operations.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *