Simple Multi-Tenancy: Get Started with Capsule

Managing multi-tenant environments in Kubernetes can quickly become a labyrinth of namespaces, RBAC rules, resource quotas, and network policies. As organizations scale and onboard more teams or customers onto a shared cluster, the complexity of isolating tenants while maintaining operational efficiency becomes a significant challenge. Traditional approaches often involve manual configuration of numerous Kubernetes primitives, leading to potential security gaps, inconsistent resource allocation, and an administrative overhead that can overwhelm even the most experienced platform teams.

Enter Capsule, a powerful open-source multi-tenancy operator for Kubernetes. Capsule addresses these complexities head-on by introducing the concept of a “Tenant” – a higher-level abstraction that aggregates multiple Kubernetes Namespaces, along with their associated resources and policies, under a single administrative domain. This approach simplifies the management of multi-tenant clusters, allowing platform administrators to delegate control to tenant owners while ensuring strict isolation and resource governance. Capsule transforms a flat Kubernetes cluster into a hierarchical structure, making it easier to manage diverse workloads and teams securely and efficiently.

This guide will walk you through the process of setting up and managing multi-tenancy with Capsule. We’ll cover everything from installation to creating tenants, assigning users, managing resource quotas, and implementing network policies. By the end of this tutorial, you’ll have a clear understanding of how Capsule can streamline your multi-tenant Kubernetes operations, enhance security, and empower your development teams with self-service capabilities.

Prerequisites

Before diving into Capsule, ensure you have the following:

• A Kubernetes Cluster: Version 1.18+ is recommended. You can use any cloud provider (AWS EKS, GKE, Azure AKS) or a local cluster like Kind or Minikube.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation.
• helm: The Kubernetes package manager, version 3+. Install it by following the Helm installation guide.
• Basic Kubernetes Knowledge: Familiarity with Namespaces, RBAC, Resource Quotas, and Custom Resource Definitions (CRDs) is beneficial.
• Administrative Privileges: You’ll need cluster-admin access to install Capsule and create Tenants.

Step-by-Step Guide: Implementing Multi-Tenancy with Capsule

Step 1: Install Capsule

The first step is to install Capsule into your Kubernetes cluster. Capsule is deployed as an operator, which means it will run as a set of pods and CRDs to manage the multi-tenancy abstractions. The easiest way to install Capsule is by using Helm.

We’ll add the Clastix Helm repository and then install Capsule into its own namespace, typically capsule-system. This keeps Capsule’s components isolated from your tenant workloads and other cluster services. The Helm chart handles the deployment of the Capsule controller, necessary CRDs, and associated RBAC roles.

# Add the Clastix Helm repository
helm repo add clastix https://clastix.github.io/charts

# Update your Helm repositories
helm repo update

# Install Capsule into the 'capsule-system' namespace
helm install capsule clastix/capsule -n capsule-system --create-namespace

Verify:
After the installation, you should see Capsule pods running in the capsule-system namespace. You can check their status using kubectl get pods.

kubectl get pods -n capsule-system

Expected Output:

NAME                                    READY   STATUS    RESTARTS   AGE
capsule-controller-manager-69f8c6f497-abcde   1/1     Running   0          2m

You should also verify that the Capsule Custom Resource Definitions (CRDs) have been installed:

kubectl get crds | grep capsule

Expected Output:

tenants.capsule.clastix.io                          2023-10-27T10:00:00Z

Step 2: Create Your First Tenant

With Capsule installed, we can now create our first Tenant. A Tenant is the core abstraction in Capsule, representing a logical isolation boundary for a group of namespaces and their resources. When you define a Tenant, you specify its name, an optional owner, and various policies that will apply to all namespaces created within that tenant.

In this example, we’ll create a tenant named dev-team-a. We’ll also define an initial owner for this tenant. The owner will have administrative privileges over all namespaces and resources within their tenant. This delegation of power is a key feature of Capsule, enabling true self-service for development teams while maintaining central governance.

# tenant-dev-team-a.yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dev-team-a
spec:
  owner:
    # This must be a valid Kubernetes User or Group name,
    # typically obtained from your OIDC provider or Kubeconfig.
    # For simplicity, we'll use a dummy user for now.
    name: "dev-team-a-admin"
    kind: User
  namespaces:
    # Define constraints for namespaces within this Tenant
    quota: 5 # Allow a maximum of 5 namespaces for this tenant
  networkPolicies:
    # Define default NetworkPolicy for all namespaces in this Tenant
    # This example denies all ingress and egress by default, requiring explicit rules
    # For more on network policies, check out our Network Policies Security Guide
    items:
      - apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: default-deny-all
        spec:
          podSelector: {}
          policyTypes:
            - Ingress
            - Egress

Apply this manifest to your cluster:

kubectl apply -f tenant-dev-team-a.yaml

Verify:
You can check the status of your new tenant using kubectl get tenant.

kubectl get tenant dev-team-a

Expected Output:

NAME         STATE     OWNER            NAMESPACE_COUNT   AGE
dev-team-a   Active    User:dev-team-a-admin   0                 1m

Notice the NAMESPACE_COUNT is 0. This is because the tenant itself doesn’t create namespaces; it only provides the framework for them. We’ll create namespaces in the next step.

Step 3: Create Namespaces within a Tenant

Once a Tenant is defined, users assigned as owners of that Tenant can create namespaces. When a namespace is created, Capsule automatically links it to the appropriate Tenant based on the requesting user’s identity. Capsule also injects the policies defined in the Tenant (like resource quotas, limit ranges, and network policies) into the new namespace.

For this step, we’ll simulate creating a namespace as the dev-team-a-admin user. In a real-world scenario, you would configure your Kubernetes authentication system (e.g., OIDC, client certificates) to map your users to the names specified in the Tenant’s owner field. For demonstration purposes, we’ll temporarily bind the dev-team-a-admin user to the cluster-admin role to allow them to create namespaces, but in production, you’d use a more restrictive approach.

# First, create a dummy RoleBinding for the dev-team-a-admin user
# This is for demonstration purposes only. In a real setup,
# your authentication provider would handle user identities.
kubectl create rolebinding dev-team-a-admin-rb --clusterrole=cluster-admin --user=dev-team-a-admin

# Now, let's create a namespace, assuming we are logged in as 'dev-team-a-admin'
# Capsule will intercept this request and associate it with the 'dev-team-a' tenant.
# Note: kubectl doesn't have a direct way to "impersonate" a user for 'create namespace'.
# For this example, we'll create it as a cluster admin, and Capsule will still link it.
# In a real scenario, the 'dev-team-a-admin' user would simply run `kubectl create ns dev-project-alpha`.

Since kubectl create ns doesn’t support impersonation directly for namespace creation that Capsule can intercept, we’ll rely on Capsule’s admission controller to link the namespace to the tenant based on the capsule.clastix.io/tenant annotation. This annotation is typically added automatically by Capsule when a tenant owner creates a namespace. If you’re creating it as a cluster admin, you can manually add the annotation:

# namespace-dev-project-alpha.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: dev-project-alpha
  labels:
    # Capsule will automatically add this label if created by a tenant owner
    # For manual creation by cluster-admin, we add it to link to the tenant
    capsule.clastix.io/tenant: dev-team-a

kubectl apply -f namespace-dev-project-alpha.yaml

Verify:
Check the tenant status again. You should see the NAMESPACE_COUNT incremented. Also, inspect the namespace to see if Capsule has injected any resources.

kubectl get tenant dev-team-a

Expected Output:

NAME         STATE     OWNER            NAMESPACE_COUNT   AGE
dev-team-a   Active    User:dev-team-a-admin   1                 5m

You can also check the labels on the namespace:

kubectl get namespace dev-project-alpha -o yaml | grep "capsule.clastix.io/tenant"

Expected Output:

    capsule.clastix.io/tenant: dev-team-a

Step 4: Configure Resource Quotas and Limit Ranges for a Tenant

One of the most critical aspects of multi-tenancy is resource governance. Capsule allows you to define cluster-wide resource quotas and limit ranges that apply to all namespaces within a specific Tenant. This prevents a single tenant from monopolizing cluster resources and ensures fair usage.

We’ll update our dev-team-a tenant to include a ResourceQuota and a LimitRange. The ResourceQuota will restrict the total CPU, memory, and pod count across all namespaces belonging to dev-team-a. The LimitRange will set default CPU and memory limits/requests for containers within those namespaces, ensuring that pods don’t consume excessive resources by default.

# tenant-dev-team-a-updated.yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dev-team-a
spec:
  owner:
    name: "dev-team-a-admin"
    kind: User
  namespaces:
    quota: 5
  resourceQuotas:
    items:
      - apiVersion: v1
        kind: ResourceQuota
        metadata:
          name: tenant-quota
        spec:
          hard:
            requests.cpu: "2"
            requests.memory: "4Gi"
            limits.cpu: "4"
            limits.memory: "8Gi"
            pods: "20"
  limitRanges:
    items:
      - apiVersion: v1
        kind: LimitRange
        metadata:
          name: tenant-limit-range
        spec:
          limits:
            - default:
                cpu: 500m
                memory: 512Mi
              defaultRequest:
                cpu: 100m
                memory: 128Mi
              type: Container
  networkPolicies:
    items:
      - apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: default-deny-all
        spec:
          podSelector: {}
          policyTypes:
            - Ingress
            - Egress

Apply the updated tenant manifest:

kubectl apply -f tenant-dev-team-a-updated.yaml

Verify:
Capsule automatically propagates these resources to all namespaces owned by the tenant. Check the dev-project-alpha namespace for the injected ResourceQuota and LimitRange.

kubectl get resourcequota -n dev-project-alpha

Expected Output:

NAME           AGE
tenant-quota   1m

kubectl get limitrange -n dev-project-alpha

Expected Output:

NAME               AGE
tenant-limit-range   1m

You can also describe these resources to see their full configuration.

Step 5: Implement Tenant-Level Network Policies

Network isolation is crucial in multi-tenant environments. Capsule allows you to define Kubernetes Network Policies that are automatically applied to all namespaces within a Tenant. This ensures consistent security posture across all of a tenant’s workloads.

In our initial tenant definition, we already included a default-deny-all network policy. This is a best practice for security, ensuring that no traffic is allowed unless explicitly permitted. Now, let’s add a more specific policy that allows ingress traffic on port 80 to pods with a specific label, within the tenant.

For more advanced networking capabilities, especially with features like encryption and eBPF-based policies, consider solutions like Cilium WireGuard Encryption or eBPF Observability with Hubble.

# tenant-dev-team-a-network-policy.yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dev-team-a
spec:
  owner:
    name: "dev-team-a-admin"
    kind: User
  namespaces:
    quota: 5
  resourceQuotas:
    items:
      - apiVersion: v1
        kind: ResourceQuota
        metadata:
          name: tenant-quota
        spec:
          hard:
            requests.cpu: "2"
            requests.memory: "4Gi"
            limits.cpu: "4"
            limits.memory: "8Gi"
            pods: "20"
  limitRanges:
    items:
      - apiVersion: v1
        kind: LimitRange
        metadata:
          name: tenant-limit-range
        spec:
          limits:
            - default:
                cpu: 500m
                memory: 512Mi
              defaultRequest:
                cpu: 100m
                memory: 128Mi
              type: Container
  networkPolicies:
    items:
      - apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: default-deny-all
        spec:
          podSelector: {}
          policyTypes:
            - Ingress
            - Egress
      - apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: allow-web-ingress
        spec:
          podSelector:
            matchLabels:
              app: webserver
          policyTypes:
            - Ingress
          ingress:
            - from:
                - podSelector: {} # Allow from any pod within the same namespace
              ports:
                - protocol: TCP
                  port: 80

Apply the updated tenant manifest:

kubectl apply -f tenant-dev-team-a-network-policy.yaml

Verify:
Check the dev-project-alpha namespace for the new NetworkPolicy. Capsule ensures these policies are present and enforced.

kubectl get networkpolicy -n dev-project-alpha

Expected Output:

NAME                 POD-SELECTOR          AGE
allow-web-ingress    app=webserver         1m
default-deny-all     <none>                5m

Step 6: Delegating Tenant Ownership and RBAC

Capsule’s true power lies in its ability to delegate administrative control to tenant owners without granting them cluster-wide privileges. When you define an owner in the Tenant CRD, Capsule automatically creates specific RBAC roles and role bindings within the tenant’s namespaces.

The capsule.clastix.io/tenant-owner role grants permissions to manage namespaces, create resources, and apply policies within the tenant’s boundaries. This allows tenant owners to onboard their teams, create new namespaces, and deploy applications without needing intervention from the central cluster administrators.

To demonstrate, we’ll configure kubectl to impersonate the dev-team-a-admin user and attempt to create a new namespace. This user should succeed if the RBAC is correctly configured by Capsule.

# Create a new namespace as if 'dev-team-a-admin' is creating it
# Note: This command uses `--as` for impersonation.
# The user 'dev-team-a-admin' must exist in your auth system or be mocked.
# The previous `cluster-admin` binding was for the initial namespace creation.
# Capsule automatically grants `capsule.clastix.io/tenant-owner` to the specified user.

# Let's create another namespace under dev-team-a as the tenant owner
kubectl create namespace dev-project-beta --as=dev-team-a-admin

Verify:
Check if the new namespace dev-project-beta was created and if it’s associated with dev-team-a.

kubectl get namespace dev-project-beta -o yaml | grep "capsule.clastix.io/tenant"

Expected Output:

    capsule.clastix.io/tenant: dev-team-a

kubectl get tenant dev-team-a

Expected Output:

NAME         STATE     OWNER            NAMESPACE_COUNT   AGE
dev-team-a   Active    User:dev-team-a-admin   2                 10m

Now, try to create a namespace as dev-team-a-admin that exceeds the tenant’s namespace quota (e.g., if quota is 2, try creating a 3rd namespace):

kubectl create namespace dev-project-gamma --as=dev-team-a-admin

Expected Output (Error):

Error from server (Forbidden): admission webhook "tenant.capsule.clastix.io" denied the request: Not enough available Namespaces for the current Tenant

This demonstrates Capsule’s effective enforcement of tenant-level quotas.

Production Considerations

Deploying Capsule in a production environment requires careful planning beyond the basic setup:

1. Authentication and Authorization Integration:

   In production, you’ll integrate Capsule with your existing identity provider (e.g., OIDC, LDAP, Active Directory). Ensure the user and group names specified in the Tenant owner field match your IDP. Capsule relies on Kubernetes’ native authentication webhook or OIDC tokens to identify users. Consider using tools like Dex or kube-rbac-proxy for robust authentication.

2. Resource Management and Cost Optimization:

   Fine-tune ResourceQuota and LimitRange for each tenant based on their actual needs. Over-provisioning can lead to wasted resources, while under-provisioning can hinder development. Monitor resource usage closely. For advanced cost optimization, especially with dynamic node provisioning, consider integrating with tools like Karpenter.

3. Network Security and Isolation:

   The default-deny-all network policy is a strong starting point. Work with tenant owners to define granular ingress and egress rules. Consider advanced CNI solutions like Cilium for enhanced network visibility, policy enforcement, and even WireGuard encryption for pod-to-pod traffic.

4. Observability and Monitoring:

   Monitor Capsule’s controller manager for any errors or performance issues. Implement cluster-wide logging and monitoring to track resource consumption per tenant. Tools like Prometheus and Grafana can provide dashboards for tenant resource usage. For deeper network observability, especially with eBPF, check out eBPF Observability with Hubble.

5. Admission Control and Policy Enforcement:

   Capsule acts as an admission controller. Be aware of the order of admission controllers if you have others deployed (e.g., Kyverno, OPA Gatekeeper). Ensure Capsule’s webhooks are correctly configured and have high availability. For supply chain security, integrating Capsule with tools like Sigstore and Kyverno can enforce image signing policies at the tenant level.

6. Backup and Disaster Recovery:

   Regularly back up your Kubernetes cluster state, including Capsule CRDs and configuration. Tools like Velero can help with this. Ensure your backup strategy accounts for the hierarchical nature introduced by Capsule.

7. Service Mesh Integration:

   If you’re using a service mesh like Istio, ensure its policies can coexist with Capsule’s network policies. Capsule operates at the Kubernetes API level, while service meshes operate at the application layer. For managing traffic within a multi-tenant service mesh, especially with newer architectures, refer to our Istio Ambient Mesh Production Guide.

8. Namespace Naming Conventions:

   Establish clear naming conventions for tenants and namespaces to maintain order and simplify management as your cluster grows. Capsule can enforce regex patterns for namespace names within a tenant.

Troubleshooting

Here are some common issues you might encounter with Capsule and their solutions.

1. Capsule Controller Pods Not Running

   Issue: The capsule-controller-manager pod is in a Pending, CrashLoopBackOff, or Error state.

   Solution:

   • Check pod logs for errors:

     kubectl logs -n capsule-system -l app.kubernetes.io/name=capsule

   • Describe the pod to check for events:

     kubectl describe pod -n capsule-system -l app.kubernetes.io/name=capsule

   • Common causes include insufficient resources (memory/CPU), incorrect RBAC for the controller, or issues with webhook certificates. Ensure your cluster has enough resources and that the capsule-controller-manager service account has the necessary permissions.
2. Tenant Owner Cannot Create Namespaces

   Issue: A user assigned as a Tenant owner receives a “Forbidden” error when trying to create a new namespace.

   Solution:

   • Verify the user’s identity: Ensure the user’s name (as seen by Kubernetes authentication) exactly matches the spec.owner.name in the Tenant CRD. Use

     kubectl auth can-i create namespace --as=<username>

     to check their permissions.

   • Check the Tenant’s namespaceQuota: If the tenant has reached its maximum allowed namespaces, new namespace creation will be blocked. Increase the quota or delete existing namespaces.
   • Inspect Capsule logs: The controller logs might provide more details on why the admission webhook denied the request.
3. Resource Quotas or Limit Ranges Not Applied

   Issue: Namespaces created within a Tenant do not have the specified ResourceQuota or LimitRange applied.

   Solution:

   • Ensure the namespace is correctly linked to the Tenant: Check if the namespace has the label capsule.clastix.io/tenant: <tenant-name>. If not, Capsule won’t manage it.
   • Verify Tenant CRD: Double-check the resourceQuotas and limitRanges sections in your Tenant definition for typos or incorrect YAML syntax.
   • Check Capsule controller logs for any errors during resource propagation.
4. Network Policies Not Enforced

   Issue: Network policies defined in the Tenant CRD are not taking effect in the tenant’s namespaces.

   Solution:

   • Confirm CNI support: Ensure your Container Network Interface (CNI) plugin (e.g., Calico, Cilium, Weave Net) supports Kubernetes Network Policies and is properly configured.
   • Verify policy presence: Check if the network policies are actually created in the target namespace:

     kubectl get networkpolicy -n <namespace>

   • Review policy rules: Incorrect podSelector or port definitions can lead to policies not matching intended traffic.
5. Cannot Delete Tenant

   Issue: Attempting to delete a Tenant CRD fails or hangs.

   Solution:

   • Capsule prevents deletion of a Tenant if there are still namespaces associated with it. You must first delete all namespaces belonging to the Tenant.
   • If the tenant is stuck in a terminating state, check for finalizers on the Tenant object:

     kubectl get tenant <tenant-name> -o yaml

     . If there are stuck finalizers, you might need to manually remove them (with extreme caution) if Capsule’s controller is not running or responsive.

6. Webhook Certificate Issues

   Issue: Admission webhooks fail, leading to errors like “Internal error occurred: failed calling webhook” when creating resources.

   Solution:

   • Capsule uses a MutatingWebhookConfiguration and ValidatingWebhookConfiguration. These require valid TLS certificates. Capsule’s Helm chart typically handles this with a cert-manager integration or a self-signing mechanism.
   • Check the logs of the capsule-controller-manager for certificate-related errors.
   • Verify the MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources in your cluster, ensuring the caBundle is correct.

FAQ Section

1. What is the difference between Capsule and Kubernetes Namespaces?

   Namespaces provide a basic level of isolation within Kubernetes, primarily for naming collision avoidance and resource scoping. Capsule introduces the concept of a “Tenant,” which is a higher-level abstraction. A Tenant aggregates multiple Namespaces under a single administrative domain, allowing cluster administrators to apply cluster-wide policies (RBAC, Resource Quotas, Network Policies) to a group of Namespaces owned by a single team or user, and delegate control over those Namespaces to the Tenant owner. This simplifies multi-tenancy management significantly.

2. Can Capsule enforce policies on existing namespaces?

   Yes, Capsule can manage existing namespaces. If you want to bring an existing namespace under a Tenant’s control, you need to label it with capsule.clastix.io/tenant: <tenant-name>. Capsule will then apply the Tenant’s policies to that namespace. However, it’s generally recommended to let Tenant owners create namespaces through Capsule for a smoother experience.

3. How does Capsule handle user authentication and authorization?

   Capsule leverages Kubernetes’ native authentication and authorization mechanisms. When you define a Tenant owner (e.g., User:dev-team-a-admin), Capsule expects that user to be authenticated by your cluster’s OIDC provider, client certificates, or other configured authentication methods. Once authenticated, Capsule’s admission webhooks intercept requests and check if the user is a Tenant owner to apply the correct policies and delegate permissions. For more on Kubernetes RBAC, refer to the official documentation.

4. Is Capsule suitable for strict security isolation between tenants?

   Capsule provides strong logical isolation by enforcing resource quotas, network policies, and RBAC boundaries at the tenant level. However, for the most stringent security requirements, especially in highly regulated environments or for untrusted multi-tenant scenarios, you might consider additional layers of isolation such as dedicated clusters per tenant, node segregation, or advanced container runtimes. Capsule significantly reduces the attack surface and administrative burden for many multi-tenant use cases.

5. Can I use Capsule with a Service Mesh like Istio or Linkerd?

   Yes, Capsule can be used alongside a service mesh. Capsule operates at the Kubernetes API level, enforcing policies on resource creation and modification. A service mesh like Istio or Linkerd operates at the application layer, managing traffic routing, mTLS, and observability between services. They complement each other. You would define your tenant-level network policies with Capsule, and then use your service mesh for fine-grained traffic management within and across namespaces. For a deeper dive into service mesh deployments, check out our

   Was this article helpful?

   Yes No