Build Multi-Tenant Apps: Capsule Simplifies

Managing multi-tenant environments in Kubernetes can quickly become a labyrinth of complexity. From isolating resources and enforcing policies to ensuring fair usage and security, the challenges are significant. Traditional approaches often involve a combination of namespaces, RBAC, quotas, and admission controllers, which, while powerful, demand meticulous configuration and ongoing maintenance. This fragmented approach can lead to configuration drift, security gaps, and a steep learning curve for platform engineers.

Enter Capsule, a powerful open-source Kubernetes operator designed to simplify multi-tenancy. Capsule introduces the concept of a “Tenant,” a higher-level abstraction that aggregates multiple Kubernetes Namespaces under a single administrative domain. This allows platform administrators to define policies, resource quotas, and access controls at the Tenant level, which Capsule then automatically propagates and enforces across all associated Namespaces. The result is a streamlined, more secure, and scalable multi-tenant Kubernetes environment, freeing up developers to focus on applications rather than infrastructure minutiae.

This guide will walk you through the process of setting up and managing multi-tenancy with Capsule. We’ll cover everything from installation and Tenant creation to advanced policy enforcement and resource management. By the end of this tutorial, you’ll have a robust understanding of how Capsule can transform your Kubernetes cluster into a true multi-tenant platform, providing strong isolation and efficient resource governance. Let’s dive in and unlock the full potential of your Kubernetes infrastructure with Capsule!

Prerequisites

Before we begin, ensure you have the following:

• A running Kubernetes cluster (v1.19+ recommended). This can be a local cluster like Kind or Minikube, or a cloud-managed cluster (EKS, GKE, AKS).
• kubectl installed and configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• helm installed (v3.0.0+). Check the Helm documentation for installation details.
• Basic understanding of Kubernetes concepts: Namespaces, RBAC, ResourceQuotas, and NetworkPolicies.
• Admin-level access to your Kubernetes cluster to install cluster-scoped resources like Capsule.

Step-by-Step Guide: Setting Up Multi-Tenancy with Capsule

Step 1: Install Capsule

First, we need to install Capsule into your Kubernetes cluster. Capsule is typically deployed as an operator via Helm. It will create a new namespace, capsule-system, and deploy all necessary components, including the Capsule controller and associated CRDs (Custom Resource Definitions).

The Capsule controller watches for Tenant and other Capsule-specific custom resources. When a Tenant is created or modified, Capsule automatically provisions and manages the underlying Kubernetes resources (like Namespaces, RBAC, ResourceQuotas, and NetworkPolicies) according to the Tenant’s specification. This automation is key to simplifying multi-tenancy management.

helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install capsule clastix/capsule -n capsule-system --create-namespace

Verify Capsule Installation

After running the Helm command, verify that Capsule pods are running in the capsule-system namespace and that the CRDs have been installed correctly.

kubectl get pods -n capsule-system

# Expected Output:
# NAME                                     READY   STATUS    RESTARTS   AGE
# capsule-5f7d8c474c-abcde                  1/1     Running   0          2m

kubectl get crd | grep capsule

# Expected Output:
# tenants.capsule.clastix.io                       2023-10-27T10:00:00Z
# tenantresourcequotas.capsule.clastix.io          2023-10-27T10:00:00Z
# ... (other Capsule CRDs)

Step 2: Create a Tenant and Assign an Owner

Now that Capsule is installed, let’s create our first Tenant. A Tenant in Capsule represents an isolated administrative domain, usually corresponding to a team, department, or customer. Each Tenant must have an owner, which is a Kubernetes User or ServiceAccount. This owner will have administrative control over the namespaces within their Tenant.

In this example, we’ll create a Tenant named dev-team-a and assign a user, dev-a-admin, as its owner. We’ll also define some initial resource quotas and network policies for this tenant. Capsule will automatically apply these configurations to all namespaces created under this tenant. This centralized approach drastically reduces the boilerplate needed for each new namespace.

# tenant-dev-team-a.yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dev-team-a
spec:
  owner:
    kind: User
    name: dev-a-admin
  namespaceOptions:
    quota: 5 # Allow a maximum of 5 namespaces for this tenant
  networkPolicies:
    - apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: deny-all-traffic
      spec:
        podSelector: {}
        policyTypes:
          - Ingress
          - Egress
    - apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-dns
      spec:
        podSelector: {}
        egress:
          - to:
              - ipBlock:
                  cidr: 0.0.0.0/0 # This should be restricted to cluster DNS or specific DNS servers
            ports:
              - port: 53
                protocol: UDP
              - port: 53
                protocol: TCP
        policyTypes:
          - Egress
  resourceQuotas:
    - apiVersion: v1
      hard:
        cpu: "10"
        memory: "20Gi"
        pods: "50"
      kind: ResourceQuota
      metadata:
        name: dev-team-a-quota
  limitRanges:
    - apiVersion: v1
      kind: LimitRange
      metadata:
        name: dev-team-a-limits
      spec:
        limits:
          - default:
              cpu: 500m
              memory: 512Mi
            defaultRequest:
              cpu: 100m
              memory: 128Mi
            type: Container

Apply the Tenant manifest:

kubectl apply -f tenant-dev-team-a.yaml

Verify Tenant Creation

Check if the Tenant has been created:

kubectl get tenant dev-team-a

# Expected Output:
# NAME         NAMESPACE QUOTA   AGE
# dev-team-a   5                 1m

You can also inspect the Tenant’s full configuration:

kubectl get tenant dev-team-a -o yaml

Step 3: Impersonate the Tenant Owner and Create Namespaces

With the Tenant created, the designated owner (dev-a-admin in our case) can now create namespaces under their administrative domain. Capsule ensures that any namespace created by or assigned to this owner will automatically inherit the policies, quotas, and limits defined in the dev-team-a Tenant.

To demonstrate this, we’ll use kubectl --as=dev-a-admin to impersonate the tenant owner. This is crucial for testing the delegated administrative capabilities that Capsule provides. Note that dev-a-admin must exist as a Kubernetes User (e.g., via a ServiceAccount or an external identity provider integrated with Kubernetes RBAC). For simplicity in this tutorial, we’re relying on Kubernetes’ built-in impersonation feature for demonstration purposes. In a real-world scenario, you’d configure actual RBAC for this user.

# Create a Namespace as dev-a-admin
kubectl create namespace dev-a-frontend --as=dev-a-admin

# Expected Output:
# namespace/dev-a-frontend created

kubectl create namespace dev-a-backend --as=dev-a-admin

# Expected Output:
# namespace/dev-a-backend created

Verify Namespace Ownership and Policies

Check if the namespaces are correctly associated with the Tenant and if the policies have been applied. Capsule adds specific labels to namespaces belonging to a Tenant.

kubectl get namespace dev-a-frontend -L capsule.clastix.io/tenant

# Expected Output:
# NAME             STATUS   AGE     TENANT
# dev-a-frontend   Active   1m      dev-team-a

kubectl get networkpolicy -n dev-a-frontend

# Expected Output:
# NAME                 POD-SELECTOR   AGE
# allow-dns            <none>         1m
# deny-all-traffic     <none>         1m

kubectl get resourcequota -n dev-a-frontend

# Expected Output:
# NAME                 AGE   REQUEST                                         LIMIT
# dev-team-a-quota     1m    cpu: 0/10, memory: 0/20Gi, pods: 0/50

kubectl get limitrange -n dev-a-frontend

# Expected Output:
# NAME                AGE
# dev-team-a-limits   1m

As you can see, the dev-a-frontend namespace is labeled with capsule.clastix.io/tenant=dev-team-a, and the NetworkPolicy, ResourceQuota, and LimitRange objects defined in our Tenant manifest have been automatically created within this new namespace. This automatic propagation is the core value proposition of Capsule, simplifying multi-tenancy management significantly. For more details on Kubernetes networking and security, refer to our Network Policies Security Guide.

Step 4: Configure RBAC for Tenant Owners

While kubectl --as is great for demonstration, in a real environment, you need to grant the actual Kubernetes user or service account the necessary permissions to manage their namespaces within the Tenant. Capsule provides specific ClusterRoles for this purpose, which you can bind to your tenant owners.

The capsule-tenant-owner ClusterRole allows users to manage namespaces, network policies, resource quotas, and limit ranges within their assigned Tenants. It’s crucial to bind this role to the correct subject (User or ServiceAccount).

# rbac-dev-a-admin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dev-a-admin-tenant-owner
subjects:
  - kind: User # Could also be ServiceAccount or Group
    name: dev-a-admin # This user must exist in your auth system
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: capsule-tenant-owner
  apiGroup: rbac.authorization.k8s.io

Apply the ClusterRoleBinding:

kubectl apply -f rbac-dev-a-admin.yaml

Verify RBAC

You can verify the permissions using kubectl auth can-i, though it’s more definitive to try creating a resource as the user after genuine authentication.

# As a cluster admin, you can check user permissions (conceptual check)
kubectl auth can-i create namespace --as=dev-a-admin

# Expected Output:
# yes

This RBAC setup ensures that dev-a-admin can create and manage namespaces, along with their associated resources like NetworkPolicies and ResourceQuotas, but only within their assigned Tenant. They cannot, for example, delete another team’s namespaces or modify cluster-wide resources.

Step 5: Advanced Tenant Configuration – Ingress and Storage Classes

Capsule allows for more advanced configurations, such as restricting Ingress hostnames, limiting allowed storage classes, and even defining custom labels and annotations for namespaces. This level of control further enhances isolation and adherence to organizational standards.

Let’s update our dev-team-a Tenant to restrict Ingress hostnames and specify allowed storage classes. This is particularly useful for ensuring that tenants only use approved domain names and storage provisions, aligning with security and cost policies. For detailed insights into networking, consider our Kubernetes Gateway API vs Ingress Guide.

# tenant-dev-team-a-advanced.yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dev-team-a
spec:
  owner:
    kind: User
    name: dev-a-admin
  namespaceOptions:
    quota: 5
    additionalLabels:
      env: development
    additionalAnnotations:
      contact: dev-a-admin@example.com
  ingressOptions:
    hostnameCollisionScope: Tenant # Prevent hostname collision within the tenant
    allowedHostnames: # Restrict ingress hostnames
      - "*.dev-a.example.com"
      - "app.dev-a.example.com"
  storageClasses: # Restrict allowed StorageClasses
    - gp2 # Example for AWS EBS
    - standard # Common for many cloud providers
  networkPolicies:
    # ... (existing network policies)
    - apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: deny-all-traffic
      spec:
        podSelector: {}
        policyTypes:
          - Ingress
          - Egress
    - apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-dns
      spec:
        podSelector: {}
        egress:
          - to:
              - ipBlock:
                  cidr: 0.0.0.0/0 # This should be restricted to cluster DNS or specific DNS servers
            ports:
              - port: 53
                protocol: UDP
              - port: 53
                protocol: TCP
        policyTypes:
          - Egress
  resourceQuotas:
    - apiVersion: v1
      hard:
        cpu: "10"
        memory: "20Gi"
        pods: "50"
      kind: ResourceQuota
      metadata:
        name: dev-team-a-quota
  limitRanges:
    - apiVersion: v1
      kind: LimitRange
      metadata:
        name: dev-team-a-limits
      spec:
        limits:
          - default:
              cpu: 500m
              memory: 512Mi
            defaultRequest:
              cpu: 100m
              memory: 128Mi
            type: Container

Apply the updated Tenant manifest:

kubectl apply -f tenant-dev-team-a-advanced.yaml

Verify Advanced Configuration

When a tenant owner creates a new namespace, it will automatically get the specified labels and annotations. Also, any Ingress created in that tenant’s namespaces will be validated against the allowedHostnames.

kubectl create namespace dev-a-data --as=dev-a-admin

# Expected Output:
# namespace/dev-a-data created

kubectl get namespace dev-a-data -o yaml | grep -E "env|contact"

# Expected Output (truncated):
#   annotations:
#     contact: dev-a-admin@example.com
#   labels:
#     env: development
#     ...

Now, let’s try to create an Ingress with an invalid hostname in dev-a-data as dev-a-admin:

# invalid-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-invalid-ingress
spec:
  rules:
  - host: www.another-domain.com # This is not allowed by the Tenant
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80

kubectl apply -f invalid-ingress.yaml -n dev-a-data --as=dev-a-admin

# Expected Output (Error from Capsule webhook):
# Error from server (Forbidden): admission webhook "validating.capsule.clastix.io" denied the request: Ingress hostname www.another-domain.com is not allowed for the current Tenant

This demonstrates Capsule’s admission control in action, enforcing the policies defined at the Tenant level. This significantly enhances security and governance in a multi-tenant environment.

Production Considerations

Deploying Capsule in a production environment requires careful planning and consideration to ensure stability, security, and scalability:

1. RBAC Integration: While we used kubectl --as for demonstration, in production, integrate Capsule’s RBAC with your organization’s identity management system (e.g., LDAP, OAuth2/OIDC via tools like Dex or Keycloak). Map your users/groups to Kubernetes Users/ServiceAccounts and bind the capsule-tenant-owner ClusterRole appropriately.
2. Resource Quotas and Limit Ranges: Define realistic and well-thought-out resource quotas and limit ranges for each Tenant. Over-provisioning can lead to wasted resources, while under-provisioning can hinder development. Regularly review and adjust these based on actual usage. Consider tools like Karpenter for cost optimization in conjunction with Capsule for node autoscaling.
3. Network Policies: Carefully design network policies at the Tenant level to enforce strict isolation between tenants and restrict egress traffic. For advanced network security and encryption, consider integrating with CNIs like Cilium, as discussed in our Cilium WireGuard Encryption Guide.
4. Monitoring and Alerting: Monitor Capsule’s own components (pods, controller logs) for health and performance. Implement alerts for failed Tenant creations, policy violations, or other critical events. Leverage tools like Prometheus and Grafana for observability. For advanced eBPF-based observability, check out eBPF Observability with Hubble.
5. Admission Controllers: Capsule heavily relies on Kubernetes admission controllers. Ensure they are correctly configured and that Capsule’s webhooks are functioning. Be mindful of the order of admission controllers if you have multiple in place.
6. Backup and Restore: Implement a robust backup strategy for your Kubernetes cluster, including Capsule’s CRDs and their data. Tools like Velero can be used to back up and restore cluster resources.
7. Security Contexts and Pod Security Standards: Enforce strong Pod Security Standards (PSS) or Pod Security Policies (PSPs – deprecated in favor of PSS) at the cluster level to ensure tenant workloads run with appropriate security contexts. Capsule can complement these by providing a tenant-specific enforcement layer. For supply chain security, integrating with tools like Sigstore and Kyverno, as detailed in Sigstore and Kyverno Security, is also vital.
8. Logging: Centralize logs from all tenant namespaces and Capsule itself to a logging solution (e.g., Elasticsearch, Loki) for auditing and troubleshooting.
9. Multi-Cluster Considerations: If you operate multiple Kubernetes clusters, decide whether to deploy Capsule on each cluster independently or to use a multi-cluster management solution to orchestrate Tenants across clusters.
10. Version Management: Stay updated with Capsule releases. Regularly review release notes for new features, bug fixes, and security patches. Plan upgrades carefully.

Troubleshooting

Here are some common issues you might encounter when working with Capsule and their solutions:

1. Issue: Capsule pods are not running or are in a CrashLoopBackOff state.

   Solution:

   1. Check the logs of the Capsule controller pod:

      kubectl logs -f -n capsule-system deploy/capsule

   2. Ensure that the capsule-system namespace has sufficient resources (CPU, memory).
   3. Verify that the Kubernetes API server is accessible and healthy.
   4. Check for any conflicting webhooks or misconfigured certificates if you’re using custom CA.
2. Issue: Namespaces are not being created under a Tenant, or policies are not being applied.

   Solution:

   1. Verify the Tenant object exists and is healthy:

      kubectl get tenant <tenant-name> -o yaml

   2. Ensure the user creating the namespace is correctly assigned as the Tenant owner and has the capsule-tenant-owner ClusterRoleBinding.
   3. Check Capsule controller logs for any errors related to namespace creation or policy propagation.
   4. Ensure the namespace quota for the Tenant has not been reached.
3. Issue: Tenant owner cannot create namespaces, even with correct RBAC.

   Solution:

   1. Double-check the ClusterRoleBinding. Ensure the name under subjects matches the actual user or service account.
   2. Verify the kind (User, ServiceAccount, Group) is correct.
   3. If using kubectl --as, ensure the user principal exists in your cluster’s authentication system for real-world scenarios.
   4. Check for any cluster-wide admission controllers that might be blocking namespace creation.
4. Issue: Ingress hostnames or StorageClasses are being rejected, even if they seem valid.

   Solution:

   1. Review the Tenant object’s ingressOptions.allowedHostnames and storageClasses fields carefully. Ensure the requested values exactly match the allowed patterns or names.
   2. Remember that allowedHostnames supports glob patterns (e.g., *.example.com) but is case-sensitive.
   3. Verify that the requested StorageClass actually exists in your cluster:

      kubectl get sc

   4. Check Capsule controller logs for admission webhook errors related to Ingress or PVC creation.
5. Issue: Network Policies defined in the Tenant are not working as expected.

   Solution:

   1. Ensure your CNI (Container Network Interface) plugin supports Network Policies (e.g., Calico, Cilium, Weave Net). Some basic CNIs like Flannel don’t support them natively without an additional component.
   2. Verify the Network Policies are correctly applied to the namespace:

      kubectl get netpol -n <namespace> -o yaml

   3. Network Policies are additive. If multiple policies apply to a pod, the most permissive rule wins. Ensure there are no conflicting policies.
   4. Test connectivity explicitly to diagnose which rules are blocking/allowing traffic.
6. Issue: ResourceQuotas are not limiting resources for pods within a Tenant’s namespace.

   Solution:

   1. Ensure that pods being deployed specify requests and limits for CPU and memory. ResourceQuotas primarily enforce these. Without them, pods might use more resources than intended until a LimitRange is also applied.
   2. Verify the ResourceQuota object exists in the target namespace:

      kubectl describe resourcequota <quota-name> -n <namespace>

   3. Check the LimitRange object as well, as it can set default requests/limits if pods don’t specify them.

FAQ Section

Here are some frequently asked questions about Capsule and multi-tenancy:

1. What is the main difference between Capsule and plain Kubernetes Namespaces for multi-tenancy?

   While Kubernetes Namespaces provide a basic level of isolation, managing policies (RBAC, ResourceQuotas, NetworkPolicies, LimitRanges) across many namespaces and ensuring consistency is complex. Capsule introduces the Tenant abstraction, which groups multiple namespaces and automatically propagates and enforces these policies from a single point. This simplifies management, reduces configuration overhead, and ensures policy consistency across an entire tenant’s domain, making multi-tenancy significantly easier to manage and scale.

2. Can Capsule manage existing namespaces?

   Yes, Capsule can adopt existing namespaces. You can label an existing namespace with capsule.clastix.io/tenant: <tenant-name>, and Capsule will then apply the policies defined in that Tenant to the namespace. Alternatively, a cluster administrator can explicitly assign an existing namespace to a tenant using the Tenant manifest.

3. How does Capsule handle cross-tenant communication?

   By default, Capsule (via Kubernetes Network Policies) can enforce strict isolation, preventing communication between pods in different tenants’ namespaces. If cross-tenant communication is required, you would typically define specific Network Policies to allow it, or use a service mesh like Istio Ambient Mesh for more granular control and secure communication channels.

4. What if a tenant owner tries to exceed their resource quotas?

   Capsule works in conjunction with Kubernetes’ built-in admission controllers. If a tenant owner attempts to deploy resources (e.g., pods, PVCs) that would exceed their Tenant’s aggregated ResourceQuota or violate a LimitRange, the Kubernetes API server, with Capsule’s webhook, will reject the request. The tenant owner will receive a “Forbidden” error, clearly indicating the quota or limit violation.

5. Is Capsule suitable for strict security requirements, like PCI DSS or HIPAA?

   Capsule provides a strong foundation for multi-tenancy by enforcing isolation and policies. However, achieving compliance with strict security standards like PCI DSS or HIPAA requires a holistic approach that goes beyond just namespace isolation. This includes robust cluster security, auditing, encryption (e.g., with Cilium WireGuard Encryption), vulnerability management, and strict access controls at all levels. Capsule significantly contributes to the “isolation” and “policy enforcement” aspects but needs to be part of a broader security strategy.

Cleanup Commands

To remove Capsule and all the resources we created during this tutorial, follow these steps:

1. Delete the Tenant and associated namespaces: Deleting the Tenant will also delete all namespaces managed by it.

   kubectl delete tenant dev-team-a

2. Delete the ClusterRoleBinding:

   kubectl delete clusterrolebinding dev-a-admin-tenant-owner

3. Uninstall Capsule via Helm:

   helm uninstall capsule -n capsule-system

4. Delete the Capsule system namespace:

   kubectl delete namespace capsule-system

5. Remove Helm repository (optional):

   helm repo remove clastix

Next Steps / Further Reading

Congratulations! You’ve successfully set up and configured multi-tenancy using Capsule. Here are some next steps and resources for deeper exploration:

• Explore Capsule’s Official Documentation: The official Capsule documentation is an excellent resource for detailed information on all features, advanced configurations, and best practices.
• Integrate with Identity Providers: Learn how to integrate Capsule’s RBAC with your organization’s identity management system (e.g., OIDC, Dex).
• Advanced Networking: Dive deeper into Kubernetes networking with tools like Cilium for advanced Network Policies, service mesh capabilities, and eBPF-based observability. Our Cilium WireGuard Encryption guide is a great starting point.
• Cost Management: Explore how to optimize your Kubernetes costs with dynamic node provisioning using tools like Karpenter. Read our Karpenter Cost Optimization guide.
• Service Mesh Integration: Consider deploying a service mesh like Istio or Linkerd within your tenants for advanced traffic management, observability, and security. Our Istio Ambient Mesh Production Guide offers deep insights.
• Policy Enforcement with Kyverno: For even more granular and custom policy enforcement beyond what Capsule offers, explore Kyverno. Our article on Securing Container Supply Chains with Sigstore and Kyverno provides a good overview.
• Observability: Enhance your cluster’s observability with tools like Prometheus, Grafana, and Loki. Understand how eBPF Observability with Hubble can provide deep insights into your network.

Conclusion

Capsule fundamentally changes how multi-tenancy is approached in Kubernetes. By introducing the Tenant abstraction, it elevates the management of shared clusters from a complex, error-prone, per-namespace configuration task to a streamlined, policy-driven process. We’ve seen how easy it is to install Capsule, define Tenants with comprehensive policies for resource quotas, network isolation, and ingress rules, and delegate namespace creation to tenant owners.

The automation provided by Capsule ensures consistency, reduces operational overhead, and empowers development teams while maintaining strict control and governance for platform administrators. This makes Kubernetes more accessible, secure, and scalable for organizations with diverse teams and applications. Embracing Capsule is a significant step towards building a truly robust and efficient multi-tenant Kubernetes platform.