Kubernetes Multi-Tenancy: Master vClusters

Introduction

Managing multi-tenancy in Kubernetes has long been a complex challenge. Organizations often struggle to balance isolation, security, and resource efficiency when multiple teams, departments, or even external customers share a single Kubernetes cluster. Traditional approaches, such as namespace-based tenancy, often fall short when stronger isolation is required, leading to “noisy neighbor” problems, complex RBAC configurations, and the potential for security breaches across tenant boundaries. While creating entirely separate clusters for each tenant offers maximum isolation, it introduces significant operational overhead, cost, and resource underutilization.

Enter vCluster, a game-changer in the multi-tenancy landscape. vCluster allows you to create lightweight, fully functional Kubernetes clusters inside a host Kubernetes cluster. Each vCluster behaves like a regular Kubernetes cluster, complete with its own API server, controller manager, and scheduler, but it leverages the underlying host cluster’s nodes, network, and storage. This innovative approach provides strong isolation for tenants at a fraction of the cost and complexity of dedicated clusters, making it an ideal solution for SaaS providers, development environments, and large enterprises looking to streamline their Kubernetes operations. This guide will walk you through setting up and managing multi-tenancy using vCluster, providing you with the tools and knowledge to unlock a new level of efficiency and isolation.

Prerequisites

Before diving into vCluster, ensure you have the following:

• A Kubernetes Cluster: An existing Kubernetes cluster (e.g., Minikube, Kind, GKE, EKS, AKS) where you have administrative access. vCluster does not run on bare metal directly but within another Kubernetes cluster. You can check your connection with kubectl get nodes.
• kubectl: The Kubernetes command-line tool, configured to connect to your host cluster. Refer to the official Kubernetes kubectl installation guide for instructions.
• Helm: The Kubernetes package manager. While not strictly required for every vCluster operation, it’s often used for advanced configurations and installing vCluster itself. Install it from the Helm documentation.
• vCluster CLI: The dedicated command-line interface for managing vClusters. This will be installed in the first step.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, Namespaces, and RBAC is assumed.

Step-by-Step Guide: Setting up Multi-Tenancy with vCluster

Step 1: Install the vCluster CLI

The vCluster CLI is your primary tool for interacting with virtual clusters. It simplifies creation, connection, and management tasks. It’s a single binary that you can download and add to your system’s PATH.

The installation process is straightforward, typically involving a curl command to fetch the binary and make it executable. Once installed, you can verify its version to ensure it’s correctly set up. The CLI handles much of the complexity, such as generating kubeconfigs and managing the underlying resources in the host cluster.

# Download and install the vCluster CLI
curl -sL https://cli.k0s.io/get | bash

# Move the binary to a directory in your PATH (e.g., /usr/local/bin)
sudo mv vcluster /usr/local/bin/

# Verify the installation
vcluster version

Verify:

vcluster version

Expected Output:

vcluster version: v0.18.0 # (version might vary)

Step 2: Create Your First Virtual Cluster

Creating a vCluster is as simple as running a single command. When you create a vCluster, it deploys a Pod in your host cluster that runs the control plane components (API server, controller manager, scheduler) of your virtual cluster. This Pod also includes a syncer component that synchronizes resources from the vCluster to the host cluster, such as Pods and PersistentVolumeClaims, which then get scheduled and provisioned on the host. The --expose flag makes the vCluster API server accessible from outside the host cluster, which is useful for local development or CI/CD pipelines.

The -n flag specifies the namespace in the host cluster where the vCluster’s components will reside. It’s good practice to create a dedicated namespace for each vCluster in the host cluster for better organization and isolation. For more advanced networking configurations and isolation between vClusters, consider implementing Kubernetes Network Policies in the host cluster.

# Create a dedicated namespace for your vcluster in the host cluster
kubectl create namespace vcluster-tenant-a

# Create a virtual cluster named 'tenant-a-vcluster' within the 'vcluster-tenant-a' namespace
# --expose allows external access to the vCluster API server (e.g., from your local machine)
vcluster create tenant-a-vcluster -n vcluster-tenant-a --expose

Verify:

Check if the vCluster Pod is running in the host cluster namespace. This Pod is the heart of your virtual cluster.

kubectl get pods -n vcluster-tenant-a

Expected Output:

NAME                                             READY   STATUS    RESTARTS   AGE
tenant-a-vcluster-0                              1/1     Running   0          2m

Step 3: Connect to the Virtual Cluster

Once your vCluster is running, you need to connect to it. The vcluster connect command automatically generates a kubeconfig file that points to your virtual cluster’s API server and sets your current kubectl context to use it. This allows you to interact with the virtual cluster as if it were a standalone Kubernetes cluster, without affecting your host cluster’s context.

This seamless context switching is one of vCluster’s most powerful features, enabling developers and operators to work within their isolated environments without manual kubeconfig management. You can now deploy applications, create namespaces, and manage resources within tenant-a-vcluster, completely isolated from other tenants or the host cluster’s default resources.

# Connect to the 'tenant-a-vcluster'
vcluster connect tenant-a-vcluster -n vcluster-tenant-a

Verify:

After connecting, verify that your kubectl context has switched and you are now interacting with the virtual cluster. You should see a new, empty virtual cluster.

kubectl config current-context
kubectl get namespaces

Expected Output (for kubectl config current-context):

vcluster_tenant-a-vcluster_vcluster-tenant-a # (context name might vary slightly)

Expected Output (for kubectl get namespaces):

NAME                   STATUS   AGE
default                Active   2m
kube-system            Active   2m
kube-public            Active   2m
kube-node-lease        Active   2m

Notice that the namespaces are the default ones for a fresh Kubernetes cluster, not those from your host cluster.

Step 4: Deploy an Application to the Virtual Cluster

Now that you’re connected to your vCluster, you can deploy applications just as you would to any standard Kubernetes cluster. The vCluster’s syncer component intercepts resource creation requests (e.g., Pods, Deployments) and translates them into corresponding resources in the host cluster. For instance, when you create a Pod in the vCluster, the syncer creates a “proxied” Pod in the host cluster’s namespace, which then gets scheduled on a host node.

This abstraction ensures that tenants have full control over their virtual cluster resources, while the host cluster handles the actual execution and resource allocation. This is where the magic of lightweight isolation happens. For more advanced resource optimization and cost management in your host cluster, especially when dealing with many vClusters, consider integrating tools like Karpenter for Kubernetes Cost Optimization.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

# Apply the deployment to your virtual cluster
kubectl apply -f deployment.yaml

Verify:

Check the status of your deployment and services within the virtual cluster.

kubectl get deployments
kubectl get pods
kubectl get services

Expected Output:

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           1m

NAME                             READY   STATUS    RESTARTS   AGE
nginx-deployment-7b8c7694f-abcde   1/1     Running   0          1m
nginx-deployment-7b8c7694f-fghij   1/1     Running   0          1m

NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP   5m
nginx-service   ClusterIP   10.96.123.456   <none>        80/TCP    1m

Now, switch back to your host cluster context and see how vCluster represents these resources:

# Disconnect from the vCluster to switch back to the host cluster context
vcluster disconnect

# Check pods in the host cluster's vcluster-tenant-a namespace
kubectl get pods -n vcluster-tenant-a

Expected Output (host cluster):

NAME                                             READY   STATUS    RESTARTS   AGE
nginx-deployment-7b8c7694f-abcde-x-vcluster-x-0   1/1     Running   0          2m # vCluster proxied pod
nginx-deployment-7b8c7694f-fghij-x-vcluster-x-0   1/1     Running   0          2m # vCluster proxied pod
tenant-a-vcluster-0                              1/1     Running   0          7m # The vCluster control plane itself

Notice the additional suffix (-x-vcluster-x-0) on the proxied Pods, indicating they are managed by the vCluster syncer.

Step 5: Create Another Virtual Cluster for a Second Tenant

To demonstrate true multi-tenancy, let’s create a second virtual cluster for “Tenant B.” This will showcase how vClusters provide strong isolation, as resources deployed in Tenant A’s vCluster will not be visible or accessible from Tenant B’s vCluster, and vice-versa. Each vCluster gets its own dedicated namespace in the host cluster, ensuring resource separation and simplifying management for the host cluster administrator.

This isolation extends to RBAC, network policies, and resource quotas, allowing you to tailor the environment for each tenant without impacting others. For production environments requiring advanced networking features and strong encryption between tenants or within a tenant’s network, consider solutions like Cilium WireGuard Encryption.

# Create a dedicated namespace for the second vcluster in the host cluster
kubectl create namespace vcluster-tenant-b

# Create a virtual cluster named 'tenant-b-vcluster'
vcluster create tenant-b-vcluster -n vcluster-tenant-b --expose

Verify:

Check the vCluster Pod in the host cluster’s namespace for Tenant B.

kubectl get pods -n vcluster-tenant-b

Expected Output:

NAME                                             READY   STATUS    RESTARTS   AGE
tenant-b-vcluster-0                              1/1     Running   0          1m

Now, connect to Tenant B’s vCluster and verify its isolation:

# Connect to the 'tenant-b-vcluster'
vcluster connect tenant-b-vcluster -n vcluster-tenant-b

# Verify context and check for deployments - it should be empty!
kubectl config current-context
kubectl get deployments

Expected Output (for kubectl get deployments in tenant-b-vcluster):

No resources found in default namespace.

This confirms that Tenant B’s vCluster is completely isolated and doesn’t see the Nginx deployment from Tenant A’s vCluster.

Step 6: Configure Resource Quotas and RBAC for Virtual Clusters

One of the key benefits of vCluster for multi-tenancy is the ability to apply resource quotas and RBAC policies at the vCluster level, which then translate to the host cluster. This allows host cluster administrators to set limits and permissions for each tenant’s vCluster without needing to configure complex, per-namespace policies across the entire host cluster.

You can define resource requests and limits for the vCluster’s control plane itself, and also set quotas that apply to resources created inside the vCluster. For finer-grained control and advanced security, you can also leverage Sigstore and Kyverno for Supply Chain Security within each vCluster, ensuring that only trusted images and configurations are deployed by tenants.

# vcluster-tenant-a-values.yaml
# Values to be applied during vcluster creation/update for tenant-a-vcluster
# This example sets resource limits for the vcluster control plane pod
# and a resource quota *within* the vcluster.
syncer:
  extraArgs:
    - --enforce-pod-security-standards=baseline # Example: enforce PSS within the vCluster

# Control plane resource limits
resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

# ResourceQuota to be applied INSIDE the vCluster
# This creates a ResourceQuota named 'vcluster-quota' within the vCluster
# that limits total CPU/memory requests/limits.
embeddedResources:
  - apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: vcluster-quota
    spec:
      hard:
        requests.cpu: "1"
        requests.memory: "2Gi"
        limits.cpu: "2"
        limits.memory: "4Gi"
        pods: "10"

# Example RBAC: Create a ServiceAccount and ClusterRoleBinding for a tenant user
# This would typically be managed via an external identity provider or a dedicated vcluster user management system.
# For simplicity, we'll demonstrate how to create one directly.
# This part is more complex and often involves integrating with an OIDC provider or similar.
# For now, we'll just show the concept of creating a user within the vCluster.
# (Note: actual user authentication would still depend on host cluster's auth proxy or external IdP)

# Update tenant-a-vcluster with the resource limits and quotas
# First, ensure you are disconnected from any vcluster
vcluster disconnect

# Update the vcluster using the values file
vcluster update tenant-a-vcluster -n vcluster-tenant-a -f vcluster-tenant-a-values.yaml

Verify:

Connect to the vCluster and check if the ResourceQuota has been applied.

vcluster connect tenant-a-vcluster -n vcluster-tenant-a

# Check ResourceQuota inside the vCluster
kubectl get resourcequota

Expected Output:

NAME             AGE
vcluster-quota   1m

You can also describe it to see the limits:

kubectl describe resourcequota vcluster-quota

Expected Output (snippet):

Name:         vcluster-quota
Namespace:    default
Resource      Used  Hard
--------      ----  ----
limits.cpu    0     2
limits.memory 0     4Gi
pods          2     10
requests.cpu  0     1
requests.memory 0     2Gi

Production Considerations

• Resource Management: While vCluster saves resources compared to dedicated clusters, it still consumes host cluster resources. Implement robust Karpenter Cost Optimization or similar autoscaling solutions on your host cluster to handle dynamic vCluster needs. Use Resource Quotas and Limit Ranges extensively within vClusters and for the vCluster control plane pods themselves.
• Security and Isolation:
  • Host Cluster RBAC: Restrict host cluster users from accessing vCluster namespaces directly, except for the vCluster operators.
  • Network Policies: Utilize Kubernetes Network Policies in the host cluster to isolate vCluster pods from each other and from critical host cluster services.
  • Pod Security Standards (PSS): Enforce PSS within vClusters using the vCluster configuration (as shown in Step 6) to limit tenant workloads’ capabilities.
  • Image Security: Integrate tools like Sigstore and Kyverno for Supply Chain Security within your vClusters to enforce image signing and policy.
• Observability:
  • Host Cluster Monitoring: Monitor the host cluster for overall health, resource utilization (CPU, memory, disk, network) of vCluster pods.
  • vCluster Monitoring: Deploy monitoring agents (e.g., Prometheus Node Exporter, kube-state-metrics) within each vCluster if tenants require visibility into their virtual cluster’s health. You can also leverage eBPF Observability with Hubble on the host cluster to gain deep insights into network traffic patterns generated by vClusters.
  • Logging: Centralize logs from both the host cluster and all vClusters for easier debugging and auditing.
• Backup and Restore: Implement a robust backup strategy for your host cluster, which implicitly backs up vCluster configurations. For tenant data (PersistentVolumes), ensure your storage solution has adequate backup capabilities.
• Networking:
  • Ingress/Egress: Plan how applications within vClusters will expose services to the outside world. This often involves creating Ingress resources in the vCluster that are then synced to the host cluster and managed by a host cluster Ingress controller or Kubernetes Gateway API.
  • Service Mesh: If tenants require advanced traffic management, deploy a service mesh like Istio Ambient Mesh within their vCluster.
• Upgrade Strategy: Plan for upgrading vCluster versions and the underlying Kubernetes version of the vClusters. vCluster supports in-place upgrades.
• Multi-Cloud/Hybrid: vCluster can run on any Kubernetes cluster, making it suitable for multi-cloud or hybrid environments where a central management plane can provision vClusters across different infrastructures.

Troubleshooting

1. Issue: vCluster Pod stuck in Pending/CrashLoopBackOff.

   Explanation: This usually indicates a resource constraint on the host cluster or an issue with the vCluster image. The vCluster control plane pod needs sufficient CPU and memory to start.

   Solution:

   • Check host cluster events for the vCluster pod:

     kubectl describe pod <vcluster-pod-name> -n <vcluster-namespace>

   • Look for messages about insufficient CPU/memory or image pull errors.
   • Increase resource requests/limits for the vCluster pod in its Helm chart values (e.g., vcluster create my-vcluster -f values.yaml where values.yaml contains resources: { requests: { cpu: "500m", memory: "512Mi" }, limits: { cpu: "1", memory: "1Gi" }}).
   • Ensure the host cluster has enough available capacity.
   • Verify network connectivity to image registries if image pull fails.
2. Issue: Cannot connect to vCluster (vcluster connect fails).

   Explanation: The connection process relies on the vCluster’s API server being reachable. This can fail due to network issues, the vCluster not being ready, or incorrect --expose configuration.

   Solution:

   • Ensure the vCluster pod is running and healthy:

     kubectl get pods -n <vcluster-namespace>

   • If you used --expose, check the host cluster service created for the vCluster API server:

     kubectl get svc -n <vcluster-namespace>

     and ensure it has an external IP if needed.

   • Check logs of the vCluster pod for API server startup errors:

     kubectl logs <vcluster-pod-name> -n <vcluster-namespace>

   • Try recreating the vCluster if it’s a fresh setup and still failing.
   • Ensure your local firewall isn’t blocking the connection.
3. Issue: Pods deployed in vCluster are stuck in Pending.

   Explanation: This means the vCluster’s syncer cannot create the corresponding proxied Pods in the host cluster, or the host cluster itself lacks resources or has scheduling constraints.

   Solution:

   • Check the events of the pending Pod within the vCluster:

     vcluster connect <vcluster-name> -n <vcluster-namespace>

     then

     kubectl describe pod <pod-name>

   • Disconnect and check the host cluster for issues in the vCluster’s dedicated namespace:

     kubectl get events -n <vcluster-namespace>

   • Look for messages related to resource quotas, insufficient CPU/memory, or scheduling failures on the host cluster.
   • Ensure the vCluster’s syncer is running and healthy (part of the main vCluster pod).
   • Verify that the vCluster’s service account in the host cluster has permissions to create Pods.
4. Issue: ResourceQuota or LimitRange defined in vCluster not taking effect.

   Explanation: vCluster handles resource quotas differently. Quotas for the vCluster’s control plane are set on the host cluster (as normal Pod resource requests/limits). Quotas within the vCluster are created as standard Kubernetes ResourceQuota objects inside the vCluster itself.

   Solution:

   • For quotas within the vCluster:
     • Connect to the vCluster:

       vcluster connect <vcluster-name> -n <vcluster-namespace>

     • Verify the ResourceQuota exists:

       kubectl get resourcequota

     • Ensure your deployment pods are requesting/limiting resources, as quotas only apply when resources are explicitly requested/limited.
     • If you defined it in a values file, ensure the vCluster was updated with that file:

       vcluster update <vcluster-name> -n <vcluster-namespace> -f <your-values.yaml>

   • For quotas on the vCluster control plane itself:
     • These are set as resource requests/limits on the vCluster’s deployment in the host cluster. Check the vCluster’s deployment YAML in the host cluster namespace.
5. Issue: Services in vCluster are not accessible externally.

   Explanation: Services in a vCluster are ClusterIP by default. To expose them externally, you need to create an Ingress or LoadBalancer service, which vCluster then syncs to the host cluster.

   Solution:

   • Ensure you have an Ingress controller (e.g., Nginx Ingress, Traefik) or a Cloud Load Balancer provider configured in your host cluster.
   • Create an Ingress resource within your vCluster, pointing to your vCluster service. The vCluster syncer will create a corresponding Ingress in the host cluster’s vCluster namespace.
   • Alternatively, change your vCluster service type to LoadBalancer. This will cause the vCluster syncer to create a LoadBalancer service in the host cluster, which your cloud provider’s CCM will then provision.
   • Verify the Ingress/LoadBalancer service status in the host cluster’s vCluster namespace.
   • Consider using the Kubernetes Gateway API for more advanced ingress management, which vCluster also supports.
6. Issue: vCluster CLI commands are slow or time out.

   Explanation: This can indicate high latency to the host cluster API server, an overloaded host cluster API server, or network issues preventing the CLI from communicating effectively with either the host or virtual cluster API.

   Solution:

   • Check your internet connection and latency to your Kubernetes cluster.
   • Verify the health of your host cluster’s API server.
   • Run kubectl get --raw=/healthz against your host cluster to check API server responsiveness.
   • Ensure the vCluster control plane pod is healthy and not restarting frequently.
   • If using a remote host cluster, consider if your local network or VPN is causing issues.

FAQ Section

Q1: What is the main difference between vCluster and namespaces for multi-tenancy?

A1: Namespaces provide logical isolation within a single Kubernetes control plane, primarily for resource naming and RBAC. They share the same API server, scheduler, and controller manager. vCluster, on the other hand, provides a separate, dedicated Kubernetes control plane (API server, scheduler, controller manager) for each tenant. This offers much stronger isolation, prevents “noisy neighbor” issues at the control plane level, simplifies RBAC for tenants (they get full cluster-admin within their vCluster), and allows tenants to use different Kubernetes versions or install their own CRDs without impacting other tenants or the host cluster.

Q2: Can I install Helm charts or custom CRDs inside a vCluster?

A2: Yes, absolutely! This is one of the major advantages of vCluster. Since each vCluster has its own API server, you can install any Helm chart, deploy any custom resource definitions (CRDs), and run any operator within a vCluster as if it were a standalone cluster. The vCluster’s syncer intelligently handles the underlying resources (like Pods, StatefulSets, PVCs) by proxying them to the host cluster while maintaining the illusion of a dedicated cluster for the tenant.

Q3: How does vCluster handle Persistent Volumes (PVs) and StorageClasses?

A3: vCluster usually syncs PersistentVolumeClaims (PVCs) from the vCluster to the host cluster. The host cluster’s storage provisioner then creates the actual PersistentVolume (PV) and binds it. If a StorageClass is referenced in the vCluster’s PVC, that StorageClass must exist and be configured in the host cluster. You can also configure vCluster to map StorageClasses, allowing tenants to request a StorageClass name in their vCluster that maps to a different StorageClass name in the host cluster.

Q4: Is vCluster suitable for production environments?

A4: Yes, vCluster is increasingly being adopted for production multi-tenancy. It’s used by SaaS providers and large organizations to offer isolated dev, staging, and even production environments. Key considerations for production include robust monitoring (both host and vCluster), proper resource quotas, network policies for isolation, security best practices (e.g., Pod Security Standards), and a solid backup strategy. Check out the official vCluster production readiness guide for more details.

Q5: Can I run different Kubernetes versions in my vClusters than my host cluster?

A5: Yes, this is a powerful feature of vCluster. You can specify the Kubernetes distribution and version for each vCluster during creation (e.g., k3s, k0s, vanilla Kubernetes). This allows tenants to test new Kubernetes versions, use specific features, or maintain compatibility with older applications, all while the host cluster runs a stable, potentially newer, version. The vCluster CLI and Helm chart allow you to configure the desired Kubernetes distribution and version for the virtual cluster’s control plane.

Cleanup Commands

When you’re finished experimenting, it’s crucial to clean up the resources to avoid incurring unnecessary costs on your host cluster.

# Disconnect from any active vCluster context
vcluster disconnect

# Delete the first virtual cluster and its associated host cluster namespace
vcluster delete tenant-a-vcluster -n vcluster-tenant-a --delete-namespace

# Delete the second virtual cluster and its associated host cluster namespace
vcluster delete tenant-b-vcluster -n vcluster-tenant-b --delete-namespace

# (Optional) Remove the deployment.yaml if you created it
rm deployment.yaml

Verify:

kubectl get namespaces | grep vcluster-

Expected Output: (No output should be returned, or only default namespaces)

# No output, or just the system namespaces if you didn't create any other 'vcluster-' prefixed namespaces.

Next Steps / Further Reading

• Explore advanced vCluster configurations: Learn about syncing specific resources,