Manage Infra with Crossplane: IaC on K8s

Introduction

The promise of Infrastructure as Code (IaC) has revolutionized how we manage cloud resources, bringing version control, automation, and repeatability to infrastructure provisioning. However, traditional IaC tools often operate outside the Kubernetes ecosystem, leading to a cognitive load for developers and operators who must context-switch between different paradigms and toolchains. This fragmentation can hinder agility and introduce inconsistencies, especially in modern cloud-native environments.

Enter Crossplane, a powerful open-source Kubernetes add-on that extends your cluster to manage and provision infrastructure from any cloud provider, on-premises, or even other Kubernetes clusters. Crossplane turns your Kubernetes API into a universal control plane for all your infrastructure, allowing you to define, provision, and manage external resources—like databases, message queues, and object storage—using familiar Kubernetes YAML manifests and kubectl commands. By doing so, Crossplane unifies your application and infrastructure management under a single, declarative API, bringing the benefits of the Kubernetes control plane to the entire cloud-native stack.

This guide will walk you through the process of setting up and using Crossplane to provision cloud resources. We’ll demonstrate how to install Crossplane, integrate it with a cloud provider (AWS in this example), and provision a managed database instance entirely from within Kubernetes. By the end, you’ll understand how Crossplane empowers platform teams to build powerful Internal Developer Platforms (IDPs) and allows developers to consume infrastructure as easily as deploying an application.

Prerequisites

To follow this guide, you will need:

• A running Kubernetes cluster (v1.20+). This can be a local cluster like Minikube or Kind, or a cloud-managed cluster like EKS, GKE, or AKS. For production, a cloud-managed cluster is recommended.
• kubectl installed and configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• helm installed (v3.0+). Instructions are available on the Helm website.
• An AWS account with programmatic access (Access Key ID and Secret Access Key) and sufficient permissions to create IAM roles, S3 buckets, RDS instances, etc. For security best practices, consider using an IAM Role with fine-grained permissions.
• Basic understanding of Kubernetes concepts (Pods, Deployments, Services, etc.) and YAML syntax.

Step-by-Step Guide

1. Install Crossplane into your Kubernetes Cluster

First, we need to install Crossplane into your Kubernetes cluster. Crossplane is typically installed via Helm, which simplifies the deployment of its core components. We’ll create a dedicated namespace for Crossplane to keep things organized.

The Crossplane core consists of the Crossplane controller and its Custom Resource Definitions (CRDs). These CRDs extend the Kubernetes API with new resource types that represent infrastructure concepts, such as Provider, ProviderConfig, and Composition. The controller then watches these CRDs and interacts with external cloud APIs to provision and manage the actual infrastructure. Think of it as giving your Kubernetes cluster the ability to speak the language of your cloud provider.

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace

Verify: Check if the Crossplane pods are running and healthy in the crossplane-system namespace. This might take a few moments for the pods to transition to a Running state.

kubectl get pods -n crossplane-system

Expected Output:

NAME                                       READY   STATUS    RESTARTS   AGE
crossplane-7f89b7b75b-abcde                1/1     Running   0          2m
crossplane-rbac-manager-87654321-fghij     1/1     Running   0          2m

2. Install the AWS Provider

Crossplane supports various cloud providers through “Providers.” Each Provider is a separate Kubernetes controller that understands how to interact with a specific cloud’s API. For this guide, we’ll install the AWS Provider, which enables Crossplane to manage AWS resources like RDS, S3, EC2, and more.

The AWS Provider introduces its own set of CRDs (e.g., DBInstance for RDS, S3Bucket for S3) into your Kubernetes cluster. When you create an instance of these CRDs, the AWS Provider controller translates that Kubernetes object into API calls to AWS, effectively provisioning or updating the resource in your AWS account. This separation of concerns ensures that the Crossplane core remains lightweight and extensible, while specific cloud logic resides within its respective provider.

kubectl apply -f https://raw.githubusercontent.com/crossplane/crossplane/master/docs/snippets/install/provider-aws.yaml

Verify: Confirm that the AWS Provider is installed and its controller pod is running. This will also create a Provider custom resource in your cluster.

kubectl get provider
kubectl get pods -n crossplane-system -l app=aws-provider

Expected Output:

NAME           INSTALLED   HEALTHY   PACKAGE                                           AGE
provider-aws   True        True      crossplane/provider-aws:v0.40.0                   2m

NAME                                        READY   STATUS    RESTARTS   AGE
crossplane-provider-aws-54321abcd-efghj     1/1     Running   0          1m

3. Configure AWS Provider Credentials

For Crossplane to interact with your AWS account, it needs credentials. We’ll store your AWS Access Key ID and Secret Access Key in a Kubernetes Secret within the crossplane-system namespace. This Secret will then be referenced by a ProviderConfig resource.

It’s crucial to handle credentials securely. While we’re using environment variables here for demonstration, in a production environment, consider using a secrets management solution like Kubernetes Secrets mounted as files, or integrating with a cloud provider’s secrets manager (e.g., AWS Secrets Manager, HashiCorp Vault) via an external secrets operator. For more advanced security, explore approaches like Sigstore and Kyverno for supply chain security.

Replace YOUR_ACCESS_KEY_ID and YOUR_SECRET_ACCESS_KEY with your actual AWS credentials.

AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
kubectl create secret generic aws-creds -n crossplane-system --from-literal=credentials="[default]\naws_access_key_id = $AWS_ACCESS_KEY_ID\naws_secret_access_key = $AWS_SECRET_ACCESS_KEY"

Next, define a ProviderConfig resource that tells the AWS Provider where to find these credentials. This ProviderConfig acts as a cluster-wide configuration for the AWS Provider, specifying the region and how to authenticate.

# provider-config-aws.yaml
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: credentials
  # You can also specify a default region here, or per resource
  # region: us-east-1

kubectl apply -f provider-config-aws.yaml

Verify: Check the status of your ProviderConfig. It should show as healthy.

kubectl get providerconfig

Expected Output:

NAME      AGE   HEALTHY   MESSAGE
default   1m    True      ProviderConfig is healthy

4. Provision an AWS RDS PostgreSQL Instance

Now that Crossplane and the AWS Provider are set up, we can provision an actual AWS resource. We’ll create an AWS RDS PostgreSQL instance using a DBInstance custom resource. This demonstrates how Crossplane allows you to declare desired infrastructure state directly within Kubernetes.

Notice how the YAML manifest for the DBInstance looks very much like a standard Kubernetes resource, but its spec.forProvider section contains AWS-specific configuration. Crossplane translates this into the necessary AWS API calls. The writeConnectionSecretToRef field is particularly useful; it instructs Crossplane to create a Kubernetes Secret containing connection details (like endpoint, username, password) for the provisioned database, making it easy for your applications to consume.

# rds-instance.yaml
apiVersion: rds.aws.crossplane.io/v1beta1
kind: DBInstance
metadata:
  name: my-crossplane-db
spec:
  forProvider:
    region: us-east-1 # Ensure this matches your desired AWS region
    dbInstanceClass: db.t3.micro
    masterUsername: admin
    engine: postgres
    engineVersion: "14.7"
    allocatedStorage: 20 # In GB
    skipFinalSnapshot: true # Set to false in production
    publiclyAccessible: false # Recommended for production
    tags: # Example tags
      - key: managedBy
        value: Crossplane
      - key: environment
        value: dev
  writeConnectionSecretToRef:
    name: db-connection-secret
    namespace: default # The namespace where your application will consume this secret

kubectl apply -f rds-instance.yaml

Verify: Monitor the status of your DBInstance. It will take some time for AWS to provision the database, so the status might initially show as Creating or Provisioning. Once ready, it should transition to Ready and Synced.

kubectl get dbinstance my-crossplane-db
kubectl describe dbinstance my-crossplane-db

Expected Output (after some time):

NAME               READY   SYNCED   EXTERNAL-NAME      AGE
my-crossplane-db   True    True     my-crossplane-db   10m

You can also check the AWS console to confirm the RDS instance is being created or is available.

5. Consume the Database Connection Secret

Once the RDS instance is provisioned and ready, Crossplane will create a Kubernetes Secret containing the connection details. Your applications running in the cluster can then consume this secret to connect to the database.

This pattern simplifies application deployment, as developers don’t need to manually fetch database credentials or configure them in their application manifests. They simply reference the secret created by Crossplane. This is a core benefit of Crossplane for building Internal Developer Platforms, abstracting infrastructure complexity from application developers.

kubectl get secret db-connection-secret -n default -o yaml

Expected Output:

apiVersion: v1
kind: Secret
metadata:
  name: db-connection-secret
  namespace: default
type: Opaque
data:
  endpoint: 
  password: 
  port: 
  username: 

You can then decode these values to get the actual connection string. For example:

kubectl get secret db-connection-secret -n default -o jsonpath='{.data.endpoint}' | base64 -d
kubectl get secret db-connection-secret -n default -o jsonpath='{.data.username}' | base64 -d
kubectl get secret db-connection-secret -n default -o jsonpath='{.data.password}' | base64 -d

You can now deploy an application that uses this secret to connect to your newly provisioned database. For instance, you might mount this secret as environment variables or files into your application’s Pod.

# app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: your-app-image:latest # Replace with your actual application image
        envFrom:
        - secretRef:
            name: db-connection-secret
        # ... other container configurations

Production Considerations

While Crossplane offers immense power, deploying it in production requires careful planning:

• IAM Permissions: Implement the principle of least privilege for your AWS credentials. Create specific IAM policies that grant Crossplane only the necessary permissions to manage the resources it’s responsible for. Avoid using root or administrative credentials. Consider IAM Roles for Service Accounts (IRSA) if running on EKS to avoid long-lived credentials.
• Observability: Integrate Crossplane logs and metrics into your existing observability stack. Monitor the health of Crossplane controllers and providers, as well as the status of the managed resources. Tools like Prometheus, Grafana, and ELK stack are essential. For advanced Kubernetes observability, consider solutions leveraging eBPF Observability with Hubble.
• High Availability: Run Crossplane controllers in a highly available configuration (multiple replicas) across different availability zones to ensure resilience.
• Backup and Restore: While Crossplane manages the lifecycle of external resources, the Kubernetes manifests defining those resources are critical. Implement robust backup and restore strategies for your Kubernetes cluster, including Crossplane CRDs and secrets.
• Drift Detection and Reconciliation: Crossplane continuously reconciles the desired state (defined in Kubernetes) with the actual state (in the cloud). Understand how it handles drift and ensure your team has processes for addressing discrepancies.
• Compositions for Abstraction: For production environments, leverage Crossplane’s Composition feature heavily. Compositions allow platform teams to define higher-level, opinionated abstractions (e.g., “Production PostgreSQL Database” or “Secure S3 Bucket”) that bundle multiple cloud resources and configurations. This empowers application developers to provision complex infrastructure with simple, self-service manifests, without needing deep cloud expertise.
• Network Policies: Ensure that your Crossplane controller pods have appropriate network access to the cloud provider’s API endpoints. Similarly, apply Kubernetes Network Policies to restrict traffic to and from Crossplane components within your cluster for enhanced security. For encrypted pod-to-pod communication, consider Cilium WireGuard Encryption.
• Cost Management: Crossplane itself doesn’t directly manage costs, but by standardizing resource provisioning, it enables better cost visibility and control. Integrate with cloud cost management tools. For Kubernetes node cost optimization, explore tools like Karpenter.
• Version Control: Keep all Crossplane manifests (Providers, ProviderConfigs, Compositions, XRs, and managed resources) under strict version control (e.g., Git). Implement GitOps practices for deploying and managing your infrastructure.
• Security Best Practices: Regularly audit Crossplane configurations, provider credentials, and the permissions granted to the Crossplane service account. Implement image scanning and vulnerability management for Crossplane and provider images. Consider integrating with tools like Sigstore and Kyverno for policy enforcement.

Troubleshooting

1. Crossplane Pods Not Running:

   Issue: Crossplane or provider pods in crossplane-system namespace are in Pending, CrashLoopBackOff, or Error state.

   Solution:

   • Check pod logs:

     kubectl logs -n crossplane-system <pod-name>

   • Describe the pod for events:

     kubectl describe pod -n crossplane-system <pod-name>

   • Common causes: resource constraints (CPU/memory), incorrect image pull secrets, network issues preventing image pull.
2. Provider Not Healthy:

   Issue: kubectl get provider shows HEALTHY as False.

   Solution:

   • Check the provider pod logs in crossplane-system for errors.
   • Verify the Provider resource itself:

     kubectl describe provider <provider-name>

     Look for conditions or events indicating the problem.

   • Ensure the provider’s dependencies are met (e.g., network access to the cloud API).
3. Managed Resource (e.g., DBInstance) Stuck in Creating or Failed State:

   Issue: After applying a resource like DBInstance, its status remains Creating or changes to Failed, and READY is False.

   Solution:

   • Describe the specific resource:

     kubectl describe dbinstance my-crossplane-db

     Look at the Status.Conditions and Events sections. Crossplane often provides detailed error messages from the cloud provider here.

   • Verify your ProviderConfig and the associated AWS credentials. Ensure they are correct and have the necessary IAM permissions for the resource you’re trying to create.
   • Check the logs of the corresponding provider controller pod (e.g., crossplane-provider-aws-... in crossplane-system).
   • Ensure there are no quotas or service limits hit in your AWS account for the region/resource type.
4. ProviderConfig Not Healthy:

   Issue: kubectl get providerconfig shows HEALTHY as False, or the message indicates credential issues.

   Solution:

   • Verify the Kubernetes Secret containing your AWS credentials:

     kubectl get secret aws-creds -n crossplane-system -o yaml

     Ensure the credentials key exists and its base64-decoded value is correctly formatted (INI format).

   • Double-check the secretRef in your ProviderConfig to ensure it points to the correct secret name and namespace.
   • Ensure the AWS credentials themselves are valid and not expired.
5. Resource Deletion Issues:

   Issue: When deleting a Crossplane-managed resource (e.g., DBInstance), it gets stuck in a Terminating state in Kubernetes but isn’t deleted in the cloud.

   Solution:

   • Check the logs of the relevant provider controller pod for errors during deletion.
   • Ensure the AWS credentials still have permission to delete the resource.
   • Sometimes, cloud resources have dependencies that prevent deletion (e.g., a database with active connections, a bucket with objects). Manually check the cloud console for such dependencies.
   • If a resource is stuck and you’ve confirmed it’s deleted in the cloud, you might need to manually remove the finalizer from the Kubernetes object, but this should be a last resort and used with extreme caution:

     kubectl edit dbinstance my-crossplane-db

     Remove the finalizers array from the metadata.

6. Application Cannot Connect to Database:

   Issue: The DBInstance shows READY: True, but your application fails to connect using the generated connection secret.

   Solution:

   • Decode the connection secret values and manually try to connect from within the cluster (e.g., using a temporary pod with a PostgreSQL client).
   • Check the security group rules associated with the RDS instance in AWS. Ensure your Kubernetes cluster’s egress IP range or security group is allowed to connect to the RDS instance on the correct port (e.g., 5432 for PostgreSQL).
   • Verify the RDS instance’s publiclyAccessible setting. If it’s false (recommended for production), your application must be in the same VPC or have VPC peering/VPN connectivity.
   • Ensure the username and password from the secret are being correctly used by your application.

FAQ Section

1. What is the difference between Crossplane and Terraform?

While both Crossplane and Terraform are IaC tools, their approaches differ significantly. Terraform is a command-line tool that uses its own state file and HCL language to manage infrastructure. Crossplane, on the other hand, extends the Kubernetes API, allowing you to manage infrastructure using Kubernetes YAML and kubectl. Crossplane brings infrastructure management directly into the Kubernetes control plane, enabling a unified declarative approach for both applications and infrastructure. Terraform is excellent for one-off deployments or managing infrastructure outside Kubernetes, while Crossplane excels at building self-service platforms within Kubernetes.

2. Can Crossplane manage resources across multiple cloud providers?

Yes, absolutely! Crossplane is designed for multi-cloud and hybrid-cloud scenarios. You can install multiple providers (e.g., AWS, Azure, GCP) into a single Crossplane installation and manage resources across all of them using the same Kubernetes API. This is one of Crossplane’s core strengths, enabling true cloud-agnostic infrastructure management.

3. What are Compositions in Crossplane? Why are they important?

Compositions are a powerful feature in Crossplane that allows platform teams to define higher-level, custom abstractions over raw cloud resources. Instead of developers directly provisioning an RDSInstance, S3Bucket, and IAMRole, a platform team can define a CompositeResourceDefinition (XRD) like “XPostgreSQL” that, when instantiated, provisions all these underlying resources simultaneously with predefined configurations (e.g., security groups, backup policies). Compositions are crucial for building Internal Developer Platforms (IDPs), as they simplify infrastructure consumption for application developers and enforce organizational standards and best practices.

4. Is Crossplane suitable for production environments?

Yes, Crossplane is designed for production use and is part of the Cloud Native Computing Foundation (CNCF). Many organizations use it to manage critical infrastructure. However, like any powerful tool, it requires careful planning for production readiness, including robust IAM policies, observability, high availability, and proper backup strategies, as detailed in the “Production Considerations” section.

5. How does Crossplane handle state management?

Crossplane leverages the Kubernetes control plane for state management. The desired state of your infrastructure is defined in Kubernetes custom resources (e.g., DBInstance YAML files) and stored in etcd. Crossplane controllers continuously reconcile this desired state with the actual state of resources in the external cloud provider. It doesn’t maintain an external state file like Terraform. This Kubernetes-native approach simplifies operations by centralizing state within the cluster.

Cleanup Commands

To remove the resources created in this guide and uninstall Crossplane:

1. Delete the AWS RDS instance: This will trigger Crossplane to deprovision the RDS instance in your AWS account.

kubectl delete dbinstance my-crossplane-db

Wait until the RDS instance is fully deleted in AWS before proceeding. You can check the AWS console.

2. Delete the connection secret:

kubectl delete secret db-connection-secret -n default

3. Uninstall the AWS Provider:

kubectl delete provider.pkg.crossplane.io/provider-aws
kubectl delete providerconfig.aws.crossplane.io/default

4. Uninstall Crossplane core components:

helm uninstall crossplane --namespace crossplane-system
kubectl delete namespace crossplane-system

5. Remove Helm repository:

helm repo remove crossplane-stable

Next Steps / Further Reading

• Explore Compositions: Dive deeper into Crossplane Compositions to build your own custom resource abstractions. This is where Crossplane truly shines for platform engineering.
• Try Other Providers: Experiment with other Crossplane providers like Azure, GCP, or even Kubernetes Provider to manage resources in other clusters.
• Integrate with GitOps: Learn how to integrate Crossplane with GitOps tools like Argo CD or Flux CD for declarative, automated infrastructure deployments.
• Security Best Practices: Review Crossplane’s security recommendations and consider implementing fine-grained RBAC for Crossplane resources.
• Kubezilla Internal Developer Platform (IDP) Guide: For more on building IDPs, refer to our Kubernetes Gateway API vs Ingress: The Complete Migration Guide, which touches on elements of platform building.
• Service Mesh Integration: If you’re building complex microservices, consider how Crossplane-provisioned services can integrate with a service mesh like Istio Ambient Mesh.
• Cloud-Native Cost Optimization: Explore Karpenter for Kubernetes Cost Optimization to pair with your Crossplane-managed infrastructure.

Conclusion

Crossplane fundamentally shifts how we think about infrastructure management in a cloud-native world. By extending the Kubernetes API, it provides a unified control plane that treats all infrastructure—whether it’s a Kubernetes Pod or an AWS RDS instance—as a first-class citizen within your cluster. This declarative, GitOps-friendly approach empowers platform teams to build robust, self-service infrastructure platforms, abstracting away cloud complexities for application developers and fostering greater agility and consistency. As you continue your cloud-native journey, embracing tools like Crossplane will be key to unlocking the full potential of Kubernetes as the universal control plane for your entire infrastructure landscape.