Manage Cloud Infra with Crossplane & Kubernetes

Verify:

The ProviderConfig should show as healthy, indicating Crossplane can access your AWS credentials:

NAME      HEALTHY   AGE
default   True      1m

4. Provision an AWS S3 Bucket

Now that Crossplane is installed, the AWS provider is running, and credentials are configured, we can provision our first cloud resource: an S3 bucket. We define the desired state of the S3 bucket using a Kubernetes manifest, just like you would define a Deployment or a Service. Crossplane observes this manifest and instructs the provider-aws to create the corresponding resource in AWS.

This declarative approach is a cornerstone of Kubernetes. You declare what you want, and Crossplane continuously works to reconcile the actual state with your desired state. Notice how the YAML looks very much like a standard Kubernetes resource, but with an apiVersion pointing to s3.aws.crossplane.io and a kind of Bucket. This consistency makes it easy for Kubernetes native developers to adopt Crossplane.

# s3-bucket.yaml
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: my-crossplane-bucket-12345
spec:
  forProvider:
    region: us-east-1
    acl: private
  providerConfigRef:
    name: default # References the ProviderConfig we created earlier

# Apply the S3 bucket manifest
kubectl apply -f s3-bucket.yaml

# Monitor the status of the S3 bucket
kubectl get bucket my-crossplane-bucket-12345

# Describe the bucket for more details (events, status, etc.)
kubectl describe bucket my-crossplane-bucket-12345

Verify:

Initially, the bucket might be in a "Creating" state. After a few moments, it should transition to "Ready: True" and show a synchronized status:

NAME                      READY   SYNCED   EXTERNAL-NAME              AGE
my-crossplane-bucket-12345   True    True     my-crossplane-bucket-12345   1m

You can also verify its existence directly in the AWS console or using the AWS CLI:

aws s3api head-bucket --bucket my-crossplane-bucket-12345 --region us-east-1

Expected output (if successful, no output, just exit code 0):

# If successful, this command returns nothing and exits with 0.
# If the bucket does not exist, it will return an error.

5. Introduce Compositions for Abstraction

While provisioning raw cloud resources is powerful, it can expose too much complexity to developers. This is where Crossplane's Compositions shine. Compositions allow platform teams to define higher-level, opinionated APIs that abstract away the underlying cloud specifics. Developers can then request an "XPostgreSQLInstance" or "XObjectStorage" without needing to know if it's AWS RDS, GCP Cloud SQL, or Azure Database for PostgreSQL, or S3, GCS, or Azure Blob Storage.

A Composition consists of two main parts:

1. Composite Resource Definition (XRD): Defines the new custom API that developers will interact with (e.g., XPostgreSQLInstance).
2. Composition: Maps the fields of the XRD to concrete cloud resources (e.g., an AWS RDS instance, a SubnetGroup, a SecurityGroup).

This enables self-service infrastructure for developers while maintaining governance and best practices defined by platform teams. For more advanced networking requirements, platform teams might also leverage Kubernetes Network Policies to secure communication between applications and these composed resources.

Let's create a simple Composition for an "XBucket" that provisions an S3 bucket with some default settings.

a. Define the Composite Resource Definition (XRD)

The XRD defines the schema for our custom resource, XBucket. Developers will create instances of this XBucket.

# xbucket-xrd.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xbuckets.kubezilla.io
spec:
  group: kubezilla.io
  names:
    kind: XBucket
    plural: xbuckets
  claimNames: # Optional: allows developers to claim resources in their namespace
    kind: BucketClaim
    plural: bucketclaims
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                parameters: # Parameters developers can specify
                  type: object
                  properties:
                    region:
                      type: string
                      description: "The AWS region for the bucket."
                    namePrefix:
                      type: string
                      description: "A prefix for the bucket name."
                  required:
                    - region
              required:
                - parameters

kubectl apply -f xbucket-xrd.yaml

Verify:

You should see the new CRD registered:

compositeresourcedefinition.apiextensions.crossplane.io/xbuckets.kubezilla.io created

b. Define the Composition

The Composition maps the XBucket to an actual AWS S3 Bucket. It specifies how the parameters from the XBucket (like region) are used to configure the underlying S3 resource. We can also add default values or additional configurations here that developers don't need to specify.

# xbucket-composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xbucket-aws-s3
  labels:
    provider: aws
    service: s3
spec:
  compositeTypeRef:
    apiVersion: kubezilla.io/v1alpha1
    kind: XBucket
  resources:
    - name: s3bucket
      base:
        apiVersion: s3.aws.crossplane.io/v1beta1
        kind: Bucket
        spec:
          forProvider:
            acl: private # Default ACL, developers don't need to specify
            tags:
              managed-by: crossplane
      patches:
        - fromFieldPath: "spec.parameters.region"
          toFieldPath: "spec.forProvider.region"
        - fromFieldPath: "metadata.uid" # Use UID to ensure unique bucket name
          toFieldPath: "spec.forProvider.name"
          transforms:
            - type: string
              string:
                fmt: "%s" # Just convert UID to string
        - fromFieldPath: "spec.parameters.namePrefix"
          toFieldPath: "spec.forProvider.name"
          transforms:
            - type: string
              string:
                fmt: "%s-"
            - type: string
              string:
                fmt: "%s%s"
              # Combine namePrefix with the UID-derived name.
              # This transform chain essentially says:
              # 1. Take 'namePrefix'
              # 2. Append '-' to it
              # 3. Prepend the result to the S3 bucket name derived from 'metadata.uid'
              # This ensures a unique name while allowing a developer-defined prefix.
              # Note: Crossplane v1.11+ simplifies this with `combine` transform.
              # For older versions, you might need a more complex UID transform or use `external-name` annotation.
              # For simplicity, we'll just use the UID for a unique name and add a prefix.
              # A better way is to use a `function` or `externalName` annotation.
              # Let's simplify and just use the UID.
              # For actual production, use `externalName` or a more robust naming convention.
      # Simplified naming for demonstration: just use the composite resource's name
      # For a real scenario, use `externalName` annotation or a more complex patch.
      readinessChecks:
        - type: MatchString
          fieldPath: "status.atProvider.state"
          matchString: "Available" # Or "Ready" depending on resource

kubectl apply -f xbucket-composition.yaml

Verify:

The Composition should be created successfully:

composition.apiextensions.crossplane.io/xbucket-aws-s3 created

c. Create an Instance of Your Composite Resource

Now, a developer can request an XBucket without knowing the underlying S3 details. They just specify the region and a name prefix.

# my-dev-bucket.yaml
apiVersion: kubezilla.io/v1alpha1
kind: XBucket
metadata:
  name: my-dev-bucket-for-app
spec:
  parameters:
    region: us-east-1
    namePrefix: dev-app-data
  resourceRef: # Optional: If you want to specify a specific external name
    name: dev-app-data-unique-id

kubectl apply -f my-dev-bucket.yaml

# Check the status of your XBucket
kubectl get xbucket my-dev-bucket-for-app

# Check the underlying S3 Bucket created by the Composition
kubectl get bucket

Verify:

You'll see your XBucket and the underlying S3 Bucket resource created by Crossplane:

NAME                    READY   SYNCED   AGE
my-dev-bucket-for-app   True    True     1m

NAME                                          READY   SYNCED   EXTERNAL-NAME                                AGE
bucket.s3.aws.crossplane.io/my-dev-bucket-for-app-xxxx   True    True     dev-app-data-my-dev-bucket-for-app-xxxx   1m

The actual external name of the S3 bucket will be generated by Crossplane based on the composition rules, likely including a unique suffix. This demonstrates the power of abstraction!

6. Clean Up Resources

It's crucial to clean up resources created by Crossplane to avoid incurring unnecessary cloud costs. When you delete a Crossplane resource, it automatically triggers the deletion of the corresponding cloud resource. This is a significant advantage over manual cleanup or traditional IaC tools that might require extra steps.

# Delete the composite resource (if created)
kubectl delete xbucket my-dev-bucket-for-app

# Delete the raw S3 bucket (if created directly without composition)
# kubectl delete bucket my-crossplane-bucket-12345

# Delete the Composition and XRD
kubectl delete -f xbucket-composition.yaml
kubectl delete -f xbucket-xrd.yaml

# Delete the ProviderConfig
kubectl delete providerconfig.aws.crossplane.io default

# Uninstall the Provider
kubectl delete provider.pkg.crossplane.io crossplane-provider-aws

# Uninstall Crossplane core
helm uninstall crossplane --namespace crossplane-system
kubectl delete namespace crossplane-system

Verify:

After running the cleanup commands, verify that the resources are gone from both Kubernetes and your cloud provider. For example, check your AWS S3 console or run:

aws s3api head-bucket --bucket dev-app-data-unique-id --region us-east-1
# Expected output: An error indicating the bucket does not exist.

And in Kubernetes:

kubectl get bucket
# Expected output: "No resources found"
kubectl get provider.pkg.crossplane.io
# Expected output: "No resources found"
kubectl get pods --namespace crossplane-system
# Expected output: "No resources found in crossplane-system namespace."

Production Considerations

Deploying Crossplane in a production environment requires careful planning and adherence to best practices. Here are key areas to consider:

• High Availability: Run Crossplane controllers with multiple replicas for fault tolerance. Ensure your Kubernetes cluster itself is highly available.
• Security & RBAC:
  • Least Privilege: Grant Crossplane's cloud provider credentials (e.g., AWS IAM roles) only the minimum necessary permissions to create, update, and delete the resources it manages.
  • Kubernetes RBAC: Implement strict Kubernetes RBAC policies. Developers should only have access to create and manage their Composite Resources (e.g., XBucket), not the underlying raw cloud resources (e.g., Bucket.s3.aws.crossplane.io) or Crossplane's core components.
  • Secrets Management: Use robust secret management solutions. Instead of static credentials, consider using Secrets Store CSI Driver with cloud provider secret stores (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) or HashiCorp Vault.
  • For enhanced cluster security, consider integrating with tools like Sigstore and Kyverno for policy enforcement and supply chain integrity.
• Observability:
  • Logging: Configure Crossplane controllers to log to your central logging system (e.g., Fluentd, Loki).
  • Metrics: Expose Prometheus metrics from Crossplane and its providers. Monitor resource reconciliation, API call rates, and error rates.
  • Events: Monitor Kubernetes events for Crossplane resources to track lifecycle changes and errors.
  • For deeper insights into your Kubernetes network, explore eBPF Observability with Hubble.
• GitOps Workflows:
  • Integrate Crossplane resource definitions (XRDs, Compositions, Composite Resources) into your GitOps pipelines (Argo CD, Flux CD). This ensures all infrastructure is version-controlled, auditable, and automatically reconciled.
  • Treat your Compositions as the "API" of your internal platform.
• Provider & Composition Management:
  • Version Control: Pin specific versions of Crossplane core and its Providers to ensure stability.
  • Testing: Thoroughly test your Compositions in development and staging environments before deploying to production.
  • Composition Validation: Use schema validation in your XRDs to ensure developers provide valid parameters.
• Cost Management:
  • Crossplane can provision expensive resources. Implement tagging strategies on your cloud resources through Compositions to enable better cost tracking and allocation.
  • Combine with tools like Karpenter for Kubernetes Cost Optimization to manage compute costs effectively.
• Service Mesh Integration: If you're running a service mesh like Istio, ensure Crossplane-provisioned services are correctly integrated. Check out our Istio Ambient Mesh Production Guide for best practices.
• Backup and Restore: While Crossplane manages the lifecycle of cloud resources, it doesn't back up the data within those resources. Implement separate backup and restore strategies for your databases, object storage, etc.

Troubleshooting

1. Issue: Crossplane pods are not running or in CrashLoopBackOff.

   Solution: Check the logs of the Crossplane pods for errors. Common issues include insufficient resource limits, incorrect Helm chart values, or network connectivity problems within the cluster.

   kubectl logs -n crossplane-system <crossplane-pod-name>
   kubectl describe pod -n crossplane-system <crossplane-pod-name>
   

2. Issue: Provider pods (e.g., provider-aws) are not running or in CrashLoopBackOff.

   Solution: Similar to Crossplane core, check the logs and describe the provider pod. Ensure the Provider custom resource is correctly applied and references a valid package. Network issues preventing the provider from pulling its image can also cause this.

   kubectl logs -n crossplane-system <provider-aws-pod-name>
   kubectl describe provider.pkg.crossplane.io provider-aws
   

3. Issue: Cloud resources (e.g., S3 Bucket) are not being provisioned, or are stuck in a "Creating" state.

   Solution:

   1. Check ProviderConfig: Ensure your ProviderConfig is healthy and the underlying secret has valid credentials.

   kubectl get providerconfig.aws.crossplane.io default
   kubectl get secret -n crossplane-system aws-creds -o yaml # Verify secret exists
   

   2. Check Resource Status: Describe the Crossplane custom resource (e.g., Bucket or XBucket) for events and status conditions. The Events section often reveals API errors from the cloud provider.

   kubectl describe bucket my-crossplane-bucket-12345
   # Or
   kubectl describe xbucket my-dev-bucket-for-app
   

   3. Cloud Provider Permissions: The most common cause is insufficient IAM permissions for the credentials used in your ProviderConfig. Check the cloud provider's logs (e.g., AWS CloudTrail) for "AccessDenied" errors.
4. Issue: Deleting a Crossplane resource doesn't delete the actual cloud resource.

   Solution: This usually happens if the reclaimPolicy for the managed resource is set to Retain instead of Delete. By default, most Crossplane managed resources have a Delete reclaim policy, but it can be overridden. Check the spec.reclaimPolicy of the managed resource. Also, ensure the Crossplane provider has the necessary delete permissions.

   kubectl get bucket my-crossplane-bucket-12345 -o yaml | grep reclaimPolicy
   

5. Issue: My Composed Resource (e.g., XBucket) is stuck, and the underlying managed resources (e.g., Bucket) are not created.

   Solution:

   1. Check XRD and Composition: Ensure the CompositeResourceDefinition (XRD) and Composition are correctly applied and valid.

   kubectl get xrd xbuckets.kubezilla.io
   kubectl get composition xbucket-aws-s3
   

   2. Check XBucket status: Describe your XBucket instance. Look for any errors in its status or events sections, especially related to composition or resource claims.

   kubectl describe xbucket my-dev-bucket-for-app
   

   3. Composition Patching Issues: If fields aren't being correctly mapped, there might be an issue with your patches in the Composition. Check exact field paths and transform logic.
6. Issue: Cannot connect to AWS/GCP/Azure API from Crossplane provider pod.

   Solution: This indicates a network connectivity issue from your Kubernetes cluster to the cloud provider's API endpoints.

   • Check network policies within your cluster. You might need to adjust Kubernetes Network Policies to allow egress traffic to cloud API endpoints.
   • Verify DNS resolution from within the pod.
   • If using a proxy, ensure the provider pod is configured to use it.
   • For clusters using advanced CNI like Cilium, ensure there are no unintended egress restrictions. See Cilium WireGuard Encryption for related networking topics.

FAQ Section

Q1: How does Crossplane compare to Terraform?

A1: Both Crossplane and Terraform are Infrastructure as Code tools. The key difference lies in their operational model. Terraform is an external CLI tool that applies infrastructure changes. Crossplane, on the other hand, extends the Kubernetes API, making your Kubernetes cluster the control plane for both applications and infrastructure. This enables GitOps for infrastructure, unified RBAC, and leveraging Kubernetes' reconciliation loop for continuous state management. You can even run Terraform within Crossplane using the Crossplane Terraform Provider for a hybrid approach.

Q2: Can I manage existing cloud resources with Crossplane?

A2: Yes, Crossplane supports importing existing cloud resources. You typically create a Crossplane managed resource (e.g., Bucket) with the external-name annotation matching the name of your existing cloud resource. Crossplane will then take ownership and manage its lifecycle.

Q3: What are Compositions, and why are they important?

A3: Compositions are Crossplane's mechanism for abstraction. They allow platform teams to define higher-level, opinionated APIs (Composite Resources) that bundle multiple underlying cloud resources into a single, user-friendly interface. For example, an "XDatabase" Composite Resource could provision an RDS instance, a subnet group, and a security group. This empowers developers to self-service infrastructure without needing deep knowledge of cloud provider specifics, fostering an Internal Developer Platform (IDP).

Q4: Is Crossplane secure for production use?

A4: Yes, Crossplane is designed with security in mind and is a