Manage Infra with Crossplane + Kubernetes

Verify:
Observe the status of your XPostgreSQLInstance and the underlying RDSInstance. It will take some time for the RDS instance to become available.

kubectl get xpostgresqlinstance my-app-db

Expected Output (eventually):

NAME        READY   SYNCED   AGE
my-app-db   True    True     5m

You can also check the underlying AWS RDS instance:

kubectl get rdsinstance

Expected Output (eventually):

NAME                                          READY   SYNCED   EXTERNAL-NAME                                         AGE
my-app-db-d8f8a84d-2d4e-4f5a-8b9c-0d1e2f3g4h5i   True    True     my-app-db-d8f8a84d-2d4e-4f5a-8b9c-0d1e2f3g4h5i-xxxx   10m

Once the RDS instance is ready, a connection secret will be created in the default namespace (as specified in writeConnectionSecretToRef).

kubectl get secret my-app-db-connection -n default -o yaml

Expected Output (truncated):

apiVersion: v1
kind: Secret
metadata:
  name: my-app-db-connection
  namespace: default
type: Opaque
data:
  KUBERNETES_CLUSTER_NAME: bXkta3ViZS1jbHVzdGVy # base64 encoded "my-kube-cluster"
  dbname: 
  endpoint: 
  password: 
  port: 
  username: 

Your applications can now consume this secret to connect to the PostgreSQL database. This declarative approach simplifies infrastructure provisioning and integrates seamlessly with your Kubernetes workflows. For advanced traffic management to your applications, check out our guide on the Kubernetes Gateway API.

Production Considerations

Deploying Crossplane in a production environment requires careful planning to ensure reliability, security, and scalability:

• Security and IAM Roles:
  • Least Privilege: Ensure the IAM credentials used by Crossplane (via ProviderConfig) have only the necessary permissions to provision and manage the specified resources. Regularly audit these permissions.
  • Secrets Management: Use a robust secrets management solution (e.g., Secrets Store CSI Driver, HashiCorp Vault) to inject cloud provider credentials into the crossplane-system namespace, rather than hardcoding them or storing them directly in Git.
  • Workload Identity/IRSA: On cloud providers like AWS (IRSA), GCP (Workload Identity), or Azure (AAD Pod Identity), configure Crossplane to assume an IAM role directly, avoiding long-lived static credentials. This is the most secure approach.
• High Availability:
  • Controller Replicas: Run Crossplane and provider controllers with multiple replicas to ensure high availability and resilience against node failures.
  • Cluster Reliability: Ensure your underlying Kubernetes cluster is highly available.
• Observability and Monitoring:
  • Logs: Aggregate Crossplane controller logs and provider controller logs into a centralized logging system.
  • Metrics: Monitor Crossplane's own metrics (exposed via Prometheus endpoints) to track resource reconciliation, API call rates, and error rates.
  • Status Conditions: Regularly check the status.conditions of your managed and composite resources for insights into their state and any issues. For deeper insights, consider leveraging techniques like those described in our eBPF Observability with Hubble guide.
• Backup and Disaster Recovery:
  • Etcd Backup: Crossplane stores its state in Kubernetes' etcd. Ensure your Kubernetes cluster's etcd is regularly backed up.
  • Configuration Backup: Store all your Crossplane manifests (Providers, ProviderConfigs, XRDs, Compositions, Managed Resources, XRs) in version control (Git) for disaster recovery and reproducibility.
• Scalability:
  • Resource Limits: Set appropriate resource requests and limits for Crossplane and provider pods to prevent resource exhaustion and ensure stable operation.
  • Provider Scaling: As the number of managed resources grows, the pressure on provider controllers increases. Monitor and scale provider replicas as needed.
• Drift Detection and Reconciliation:
  • Crossplane continuously reconciles the desired state (your Kubernetes manifests) with the actual state (in the cloud). This helps detect and correct configuration drift.
  • Understand the reconciliation intervals and how they impact resource updates and error recovery.
• Version Management:
  • Pin versions of Crossplane and its providers (e.g., v0.40.0) to ensure predictable behavior and avoid unexpected changes.
  • Test upgrades in a staging environment before applying them to production.
• Cost Management:
  • Crossplane provisions real cloud resources, which incur costs. Implement tagging strategies (as shown in the S3 example) to track and attribute costs.
  • Integrate with cloud cost management tools. For optimizing Kubernetes node costs, our Karpenter Cost Optimization guide might be relevant.

Troubleshooting

Here are some common issues you might encounter with Crossplane and how to resolve them:

1. Issue: Crossplane or Provider Pods are not running/healthy.

   Description: The Crossplane or provider controller pods are in Pending, CrashLoopBackOff, or Error state.

   Solution:

   • Check Pod Logs: The first step is always to check the logs of the problematic pod.

     kubectl logs -f <pod-name> -n crossplane-system

   • Describe Pod: Use kubectl describe pod <pod-name> -n crossplane-system to check for events, resource limits, and volume mount issues.
   • Resource Constraints: Ensure your cluster has enough resources (CPU, Memory) for the pods. Increase requests/limits if necessary.
   • Image Pull Issues: Verify that the image can be pulled from the registry.
2. Issue: Provider is not HEALTHY: True.

   Description: After installing a provider (e.g., provider-aws), kubectl get providers shows HEALTHY: False.

   Solution:

   • Check Provider Pod Logs: The provider pod is likely failing to start or connect to the Crossplane API server.

     kubectl get pods -n crossplane-system -l app=crossplane-provider-<provider-name>
     kubectl logs -f <provider-pod-name> -n crossplane-system

   • Verify Crossplane Core: Ensure the core Crossplane pods are running and healthy first.
   • Network Connectivity: Check if the provider can reach external registries to pull its image or external APIs (though less common for HEALTHY status).
3. Issue: Managed Resource (e.g., S3Bucket) is stuck in Provisioning or Creating, or READY: False.

   Description: The resource is not being created in the cloud, or its status isn't updating.

   Solution:

   • Check Resource Events: