Build Your Internal Developer Platform

“`html

Introduction

In today’s fast-paced software development landscape, engineering teams are constantly striving for greater agility, efficiency, and consistency. While Kubernetes has revolutionized how we deploy and manage applications, simply adopting it isn’t enough to unlock its full potential. This is where Platform Engineering comes into play, offering a strategic approach to building an Internal Developer Platform (IDP) that empowers developers to self-serve, innovate faster, and reduce operational overhead.

An Internal Developer Platform acts as a curated layer on top of your existing infrastructure, abstracting away much of the underlying complexity of Kubernetes, cloud services, and other operational tools. By providing standardized workflows, golden paths, and opinionated defaults, an IDP allows developers to focus on writing code and delivering business value, rather than getting bogged down in infrastructure configuration and deployment details. In this comprehensive guide, we’ll walk through the process of designing and implementing a foundational IDP on Kubernetes, covering key components, best practices, and practical examples to get you started.

Prerequisites

Before diving into building your Internal Developer Platform, ensure you have the following:

• Kubernetes Cluster: An existing Kubernetes cluster (e.g., Minikube, Kind, GKE, EKS, AKS). For production, a managed Kubernetes service is highly recommended.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Helm: The Kubernetes package manager. Install it from the official Helm website.
• Git: Familiarity with Git and a Git repository (GitHub, GitLab, Bitbucket) to store your platform configurations and application manifests.
• Basic Kubernetes Knowledge: Understanding of Kubernetes concepts like Pods, Deployments, Services, Ingress, and Namespaces.
• Domain Name (Optional but Recommended): For exposing applications with a custom domain.

Step-by-Step Guide: Building Your Internal Developer Platform

Building an IDP is an iterative process. We’ll focus on foundational components that provide immediate value: GitOps for continuous delivery, a service mesh for traffic management and security, and policy enforcement for governance.

Step 1: Establish GitOps with Argo CD

GitOps is the cornerstone of any modern IDP. It enables continuous delivery by using Git as the single source of truth for declarative infrastructure and application configurations. Argo CD is a popular, open-source GitOps continuous delivery tool designed for Kubernetes. It automatically synchronizes the desired state defined in Git with the actual state of applications in your cluster, ensuring consistency and providing a clear audit trail.

By adopting GitOps, developers can make changes to their application configurations (e.g., resource limits, environment variables, image versions) directly in Git. Argo CD then detects these changes and applies them to the cluster. This reduces manual errors, simplifies rollbacks, and provides a powerful mechanism for managing environments.

Install Argo CD

First, create a dedicated namespace for Argo CD and apply its installation manifests. You can find the latest stable release manifests on the Argo CD GitHub repository.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify Argo CD Installation

Check if all Argo CD pods are running in the argocd namespace.

kubectl get pods -n argocd

Expected Output:

NAME                                        READY   STATUS    RESTARTS   AGE
argocd-application-controller-xxxxxxxxx-xxxxx   1/1     Running   0          2m
argocd-dex-server-xxxxxxxxx-xxxxx             1/1     Running   0          2m
argocd-notifications-controller-xxxxxxxxx-xxxxx 1/1     Running   0          2m
argocd-redis-xxxxxxxxx-xxxxx                  1/1     Running   0          2m
argocd-repo-server-xxxxxxxxx-xxxxx            1/1     Running   0          2m
argocd-server-xxxxxxxxx-xxxxx                 1/1     Running   0          2m

Access the Argo CD UI

To access the Argo CD web UI, you’ll need to forward the port for the argocd-server service and retrieve the initial admin password.

# Port-forward the argocd-server service
kubectl port-forward svc/argocd-server -n argocd 8080:443 &

# Get the initial admin password (replace 'argocd-server-xxxxxxxxx-xxxxx' with your actual pod name if needed)
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Navigate to https://localhost:8080 in your browser. Log in with username admin and the password retrieved above.

Deploy a Sample Application with Argo CD

Create a Git repository (e.g., https://github.com/your-org/platform-apps.git) and add a simple Nginx deployment manifest. This will serve as our first application managed by GitOps.

nginx-app.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Commit this file to your Git repository.

Now, create an Argo CD Application resource that tells Argo CD where to find your application manifests in Git and where to deploy them in your cluster.

argocd-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-nginx-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/platform-apps.git # Replace with your Git repository URL
    targetRevision: HEAD
    path: . # Path within the repository to your application manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: default # Deploy to the 'default' namespace
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply this Application resource:

kubectl apply -f argocd-app.yaml -n argocd

Verify Application Deployment

Watch Argo CD sync the application. You can observe this in the Argo CD UI or via the CLI.

argocd app get my-nginx-app --refresh --hard
kubectl get deployment nginx-deployment
kubectl get service nginx-service

Expected Output:

# From `argocd app get my-nginx-app` (shortened)
Name:               my-nginx-app
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://localhost:8080/applications/my-nginx-app
Repo:               https://github.com/your-org/platform-apps.git
Target:             HEAD
Path:               .
Sync Status:        Synced
Health Status:      Healthy

# From `kubectl get deployment nginx-deployment`
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           1m

# From `kubectl get service nginx-service`
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
nginx-service   ClusterIP   10.100.100.100   <none>        80/TCP    1m

Congratulations! You’ve successfully set up your first GitOps-managed application. For more advanced GitOps patterns and managing multiple clusters, explore Argo Rollouts for progressive delivery and Argo CD multi-cluster support.

Step 2: Implement Service Mesh with Istio

A service mesh provides critical capabilities for an IDP, offering traffic management, observability, security, and reliability features without requiring changes to application code. Istio is a powerful, open-source service mesh that integrates seamlessly with Kubernetes. It allows platform engineers to define routing rules, enforce policies, collect telemetry, and secure communication between services across your platform.

For developers, Istio simplifies tasks like A/B testing, canary deployments, and circuit breaking. For operations, it provides deep insights into service behavior and network health. If you’re interested in a lightweight alternative to Istio’s full sidecar model, consider exploring Istio Ambient Mesh, which offers a sidecar-less approach.

Install Istio

Download and install the Istio command-line tool (istioctl), then use it to install Istio onto your cluster. We’ll use the default profile for a good balance of features and resource consumption.

curl -L https://istio.io/downloadIstio | sh -
cd istio-*
./bin/istioctl install --set profile=default -y

Verify Istio Installation

Check if all Istio control plane pods are running in the istio-system namespace.

kubectl get pods -n istio-system

Expected Output:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-xxxxxxxxx-xxxxx    1/1     Running   0          2m
istiod-xxxxxxxxx-xxxxx                  1/1     Running   0          2m

Enable Istio Sidecar Injection for a Namespace

To have Istio manage services in a particular namespace, you need to label that namespace for automatic sidecar injection. We’ll label the default namespace where our Nginx app is deployed.

kubectl label namespace default istio-injection=enabled --overwrite

Restart Application Pods to Inject Sidecars

For existing pods in an injected namespace, you need to restart them for the Istio proxy (sidecar) to be injected.

kubectl rollout restart deployment nginx-deployment

Verify Sidecar Injection

Check the pods again; they should now show 2/2 containers (one for Nginx, one for the Istio proxy).

kubectl get pods -l app=nginx

Expected Output:

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-xxxxxxxxx-xxxxx    2/2     Running   0          1m
nginx-deployment-xxxxxxxxx-xxxxx    2/2     Running   0          1m
nginx-deployment-xxxxxxxxx-xxxxx    2/2     Running   0          1m

Expose the Nginx Application via Istio Gateway

Instead of a standard Kubernetes Ingress, we’ll use an Istio Gateway and VirtualService to expose our application. This provides more advanced traffic management capabilities. For a deeper dive into modern Kubernetes networking, check out our guide on the Kubernetes Gateway API vs Ingress.

nginx-gateway.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-gateway
spec:
  selector:
    istio: ingressgateway # Use the default Istio ingress gateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "nginx.example.com" # Replace with your domain or use a placeholder
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-virtualservice
spec:
  hosts:
  - "nginx.example.com" # Must match the Gateway host
  gateways:
  - nginx-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx-service # Name of your Kubernetes Service
        port:
          number: 80

Apply these resources:

kubectl apply -f nginx-gateway.yaml

Get the Ingress IP/Hostname

To access your application, you need the external IP or hostname of the Istio Ingress Gateway. This will vary depending on your Kubernetes environment.

# For LoadBalancer service (GCP, AWS, Azure)
kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# For NodePort (Minikube, Kind, or bare-metal)
# Minikube:
# minikube ip
# kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'
#
# Kind:
# kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'
# (Then find your Kind cluster's external IP or use `localhost` if port-forwarded)

Once you have the IP/hostname, update your local /etc/hosts file (or DNS) to point nginx.example.com to this IP/hostname. Then, access your application via http://nginx.example.com.

Step 3: Enforce Policies with Kyverno

Governance and security are paramount in any production environment. An IDP needs mechanisms to ensure that deployed applications adhere to organizational standards, security policies, and best practices. Kyverno is a Kubernetes native policy engine that can validate, mutate, and generate configurations. Unlike other policy engines, Kyverno policies are simply Kubernetes resources, so no new language is required.

With Kyverno, platform engineers can define policies that:

• Require specific labels or annotations on all resources.
• Enforce resource limits and requests.
• Restrict image registries to trusted sources.
• Add default sidecars (e.g., for observability agents).
• Prevent deployments from privileged containers.

For more advanced supply chain security, integrate Kyverno with Sigstore and Kyverno to verify container image signatures.

Install Kyverno

Install Kyverno using Helm. It’s recommended to install it in its own namespace.

helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Verify Kyverno Installation

Check if Kyverno pods are running in the kyverno namespace.

kubectl get pods -n kyverno

Expected Output:

NAME                                     READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-xxxxxxxxx-xxxxx   1/1     Running   0          1m
kyverno-background-controller-xxxxxxxxx-xxxxx  1/1     Running   0          1m
kyverno-cleanup-controller-xxxxxxxxx-xxxxx     1/1     Running   0          1m

Create a Sample Policy: Enforce Resource Limits

Let’s create a ClusterPolicy that requires all containers in new deployments to have CPU and memory requests and limits defined. This is a critical best practice for stable and cost-efficient clusters. For further cost optimization, consider tools like Karpenter for Kubernetes Cost Optimization.

resource-limits-policy.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce # Can be Audit or Enforce
  rules:
  - name: check-container-resources
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Containers must have CPU and memory requests and limits."
      pattern:
        spec:
          containers:
            - resources:
                requests:
                  memory: "?*"
                  cpu: "?*"
                limits:
                  memory: "?*"
                  cpu: "?*"

Apply the policy:

kubectl apply -f resource-limits-policy.yaml

Test the Policy

Try deploying a pod without resource limits. Kyverno should block it.

bad-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
    # No resources defined

kubectl apply -f bad-pod.yaml

Expected Output (Error):

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/bad-pod was blocked due to the following policies
require-resource-limits:
  check-container-resources: 'Validation error: Containers must have CPU and memory requests and limits.

Now, deploy a pod with resource limits, and it should succeed.

good-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: good-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"

kubectl apply -f good-pod.yaml

Expected Output:

pod/good-pod created

Step 4: Observability (Monitoring, Logging, Tracing)

A robust IDP isn’t complete without comprehensive observability. Developers need to understand how their applications are performing, identify issues quickly, and gain insights into user behavior. This typically involves a combination of monitoring, logging, and tracing solutions.

• Monitoring: Prometheus for collecting metrics and Grafana for visualization.
• Logging: Loki for aggregating logs, often combined with Promtail or Fluent Bit as agents.
• Tracing: OpenTelemetry for standardized tracing, with backends like Jaeger or Tempo.

While a full setup is beyond this guide’s scope, here’s how you might install a basic Prometheus and Grafana stack via Helm.

Install Prometheus and Grafana

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace
helm install grafana grafana/grafana --namespace monitoring

Access Grafana

Get the admin password for Grafana:

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Port-forward to access Grafana UI:

kubectl port-forward svc/grafana -n monitoring 3000:80 &

Access Grafana at http://localhost:3000. Log in with admin and the password you retrieved.

For advanced eBPF-based observability, particularly for network insights, explore eBPF Observability: Building Custom Metrics with Hubble.

Production Considerations

Building an IDP for production requires careful planning and implementation beyond the basic setup. Here are key areas to consider:

1. Security Best Practices:
   • RBAC: Implement strict Role-Based Access Control for all components and developer access. Use different roles for different teams and environments.
   • Network Policies: Secure inter-service communication and control ingress/egress traffic within your cluster. Our Kubernetes Network Policies Security Guide offers comprehensive hardening strategies.
   • Secrets Management: Use a dedicated solution like HashiCorp Vault, Kubernetes Secrets Store CSI Driver, or cloud provider secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) instead of raw Kubernetes Secrets.
   • Image Scanning & Signing: Integrate container image scanning into your CI/CD pipeline and enforce image signing with tools like Sigstore to ensure only trusted images are deployed.
2. Scalability & High Availability:
   • Cluster Autoscaling: Implement a cluster autoscaler (e.g., Cluster Autoscaler or Karpenter) to dynamically adjust node capacity based on demand.
   • Horizontal Pod Autoscaling (HPA): Configure HPA for your applications to scale pods based on CPU, memory, or custom metrics.
   • Multi-Zone/Multi-Region Deployments: Design your IDP and applications for resilience across multiple availability zones or even regions.
3. Cost Optimization:
   • Resource Quotas & Limit Ranges: Enforce resource quotas at the namespace level and use limit ranges to set default resource requests/limits for pods.
   • Right-Sizing: Continuously monitor resource usage and right-size your application pods and nodes. Tools like Vertical Pod Autoscaler (VPA) can help.
   • Spot/Preemptible Instances: Leverage spot instances for fault-tolerant workloads to significantly reduce costs, especially with intelligent schedulers like Karpenter.
4. Developer Experience (DX):
   • Standardized Templates: Provide golden path templates for common application types (e.g., microservice, web app, database) that include pre-configured manifests, CI/CD pipelines, and observability.
   • Internal Documentation: Maintain comprehensive and up-to-date documentation for all IDP components, APIs, and workflows.
   • Self-Service Portals: Consider building or adopting a self-service portal (e.g., Backstage.io) that acts as a single pane of glass for developers to provision resources, deploy applications, and view dashboards.
5. Continuous Improvement:
   • Feedback Loops: Establish clear channels for developers to provide feedback on the IDP.
   • Metrics & KPIs: Define key performance indicators for your platform (e.g., deployment frequency, lead time for changes, MTTR, developer satisfaction) and track them regularly.
   • Automated Testing: Implement automated tests for your platform components and configurations to ensure reliability and prevent regressions.

Troubleshooting

Even with careful planning, issues will arise. Here are common troubleshooting scenarios when building an IDP on Kubernetes:

1. Argo CD Application Not Syncing / OutOfSync Status

   Issue: Your Argo CD application shows OutOfSync or fails to sync.

   Solution:

   • Check Argo CD UI: The UI provides detailed error messages and diffs between desired and live states.
   • Verify Git Repo & Path: Ensure repoURL, targetRevision, and path in your Application resource are correct and accessible by Argo CD.
   • Kubernetes Permissions: Argo CD needs appropriate RBAC permissions to create/update resources in the target namespace. Check the service account associated with the argocd-application-controller.
   • Resource Conflicts: If resources already exist in the cluster but are not managed by Argo CD, you might need to manually delete them or use Argo CD’s replace or prune options carefully.

   
   argocd app logs my-nginx-app
   argocd app diff my-nginx-app
   

2. Istio Sidecar Not Injected

   Issue: Your application pods are running with 1/1 containers instead of 2/2, indicating the Istio sidecar wasn’t injected.

   Solution:

   • Namespace Label: Ensure the target namespace is labeled with istio-injection=enabled. Remember to restart pods after labeling.
   • Istiod Health: Check the logs and status of the istiod pod in the istio-system namespace.
   • Admission Webhook: Istio uses a mutating admission webhook for injection. Ensure it’s healthy and not blocked by network policies or other admission controllers.

   
   kubectl get ns default --show-labels
   kubectl describe pod <your-app-pod>
   kubectl logs -n istio-system $(kubectl get pod -n istio-system -l app=istiod -o jsonpath='{.items[0].metadata.name}')
   

3. Application Not Accessible via Istio Gateway/VirtualService

   Issue: You’ve configured a Gateway and VirtualService, but your application is unreachable.

   Solution:

   • Gateway IP/Hostname: Double-check the external IP/hostname of the istio-ingressgateway service and your DNS//etc/hosts configuration.
   • Host Match: Ensure the hosts in your Gateway and VirtualService exactly match the hostname you’re trying to access.
   • Service Name: Verify the destination.host in your VirtualService points to the correct Kubernetes Service name.
   • Port Configuration: Check that the port in the Gateway, VirtualService, and your Kubernetes Service match.
   • Firewall: If on a cloud provider, ensure the necessary ports (e.g., 80, 443) are open in your cloud firewall rules for the ingress gateway.

   
   kubectl get gateway,virtualservice -o yaml
   istioctl analyze
   

4. Kyverno Policy Not Enforcing/Mutating

   Issue: You’ve created a Kyverno policy, but it’s not blocking or modifying resources as expected.

   Solution:

   • Policy Status: Check the status of your ClusterPolicy or Policy resource. Ensure it’s Active.
   • validationFailureAction: Confirm validationFailureAction is set to Enforce for validation policies you want to block. For mutation, ensure the rule is configured correctly.
   • match/exclude Rules: Carefully review the match and exclude blocks in your policy to ensure it targets the correct resources and namespaces.
   • Kyverno Logs: Check the logs of the Kyverno admission controller for any errors or policy evaluations.
   • PolicyReport: Kyverno generates PolicyReport resources. Check these for audit results even if validationFailureAction is Audit.

   
   kubectl get clusterpolicy require-resource-limits -o yaml
   kubectl logs -n kyverno $(kubectl get pod -n kyverno -l app.kubernetes.io/component=kyverno-admission-controller -o jsonpath='{.items[0].metadata.name}')
   kubectl get policyreport,clusterpolicyreport -A
   

5. Resource Exhaustion / Performance Issues

   Issue: Cluster components or applications are crashing, running slowly, or experiencing OOMKills.

   Solution:

   • Resource Requests/Limits: Ensure all application pods have appropriate CPU and memory requests and limits. Missing limits can lead to “noisy neighbor” problems.
   • Node