Orchestration

kubectl Deep Dive: What Really Happens When You Press Enter

October 10, 2026 Kubezilla Team 13 min read
kubectl Deep Dive: What Really Happens When You Press Enter, cover image with a terminal running kubectl apply

You type kubectl get pods dozens of times a day. It feels like a simple command. But in the half second before that table shows up, kubectl reads a config file, picks a cluster, proves who you are, asks the API server what resources exist, builds an HTTP request, and turns raw JSON into a neat table.

Most people never look at any of that. And that is fine, until something breaks. A context points at the wrong cluster. An apply wipes out a field someone else set. A command hangs for 30 seconds before it works. When that happens, knowing how kubectl works inside is the difference between guessing and fixing.

This post opens the hood. We will follow a single command from your terminal to the API server and back, and pick up the features that make you faster along the way.

kubectl Is Just an HTTP Client

This is the most useful fact about kubectl: it does not talk to nodes, pods, or etcd. It only talks to the Kubernetes API server, over HTTPS, using plain REST calls. Everything else in the cluster reacts to what is stored behind that API.

You can see this for yourself. Turn up the verbosity:

kubectl get pods -n default -v=6

You will see lines like this:

I1010 10:12:04.118  loader.go:395] Config loaded from file:  /home/ajeet/.kube/config
I1010 10:12:04.231  round_trippers.go:553] GET https://127.0.0.1:6443/api/v1/namespaces/default/pods?limit=500 200 OK in 14 milliseconds

That is the whole story in two lines. A config file was loaded, and one GET request was sent. Go further with -v=8 to see request and response headers and the body, or -v=9 to see the full response with no truncation. Level 9 also prints the equivalent curl command, which is handy when you want to replay a call outside kubectl.

You can even skip the friendly layer and call the API directly:

kubectl get --raw /api/v1/namespaces/default/pods | jq '.items[].metadata.name'
kubectl get --raw /readyz?verbose
kubectl get --raw /metrics | head

Once you see kubectl as a REST client, a lot of behavior stops being mysterious. Slow commands are slow HTTP calls. “Forbidden” errors are HTTP 403s. Missing resources are 404s. Now let us walk through each step of a request.

Step 1: Finding the Cluster with kubeconfig

Before kubectl can send anything, it needs to know where to send it and who you are. That comes from kubeconfig. By default it reads ~/.kube/config, but the KUBECONFIG environment variable can point to one or more files.

A kubeconfig has three lists:

  • clusters: API server URLs and the CA certificate to trust them
  • users: credentials, such as client certificates, tokens, or an exec plugin
  • contexts: a named pairing of one cluster, one user, and an optional default namespace

The current-context field decides which pairing is active. That is it. There is no magic “login” state. Switching clusters is just changing one string in a file.

kubectl config get-contexts
kubectl config use-context kind-dev
kubectl config set-context --current --namespace=payments
kubectl config view --minify          # only the active context
kubectl config view --minify --raw    # include secrets, careful

Merging multiple files

When KUBECONFIG lists several files, kubectl merges them. The first file to set a value wins. This lets you keep each cluster in its own file and avoid one giant config:

export KUBECONFIG=~/.kube/config:~/.kube/eks-prod.yaml:~/.kube/kind-dev.yaml
kubectl config get-contexts

A safety tip: keep production configs out of your default path. Load them only in a shell where you mean to touch production. A wrong context is one of the most common causes of real outages, and no RBAC rule can save you from running the right command on the wrong cluster.

Step 2: Proving Who You Are

Every request carries credentials. Common types are:

  • Client certificates, used by kubeadm, kind, and many local clusters
  • Bearer tokens, such as ServiceAccount tokens
  • Exec credential plugins, used by EKS, GKE, AKS, and OIDC setups. kubectl runs an external binary (for example aws eks get-token or kubelogin) and uses the short-lived token it returns.

Exec plugins explain why a command sometimes pauses or opens a browser. It is not kubectl being slow. It is your cloud login tool fetching a fresh token.

Two commands help a lot here. The first tells you who the API server thinks you are:

kubectl auth whoami

The second checks what you are allowed to do, before you try it:

kubectl auth can-i create deployments -n payments
kubectl auth can-i '*' '*' --all-namespaces      # am I cluster admin?
kubectl auth can-i --list -n payments
kubectl auth can-i get secrets --as=system:serviceaccount:payments:api

The --as flag uses impersonation. It is the fastest way to test the RBAC rules you wrote for a ServiceAccount without creating a pod to try them.

Step 3: Discovery, or How kubectl Knows What “pods” Means

You type pods, po, or deploy. The API server only understands paths like /apis/apps/v1/namespaces/default/deployments. Something has to translate.

That is the job of API discovery. kubectl asks the server which API groups, versions, and resources exist, along with their short names and verbs. It then uses a component called the RESTMapper to turn your words into a URL.

kubectl api-resources
kubectl api-resources --api-group=apps -o wide
kubectl api-versions

Discovery results are cached under ~/.kube/cache/discovery. This is why a CRD you just installed sometimes shows “the server does not have a resource type”. The cache is stale. kubectl refreshes it on its own, but you can force it:

rm -rf ~/.kube/cache/discovery
# or
kubectl api-resources --cached=false

Discovery also powers kubectl explain, which reads the OpenAPI schema straight from your cluster. It is better than any web page because it matches your exact version and your installed CRDs:

kubectl explain deployment.spec.strategy
kubectl explain pod.spec.containers.resources --recursive

Step 4: Sending the Request

Now kubectl builds the HTTP call. A get becomes a GET, a delete becomes a DELETE, and so on. The interesting part is what happens on the server side for writes.

When a create or update reaches the API server, it goes through this pipeline:

  • Authentication: who are you?
  • Authorization: are you allowed to do this? (usually RBAC)
  • Mutating admission: webhooks and built-in plugins may change your object, for example to inject a sidecar or set defaults
  • Schema validation
  • Validating admission: policies such as ValidatingAdmissionPolicy, Kyverno, or Gatekeeper may reject it
  • Persist to etcd

Only after the object is stored do controllers and the scheduler notice and start acting. kubectl returning “created” means the object is saved, not that your pods are running.

Dry runs that actually test something

There are two kinds of dry run, and they are very different:

kubectl apply -f app.yaml --dry-run=client   # local only, no server checks
kubectl apply -f app.yaml --dry-run=server   # full pipeline, nothing saved

Client dry run only checks that your YAML parses. Server dry run sends the request through authentication, RBAC, admission webhooks, and validation, then throws the result away. If a policy will block your deploy, --dry-run=server tells you now instead of in your CI logs.

Even better is kubectl diff, which uses a server dry run under the hood and shows exactly what will change:

kubectl diff -f app.yaml

Make it a habit to run diff before every apply on a shared cluster.

Apply, Explained: Client-Side vs Server-Side

kubectl apply is declarative. You describe what you want, and kubectl works out the change. But how it works that out matters a lot when more than one tool touches the same object.

Client-side apply

The classic mode stores your last applied config in an annotation called kubectl.kubernetes.io/last-applied-configuration. On the next apply, kubectl compares three things: that annotation, your new file, and the live object. The math happens on your laptop.

The problem is that only kubectl knows about that annotation. If an autoscaler, a controller, or a teammate changes the same field, client-side apply can quietly overwrite it.

Server-side apply

Server-side apply moves the merge logic into the API server and tracks field ownership. Every field records which “manager” set it. If you try to change a field someone else owns, you get a conflict instead of a silent overwrite.

kubectl apply --server-side -f app.yaml
kubectl apply --server-side --field-manager=ci-pipeline -f app.yaml
kubectl apply --server-side --force-conflicts -f app.yaml   # take ownership on purpose

You can see who owns what:

kubectl get deploy web -o yaml --show-managed-fields

A typical conflict looks like this. Your file sets replicas: 3, but the HorizontalPodAutoscaler owns spec.replicas. Server-side apply refuses, and you learn that replicas should not be in your manifest at all. That one lesson prevents a whole class of “why did my app scale down after deploy” incidents.

For new work, server-side apply is the better default. GitOps tools like Argo CD and Flux support it too.

Watching Instead of Polling

The Kubernetes API is built around watches. A client opens a long-running request and the server streams every change. kubectl exposes this in several useful ways:

kubectl get pods -w                       # stream changes
kubectl get pods -w --output-watch-events  # show ADDED / MODIFIED / DELETED
kubectl rollout status deploy/web          # wait for a rollout to finish
kubectl events --watch -n payments         # stream events

The most underused one is kubectl wait. Stop writing sleep 30 in your scripts:

kubectl wait --for=condition=Available deploy/web --timeout=120s
kubectl wait --for=condition=Ready pod -l app=web --timeout=60s
kubectl wait --for=delete pod/old-job-xyz --timeout=60s
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/db-0
kubectl wait --for=create secret/tls-cert --timeout=90s

These commands return as soon as the condition is true, and fail with a clear exit code if it never is. Your CI pipeline becomes both faster and more honest.

Output: Getting Exactly the Data You Want

The API returns JSON. The table you normally see is a server-side “Table” view that kubectl prints. You can ask for any shape you like.

JSONPath

# Image of every container in every pod
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

# Node internal IPs
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'

# Decode a secret value
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d

Custom columns

When you want a table but with your own fields:

kubectl get pods -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount'

Sorting and filtering

kubectl get pods --sort-by=.status.startTime
kubectl get events --sort-by=.lastTimestamp
kubectl get pods --field-selector=status.phase!=Running -A
kubectl get pods -l 'app in (web,api),tier!=cache'

Field selectors and label selectors are filtered by the API server, not by your shell. On a big cluster that means less data over the wire and faster results than piping to grep.

KYAML

Recent kubectl releases add a new output format called KYAML. It is still valid YAML, but it always quotes strings and uses flow-style braces, so whitespace mistakes and the classic “Norway problem” (where NO turns into false) go away.

kubectl get deploy web -o kyaml

On v1.34 this needed KUBECTL_KYAML=true. Check kubectl get --help on your version to see if it is on by default.

Debugging Like You Mean It

This is where kubectl skills pay off the most. Here is a practical order for a pod that is not working.

1. Look at events first

kubectl describe pod web-7d9f8 -n payments
kubectl events --for pod/web-7d9f8 -n payments

Image pull errors, failed scheduling, failing probes, and OOM kills all show up here. The kubectl events command is newer than get events and sorts by time by default.

2. Read the logs, including the crashed ones

kubectl logs web-7d9f8 -c app
kubectl logs web-7d9f8 --previous        # logs from before the last restart
kubectl logs deploy/web --all-containers --since=15m
kubectl logs -l app=web --prefix --max-log-requests=10 -f

If a pod is in CrashLoopBackOff, --previous is the flag you want. The current container has just started and has nothing useful to say yet.

3. Get inside, even when there is no shell

Distroless and hardened images often have no shell, so kubectl exec fails. Ephemeral debug containers solve this. They attach a new container to the running pod and can share its process namespace:

kubectl debug -it web-7d9f8 --image=busybox:1.36 --target=app
kubectl debug web-7d9f8 -it --image=nicolaka/netshoot --copy-to=web-debug --share-processes
kubectl debug node/worker-2 -it --image=ubuntu    # host filesystem at /host

Recent versions also support --profile (for example general, netadmin, or sysadmin) to give the debug container the right privileges for the job.

4. Reach the service from your laptop

kubectl port-forward svc/web 8080:80 -n payments
kubectl port-forward pod/db-0 5432:5432 --address 127.0.0.1

Port-forward tunnels through the API server, not directly to the node, so it works even when your laptop cannot reach the pod network.

5. Check resource use

kubectl top pods -n payments --sort-by=memory
kubectl top nodes

This needs metrics-server installed. If top says “Metrics API not available”, that is the reason.

Extending kubectl with Plugins

kubectl has a very simple plugin system. Any executable on your PATH named kubectl-something becomes kubectl something. That is the whole contract.

cat > /usr/local/bin/kubectl-podnodes <<'EOF'
#!/usr/bin/env bash
kubectl get pods "$@" -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase'
EOF
chmod +x /usr/local/bin/kubectl-podnodes

kubectl podnodes -n payments
kubectl plugin list

For ready-made plugins, use Krew, the plugin manager from Kubernetes SIG CLI. A few worth trying:

  • kubectl krew install ctx ns: fast context and namespace switching
  • kubectl krew install neat: strips noisy fields from YAML so you can reuse it
  • kubectl krew install tree: shows owner relationships, like Deployment to ReplicaSet to Pod
  • kubectl krew install who-can: the reverse of auth can-i; who is allowed to do this?

Your Own Defaults with kuberc

For years, people customized kubectl with shell aliases. That works, but aliases cannot change default flags, and they mix personal taste into kubeconfig sharing. kuberc fixes this.

kuberc is a separate file, ~/.kube/kuberc, just for your personal preferences. It has been beta since Kubernetes v1.34. It keeps cluster details in kubeconfig and your habits in kuberc, so you never have to share one to share the other.

apiVersion: kubectl.config.k8s.io/v1beta1
kind: Preference
defaults:
  # Always use server-side apply
  - command: apply
    options:
      - name: server-side
        default: "true"
  # Ask before deleting
  - command: delete
    options:
      - name: interactive
        default: "true"
aliases:
  # "kubectl getn" lists namespaces as JSON
  - name: getn
    command: get
    prependArgs:
      - namespace
    options:
      - name: output
        default: json

That delete --interactive default alone might save your weekend. You can point at a different file with --kuberc or the KUBERC environment variable. In v1.36, kubectl also has a kubectl kuberc command with view and set subcommands, so you can manage the file without editing YAML by hand.

Small Habits That Add Up

A few things that make daily work smoother:

  • Turn on completion. source <(kubectl completion bash) or the zsh/fish versions. It completes resource names from your live cluster.
  • Generate YAML instead of writing it. kubectl create deploy web --image=nginx --dry-run=client -o yaml > web.yaml gives you a clean starting point.
  • Show your context in your prompt. Seeing “prod” in red before every command is cheap insurance.
  • Use -o name in scripts. It prints pod/web-7d9f8, which you can pipe straight into other kubectl commands.
  • Keep kubectl close to your cluster version. kubectl supports one minor version older or newer than the API server. Outside that window, odd things happen.

Quick Reference

I want to…Command
See the raw HTTP callskubectl get pods -v=8
Check who I amkubectl auth whoami
Check a permissionkubectl auth can-i delete pods -n prod
Test a change against policieskubectl apply -f x.yaml --dry-run=server
Preview a changekubectl diff -f x.yaml
Avoid overwriting other toolskubectl apply --server-side -f x.yaml
Wait for something in a scriptkubectl wait --for=condition=Ready pod -l app=web
Read logs from a crashed containerkubectl logs pod/x --previous
Debug a pod with no shellkubectl debug -it pod/x --image=busybox --target=app
Learn a fieldkubectl explain deploy.spec.strategy

Wrapping Up

kubectl looks like a simple CLI, but it is really a well-designed API client. It reads your kubeconfig, gets a token, discovers what the cluster supports, sends a REST call, and formats the answer. Each of those steps has tools you can use: -v=8 to see the traffic, auth can-i to check access, explain to read the schema, server-side apply to avoid fights over fields, wait to replace sleeps, and debug to get into any pod.

Pick two or three of these and use them this week. The next time something breaks, you will know exactly where to look.

What is your favorite kubectl trick? Share it with the community on the Collabnix Slack or Discord.

Leave a comment