You type kubectl get pods dozens of times a day. It feels like a simple command. But in the half second before that table shows up, kubectl reads a config file, picks a cluster, proves who you are, asks the API server what resources exist, builds an HTTP request, and turns raw JSON into a neat table.
Most people never look at any of that. And that is fine, until something breaks. A context points at the wrong cluster. An apply wipes out a field someone else set. A command hangs for 30 seconds before it works. When that happens, knowing how kubectl works inside is the difference between guessing and fixing.
This post opens the hood. We will follow a single command from your terminal to the API server and back, and pick up the features that make you faster along the way.
kubectl Is Just an HTTP Client
This is the most useful fact about kubectl: it does not talk to nodes, pods, or etcd. It only talks to the Kubernetes API server, over HTTPS, using plain REST calls. Everything else in the cluster reacts to what is stored behind that API.
You can see this for yourself. Turn up the verbosity:
kubectl get pods -n default -v=6
You will see lines like this:
I1010 10:12:04.118 loader.go:395] Config loaded from file: /home/ajeet/.kube/config
I1010 10:12:04.231 round_trippers.go:553] GET https://127.0.0.1:6443/api/v1/namespaces/default/pods?limit=500 200 OK in 14 milliseconds
That is the whole story in two lines. A config file was loaded, and one GET request was sent. Go further with -v=8 to see request and response headers and the body, or -v=9 to see the full response with no truncation. Level 9 also prints the equivalent curl command, which is handy when you want to replay a call outside kubectl.
You can even skip the friendly layer and call the API directly:
kubectl get --raw /api/v1/namespaces/default/pods | jq '.items[].metadata.name'
kubectl get --raw /readyz?verbose
kubectl get --raw /metrics | head
Once you see kubectl as a REST client, a lot of behavior stops being mysterious. Slow commands are slow HTTP calls. “Forbidden” errors are HTTP 403s. Missing resources are 404s. Now let us walk through each step of a request.
Step 1: Finding the Cluster with kubeconfig
Before kubectl can send anything, it needs to know where to send it and who you are. That comes from kubeconfig. By default it reads ~/.kube/config, but the KUBECONFIG environment variable can point to one or more files.
A kubeconfig has three lists:
- clusters: API server URLs and the CA certificate to trust them
- users: credentials, such as client certificates, tokens, or an exec plugin
- contexts: a named pairing of one cluster, one user, and an optional default namespace
The current-context field decides which pairing is active. That is it. There is no magic “login” state. Switching clusters is just changing one string in a file.
kubectl config get-contexts
kubectl config use-context kind-dev
kubectl config set-context --current --namespace=payments
kubectl config view --minify # only the active context
kubectl config view --minify --raw # include secrets, careful
Merging multiple files
When KUBECONFIG lists several files, kubectl merges them. The first file to set a value wins. This lets you keep each cluster in its own file and avoid one giant config:
export KUBECONFIG=~/.kube/config:~/.kube/eks-prod.yaml:~/.kube/kind-dev.yaml
kubectl config get-contexts
A safety tip: keep production configs out of your default path. Load them only in a shell where you mean to touch production. A wrong context is one of the most common causes of real outages, and no RBAC rule can save you from running the right command on the wrong cluster.
Step 2: Proving Who You Are
Every request carries credentials. Common types are:
- Client certificates, used by kubeadm, kind, and many local clusters
- Bearer tokens, such as ServiceAccount tokens
- Exec credential plugins, used by EKS, GKE, AKS, and OIDC setups. kubectl runs an external binary (for example
aws eks get-tokenorkubelogin) and uses the short-lived token it returns.
Exec plugins explain why a command sometimes pauses or opens a browser. It is not kubectl being slow. It is your cloud login tool fetching a fresh token.
Two commands help a lot here. The first tells you who the API server thinks you are:
kubectl auth whoami
The second checks what you are allowed to do, before you try it:
kubectl auth can-i create deployments -n payments
kubectl auth can-i '*' '*' --all-namespaces # am I cluster admin?
kubectl auth can-i --list -n payments
kubectl auth can-i get secrets --as=system:serviceaccount:payments:api
The --as flag uses impersonation. It is the fastest way to test the RBAC rules you wrote for a ServiceAccount without creating a pod to try them.
Step 3: Discovery, or How kubectl Knows What “pods” Means
You type pods, po, or deploy. The API server only understands paths like /apis/apps/v1/namespaces/default/deployments. Something has to translate.
That is the job of API discovery. kubectl asks the server which API groups, versions, and resources exist, along with their short names and verbs. It then uses a component called the RESTMapper to turn your words into a URL.
kubectl api-resources
kubectl api-resources --api-group=apps -o wide
kubectl api-versions
Discovery results are cached under ~/.kube/cache/discovery. This is why a CRD you just installed sometimes shows “the server does not have a resource type”. The cache is stale. kubectl refreshes it on its own, but you can force it:
rm -rf ~/.kube/cache/discovery
# or
kubectl api-resources --cached=false
Discovery also powers kubectl explain, which reads the OpenAPI schema straight from your cluster. It is better than any web page because it matches your exact version and your installed CRDs:
kubectl explain deployment.spec.strategy
kubectl explain pod.spec.containers.resources --recursive
Step 4: Sending the Request
Now kubectl builds the HTTP call. A get becomes a GET, a delete becomes a DELETE, and so on. The interesting part is what happens on the server side for writes.
When a create or update reaches the API server, it goes through this pipeline:
- Authentication: who are you?
- Authorization: are you allowed to do this? (usually RBAC)
- Mutating admission: webhooks and built-in plugins may change your object, for example to inject a sidecar or set defaults
- Schema validation
- Validating admission: policies such as ValidatingAdmissionPolicy, Kyverno, or Gatekeeper may reject it
- Persist to etcd
Only after the object is stored do controllers and the scheduler notice and start acting. kubectl returning “created” means the object is saved, not that your pods are running.
Dry runs that actually test something
There are two kinds of dry run, and they are very different:
kubectl apply -f app.yaml --dry-run=client # local only, no server checks
kubectl apply -f app.yaml --dry-run=server # full pipeline, nothing saved
Client dry run only checks that your YAML parses. Server dry run sends the request through authentication, RBAC, admission webhooks, and validation, then throws the result away. If a policy will block your deploy, --dry-run=server tells you now instead of in your CI logs.
Even better is kubectl diff, which uses a server dry run under the hood and shows exactly what will change:
kubectl diff -f app.yaml
Make it a habit to run diff before every apply on a shared cluster.
Apply, Explained: Client-Side vs Server-Side
kubectl apply is declarative. You describe what you want, and kubectl works out the change. But how it works that out matters a lot when more than one tool touches the same object.
Client-side apply
The classic mode stores your last applied config in an annotation called kubectl.kubernetes.io/last-applied-configuration. On the next apply, kubectl compares three things: that annotation, your new file, and the live object. The math happens on your laptop.
The problem is that only kubectl knows about that annotation. If an autoscaler, a controller, or a teammate changes the same field, client-side apply can quietly overwrite it.
Server-side apply
Server-side apply moves the merge logic into the API server and tracks field ownership. Every field records which “manager” set it. If you try to change a field someone else owns, you get a conflict instead of a silent overwrite.
kubectl apply --server-side -f app.yaml
kubectl apply --server-side --field-manager=ci-pipeline -f app.yaml
kubectl apply --server-side --force-conflicts -f app.yaml # take ownership on purpose
You can see who owns what:
kubectl get deploy web -o yaml --show-managed-fields
A typical conflict looks like this. Your file sets replicas: 3, but the HorizontalPodAutoscaler owns spec.replicas. Server-side apply refuses, and you learn that replicas should not be in your manifest at all. That one lesson prevents a whole class of “why did my app scale down after deploy” incidents.
For new work, server-side apply is the better default. GitOps tools like Argo CD and Flux support it too.
Watching Instead of Polling
The Kubernetes API is built around watches. A client opens a long-running request and the server streams every change. kubectl exposes this in several useful ways:
kubectl get pods -w # stream changes
kubectl get pods -w --output-watch-events # show ADDED / MODIFIED / DELETED
kubectl rollout status deploy/web # wait for a rollout to finish
kubectl events --watch -n payments # stream events
The most underused one is kubectl wait. Stop writing sleep 30 in your scripts:
kubectl wait --for=condition=Available deploy/web --timeout=120s
kubectl wait --for=condition=Ready pod -l app=web --timeout=60s
kubectl wait --for=delete pod/old-job-xyz --timeout=60s
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/db-0
kubectl wait --for=create secret/tls-cert --timeout=90s
These commands return as soon as the condition is true, and fail with a clear exit code if it never is. Your CI pipeline becomes both faster and more honest.
Output: Getting Exactly the Data You Want
The API returns JSON. The table you normally see is a server-side “Table” view that kubectl prints. You can ask for any shape you like.
JSONPath
# Image of every container in every pod
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
# Node internal IPs
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'
# Decode a secret value
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
Custom columns
When you want a table but with your own fields:
kubectl get pods -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount'
Sorting and filtering
kubectl get pods --sort-by=.status.startTime
kubectl get events --sort-by=.lastTimestamp
kubectl get pods --field-selector=status.phase!=Running -A
kubectl get pods -l 'app in (web,api),tier!=cache'
Field selectors and label selectors are filtered by the API server, not by your shell. On a big cluster that means less data over the wire and faster results than piping to grep.
KYAML
Recent kubectl releases add a new output format called KYAML. It is still valid YAML, but it always quotes strings and uses flow-style braces, so whitespace mistakes and the classic “Norway problem” (where NO turns into false) go away.
kubectl get deploy web -o kyaml
On v1.34 this needed KUBECTL_KYAML=true. Check kubectl get --help on your version to see if it is on by default.
Debugging Like You Mean It
This is where kubectl skills pay off the most. Here is a practical order for a pod that is not working.
1. Look at events first
kubectl describe pod web-7d9f8 -n payments
kubectl events --for pod/web-7d9f8 -n payments
Image pull errors, failed scheduling, failing probes, and OOM kills all show up here. The kubectl events command is newer than get events and sorts by time by default.
2. Read the logs, including the crashed ones
kubectl logs web-7d9f8 -c app
kubectl logs web-7d9f8 --previous # logs from before the last restart
kubectl logs deploy/web --all-containers --since=15m
kubectl logs -l app=web --prefix --max-log-requests=10 -f
If a pod is in CrashLoopBackOff, --previous is the flag you want. The current container has just started and has nothing useful to say yet.
3. Get inside, even when there is no shell
Distroless and hardened images often have no shell, so kubectl exec fails. Ephemeral debug containers solve this. They attach a new container to the running pod and can share its process namespace:
kubectl debug -it web-7d9f8 --image=busybox:1.36 --target=app
kubectl debug web-7d9f8 -it --image=nicolaka/netshoot --copy-to=web-debug --share-processes
kubectl debug node/worker-2 -it --image=ubuntu # host filesystem at /host
Recent versions also support --profile (for example general, netadmin, or sysadmin) to give the debug container the right privileges for the job.
4. Reach the service from your laptop
kubectl port-forward svc/web 8080:80 -n payments
kubectl port-forward pod/db-0 5432:5432 --address 127.0.0.1
Port-forward tunnels through the API server, not directly to the node, so it works even when your laptop cannot reach the pod network.
5. Check resource use
kubectl top pods -n payments --sort-by=memory
kubectl top nodes
This needs metrics-server installed. If top says “Metrics API not available”, that is the reason.
Extending kubectl with Plugins
kubectl has a very simple plugin system. Any executable on your PATH named kubectl-something becomes kubectl something. That is the whole contract.
cat > /usr/local/bin/kubectl-podnodes <<'EOF'
#!/usr/bin/env bash
kubectl get pods "$@" -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase'
EOF
chmod +x /usr/local/bin/kubectl-podnodes
kubectl podnodes -n payments
kubectl plugin list
For ready-made plugins, use Krew, the plugin manager from Kubernetes SIG CLI. A few worth trying:
kubectl krew install ctx ns: fast context and namespace switchingkubectl krew install neat: strips noisy fields from YAML so you can reuse itkubectl krew install tree: shows owner relationships, like Deployment to ReplicaSet to Podkubectl krew install who-can: the reverse ofauth can-i; who is allowed to do this?
Your Own Defaults with kuberc
For years, people customized kubectl with shell aliases. That works, but aliases cannot change default flags, and they mix personal taste into kubeconfig sharing. kuberc fixes this.
kuberc is a separate file, ~/.kube/kuberc, just for your personal preferences. It has been beta since Kubernetes v1.34. It keeps cluster details in kubeconfig and your habits in kuberc, so you never have to share one to share the other.
apiVersion: kubectl.config.k8s.io/v1beta1
kind: Preference
defaults:
# Always use server-side apply
- command: apply
options:
- name: server-side
default: "true"
# Ask before deleting
- command: delete
options:
- name: interactive
default: "true"
aliases:
# "kubectl getn" lists namespaces as JSON
- name: getn
command: get
prependArgs:
- namespace
options:
- name: output
default: json
That delete --interactive default alone might save your weekend. You can point at a different file with --kuberc or the KUBERC environment variable. In v1.36, kubectl also has a kubectl kuberc command with view and set subcommands, so you can manage the file without editing YAML by hand.
Small Habits That Add Up
A few things that make daily work smoother:
- Turn on completion.
source <(kubectl completion bash)or the zsh/fish versions. It completes resource names from your live cluster. - Generate YAML instead of writing it.
kubectl create deploy web --image=nginx --dry-run=client -o yaml > web.yamlgives you a clean starting point. - Show your context in your prompt. Seeing “prod” in red before every command is cheap insurance.
- Use
-o namein scripts. It printspod/web-7d9f8, which you can pipe straight into other kubectl commands. - Keep kubectl close to your cluster version. kubectl supports one minor version older or newer than the API server. Outside that window, odd things happen.
Quick Reference
| I want to… | Command |
|---|---|
| See the raw HTTP calls | kubectl get pods -v=8 |
| Check who I am | kubectl auth whoami |
| Check a permission | kubectl auth can-i delete pods -n prod |
| Test a change against policies | kubectl apply -f x.yaml --dry-run=server |
| Preview a change | kubectl diff -f x.yaml |
| Avoid overwriting other tools | kubectl apply --server-side -f x.yaml |
| Wait for something in a script | kubectl wait --for=condition=Ready pod -l app=web |
| Read logs from a crashed container | kubectl logs pod/x --previous |
| Debug a pod with no shell | kubectl debug -it pod/x --image=busybox --target=app |
| Learn a field | kubectl explain deploy.spec.strategy |
Wrapping Up
kubectl looks like a simple CLI, but it is really a well-designed API client. It reads your kubeconfig, gets a token, discovers what the cluster supports, sends a REST call, and formats the answer. Each of those steps has tools you can use: -v=8 to see the traffic, auth can-i to check access, explain to read the schema, server-side apply to avoid fights over fields, wait to replace sleeps, and debug to get into any pod.
Pick two or three of these and use them this week. The next time something breaks, you will know exactly where to look.
What is your favorite kubectl trick? Share it with the community on the Collabnix Slack or Discord.
