Kubernetes Security

Overview

Kubernetes security rests on three independent mechanisms that each surprise people used to a simpler model: RBAC is default-deny and purely additive, permissions can only be granted, never explicitly denied, so restricting access means removing a binding, not layering a deny rule on top. Pod Security Standards define three cumulative levels, Privileged, Baseline, Restricted, each adding real hardening (non-root, no privilege escalation, dropped capabilities) on top of the last. NetworkPolicy inverts the surprise the other way: pods are wide open by default with no policy at all, and only become isolated the moment any policy selects them, which is why the very first NetworkPolicy applied to a namespace is the one most likely to unexpectedly break something.

I start with Kubernetes Fundamentals if Deployment ownership, Pod readiness, Service selectors, or EndpointSlices are unfamiliar. Security troubleshooting depends on proving the normal workload and network path first; otherwise an empty Service backend can be mistaken for a NetworkPolicy denial, or an admission rejection for an application crash.

Quick Reference

Syntax

# RBAC - grants are additive; to restrict access, remove the
# binding, there's no "deny" rule to add on top of it.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
  - kind: ServiceAccount
    name: ci-deploy
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Examples

# NetworkPolicy - the moment this selects role=db pods, they
# become isolated for ingress; only traffic matching this rule
# is allowed in, everything else that used to reach them is cut off.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-allow-api-only
spec:
  podSelector:
    matchLabels: { role: db }
  policyTypes: ['Ingress']
  ingress:
    - from:
        - podSelector:
            matchLabels: { role: api }

Verify authorization before changing RBAC

kubectl auth can-i get pods -n production
kubectl auth can-i get pods --subresource=log -n production

# Administrators can test a ServiceAccount, but impersonation itself
# requires permission.
kubectl auth can-i list secrets \
  --as=system:serviceaccount:dev:api \
  -n production

Evaluate Pod Security admission without enforcing it

kubectl label --dry-run=server --overwrite namespace production \
  pod-security.kubernetes.io/enforce=restricted

The server-side dry run evaluates the proposed policy and returns violations without changing the namespace. I use audit and warn modes to discover incompatible workloads before enforcement, then pin the policy version your cluster has tested rather than silently inheriting new rules.

Troubleshooting Security Controls

kubectl get endpointslices \
  -l kubernetes.io/service-name=<service> \
  -n <destination-namespace>
kubectl get networkpolicy -n <source-namespace>
kubectl get networkpolicy -n <destination-namespace>
kubectl run -it --rm policy-test \
  -n <source-namespace> \
  --restart=Never \
  --image=busybox:1.36 -- sh

Visual Diagram

Common Mistakes

• Trying to write an RBAC "deny" rule; RBAC has no such concept, restricting access means removing the grant, not layering a denial on top of it.
• Applying the Restricted Pod Security Standard cluster-wide without testing, breaking workloads that legitimately needed a capability Baseline still allowed.
• Assuming pods are isolated by default and skipping NetworkPolicy entirely, when the actual default is fully open pod-to-pod traffic.
• Applying a single NetworkPolicy to a namespace and being surprised when unrelated traffic breaks, not realizing that policy selected pods well beyond its intended scope.
• Checking only one RoleBinding and missing a broader ClusterRoleBinding that grants the same subject more access.
• Testing policy from an arbitrary debug Pod whose namespace and labels do not match the real caller, then drawing the wrong conclusion from the result.

Performance

• RBAC authorization checks happen on the API request path but are lightweight lookups against already-loaded bindings; the real cost is the audit and design effort of keeping bindings correct, not runtime latency.
• NetworkPolicy enforcement is implemented by the cluster's CNI plugin, not the API server itself, and its performance characteristics (rule-matching overhead per packet) vary by CNI implementation and policy count.

Best Practices

• Scope RBAC bindings as narrowly as possible (a Role in one namespace) and reserve ClusterRole/ClusterRoleBinding for genuinely cluster-wide needs.
• I use kubectl auth can-i in deployment checks and runbooks to verify effective permissions rather than inferring them from one binding.
• I avoid wildcard permissions and routine cluster-admin use; new API resources can make an old wildcard grant more powerful over time.
• Apply the Restricted Pod Security Standard to namespaces running security-critical or lower-trust workloads, and Baseline as the sane default everywhere else.
• Introduce Pod Security admission through audit/warn evaluation and server-side dry runs before switching a namespace to enforce.
• Introduce NetworkPolicy deliberately and incrementally, starting with a namespace's own traffic map, rather than applying a broad policy and reacting to what breaks.
• Default every namespace to at least a basic deny-by-default NetworkPolicy plus explicit allows, rather than leaving pod-to-pod traffic fully open indefinitely.