Kubernetes: Zero to Production

Deploying to production is not just kubectl set image and hoping for the best. Production deployments must be safe, reversible, and observable. The strategy you choose determines your blast radius when things go wrong, your deployment speed, and the experience your users have during releases. This chapter covers every major deployment strategy used by top engineering teams worldwide.

Deployment Strategies Comparison

Rolling Updates: Tuning for Zero-Downtime

# Rolling update is the Kubernetes default — but the defaults are often wrong
# Default: maxUnavailable=25%, maxSurge=25% — this means up to 25% downtime!

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0      # NEVER have fewer than 10 running pods (zero downtime!)
      maxSurge: 3            # Allow 3 extra pods during the rollout (13 max total)
      # With these settings:
      # 1. 3 new pods start (total: 10 old + 3 new = 13)
      # 2. Wait until new pods pass readiness probes
      # 3. Terminate 3 old pods (total: 7 old + 3 new = 10)
      # 4. Repeat until all 10 are new version
  minReadySeconds: 30          # New pod must be Ready for 30s before counting as available
                               # Prevents "too fast" rollout hiding crashes
  progressDeadlineSeconds: 600 # Fail if rollout doesn't complete in 10 minutes
  selector:
    matchLabels: { app: api-server }
  template:
    metadata:
      labels: { app: api-server }
    spec:
      containers:
      - name: api
        image: mycompany/api:v2.0
        readinessProbe:              # CRITICAL: Rolling update waits for this before moving on
          httpGet:
            path: /ready
            port: 3000
          initialDelaySeconds: 15    # Wait 15s after container starts
          periodSeconds: 5           # Check every 5 seconds
          successThreshold: 2        # Must be healthy twice in a row before counted as Ready
          failureThreshold: 3        # 3 failures = mark as not ready, pause rollout
        livenessProbe:               # Restarts the container if unhealthy
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
              # Wait 15s before stopping — allows load balancer to drain connections
              # Prevents 502 errors during pod termination
      terminationGracePeriodSeconds: 60   # Give the app 60s to finish in-flight requests

# Monitor the rollout
kubectl rollout status deployment/api-server -n production -w
# Waiting for deployment "api-server" rollout to finish: 3 out of 10 new replicas have been updated...
# Waiting for deployment "api-server" rollout to finish: 6 out of 10 new replicas have been updated...
# Waiting for deployment "api-server" rollout to finish: 9 out of 10 new replicas have been updated...
# deployment "api-server" successfully rolled out

# Immediate rollback if issues detected
kubectl rollout undo deployment/api-server -n production
# Rolls back to previous revision instantly

# Rollback to specific revision
kubectl rollout history deployment/api-server -n production
kubectl rollout undo deployment/api-server --to-revision=3 -n production

Blue/Green Deployments

# Blue/Green: Run two complete environments simultaneously
# "Blue" = current production version
# "Green" = new version (fully deployed but not receiving traffic)
# Switch traffic: update the Service selector from blue to green (instant!)
# Rollback: update selector back to blue (instant!)

# =============================================
# 1. Blue Deployment (current production)
# =============================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-blue
  namespace: production
  labels:
    app: api
    version: blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: api
      version: blue
  template:
    metadata:
      labels:
        app: api
        version: blue
    spec:
      containers:
      - name: api
        image: mycompany/api:v1.5.0   # Current production version
        ports:
        - containerPort: 3000

---
# =============================================
# 2. Service — currently pointing to BLUE
# =============================================
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: production
spec:
  selector:
    app: api
    version: blue              # ← This is the key! Points to blue pods
  ports:
  - port: 80
    targetPort: 3000

---
# =============================================
# 3. Green Deployment (new version — not receiving traffic yet)
# =============================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-green
  namespace: production
  labels:
    app: api
    version: green
spec:
  replicas: 5
  selector:
    matchLabels:
      app: api
      version: green
  template:
    metadata:
      labels:
        app: api
        version: green
    spec:
      containers:
      - name: api
        image: mycompany/api:v2.0.0   # New version — fully deployed, not live yet
        ports:
        - containerPort: 3000

# =============================================
# 4. Test green deployment (no real user traffic)
# =============================================
kubectl port-forward deployment/api-green 8080:3000 -n production
curl http://localhost:8080/health    # Test new version directly

# Run integration tests against green
kubectl run test --image=mycompany/integration-tests --rm -it \
  --env="API_URL=http://api-green.production.svc.cluster.local" \
  -- pytest tests/integration/

# =============================================
# 5. SWITCH TO GREEN — update Service selector
# This is an atomic operation — instant traffic switch!
# =============================================
kubectl patch service api-service -n production \
  -p '{"spec": {"selector": {"app": "api", "version": "green"}}}'

# All traffic immediately flows to green pods
# Zero downtime — the Service IP doesn't change

# =============================================
# 6. Rollback — just patch selector back to blue
# =============================================
kubectl patch service api-service -n production \
  -p '{"spec": {"selector": {"app": "api", "version": "blue"}}}'
# Instant rollback! Users never notice.

# =============================================
# 7. After validation — clean up blue
# =============================================
kubectl delete deployment api-blue -n production

PodDisruptionBudget: Protecting Availability During Node Maintenance

# PodDisruptionBudget (PDB) tells Kubernetes how many pods from a Deployment
# can be voluntarily disrupted at once (node drain, cluster upgrade, etc.)
# Without a PDB, a node drain can terminate ALL pods of a service simultaneously!

# =============================================
# PDB: At least 80% of pods must be available at all times
# =============================================
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: production
spec:
  minAvailable: "80%"    # With 10 replicas: at least 8 must always be running
  # OR:
  # maxUnavailable: 2    # At most 2 can be unavailable at once
  selector:
    matchLabels:
      app: api-server

# =============================================
# Real-world scenario: EKS node group upgrade
# =============================================
# Without PDB: kubectl drain node drains ALL pods on a node instantly
# If all pods of a service land on the same node = complete outage

# With PDB: kubectl drain respects PDB
# It will only evict pods if minAvailable is maintained
# If it would violate the PDB, the drain pauses and waits

# Drain a node (for maintenance/upgrade)
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
# evicting pod production/api-server-xxxxx
# evicting pod production/api-server-yyyyy
# error when evicting pods/api-server-zzzzz: Cannot evict pod as it would violate PDB
# ← Kubernetes waits for another pod to start on a different node first

# =============================================
# HPA: Horizontal Pod Autoscaler
# Scale based on CPU, memory, or custom metrics
# =============================================
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 5           # Never scale below 5 (for reliability)
  maxReplicas: 50          # Never scale above 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60    # Scale up when avg CPU > 60%, down when < 60%
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods                    # Custom metric: pending jobs in queue
    pods:
      metric:
        name: jobs_pending_per_pod
      target:
        type: AverageValue
        averageValue: "30"        # Scale when > 30 pending jobs per pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60    # Wait 60s before scaling up again
      policies:
      - type: Pods
        value: 5                        # Add at most 5 pods per scale event
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # Wait 5 minutes before scaling down
      policies:
      - type: Percent
        value: 10                       # Remove at most 10% of pods per scale event
        periodSeconds: 60

# =============================================
# 1. PodDisruptionBudget — never < 90% pods available
# =============================================
kubectl apply -f - <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api-pdb
  namespace: production
spec:
  minAvailable: "90%"
  selector:
    matchLabels:
      app: payment-api
EOF

# =============================================
# 2. HPA with custom metrics from Prometheus
# =============================================
kubectl apply -f - <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  minReplicas: 10
  maxReplicas: 100
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50      # Aggressive — scale earlier for payment service
  - type: External
    external:
      metric:
        name: http_requests_per_second
        selector:
          matchLabels:
            service: payment-api
      target:
        type: AverageValue
        averageValue: "500"         # Scale when > 500 req/s per pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30    # Respond faster to traffic spikes
      policies:
      - type: Percent
        value: 100                      # Double pods if needed
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 600   # 10 min cooldown — don't thrash
EOF

# =============================================
# 3. Canary deployment with automated analysis
# Using Flagger for progressive delivery
# =============================================
helm repo add flagger https://flagger.app
helm install flagger flagger/flagger \
  --namespace flagger-system \
  --create-namespace \
  --set prometheus.install=true \
  --set meshProvider=kubernetes

kubectl apply -f - <<'EOF'
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: payment-api
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  progressDeadlineSeconds: 600
  service:
    port: 80
    targetPort: 3000
  analysis:
    interval: 1m              # Analyze metrics every 1 minute
    threshold: 5              # Auto-rollback after 5 consecutive failures
    maxWeight: 50             # Maximum 50% canary traffic
    stepWeight: 10            # Increase canary by 10% per interval
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99               # Must have > 99% success rate
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500              # P99 latency must be < 500ms
      interval: 1m
    webhooks:
    - name: load-test
      url: http://flagger-loadtester.test/
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://payment-api-canary.production/"
EOF

# How Flagger works:
# 1. You update the Deployment image tag
# 2. Flagger detects the change
# 3. Creates payment-api-canary deployment with new version
# 4. Sends 10% traffic to canary
# 5. Checks Prometheus metrics every 1 minute
# 6. If success-rate > 99% and latency < 500ms: increase to 20%, then 30%...
# 7. If any check fails: AUTO-ROLLBACK → 0% to canary, alert sent

# Monitor canary progress
kubectl get canary payment-api -n production -w
# NAME          STATUS        WEIGHT   LASTTRANSITIONTIME
# payment-api   Progressing   10       2024-01-15T10:00:00Z
# payment-api   Progressing   20       2024-01-15T10:01:00Z
# payment-api   Progressing   30       2024-01-15T10:02:00Z
# payment-api   Succeeded     0        2024-01-15T10:07:00Z  ← Full rollout!

# =============================================
# 4. On-Call Runbook (in the cluster as ConfigMap)
# =============================================
kubectl create configmap payment-api-runbook -n production \
  --from-literal=runbook.md="
# Payment API On-Call Runbook

## Pod CrashLoopBackOff
1. kubectl logs pod-name --previous
2. kubectl describe pod pod-name
3. Check OOM: kubectl top pods -n production
4. Rollback if recent deploy: kubectl rollout undo deploy/payment-api

## High Latency Alert (P99 > 500ms)
1. Check HPA: kubectl get hpa -n production
2. Check DB connections: kubectl exec -n production deploy/payment-api -- env | grep DB
3. Check upstream: kubectl exec -n production deploy/payment-api -- curl -s payment-gateway/health
4. If DB issue: kubectl get pods -n data

## Scaling Emergency (traffic spike)
1. kubectl scale deployment payment-api --replicas=50 -n production
2. Verify nodes available: kubectl get nodes | grep Ready
3. If nodes insufficient: (Karpenter auto-provisions — wait 90s)
"

Chapters 11–15 Covered

You've mastered Security (RBAC, PSA, OPA Gatekeeper), Monitoring (Prometheus, Grafana, SLOs), GitOps with ArgoCD, AWS EKS with IRSA and Karpenter, and Production Deployment Strategies including canary analysis with Flagger. You are now equipped to operate production Kubernetes clusters at scale.