KEDA at 5M Users: Why We Built a Custom Session-Aware Eviction Controller

Our Kubernetes autoscaler was evicting pods mid-game. Players were getting disconnected at the worst possible moment: tournament finals, right after a winning move. We needed a smarter scaler. What we built changed how I think about infrastructure decisions entirely.

The platform served 5 million concurrent users at peak. Every night at 8pm IST, when the Indian tournament schedule kicked in, concurrent users would spike — sometimes doubling in under 60 seconds. Standard Kubernetes HPA was trying to help. It was, in fact, making things worse.

The Problem: HPA Doesn't Know What a Game Session Is

Here's the thing about gaming workloads that makes them different from typical web APIs: a pod serving 300 active game sessions uses almost identical CPU to an idle pod. After the initial WebSocket handshake, game state is just waiting. Tiny heartbeats, occasional moves. CPU stays flat. HPA sees "low CPU" and thinks "scale down." Then it evicts a pod with 300 live sessions on it.

Three hundred players, simultaneously disconnected, mid-game. That's not a bug. That's a feature of an autoscaler that has no idea what it's actually running.

• Spike patterns at 8pm IST: concurrent users hit 5M, HPA reaction time was 3-5 minutes.
• During off-peak: CPU drops, HPA scales down, real sessions evicted.
• After tournaments: traffic cliff, HPA aggressively scales down — evicting whoever's still logged in.

The Solution: KEDA with Session-Aware Metrics

KEDA (Kubernetes Event Driven Autoscaler) lets you scale on whatever metrics actually matter to your workload. For gaming, that meant active session counts, not CPU.

We exposed a custom Prometheus metric from each session service pod: active_sessions_total. Then we told KEDA: keep one replica per 250 active sessions. Sessions drain off-peak, replicas go down. Sessions spike, replicas go up. The autoscaler finally knew what it was scaling.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: session-service
  namespace: platform
spec:
  scaleTargetRef:
    name: session-service
  minReplicaCount: 3
  maxReplicaCount: 80
  pollingInterval: 15
  cooldownPeriod: 180   # don't be hasty about scaling down
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://thanos-query:9090
        metricName: active_sessions_total
        threshold: "250"
        query: sum(active_sessions_total{namespace="platform"})

For event-driven services (game result processing, leaderboard updates), we used KEDA's Kafka scaler — scale based on consumer group lag, not CPU. When a tournament ended and 100K results hit the queue simultaneously, processing pods scaled to clear the backlog and scaled back down when the queue emptied.

The Python Controller: Teaching Kubernetes Which Pod to Kill

KEDA decides how many pods to run. Kubernetes decides which pods to evict when scaling down. And Kubernetes, left to its own devices, just picks one at random.

We wrote a Python controller that intercepts scale-down decisions and picks the eviction candidate with the fewest active sessions, not just any pod.

def score_pod(pod_ip: str, port: int = 8080) -> float:
    """Lower score = safer to evict."""
    try:
        r = requests.get(f"http://{pod_ip}:{port}/metrics", timeout=2)
        sessions = parse_metric(r.text, "active_sessions_total")
        pending  = parse_metric(r.text, "pending_requests_total")
        return sessions * 1.5 + pending * 0.5
    except Exception:
        return 0.0  # unreachable pod = safe to evict

def evict_least_loaded(pods: list) -> str:
    scored = [(pod.metadata.name, score_pod(pod.status.pod_ip))
              for pod in pods]
    scored.sort(key=lambda x: x[1])
    return scored[0][0]   # minimum score = fewest live sessions

Once the candidate is selected, the controller patches it with drain.platform/scheduled: "true", which the app reads during its SIGTERM handler. The app stops accepting new sessions, waits for existing ones to finish (up to the configured drain timeout), then terminates cleanly. Players finish their games. Nobody gets disconnected.

Shadow Mode: Rolling It Out Without the Risk

We didn't trust ourselves enough to deploy this directly to production. We ran the controller in shadow mode for two weeks: it logged which pod it would have evicted and compared that to what Kubernetes actually evicted. The delta was enormous. Kubernetes had been regularly choosing pods with 200+ active sessions when empty pods were sitting right there.

After shadow mode validated the scoring logic, we enabled the controller for one service, watched it for a week, then rolled it across the fleet. No incidents. No disconnections. The on-call rotation got noticeably quieter.

The Results

The Catch

KEDA is genuinely powerful and the ecosystem is mature. But a few things that bit us:

• cooldownPeriod is critical. Too short and you get scale-up/scale-down thrashing during irregular traffic. We settled on 180 seconds for session services, 120 for stateless workers.
• minReplicaCount = 0 is aggressive. Cold start latency is real. Only set it to zero for batch workloads where latency doesn't matter.
• Drain timeout must match your session length. Our average session was 15 minutes. A 2-minute drain timeout would still have killed sessions. We set it to 20 minutes and accepted the slower scale-down.
• Test your scaler logic with chaos. Deliberately trigger scale-down events in staging and verify no active sessions drop before you touch production.

The deeper lesson: cost optimization in stateful infrastructure isn't about scaling down faster. It's about scaling down correctly. Generic CPU metrics will never tell you when it's safe to evict a pod. Only your business metrics will. Build session awareness into your controllers and users will never know a scale-down happened.