How to Scale WebSocket Infrastructure for Realtime Games

If your WebSocket layer crumples when a featured streamer brings 30,000 players into one lobby, you don’t just drop packets — you drop revenue and community trust. Realtime games don’t get retries; they get uninstalls.

You can scale WebSockets for realtime games by sharding by room/instance, using sticky load balancing, putting a fast pub/sub bus (Redis or NATS) behind gateways, adopting a compact binary protocol, enforcing backpressure and rate limits, and planning for draining, failover, and thundering-herd reconnects. This article lays out the architecture, the production pitfalls, and the actual ops moves we implement as a studio.

The constraints that actually matter for WebSocket game traffic

Realtime game traffic isn’t generic chat. The constraints push your architecture:

• Latency budget: 50–150 ms end-to-end is what most casual and midcore web games tolerate; competitive twitch games want <50 ms regional RTT. That budget includes client frames, JS GC, TLS, network, and server processing. Every extra hop or queue eats budget.
• Message patterns: small and frequent (position deltas, inputs), occasional bursts (round start, inventory sync), and rare large (map or replay chunks). Fanout to N players in a room amplifies CPU and egress.
• Delivery semantics: input streams and state deltas are often best-effort; critical events (match start, rewards) are reliable and ordered. Over WebSockets you’ll implement reliability at the app layer.
• Connection characteristics: long-lived TLS connections, NAT timeouts on mobile, intermittent connectivity. Your infra must handle churn and resumption without storming the backend.
• Server state: rooms are inherently stateful. Pretending your WebSocket layer is stateless quickly collapses into chaos.

A small dry-wit aside: if you ignore backpressure, players will discover a new speedrun category — “from login to alt-F4” — powered entirely by your GC pauses.

A baseline architecture that scales past the first spike

Here’s a pragmatic blueprint we deploy for browser-based and cross-platform games using WebSockets.

• Edge/load balancer: Cloudflare/Cloudfront + regional L4/7 (Envoy/HAProxy/Nginx) with sticky sessions based on a roomId or matchId hash.
• Gateway pool (stateful WebSocket nodes): Handles connections, authentication, rate limits, binary serialization, fanout to room members, and minimal game logic (ideally zero). We like uWebSockets.js or Go for efficiency.
• Matchmaker/service API: Places players into rooms, chooses region + gateway partition, returns a signed token with target node hint.
• Pub/sub bus: Redis (single-region, low-latency fanout) or NATS (multi-region, higher throughput) to propagate events across gateways or to authoritative game workers.
• Authoritative game workers: Run deterministic game simulation (tick-based), generate state deltas; colocated with gateway shards for low latency.
• State store: Redis for presence, room membership, transient state; Postgres for persistence (leaderboards, inventory). Kafka for analytics/telemetry, not for hot-path gameplay.
• Observability: Prometheus + Grafana, OpenTelemetry traces from gateway to workers, log sampling to avoid I/O stalls. Synthetic pings from clients.

Data flow (simplified):

1. Client hits edge, LB hashes roomId to a gateway partition and maintains stickiness.
2. Gateway authenticates JWT, registers presence in Redis, subscribes to room:<id>.
3. Client sends input events; gateway enforces rate/size, forwards to worker via NATS/Redis.
4. Worker runs tick, publishes state deltas to room:<id>.
5. Gateway fanouts binary deltas to connected players; slow consumers are buffered within limits then dropped.

Minimal gateway skeleton with backpressure (TypeScript)

// uWebSockets.js example with Redis fanout
import uWS from 'uWebSockets.js';
import { createClient } from 'redis';
import { encodeDelta, decodeClientMsg } from './codec';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

// roomId -> Set of sockets
const rooms = new Map<string, Set<uWS.WebSocket>>();

function joinRoom(ws: uWS.WebSocket, roomId: string) {
  if (!rooms.has(roomId)) rooms.set(roomId, new Set());
  rooms.get(roomId)!.add(ws);
  ws.getUserData().roomId = roomId;
}

// Subscribe to room channels and fanout efficiently
const sub = redis.duplicate();
await sub.connect();
await sub.pSubscribe('room:*', (channel, payload) => {
  const roomId = channel.split(':')[1];
  const set = rooms.get(roomId);
  if (!set) return;
  const buf = Buffer.from(payload, 'binary');
  for (const sock of set) {
    // Backpressure check; skip slow consumers
    if (sock.getBufferedAmount() > 256 * 1024) {
      sock.end(1011, 'Backpressure');
      continue;
    }
    sock.send(buf, true, true);
  }
});

uWS.App()
  .ws('/*', {
    idleTimeout: 60,
    maxPayloadLength: 64 * 1024,
    compression: uWS.SHARED_COMPRESSOR,
    upgrade: (res, req, ctx) => {
      // Verify JWT, extract roomId & userId
      const token = req.getHeader('sec-websocket-protocol');
      const { roomId, userId } = verify(token);
      res.upgrade({ userId, roomId },
        req.getHeader('sec-websocket-key'),
        req.getHeader('sec-websocket-protocol'),
        req.getHeader('sec-websocket-extensions'),
        ctx
      );
    },
    open: (ws) => {
      joinRoom(ws, ws.getUserData().roomId);
      // Optionally notify presence
      redis.sAdd(`presence:${ws.getUserData().roomId}`, ws.getUserData().userId);
    },
    message: (ws, message, isBinary) => {
      // Decode input, rate limit
      const data = Buffer.from(message);
      if (data.length > 2048) return; // hard cap
      if (!isBinary) return;
      const evt = decodeClientMsg(data);
      // Queue to worker via Redis Stream or NATS
      redis.xAdd(`inputs:${ws.getUserData().roomId}`, '*', { u: ws.getUserData().userId, p: data });
    },
    close: (ws) => {
      const roomId = ws.getUserData().roomId;
      rooms.get(roomId)?.delete(ws);
      if (rooms.get(roomId)?.size === 0) rooms.delete(roomId);
      redis.sRem(`presence:${roomId}`, ws.getUserData().userId);
    }
  })
  .listen(9001, (token) => {
    if (!token) throw new Error('Failed to bind');
    console.log('Gateway on :9001');
  });

function verify(token?: string): { roomId: string; userId: string } {
  // Stub. In production, verify HMAC/JWT, include region partition hint.
  return { roomId: 'r-123', userId: 'u-xyz' };
}

This skeleton demonstrates: per-room fanout, binary payloads, backpressure-based drops, and separation of input ingestion vs. state delta distribution.

Sharding, stickiness, and fanout math

• Shard by room: The only scalable unit for realtime games is the room/match/instance. Assign rooms to gateway partitions via consistent hashing. Keep all members of a room on the same node for fanout locality.
• Sticky sessions: The load balancer must route all connections of a given room to the same node. Use a hash policy (Envoy: hash_policy, Nginx: hash $room_id with map from JWT to header). IP stickiness is not sufficient.
• Avoid cross-node fanout: Publishing state to N nodes and N×M sockets will multiply CPU and egress. If a room must exceed a single node, you’re in “mega-room” territory; consider tree fanout (gateway -> fanout layer -> leaves) or dedicated fanout services written in Go/Rust.
• Memory budgeting: Rough numbers in our experience as builders:
  • Node ws: 30–80 KB per connection (headers, buffers, JS objects) + your app state.
  • uWebSockets.js: ~10–40 KB per connection.
  • TLS contexts and compression inflate memory per connection.
  • 50k CCU on a single gateway with uWS is doable on a 16–32 GB instance if messages are small and you manage backpressure. Always benchmark with production-like payloads.
• Fanout CPU cost: fanout_cost ≈ room_size × serialization_cost. Serialize once per tick per room, reuse buffers when possible; avoid per-recipient JSON.stringify. Prefer FlatBuffers/Protobuf or custom binary. Compress only if payloads are large (>500 B) and CPU allows.

Choosing a pub/sub backbone

You need a bus to relay inputs to workers and state deltas to gateways when they’re not co-located. Here’s a succinct comparison:

For most realtime rooms under a few thousand players, Redis Pub/Sub or NATS Core hits the sweet spot. Use Streams or JetStream when you require replayable inputs for audit or anti-cheat — but understand the flow control implications.

Protocol: bytes over JSON, deltas over snapshots

• Binary over text: JSON costs you bandwidth and GC time. Use ArrayBuffer/DataView, FlatBuffers, or Protobuf. Keep payloads <1 KB where possible.
• Delta-then-snapshot: Send small per-tick deltas; periodically send a snapshot (e.g., every 2–5 seconds) so late joiners rebase quickly. Server keeps last snapshot to rehydrate reconnects.
• Compression: Enable permessage-deflate or SHARED_COMPRESSOR sparingly. It helps for big payloads; for <200 B deltas, CPU cost dominates savings.
• Ordering: Sequence IDs per room stream; drop out-of-order deltas on client; ack only for critical events.
• Rate caps: Hard cap messages/sec per client and bytes/sec per client to protect rooms from a single bad actor.

Alternatives and fallbacks: WebTransport, SSE, and HTTP/3

• WebSocket remains the most broadly available bidirectional choice in browsers and native. WebTransport (HTTP/3, QUIC) unlocks unreliable datagrams and better head-of-line behavior but has partial ecosystem support.
• SSE is viable for one-way fanout (spectators, live telemetry) without per-connection WS overhead. If you need a deep dive on resilient SSE, see our guide to resumable SSE streams.
• Native apps can mix: WS for control/state, UDP/QUIC for inputs. For pure web games, keep it WS for simplicity.

Operations playbook: stickiness, autoscaling, and zero-downtime drains

• Load balancer

  • Terminate TLS at edge; try to keep regional L7 as a TCP/WS pass-through to avoid double terminates.
  • Use a hash policy on roomId or a stable partitionId. Validate the hash on the server to prevent targeted load skews.
  • Enable PROXY protocol if you need client IP at the gateway.

• Instance sizing and bin-packing

  • Prefer larger nodes up to the point where a single node failure impacts too many rooms. A blend of medium/large instances often balances efficiency and blast radius.
  • Pin CPU affinity where applicable. Avoid noisy neighbors.

• Autoscaling

  • Scale on connection count and egress, not just CPU. CPU often looks fine until GC spikes under backpressure.
  • Warm standby nodes: keep a few pre-booted nodes to absorb spikes.

• Draining for deploys

  • Mark a node as draining: new rooms not assigned; existing rooms continue until match end. For persistent lobbies, instruct clients to reconnect during a maintenance window or implement mid-room migration (hard).
  • Blue/green deployments with connection draining windows prevent mass reconnect storms.

• Reconnect storms

  • Use randomized backoff jitter (50–2500 ms) on clients.
  • Gatekeeper responses: during spikes return a quick 503 with Retry-After and a backoff hint.
  • Token-based admission: only allow reconnects with recent room tokens; queue others.

• Health and SLOs

  • Export: connections per node, p50/p95/p99 end-to-end E2E latency, message queue depth, dropped messages, backpressure closes, GC durations, egress Mbps.
  • Alert on rising getBufferedAmount, increasing reconnects, and rising tail latencies.

Graceful connection draining example (Go)

// Sketch of a drain-aware server in Go
var draining atomic.Bool

func drainHandler(w http.ResponseWriter, r *http.Request) {
  if r.Method == http.MethodPost {
    draining.Store(true)
    // stop taking new rooms; existing sockets continue
    w.WriteHeader(200)
    w.Write([]byte("draining"))
  }
}

func upgradeWS(w http.ResponseWriter, r *http.Request) {
  roomID := r.URL.Query().Get("room")
  if draining.Load() && !roomExists(roomID) { // allow only existing rooms
    http.Error(w, "server draining", http.StatusServiceUnavailable)
    return
  }
  // ... proceed with upgrade
}

What breaks in production

• Slow consumers: One lagging client can balloon buffered bytes and hurt the whole process. Always enforce per-socket backpressure caps and drop when exceeded.
• GC pauses and event loop stalls: In Node, allocating big Buffers per send or using JSON can induce stalls. Reuse buffers, pool objects, and move heavy work off the main thread.
• Presence storms: When a shard dies and thousands reconnect, your Redis presence keys thrash. Use batched writes and TTLs; design presence reads to be eventually consistent.
• Thundering herd at match start: Broadcasting full snapshots to 5,000 spectators plus 100 players can crush a node. Precompute and cache snapshot buffers; stage fanout in micro-batches (e.g., 1 ms gaps) to avoid bursts.
• Kernel limits: ulimit -n, TCP ephemeral ports, somaxconn, and keepalive defaults will cap your max connections. Tune sysctls and test.
• Mobile networks: NAT rebinding and idle timeouts close connections unexpectedly. Send lightweight heartbeats (e.g., 20–30 s) and support rapid resumption with room snapshots.
• Load balancer stickiness drift: Misconfigured hashing or rotation breaks room locality. Log and verify the shard key end-to-end.
• Cross-region chat: Latency balloons when rooms span regions. Keep authoritative game + gateway in the same region; provide region selection in the client.

Cost and ROI: what to expect and where money actually goes

This is a realistic envelope to inform budgets; exact numbers depend on your stack and player behavior.

• Compute

  • Gateway nodes: With uWebSockets.js or Go, expect 10–50k CCU per node depending on message rates. A pool of 10 nodes can support 100–200k CCU with sane deltas. Instance costs: $150–$600/month per node for beefy VMs.
  • Authoritative workers: More CPU-bound; 1–4k active players per worker at 10–20 Hz tick, heavily dependent on game complexity.

• Memory

  • Plan 20–80 KB/connection at the gateway excluding app state. 100k CCU can be 2–8 GB just in connection overhead. Factor snapshots/cache separately.

• Network egress (the silent bill)

  • Example: 2 KB/s per client average (small deltas, some chat). At 100k CCU, that’s 200 MB/s = 17 TB/day. At $0.05–$0.12/GB egress, you’re looking at roughly $850–$2,000/day. Optimizing payload size yields meaningful savings.

• Pub/sub

  • Managed Redis: $500–$3,000/month for high-throughput plans with replicas.
  • NATS: Self-managed 3–5 node cluster on mid-size VMs is common; $300–$1,500/month infra. Managed options exist at higher price points.

• Ops and engineering

  • Tooling (observability, error tracking): $300–$2,000/month depending on volume and retention.
  • Team time to build correctness, migrations, and hardening pays for itself vs. outage costs.

Where teams overspend:

• Over-compressing tiny packets (CPU > bandwidth savings).
• Using Kafka on the hot path (latency overhead + complexity) when Redis/NATS suffice.
• Scaling by connection count only; the true limiter is fanout × serialization.
• Running everything managed + cross-region without calculating egress and latency penalties.

Where teams under-invest:

• Draining and reconnect control paths — the first spike exposes this immediately.
• Binary serialization and delta design — the cheapest bandwidth is the byte you never send.
• Load testing that mirrors real player patterns (bursts, spectating, background tabs).

If you’re exploring serverless to avoid ops, note that cold starts and connection limits make most serverless platforms a poor fit for stateful WebSockets at scale. For patterns that can work in serverless environments, see our notes on serverless at extreme scale — but treat it as an exception, not the default, for realtime gaming.

Putting it together: a deployment plan you can execute

1. Choose regions close to your players. Start with 2–3 regions, each a full stack (LB, gateways, workers, Redis, Postgres). Avoid cross-region hot paths.
2. Implement gateway stickiness using a signed partitionId in the JWT. LB hashes this value. Verify on the server to prevent abuse.
3. Adopt binary protocol early. Define message schemas, deltas, and snapshots. Measure average payload sizes.
4. Pick a pub/sub: Redis Pub/Sub for low-latency fanout; NATS if you need service mesh patterns or multi-region topo. Keep Kafka out of the gameplay loop.
5. Backpressure and rate limits from day one: bytes per second and messages per second per socket. Decide drop policies.
6. Observability with real SLOs: track p95/p99, backpressure closes, and snapshot/delta ratios.
7. Load-test with production-like clients: realistic tick rates, join/leave bursts, packet loss simulation, background tabs throttled by browsers.
8. Ops: tune kernel, implement draining endpoints, blue/green deployments, and a reconnect control plane.

FAQ

• How many players can a single WebSocket node handle?

  • With efficient frameworks (uWebSockets.js, Go) and small binary deltas, 10–50k concurrent connections per node is a practical range. The real limiter is fanout serialization and egress, not just connection count.

• Should we use Redis or NATS for pub/sub?

  • Redis Pub/Sub is simple and excellent for single-region, low-latency fanout. NATS adds better multi-region topologies and request/reply semantics. If you need durable replay, use Redis Streams or NATS JetStream — but understand their flow control.

• Is Kafka good for realtime gameplay messages?

  • Not in the hot path. Kafka shines for analytics, event sourcing, and offline processing. Its batching and commit model add latency that’s rarely acceptable for gameplay deltas.

• Can we migrate rooms between nodes without disconnecting players?

  • It’s possible but complex: you need state transfer, sequence synchronization, and dual-publish windows. Most teams avoid live migration and rely on draining at match boundaries plus fast reconnect with snapshots.

• What about WebTransport/QUIC for browsers?

  • Promising for unreliable datagrams and better HoL behavior, but ecosystem maturity varies. For now, WebSockets remain the pragmatic default for broad compatibility.

• How do we protect against reconnect storms?

  • Use token-based admission, randomized backoff, edge-level 503 with Retry-After, and warmed standby nodes. Consider gating reconnects per room and prioritize in-progress matches.

Key takeaways

• Shard by room and enforce sticky load balancing; cross-node fanout kills scale.
• Use binary protocols and deltas; serialize once per tick per room and reuse buffers.
• Choose Redis or NATS for the hot path; keep Kafka out of gameplay fanout.
• Enforce backpressure and per-socket rate limits; drop slow consumers deterministically.
• Plan for draining, reconnect control, and kernel tuning before your first spike.
• Watch egress costs — optimizing payload size is a direct, recurring ROI.

If you’re building or re-architecting a realtime game backend, MTBYTE can map your current stack to a resilient, cost-aware WebSocket architecture and help you ship it. Reach out at /contact and let’s design it with production in mind.