Building Sentinel: Sharding, Gateways, and Why WebSockets Lie to You

Everyone's first Discord bot is a REST wrapper: respond to a slash command, make an API call, done. The moment you want to react to things happening in real time, a member joining, a message getting deleted, a role changing, you're not making requests anymore, you're holding open a persistent connection and listening, and that's a completely different engineering problem with its own failure modes that don't show up until you've been running in production for a while.

The gateway isn't just 'a websocket'

Discord's gateway is a WebSocket connection with a defined lifecycle on top of the raw protocol, and getting the lifecycle wrong is where almost every 'my bot randomly stops working' bug actually lives. On connect, you send an IDENTIFY payload with your token and intents, and Discord starts streaming events at you. Discord also tells you, in the first payload it sends, a heartbeat interval, and you have to send a heartbeat on that exact cadence or the connection gets killed as dead, no warning. Miss a couple of heartbeats because your event loop was busy doing something else and you're reconnecting from scratch, which for a bot of any size means a burst of API calls just to rebuild cache state.

The detail that actually matters in production is RESUME. Every event Discord sends includes a sequence number. If your connection drops, and it will, Discord restarts gateway nodes, your network blips, whatever, you don't have to IDENTIFY again and eat a fresh full state sync. You can RESUME with your session ID and the last sequence number you saw, and Discord replays everything you missed since then. Implementing this properly means tracking that sequence number on every single event, not just the ones you care about, and handling the case where Discord rejects the resume by falling back to a clean IDENTIFY instead of just hanging:

ws.on('close', (code) => {
  if (canResume(code) && sessionId && lastSequence) {
    reconnectAndResume(sessionId, lastSequence);
  } else {
    reconnectAndIdentify();
  }
});

Getting this wrong doesn't crash the bot. It just makes it quietly worse, more full resyncs than necessary, more load on your own cache rebuild logic, occasional gaps where an event genuinely got missed during the reconnect window. It's the kind of bug that shows up as 'the bot felt laggy yesterday around 3pm' rather than an exception in your logs, which makes it miserable to track down after the fact if you didn't build the instrumentation for it up front.

Rate limits are a bucket problem, not a sleep(1000) problem

Discord's REST API rate limits per route, not globally, and the limit for a given route comes back to you in response headers on every single call, X-RateLimit-Remaining, X-RateLimit-Reset-After, a bucket hash that groups routes sharing the same limit. The naive approach, catch a 429, sleep, retry, works until you're making enough calls that you're eating 429s constantly, and enough of those in a short window gets your whole application temporarily banned from the API, which takes down every guild the bot's in, not just the one that triggered it.

The actual fix is a token-bucket queue per route bucket: track remaining calls and reset time from the headers Discord already gives you, and queue outgoing requests against that instead of firing them and hoping. A request for a bucket that's exhausted waits until reset instead of going out and immediately eating a 429. It's maybe 150 lines of actual logic, but it's the difference between a bot that degrades gracefully under load and one that face-plants the first time a server does something that triggers a burst of moderation actions at once, like, say, exactly the situation Sentinel exists to handle.

Sliding-window detection for anti-raid

The anti-raid system needed to answer one question fast: 'are joins happening faster than they plausibly would organically?' The naive version, count joins in the last N seconds, alert if over threshold, has an obvious failure mode: a legitimate traffic spike (server gets shouted out somewhere, a big event starts) looks identical to the early stage of a raid until it's either very over or very not.

What actually works is a ring buffer of join timestamps per guild, checked on every new join rather than on a fixed timer, so detection latency is 'as fast as the next join event' rather than 'up to N seconds behind.' The check counts how many timestamps fall inside the trailing window and compares against a threshold that scales slightly with the guild's normal join rate, a 50-person server and a 50,000-person server should not share a flat numeric threshold, because what's suspicious for one is a normal Tuesday for the other. Tuning that scaling factor took running the detector against real join-burst data from test servers before the false-positive rate on legitimate spikes came down to something that wouldn't annoy actual communities.

When it fires, the response has to be fast and automatic, revoke invites, bump verification level, drop an alert in the mod log, because the entire value proposition is reacting before a human has opened the app, not after. That's also why this logic runs server-side close to the gateway event itself rather than being computed lazily when someone checks a dashboard: by the time someone looks, the raid's already happened.

The common thread across all of this, the resume logic, the rate limit queue, the sliding window, is that none of it is hard in the 'hard algorithm' sense. It's all straightforward once you've seen it. The actual difficulty is that none of these problems show up in a demo. They show up three weeks into production, under load, at the exact moment you'd most like your infrastructure to not be the thing failing.