We had 50 cron jobs, 30 webhook receivers, and a Slack channel called #on-call-nightmares that had become a living monument to everything brittle about our event processing. Then we found Argo Events. Over three months, we replaced almost all of it. Here's what that actually looked like.

The Before State (It Was Bad)

Before Argo Events, our event processing looked like this: Bash scripts on cron, fired by Kubernetes CronJob objects. Webhook receivers written by whoever needed them, deployed as long-running Deployments with no shared reliability guarantees. Race conditions. Missed events when pods restarted mid-fire. No retry logic. No dead-letter handling.

The canonical failure mode: a game result would come in via Kafka, trigger a leaderboard update job, and if the job happened to restart mid-run (OOM, eviction, anything), the event was gone. The leaderboard wouldn't update. Players would notice. Someone would get paged. Someone would manually re-run a script. This happened, on average, twice a week.

The deeper problem was coordination. We had 30 different webhook receivers with no shared routing layer. Adding a new event type meant writing a new receiver, deploying it, figuring out where its logs were, and hoping nobody else was already listening to the same source for a different purpose.

A Slack channel full of "why did X break at 3am" messages is a symptom, not the problem. The problem is infrastructure that makes failure the default and reliability the exception.

What Argo Events Actually Is

Argo Events has three building blocks. Once you internalize these, everything else makes sense.

EventSource is where events enter the system. It listens to something: a Kafka topic, a webhook endpoint, an S3 bucket notification, a cron schedule, a GitHub push. It normalizes the event into a standard format and publishes it to the EventBus.

EventBus is the message backbone. By default it's NATS JetStream, a durable, at-least-once message bus that persists events even when consumers are temporarily unavailable. This is the piece that changed our reliability story.

Sensor subscribes to the EventBus, watches for specific events, and fires triggers: submit an Argo Workflow, make an HTTP call, apply a Kubernetes resource, post to Slack. One event can trigger many sensors. Many events can trigger one sensor with dependency conditions.

Our First EventSource: Kafka Game Results

The first thing we ported was our game result processing pipeline. A Kafka topic received game completion events; a cron job would batch-process them every 5 minutes. This meant leaderboards were always stale by up to 5 minutes, and if the batch job missed events, they were gone. 

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: game-results
  namespace: argo-events
spec:
  kafka:
    game-result-topic:
      url: kafka-broker.platform:9092
      topic: game-results
      partition: "0"
      consumerGroup: argo-events-consumer
      sasl:
        mechanism: PLAIN
        userSecret:
          name: kafka-creds
          key: username
        passwordSecret:
          name: kafka-creds
          key: password

The Sensor that reacts to this triggers an Argo Workflow for each game result event: leaderboard update, prize credit, notifications. The whole pipeline is now event-driven, sub-second, and durable. If the Sensor pod restarts, NATS JetStream replays the event from the last committed offset. Nothing is lost.

The EventBus: Why NATS Changed Our Reliability Story

The old architecture had no message bus. Events were fire-and-forget HTTP calls. If the receiver was down, the event was gone.

EventBus with NATS JetStream gives you persistence, replay, and consumer groups. Events sit in the bus until a Sensor acknowledges them. If the Sensor crashes mid-processing, the event gets redelivered when it comes back. We went from "events are lost on pod restart" to "events are processed at least once, eventually, regardless of transient failures." 

apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
  namespace: argo-events
spec:
  nats:
    native:
      replicas: 3         # HA — survives single node failure
      auth: token
      persistence:
        storageClassName: ssd
        accessMode: ReadWriteOnce
        volumeSize: 20Gi  # keep events for replay

The Double-Processing Bug: An Incident It Solved Retroactively

Before Argo Events, we had a leaderboard corruption bug that happened roughly once a month. A game result would be processed twice — once by the cron batch, once by a webhook receiver that was also watching the same Kafka topic for a different reason. Two writes to the same leaderboard entry, in rapid succession, with a race condition on the score update. Leaderboard corrupted. Manual rollback required.

Argo Events doesn't solve race conditions by magic. But it forces you to have one canonical consumer per event type. The EventSource consumes from Kafka. The Sensor subscribes to the EventBus. One processing path, one consumer group, NATS handling delivery. We also enabled leaderElection on the Sensor to prevent duplicate triggers from multiple Sensor replicas. The double-processing bug stopped occurring. Just gone.

leaderElection on Sensors is not the default and it's not well-documented. Enable it in production. Without it, if you have multiple Sensor replicas, they will all fire on the same event.

Rolling It Out Across 200+ Services

We didn't migrate everything at once. We started with the game result pipeline: highest pain, most visible. Ran it in parallel with the old cron system for two weeks, comparing outputs. When the outputs matched and reliability improved, we killed the cron.

Then we did the webhook receivers. Thirty of them, over 6 weeks, one team at a time. Each team got a template for their EventSource and Sensor, a migration guide, and ownership of their own event namespace. The shared EventBus meant they didn't have to run their own message infrastructure. They just wrote the EventSource and Sensor YAML.

The Numbers

50
Cron jobs replaced
2M+
Events processed daily
8mo
Zero missed events in production
0
Leaderboard corruptions post-migration

The Catch: YAML Complexity Grows Fast

Argo Events is powerful and production-ready. It's also verbose. A simple "Kafka event triggers a workflow" setup is 3 YAML files with ~80 lines total. When you have dependencies between events (trigger only if event A AND event B arrive within 30 seconds), the Sensor YAML gets complex fast. I'm not going to pretend it's clean.

Template everything. We built a Helm chart with sensible defaults for our most common patterns. New teams fill in 5 variables and get a production-ready EventSource + Sensor. Without that abstraction, the raw YAML would have killed adoption.

The #on-call-nightmares Slack channel still exists. It's mostly quiet now. Someone suggested archiving it. We're keeping it as a monument to what we used to have.