Do You Need Kafka? A Guide for Startups
Most startups do not need Kafka. Apache Kafka is built for high-volume event streams that several independent services need to read, replay and process in order. If what you actually need is background jobs, retries or notifying a couple of services when something happens, a Redis-based queue, a managed queue such as Amazon SQS, or Postgres will do the job with far less to operate. This guide explains what event-driven architecture is, when it pays off, what to use instead of Kafka and the signs that Kafka has become the right call.
It is written for founders and engineering leads making an architecture decision, not a Kafka tutorial. I have used Kafka in production, on a video streaming platform where real-time events fed search and caching, so this is a view from someone who likes Kafka and still recommends against it most of the time.
What Is Event-Driven Architecture?
In an event-driven architecture, parts of a system communicate by publishing events (facts that something happened, such as OrderPlaced or VideoUploaded) instead of calling each other directly. Other parts subscribe to the events they care about and react in their own time.
The benefits are real:
- Decoupling. The service that places an order does not need to know that email, analytics and inventory all react to it.
- Resilience. If the email service is down, events wait; the order still goes through.
- Scalability. Slow work moves off the request path, so users are not kept waiting.
But you can get most of these benefits without Kafka. Event-driven is a pattern; Kafka is one, fairly heavy, way to implement it.
Kafka vs a Message Queue: What Is the Difference?
| Message queue (RabbitMQ, SQS, BullMQ) | Kafka | |
|---|---|---|
| Model | Queue of messages to process | Append-only log of events |
| After processing | Message is usually removed | Event is kept for the retention period |
| Multiple readers | Each message typically goes to one consumer | Any number of consumer groups read the same stream independently |
| Replay | Not built in | Consumers can re-read from an earlier offset |
| Ordering | Per queue, with limits | Guaranteed within a partition |
| Typical job | Distribute work: send email, resize image, call an API | Record a stream: activity feeds, analytics, syncing many systems |
| Operational weight | Low to moderate; fully managed options exist | High if self-hosted; managed options reduce it |
The short version: queues distribute work, Kafka records history. Many teams adopt Kafka for a job that is really just distributing work.
What Should a Startup Use Instead of Kafka?
Match the tool to the actual job:
- Background jobs and retries (emails, PDF generation, webhooks, AI calls): a Redis-based queue such as BullMQ in Node.js, or Amazon SQS. You get retries, delays, concurrency limits and dead-letter handling with very little setup.
- Notify a few services when something happens: Redis Pub/Sub for fire-and-forget, Redis Streams when you need consumer groups and some history, or SNS + SQS on AWS for durable fan-out.
- Publish an event reliably after a database write: the transactional outbox pattern. Write the event to an
outboxtable in the same Postgres transaction as the business change, and have a worker publish it. This avoids the classic bug where the database commit succeeds but the event is lost, or the reverse. - Real-time updates to users: WebSockets or server-sent events backed by Redis Pub/Sub.
For most products, a Postgres database, Redis and a job queue cover everything until the business is well past product-market fit. That is the same "boring technology" approach I recommend in system design for AI-powered SaaS.
When Is Kafka the Right Choice?
Kafka starts to earn its cost when two or more of these are true:
- Many independent consumers need the same stream. Search indexing, analytics, recommendations and notifications all read the same user activity events, owned by different teams.
- You need replay. You want to rebuild a read model, backfill a new service from months of history, or recover from a bug by reprocessing events.
- Volume is high and sustained. Event rates where a single queue or database table becomes the bottleneck.
- Ordering per key matters at scale. All events for one account or one device must be processed in order, across many parallel consumers.
- Real-time data pipelines. Feeding a data warehouse, search index and cache from one source of truth as events happen.
On the streaming platform, Kafka made sense for that last reason: it carried real-time events in a system that also relied on Redis caching and fast search across actors, genres and directors, while serving many concurrent streams. For a typical SaaS MVP, none of these apply on day one.
What Does Kafka Actually Cost to Run?
The licence is free; the operation is not.
- Cluster operations. Brokers, replication, upgrades, disk usage, rebalancing and monitoring. Since Kafka 4.0, clusters run in KRaft mode without ZooKeeper, which simplifies things, but it is still a distributed system to look after.
- Design decisions that are hard to change. Topic names, partition counts and message schemas are difficult to change once many consumers depend on them.
- New failure modes. Consumer lag, poison messages, rebalancing storms and duplicate processing all need handling in your code.
- Learning curve. Every developer who touches it needs to understand offsets, consumer groups and delivery semantics.
If you do need Kafka, use a managed service such as Confluent Cloud, Amazon MSK or a Kafka-compatible platform like Redpanda. That removes most of the cluster work, so your team only owns the design.
How Do You Keep the Option Open?
You do not have to choose Kafka now to be able to adopt it later. Three habits make a future move easy:
- Name events as facts.
InvoicePaid, notsendInvoiceEmail. Facts can have many consumers later; commands cannot. - Publish through one small module. Code calls
publish(event), and only that module knows whether it goes to BullMQ, SNS or Kafka. - Use the outbox pattern from the start for events that matter. Switching the outbox worker from a queue to Kafka is a small change.
This also matters if you ever face a bigger architecture decision. A system with clear events and boundaries is far easier to evolve piece by piece, as I describe in when to rewrite your MVP and when not to.
The Short Version
- Start with Postgres, Redis and a job queue. They cover most products for a long time.
- Use the outbox pattern for events that must not be lost.
- Move to Kafka (managed) when several consumers need the same high-volume stream, or when you need replay.
- Keep events as facts behind one publishing module, so the move is cheap when it comes.
Every extra moving part is something to monitor at 2am. If you want a second opinion on whether your architecture needs Kafka yet, that is exactly the kind of decision a short review from a senior full stack developer can settle quickly. For more on how requests and data flow through a production system, see what actually happens when you hit enter after typing a URL.
Frequently Asked Questions
Does my startup need Kafka? Probably not yet. Kafka is built for high-volume event streams that many independent services read, replay or process in order. Most early products only need background jobs or simple pub/sub, which BullMQ, Amazon SQS or Postgres handle with far less operational work.
What is the difference between Kafka and a message queue? A message queue delivers each message to a consumer and usually removes it after processing. Kafka is an append-only log: events are kept for a retention period, many consumer groups can read the same stream, and consumers can replay. Queues distribute work; Kafka records a stream of events.
What should a startup use instead of Kafka? BullMQ or Amazon SQS for background jobs, Redis Pub/Sub, Redis Streams or SNS with SQS for fan-out, and the transactional outbox pattern on Postgres for publishing events reliably after a database write.
When is Kafka the right choice? When several independent consumers need the same high-volume stream, when you need replay, when strict per-key ordering matters at scale, or when you feed real-time analytics and search from one source of truth.
Is managed Kafka a good option for startups? If you need Kafka, yes. Confluent Cloud, Amazon MSK or Kafka-compatible platforms like Redpanda remove most of the work of running the cluster, so your team only owns topic and consumer design.