Understanding Kafka Producers, Partitions, and Consumers in Practice

Introduction

Kafka is useful when many parts of a system need to publish data and other parts need to consume that data without being tightly connected to each other. A service can produce an event, Kafka can store it in an ordered log, and one or more consumers can pull it later at a rate they can handle.

Kafka was built at LinkedIn to handle log processing demands. The design combines ideas from traditional log aggregation and publish/subscribe messaging systems. That combination matters for developers because Kafka is not only a queue and not only a file log. It gives applications a messaging-style API while storing real-time event data in logs that can be split, replicated, and consumed by different applications.

This post explains the practical path of a Kafka message: how it is created by a producer, how Kafka chooses a partition, how Kafka stores the data efficiently, how consumers read with offsets, and how consumer groups create parallel processing.

Context and Scope

A typical Kafka-based system has a few moving parts:

• A producer creates messages and publishes them to Kafka.
• A message is Kafka's unit of data. It is an array of bytes and may include an optional key.
• A topic is a named stream of related messages.
• A partition is a split of a topic. Each partition is a logical log.
• A broker receives records from producers, stores them, and serves them to consumers.
• A consumer pulls messages from Kafka.
• A consumer group is one or more consumers working together to consume subscribed topics.

The expected outcome is simple: producers should send records safely and efficiently, Kafka should place them in the right topic partition, and consumers should process them without being overwhelmed.

Producer
  |
  | creates and sends messages
  v
Kafka broker
  |
  | stores messages in topic partitions
  v
Consumer group
  |
  | consumers pull records by offset
  v
Application processing

The hard parts are the tradeoffs: throughput versus latency, safety versus speed, ordering versus parallelism, and consumer capacity versus message volume.

Messages, Topics, and Partitions

Kafka's smallest data unit is a message. The message body is just bytes. The optional key is also bytes. That means Kafka does not need to understand your domain object directly. Your application decides how to serialize a value, and the consumer must know how to interpret it.

A useful way to think about a Kafka message is like a row in a table, but stored in a log instead of a relational table. A message can have a schema in your application, but Kafka itself stores the key and value as byte arrays. Each message also has an offset, which identifies its position in a partition.

Messages are organized into topics. A topic can be split into multiple partitions. Each partition is a logical log, and physically that log is stored as a set of segment files of approximately the same size. New messages are appended to the active segment file.

Topic: order-events

Partition 0 log
  offset 0 -> message
  offset 1 -> message
  offset 2 -> message

Partition 1 log
  offset 0 -> message
  offset 1 -> message

Partition 2 log
  offset 0 -> message
  offset 1 -> message
  offset 2 -> message
  offset 3 -> message

This structure explains many Kafka behaviors:

• A topic is the developer-facing name for a stream.
• A partition is the unit that stores ordered records.
• An offset only has meaning inside one partition.
• Adding partitions gives Kafka more places to write and more units for consumers to process in parallel.
• The partitioning decision affects which records are processed together.

Why Kafka Can Be Fast While Using the Filesystem

A common first reaction is to ask why Kafka can be fast if it stores data on disk. The important detail is the access pattern.

Random disk access is slow compared with memory. Kafka is designed around sequential access. New messages are appended to log segment files instead of being written to random locations. Sequential access lets the storage layer work efficiently.

Kafka also lets the operating system filesystem handle much of the storage work. Data goes through the operating system page cache before it is flushed to disk. On reads, if the requested data is already in the page cache, it can be returned from there. On writes, the page can be marked dirty and flushed to disk later by the operating system.

Write path, simplified

Producer request
  |
  v
Broker receives messages
  |
  v
Operating system page cache
  |
  v
Dirty pages eventually flushed
  |
  v
Disk segment files

Kafka also benefits from zero-copy transfer. With zero-copy, data can move directly from the page cache to the socket buffer, avoiding redundant copies through application-level buffers. This matters when consumers are reading large amounts of data because each unnecessary copy costs CPU and memory bandwidth.

Batching is another important optimization. Kafka's protocol includes a message set abstraction that lets messages be grouped together. Sending many single-message requests creates network round-trip overhead. Grouping messages into batches makes client-to-broker communication more efficient and also helps the broker write messages more efficiently.

The practical lesson is that Kafka throughput is not based on one trick. It comes from several design choices working together:

• Append records sequentially.
• Use the operating system page cache.
• Avoid redundant copies with zero-copy transfer.
• Batch messages instead of sending everything one at a time.

Producer Workflow

When an application uses the producer API, the message does not immediately become a file on disk. It goes through a pipeline.

1. The application creates a ProducerRecord.
2. The producer serializes the record key and value into byte arrays.
3. If the application did not choose a partition, the record is sent to a partitioner.
4. After the destination partition is known, the record is added to a batch.
5. A separate thread sends the batch to the broker.
6. The broker returns metadata when the send succeeds, or an error when it fails.

The returned metadata includes the topic, partition, and record offset. That metadata is useful because it tells the producer where Kafka stored the message.

# Conceptual producer flow, not tied to one client library

record = ProducerRecord(
    topic="payment-events",
    key=payment_id_bytes,
    value=payment_event_bytes
)

serialized_record = serialize_key_and_value(record)

if record.partition is None:
    destination = choose_partition(serialized_record.key)
else:
    destination = record.partition

batch = get_batch_for(destination)
batch.add(serialized_record)

# A sender thread later sends the batch to the broker.
metadata_or_error = send_batch_to_broker(batch)

This flow is important when debugging producer behavior. A slow producer may not be slow because Kafka is writing slowly. The producer may be waiting for synchronous responses, sending too many small requests, or routing records in a way that creates uneven partition load.

Producer Send Styles

A producer can send messages in different ways. The right choice depends on how much feedback the application needs and how much latency it can tolerate.

For most practical systems, asynchronous sending is the balanced option. It avoids blocking on every message while still giving the application a place to handle errors.

Acknowledgments: Deciding When a Message Counts as Delivered

Producer acknowledgments, configured through acks, define when Kafka tells the producer that a send succeeded.

acks=0
  Producer does not wait for a broker reply.

acks=1
  Producer receives success after the leader receives the message.

acks=all
  Producer receives success after all replicas receive the message.

This choice should be made from business requirements. A clickstream event may tolerate less confirmation than a financial event. A system that must preserve important events should not choose the fastest setting simply because it has lower latency.

Partitioning: How Kafka Distributes Messages

Kafka uses partitions to distribute messages inside a topic. The message key is central to this decision.

A message key is null by default. If the key is null and the application has not defined a custom partitioner, Kafka can distribute records across partitions. Two common strategies are round-robin and sticky partitioning:

• Round-robin partitioning assigns messages cyclically, one partition after another.
• Sticky partitioning sends as many records as possible to the same partition until a condition is met, then moves to another partition.

If the key is not null, Kafka hashes the key and uses the result to map the message to a specific partition. This means records with the same key are routed to the same partition.

Messages with keys

key=A -> hash -> Partition 0
key=B -> hash -> Partition 2
key=C -> hash -> Partition 1
key=A -> hash -> Partition 0
key=B -> hash -> Partition 2

That behavior is useful when related events should travel through the same partition. For example, if all events for the same payment ID use the payment ID as the key, they will be mapped to the same partition. Since a consumer reads a particular partition sequentially, that gives the application a predictable processing path for related records inside that partition.

The tradeoff is that keys affect distribution. A poor key choice can concentrate too much traffic on one partition. A missing key can improve distribution, but it may separate related records across partitions.

Consumer Workflow: Pulling Records by Offset

Kafka consumers use a pull model. Instead of Kafka pushing messages as fast as possible, consumers request data from the broker. This lets consumers read at a rate that fits their capacity and helps avoid flooding a slow consumer.

A consumer always consumes messages from a particular partition sequentially. Each request includes the offset where consumption should begin. The Consumer API behaves like an infinite loop that keeps polling the broker for more data. The pull requests are asynchronous, but each request still says which offset the consumer wants next.

# Conceptual consumer loop

next_offset = load_starting_offset(partition)

while application_is_running:
    records = poll_broker(
        partition=partition,
        offset=next_offset
    )

    for record in records:
        process(record)
        next_offset = record.offset + 1

    acknowledge_offset(next_offset)

Offset acknowledgment is an important detail. If a consumer acknowledges an offset, the broker can treat the previous messages in that partition as received by that consumer. This is why offset handling is part of correctness, not just bookkeeping.

A practical consumer should avoid processing records blindly and acknowledging too early. If the application acknowledges progress before processing is complete, a failure may make recovery harder. If it never acknowledges progress, it may repeat work.

Consumer Groups and Parallelism

A consumer group lets multiple consumers share the work for subscribed topics. The smallest unit of parallelism is a topic partition. Inside one consumer group, all messages from one partition are consumed by only one consumer.

That rule has direct scaling consequences:

Topic with 4 partitions, consumer group with 2 consumers

Partition 0 -> Consumer 1
Partition 1 -> Consumer 1
Partition 2 -> Consumer 2
Partition 3 -> Consumer 2

This is a common source of confusion. Adding more consumers does not automatically increase throughput forever. Once the group has more consumers than partitions, the extra consumers cannot be assigned unique partitions, so they sit idle.

The practical design rule is to think about partitions and consumer groups together. If a topic has too few partitions, the consumer group cannot scale beyond that number of active consumers. If a topic has many partitions, the system has more parallel units to assign, but the application must still choose keys carefully so messages are distributed in a useful way.

Practical Workflow for Designing a Kafka Stream

When adding Kafka to an application, use this workflow before writing client code.

1. Define the event stream. Decide what the topic represents. A topic should group messages that consumers can understand as the same kind of stream.
2. Define the message shape. Kafka stores bytes, so decide how producers serialize the value and how consumers interpret it. Treat schema decisions as application-level contracts.
3. Choose the key. Use a key when related records should map to the same partition. Leave it null only when distribution matters more than grouping by entity.
4. Estimate partition needs. Partitions define the parallelism available to a consumer group. A group cannot actively use more consumers than the topic has partitions.
5. Pick the send style. Use fire-and-forget only when missed failures are acceptable. Use synchronous sends carefully. Prefer asynchronous sends when throughput and error handling both matter.
6. Pick the acknowledgment level. Use acks=all when the event must survive a broker crash. Accept the latency tradeoff deliberately.
7. Batch messages. Avoid designing producers that send too many single-message requests. Batching reduces network round trips and helps broker writes.
8. Design the consumer loop. Poll by offset, process records, and acknowledge progress only when the application is ready to treat earlier records as received.
9. Size consumer groups around partitions. Add consumers when there are partitions to assign. Do not expect idle consumers to increase throughput.

Common Mistakes to Watch For

Treating messages as rich objects inside Kafka

Kafka stores message keys and values as bytes. Your services can use schemas and domain objects, but serialization and deserialization are your responsibility.

Ignoring the partition key

A null key lets Kafka distribute messages, but related records may not stay together. A non-null key maps the same key to the same partition, but a poor key can overload one partition.

Using synchronous sends for every message

Synchronous sending is easy to reason about, but waiting for a response on every send can impact performance. Asynchronous sending with error callbacks is usually a better fit for high-throughput producers.

Assuming all acknowledgment settings provide the same safety

acks=0, acks=1, and acks=all mean different things. The safest setting waits for all replicas, but it also increases latency.

Adding more consumers than partitions

A consumer group can only parallelize work at the partition level. If a topic has four partitions, a fifth consumer in the same group has no partition to own.

Acknowledging offsets without understanding the meaning

Acknowledging an offset tells Kafka that previous messages in that partition have been received. Acknowledge only when that matches the application's processing state.

Checklist

Use this checklist when reviewing a Kafka design:

• The topic has a clear purpose.
• Producers and consumers agree on how message bytes are serialized and interpreted.
• The key strategy is intentional, not accidental.
• The partition count supports the desired consumer group parallelism.
• The producer send style matches the required feedback and throughput.
• The acks setting matches the business value of the event.
• Producers batch messages instead of sending excessive single-message requests.
• Consumers pull at a rate they can handle.
• Consumers process each assigned partition sequentially.
• Offset acknowledgments happen only after the application is ready to mark progress.
• Extra consumers are not expected to help once every partition is assigned.

Conclusion

Kafka is easiest to understand as a system built around partitioned logs. Producers create byte-based messages, Kafka stores them in topic partitions, and consumers pull records by offset. The practical power comes from making deliberate choices: use keys when related records should share a partition, batch messages for efficient writes, choose acknowledgments based on safety requirements, and size consumer groups around partition-level parallelism.