Apache Kafka interview questions

What a strong answer covers

• Define a Kafka Topic as a category or feed name to which records are published.
• Explain that topics are logical constructs, while partitions are the physical units of storage and parallelism.
• Describe how a topic is divided into one or more ordered, immutable partitions.
• Mention that each record within a partition has a unique, sequential offset.

Where people lose the point

• ×Confusing a topic with a queue, failing to mention its log-like nature.
• ×Not clearly distinguishing between a topic (logical) and a partition (physical/parallelism unit).
• ×Incorrectly stating that partitions are unordered or mutable.

What a strong answer covers

• Define a Consumer Group as a set of consumers that cooperate to consume messages from one or more topics.
• Explain that within a group, each partition is consumed by exactly one consumer instance.
• Describe how consumer groups enable scalable and fault-tolerant consumption.
• Mention that if a consumer fails, its partitions are redistributed among other active consumers in the group (rebalancing).

Where people lose the point

• ×Stating that all consumers in a group read all messages from all partitions.
• ×Failing to explain the role of rebalancing in fault tolerance and scalability.
• ×Not mentioning that different consumer groups can consume the same topic independently.

What a strong answer covers

• Define an offset as a unique, sequential ID number given to each record within a partition.
• Explain that offsets are local to each partition and start from 0.
• Describe its purpose: consumers use offsets to track their position in a partition, indicating which messages have been processed.
• Mention that committed offsets allow consumers to resume processing from the last successfully processed message after a restart or rebalance.

Where people lose the point

• ×Confusing offsets with message keys or timestamps.
• ×Incorrectly stating that offsets are global across a topic or cluster.
• ×Failing to explain how offsets enable fault-tolerant consumption.

What a strong answer covers

• Define a Kafka Broker as a server that forms part of the Kafka cluster.
• Explain its primary role: storing topic partitions and handling read/write requests from producers and consumers.
• Mention that brokers also manage replication of partitions to ensure data durability and fault tolerance.
• Describe how a cluster consists of multiple brokers working together, with one acting as the controller (historically Zookeeper, now KRaft).

Where people lose the point

• ×Confusing a broker with a producer or consumer.
• ×Omitting the role of replication and fault tolerance.
• ×Not mentioning that brokers store the actual message data.

What a strong answer covers

• Explain that producers create `ProducerRecord` objects containing a topic, value, and optionally a key and partition.
• Describe how the producer client library handles serialization, partitioning, and sending records to the appropriate broker.
• Mention the role of the partitioner (default or custom) in determining which partition a record goes to.
• Discuss batching and compression as common optimizations for sending messages efficiently.

Where people lose the point

• ×Overlooking the role of message keys in partitioning.
• ×Not mentioning the concept of batching or compression.
• ×Incorrectly stating that producers directly interact with all brokers for a topic.

What a strong answer covers

• Define `acks` as a producer configuration that controls the level of acknowledgment required from the Kafka brokers before a write is considered successful.
• Detail `acks=0`: producer doesn't wait for any acknowledgment, lowest latency, highest risk of data loss.
• Detail `acks=1`: producer waits for the leader broker to acknowledge the write, moderate latency and durability.
• Detail `acks=all` (or `-1`): producer waits for all In-Sync Replicas (ISRs) to acknowledge the write, highest durability, highest latency.

Where people lose the point

• ×Confusing `acks` with consumer offset commits.
• ×Incorrectly associating `acks=0` with guaranteed delivery.
• ×Failing to explain the trade-off between durability and latency for each `acks` setting.

What a strong answer covers

• Explain that rebalancing occurs when consumers join or leave a consumer group, or when topic partitions are added/removed.
• Describe the goal: to redistribute partition ownership among the active consumers in the group.
• Detail the steps: consumers stop processing, group coordinator (broker) assigns partitions, consumers resume processing.
• Mention the impact: a temporary pause in consumption during the rebalance, which can affect latency.

Where people lose the point

• ×Incorrectly stating that rebalancing happens for every message.
• ×Failing to mention the temporary pause in consumption during a rebalance.
• ×Not explaining that rebalancing ensures each partition is consumed by only one consumer in the group.

What a strong answer covers

• Define ISRs as the set of replicas (including the leader) that are fully caught up with the leader's log for a given partition.
• Explain their importance for data durability: a message is considered committed only when it has been successfully replicated to all ISRs.
• Describe how ISRs are crucial for fault tolerance: if the leader fails, a new leader is elected from the ISRs, ensuring no data loss.
• Mention that a replica falling out of sync (e.g., due to network issues or slowness) will be removed from the ISR set until it catches up.

Where people lose the point

• ×Confusing ISRs with all replicas, regardless of their sync status.
• ×Incorrectly stating that a new leader can be elected from any replica, even if not in sync.
• ×Failing to connect ISRs directly to the `acks=all` producer setting.

What a strong answer covers

• Define at-most-once: messages might be lost but are never redelivered. Producer doesn't wait for acknowledgment (e.g., `acks=0`).
• Define at-least-once: messages might be redelivered but are never lost. Achieved with `acks=all` and proper consumer offset management.
• Explain the trade-offs: at-most-once offers higher throughput/lower latency but less reliability; at-least-once offers higher reliability but potentially lower throughput/higher latency.
• Mention that at-least-once often requires consumers to be idempotent to handle duplicate messages gracefully.

Where people lose the point

• ×Confusing the definitions or their implications for data loss/duplication.
• ×Incorrectly associating `acks=all` with at-most-once.
• ×Failing to mention the need for idempotency with at-least-once processing.

What a strong answer covers

• Explain that Kafka guarantees message order *within a single partition*.
• Describe how producers can ensure messages with the same key go to the same partition (e.g., using a default hash partitioner).
• Mention that consumers within a consumer group will process messages from a given partition in the order they were written.
• Clarify that there is no global ordering guarantee across different partitions of a topic.

Where people lose the point

• ×Claiming global ordering across an entire topic.
• ×Not emphasizing the role of message keys in maintaining order for related events.
• ×Failing to explain that ordering is a partition-level guarantee.

What a strong answer covers

• Define exactly-once: each message is processed exactly one time, even in the face of producer or consumer failures.
• Describe the three components: idempotent producers (prevent duplicate writes), transactional producers (atomic writes to multiple partitions and offset commits), and transactional consumers (read only committed data).
• Explain how transactional producers use a `transactional.id` to ensure atomicity across multiple writes and offset commits.
• Mention that consumers must be configured to `isolation.level=read_committed` to only read messages from committed transactions.

Where people lose the point

• ×Confusing exactly-once with at-least-once or at-most-once.
• ×Omitting any of the three key components (idempotent producers, transactional producers, transactional consumers).
• ×Failing to explain the role of `transactional.id` or `isolation.level`.

What a strong answer covers

• Identify key considerations: high throughput, order of events for a single order, potential for hot partitions.
• Propose using `order_id` as the message key to ensure all events related to a specific order (e.g., created, paid, shipped) land in the same partition, preserving order.
• Discuss the number of partitions: start with a reasonable number (e.g., 2-3x the number of brokers or expected consumer instances) and monitor for hot spots.
• Address potential issues: if a few `order_id`s are extremely active, they could create hot partitions; suggest strategies like re-keying or using a composite key if necessary, or monitoring for skew.

Where people lose the point

• ×Suggesting random partitioning, which would break order for individual orders.
• ×Ignoring the potential for hot partitions or not providing a strategy to mitigate them.
• ×Not considering the relationship between partitions and consumer parallelism.

What a strong answer covers

• **Partitions:** Increasing partitions increases parallelism for producers and consumers, leading to higher throughput. However, it increases broker overhead (more file handles, metadata) and consumer rebalancing frequency/duration.
• **Replication Factor:** Increasing replication factor enhances data durability and fault tolerance (more copies of data). However, it increases storage requirements and network I/O for replication.
• Explain that partitions scale throughput, while replication scales durability and availability.
• Discuss how both impact resource usage: more partitions mean more open files and potential for more consumer groups, while higher replication means more disk space and network traffic between brokers.

Where people lose the point

• ×Confusing the purpose of partitions (scalability) with replication (durability).
• ×Failing to mention the overhead associated with too many partitions (broker/consumer).
• ×Not discussing the increased resource consumption (storage, network) with higher replication.

What a strong answer covers

• **Diagnose:** Check consumer group status (number of active consumers vs. partitions), monitor consumer CPU/memory usage, check network latency between consumers and brokers, analyze consumer application logs for errors or slow processing logic.
• **Resolve (Consumer-side):** Increase the number of consumer instances in the group (up to the number of partitions), optimize consumer processing logic (e.g., batching, parallel processing within a consumer), adjust `max.poll.records` or `fetch.min.bytes`.
• **Resolve (Broker/Topic-side):** Check broker health (CPU, disk I/O, network), ensure sufficient partitions for the topic, verify no hot partitions are causing bottlenecks.
• Consider external factors: upstream producer spikes, database bottlenecks if consumers are writing to a DB.

Where people lose the point

• ×Jumping directly to solutions without proper diagnosis.
• ×Suggesting increasing partitions without considering the number of consumers.
• ×Ignoring the possibility of slow consumer processing logic as a root cause.

What a strong answer covers

• **Use a Schema Registry:** Recommend using a Schema Registry (e.g., Confluent Schema Registry) to store and manage schemas (e.g., Avro, Protobuf, JSON Schema).
• **Schema Compatibility:** Explain the importance of defining compatibility rules (e.g., backward, forward, full) to ensure producers and consumers can evolve independently.
• **Serialization/Deserialization:** Describe how producers serialize messages with schema IDs and consumers use the schema ID to fetch the correct schema for deserialization.
• **Strategies for Changes:** Discuss strategies like adding optional fields (backward compatible), deprecating fields (forward compatible), or using schema versioning for breaking changes.

Where people lose the point

• ×Suggesting manual schema management without a Schema Registry.
• ×Ignoring the concept of schema compatibility and its importance for evolving systems.
• ×Not explaining how schema IDs facilitate dynamic schema resolution.