A Simple Red Hat AMQ Streams (Kafka) Sizing Calculator

Version 0.9

The best way to accurately size your Kafka cluster is by simulating the target load on your Kafka cluster. If you are planning to do that, you may want to use the load generation tools: kafka-producer-perf-test and kafka-consumer-perf-test, that come with Kafka, to simulate your target load. Documentation can be found here.

This calculator uses simple parameters to size and recommend your Red Hat AMQ Streams (Kafka) configuration, including the number of Zookeeper nodes, the number of Kafka broker nodes and the number of subscription cores required, among other things.

Broker Parameters

All input fields are mandatory. Some fields have been pre-populated for you with common values. You can change the default values to suit your requirements.

Target Message Rate (msgs/s):

The message rate your Kafka cluster is to handle.

Average Message Size (number of bytes):

The average message size.

Number of Replicas:

Replicas are used for redundancy. It affects the amount of storage as well as the performance of your Kafka cluster

Network Adapter Speed (Gbps):

The network adapter is assumed to be working in full-duplex mode. Standard Gigabit network adapter is 1 Gbps. It is likely that either the network adapter speed (or the max. disk throughput) is going to be the bottleneck of your system. You may want to use a higher speed network adapter if that is the case.

Max. Disk Throughput (MB/s):

For hard disks, it is around 125 MB/s. For SSDs, it could be 400MB/s or higher.

Max. Utilisation Allowed for Network Adapter and Disk Subsystem (between 0.01 and 1.00):

You do not want to have your network adapter and disk system running at 100% as this may impact the performance of the system during a sudden temporary surge in message rate.

Number of Consumer Groups:

A consumer group is a group of related consumers that perform a task, like sending messages to a service. Different consumer groups can read from different offsets in a partition.

Number of Lagging Consumers:

Caching of data reduces the need to read data from disk when readers (consumers in consumer groups) read data. However, the size of the cache is finite. When a slow reader tries to read the data, the data may have already been evicted from the cache necessitating a disk read which impacts performance. The more lagging consumers, the more impact on performance.

0 is the best case scenario. 0 means that there are no lagging readers ie, all data are in the cache and the reads do not incur the penalty of reading from disk.

{# of Consumer Groups} + ({# of Replicas} - 1) is the worst case scenario. This implies that all reads from readers of the consumer groups as well as the followers trying to read the data from the leader to store them on their local disk incur the penalty of reading from disk.

Data Retention Period (number of days):

This parameter is used to estimate the total storage required for the solution.

Number of Zookeeper Node Failures that can be Tolerated (1 or 2):

A Zookeeper cluster keeps track of status of the Kafka cluster nodes and Kafka topics, partitions etc. The number of Zookeeper nodes is always an odd number: 3, 5, 7, etc. This parameter determines the size of your Zookeeper cluster.

Partition Parameters

The following are optional parameters for estimating the minimum number of partitions required for a topic. This estimation will be skipped if any of the following fields are left at zero (0).

Target Throughput of a Topic (MB/s):

This parameter specifies the target throughput for a topic.

Throughput of the Slowest Producer (MB/s):

This parameter is used to estimate the minimum number of partitions needed for producers to produce the target message throughput.

Throughput of the Slowest Consumer (MB/s):

This parameter is used to estimate the minimum number of partitions needed for consumers to consume at the target message throughput.