WHERE DO I START?
You are here:Integrations > Kafka
  • Top ↑

Kafka

Summary

Kafka is a publish-subscribe message queuing system that's designed like a distributed commit log. Netuitive can help monitor the performance and throughput of your Kafka server using our Kafka collector for the Linux agent. Kafka, Kafka Consumer Lag, and Zookeeper metrics are all collected using this collector.

Configuration

Important   If you haven't installed the Linux agent already, see the instructions on this page for how to install it. If you need to disable the Linux integration or view the unique API key assigned to your account, navigate to the Integrations page under the user account drop-down menu and click the integration designated as Infrastructure under the Integration column.

Collector Configuration

  1. Download the Jolokia JVM JAR file.
    1. Move the downloaded file to the /opt/netuitive-agent/ directory.
    2. Run the following at the command line to set an environment variable:
      export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar"
      Note   If you're running Kafka in a container, you'll need to complete an additional step to get Jolokia working in the container. See the detailed instructions below.
    3. Tip   Another option would be to add the following line to the very end of the kafka-server-start.sh file (typically located in /etc/kafka/bin, or /opt/kafka/bin — sometimes the Kafka directory contains the version number as well). This option is more portable.
      JVM_OPTS="$JVM_OPTS -javaagent:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar"
      Important   When adding this line to your kafka-server-start.sh file, ensure that the quotes are not automatically formatted to be smart quotes. Smart quotes will prevent Kafka from starting.
    4. Restart Kafka, and confirm Jolokia is running by accessing http://localhost:8778/jolokia/
  2. Navigate to the collectors folder.
    Note   The default location is at /opt/netuitive-agent/conf/collectors.
  3. Open the KafkaJolokiaCollector.conf file.
  4. Change the enabled setting to True.
  5. Update version to be the version of Kafka running on your machine and uncomment the line.
  6. Optionally, update host and port to the correct settings.
  7. Save the file, and restart the Linux Agent.
  8. Note   This integration's package (computed metrics, dashboards, and policies that will give you important events and alerts) will be automatically enabled and provisioned to your account as soon as Netuitive receives data from the integration. The PACKAGES button on the integration setup page will become active once data is received, so you'll be able to disable and re-enable the package at will.

Additional Configuration Options

Collector Options

Important   The path and jolokia_path settings should not be changed without consulting Netuitive support first.
Option Default Description
enabled False Enable collecting Kafka metrics.
path kafka The metric prefix, e.g., how you want the metrics to show up in Netuitive.
jolokia_path jolokia Part of the URL path that points to where your application serves metrics. Typically jmx or jolokia.
metrics_blacklist .*Percentile$|.*FifteenMinuteRate.*|.*FiveMinuteRate.*|.*\.MeanRate$|.*\.OneMinuteRate.*|.*\.StdDev$|.*Metadata.*|.*__consumer_offsets.*|.*\.(Min|Max)$ Regex list to match metrics to block. Mutually exclusive with metrics_whitelist option.
host localhost Hostname/IP address by which the Kafka instance can be reached by the Linux agent.
port 8778 The port that the Jolokia JAR file is listening on.
version 0.8

Specifies the Kafka major release being used on the host. Currently, Netuitive supports 0.8, 0.9, and 0.10.

zookeeper localhost:2181 The hostname / IP address as well as the port by which the Zookeeper instance can be reached by the Linux agent.
bin /opt/kafka/bin/kafka-run-class.sh The location of the kafka-run-class.sh file. Certain metrics must be retrieved via command line calls, thus the location of kafka-run-class.sh is a necessity. The exception to this being if you're running Kafka in a Docker container; see the args setting below for more information.
args exec {kafka-container-id} {kafka-run-class.sh}

The argument passed to Kafka running in a docker container.

Note   If you are running Kafka in a docker container, you'll need to set the bin setting to the location of the docker binary. Then, you'll need to replace {kafka-container-id} with the ID of your Kafka container and {kafka-run-class.sh} with the location of the kafka-run-class.sh file within the container.
consumer_groups Group1, Group2, Group3

For Kafka 8 or earlier: This setting specifies a list of consumer groups that you want to receive consumer lag metrics. If you do not specify this, no consumer lag metrics will be collected.

For Kafka 9 or later: This setting is ignored as consumer groups can be auto-discovered.

topics Topic1, Topic2, Topic3

For Kafka 8 or earlier: This setting specifies a list of topics associated with the consumer groups. If you do not specify this, all topics are considered.

For Kafka 9 or later: this setting is ignored.

byte_unit   Default numeric output(s).
measure_collector_time   Measure the collector's run time in milliseconds.
metrics_whitelist   Regex list to match metrics to transmit. Mutually exclusive with metrics_blacklist option.

Dependencies

OS

Redhat 6 and 7
CentOS 6 and 7
Amazon Linux (latest)
Ubuntu 12, 14, 15, and 16
Debian 7, 8, and 9

Miscellaneous

Linux Agent
urllib2
Kafka (latest version is ideal)
xml.etree

Metrics

Collected

Due to the sheer volume of Kafka metrics, the individual metrics won't be documented here. Instead, here are some general properties of the groups of metrics:

  • All metrics share the following properties:
    • Statistic: average
    • Min: 0
    • CORR:
    • UTIL:
  • BASE
    • Will be on for most metrics
    • Will be off for metrics matching the following regexes:
      • ^kafka.cluster.*
      • ^kafka\.utils\..*
      • ^kafka\.log\..*(LogEndOffset|LogStartOffset)$
      • ^kafka\.controller\.((?!broker-\d).).*
      • ^kafka\.consumer_group\.((?!consumer_lag$).)*$
  • For metrics that end with percent:
    • Unit = percent
    • Max = 1
  • For metrics that describe bytes per second (those containing Bytes.*PerSec or ending with byte-rate):
    • Unit = Bps (bytes per second)
    • SDS = ReplaceWithZero
  • There are a number of metrics that describe a unit of time:
    • Unit
      • If the metric name ends with Ms.Mean, time-avg, or time-max then the units are ms (milliseconds)
      • If the metric name contains time-ns, then the units are ns (nanoseconds)
      • If the metric name contains time-secs, then the units are seconds
    • SDS = ReplaceWithZero
  • For metrics matching kafka\.(server|network)\..*Metrics\..*\.Count or kafka.zookeeper.zk_packets.*:
    • Type = COUNTER
  • The Zookeeper has_owner metric:
    • Max = 1
  • All Zookeeper latency metrics (zk_.*latency):
    • Unit = ms (milliseconds)
  • All Zookeeper data size metrics (zk_.*data_size):
    • Unit = bytes

Additional Information

Running Kafka in a Docker Container

Getting the Jolokia agent running in a Kafka container requires three additional modifications to the docker run command. To illustrate the modifications needed, we're going to assume that your docker run command looks like this initially:

docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip\`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka

Add the following changes to the initial docker run command to get Jolokia running in a Kafka container:

  • Make the Jolokia JAR file available to the container by passing it in as a mounted volume. As in the instructions above, we recommend that you download the JAR and move it to the /opt/netuitive-agent/ directory.
    -v /opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar
  • Pass in the KAFKA_OPTS environment variable to Kafka so it starts with the Jolokia agent included.
    -e KAFKA_OPTS=-javaagent:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar=port=8778,host=0.0.0.0
  • Expose the Jolokia port.
    -p 8778:8778

Here's what the full command looks like with the three additional portions added:

docker run -p 2181:2181 -p 9092:9092 -p 8778:8778 -e ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` -e ADVERTISED_PORT=9092 -e KAFKA_OPTS=-javaagent:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar=port=8778,host=0.0.0.0 -v /opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar spotify/kafka