Monitor Confluent Platform Kafka Streams Applications with Unified Stream Manager
With Unified Stream Manager (USM), you can monitor Kafka Streams application metrics for your Confluent Platform clusters in the Confluent Cloud Console.
Kafka Streams applications push observability data directly to the Apache Kafka® brokers using the KIP-714 client metrics push mechanism, the same Kafka standard used for producer and consumer monitoring. The USM Agent, which runs in your on-premises environment, collects this data and visualizes it in Confluent Cloud alongside your other cluster metrics.
When you enable client monitoring on your Confluent Platform cluster, the brokers collect Kafka Streams application metrics in addition to producer and consumer metrics. You do not need to configure Kafka Streams metrics separately.
Use Kafka Streams monitoring to track the health and performance of your applications, including throughput, consumer lag, end-to-end latency, and the state of individual stream threads. For the complete list of Kafka Streams metrics and their definitions, see Kafka Streams Metrics.
Supported versions
Kafka Streams application monitoring is available for Confluent Platform clusters running KRaft mode on the following versions:
Confluent Platform 8.2.1 or later within the 8.2.x series
Confluent Platform 8.3.0 or later
Earlier Confluent Platform versions support producer and consumer client monitoring, but not Kafka Streams application monitoring in the Confluent Cloud Console.
Before you begin
Complete the following before monitoring Kafka Streams applications in the Confluent Cloud Console:
Ensure your Confluent Platform cluster meets the version requirements in Supported versions and is running in KRaft mode.
Ensure your Kafka Streams applications use a Kafka Streams Java library based on Kafka 4.2.0 or later. USM identifies active Kafka Streams applications using the application ID label, which Kafka adds starting in version 4.2.0. Applications that use earlier versions do not appear in the monitoring views.
Enable client monitoring on your Confluent Platform cluster. Enabling client monitoring also collects Kafka Streams application metrics. For configuration steps using Confluent for Kubernetes (CFK), Confluent Ansible, or manual broker configuration, see Configure Kafka Streams monitoring.
View Kafka Streams applications in the Confluent Cloud Console
Follow these steps to navigate to the Kafka Streams page for your Confluent Platform cluster in the Confluent Cloud Console.
In the Confluent Cloud Console, navigate to the Environments page and click your environment.
In the navigation menu, click Clusters, then click your Confluent Platform cluster.
In the cluster’s navigation menu, click Kafka Streams.
What the Kafka Streams page displays depends on whether any Kafka Streams applications are connected to the cluster. Metric collection can take a few minutes after an application starts.
Create your first Kafka Streams application
If no Kafka Streams applications are connected to the cluster, the Kafka Streams page shows a getting-started view that helps you create your first application.
The view includes the following actions:
Get started: Opens the documentation for building and running a Kafka Streams application.
Create new app in VS Code: Uses Confluent for VS Code to scaffold, configure, and run a new Kafka Streams application against your cluster. For a step-by-step walkthrough, see Kafka Streams Quick Start.
Learn about Kafka Streams in Confluent Platform: Opens the Kafka Streams for Confluent Platform documentation.
If your Confluent Platform cluster is running a version earlier than 8.2.1, the page shows an upgrade prompt instead of the getting-started view.
Monitor all Kafka Streams applications
The Kafka Streams page lists every Kafka Streams application connected to the cluster, with metrics aggregated across all applications in the header.
The header shows the following aggregate metrics:
Total applications: The number of Kafka Streams applications connected to the cluster.
Running threads: The total number of stream threads running across all applications.
Total production (last min): The total volume of data the applications produced in the last minute.
Total consumption (last min): The total volume of data the applications consumed in the last minute.
The applications table shows the following columns:
Application: The name of the Kafka Streams application. Click the name to open the application’s detail page.
Client version: The Kafka client library and version the application uses, for example,
apache-kafka-java-4.2.0.Status: The current state of the application, for example, Running.
Threads: The number of stream threads running in the application.
Total production (last min): The volume of data the application produced in the last minute.
Total consumption (last min): The volume of data the application consumed in the last minute.
Total lag: The total consumer lag for the application.
To find a specific application, enter a full or partial application name in the Search box. The table filters as you type.
Monitor a Kafka Streams application
On the Kafka Streams page, click an application name to open its detail page. The detail page shows the application name, its Status, and its client version, followed by header metrics and performance charts.
The header shows the following metrics:
Running threads: The number of stream threads running in the application.
Total production (last min): The volume of data the application produced in the last minute.
Total consumption (last min): The volume of data the application consumed in the last minute.
Size of memtables: The approximate size of the active, unflushed immutable, and pinned immutable memtables.
Estimated number of keys: The estimated number of keys in the active and unflushed immutable memtables and storage.
Block cache usage: The memory size of the entries residing in the block cache.
The detail page includes the following charts. Use the Time selector on each chart to switch between the last hour and the last six hours.
End-to-end latency (ms): The maximum, average, and minimum end-to-end latency of a record, measured by comparing the record timestamp with the system time when the record has been fully processed.
CPU ratio: The fraction of time the application’s threads spend on each operation, broken down into Commit, Poll, Process, and Punctuate.
A metric with no data appears as a dash (-) rather than a zero.
Note
Metrics for the last minute apply a short padding to ensure data completeness, so the most recent activity can take a few minutes to appear.
Monitor stream threads
The application detail page lists the stream threads the application runs. The threads table shows the following columns:
Thread: The name of the stream thread, for example,
rocksdb-example-StreamThread-1.Process ID: The identifier of the process the thread belongs to.
Status: The current state of the thread, for example, Running.
Average records processed (last min): The average number of records the thread processed in the last minute.
To inspect a single thread, click its name in the Thread column. The Thread panel opens and shows the following:
Client ID: The client identifier for the thread, which you can copy.
End-to-end latency (ms): The end-to-end latency for the thread, using the same time selector as the application-level chart.
CPU ratio: The commit, poll, process, and punctuate ratios for the thread.