Documentation Grafana Cloud Instrument and send data Fleet Management Introduction Architecture

Fleet Management architecture

Grafana Fleet Management is a Grafana Cloud application that lets you configure, monitor, and manage your collector fleet remotely and at scale. This topic describes the architecture of the Fleet Management service.

Core concepts

Fleet Management is built on the concepts of collectors and configuration pipelines.

Collectors

Collectors are individual Grafana Alloy or OpenTelemetry Collector deployments registered in the Fleet Management service. Upon startup, the registered collector appears in the Fleet Management application, where you can monitor its health and assign configuration remotely.

Grafana Alloy collectors are configured to connect to Fleet Management using a remotecfg block in their local configuration files. The unique collector ID can be set by the id field in the remotecfg block. If not provided, it uses a generated UUID. OpenTelemetry Collector deployments connect to Fleet Management through Open Agent Management Protocol (OpAMP). The unique collector ID is set by the InstanceUid field of the OpAMP client configuration. If unset, the ID is auto-generated. You can configure the OpAMP Supervisor or OpAMP extension as the client, but you must connect with the OpAMP Supervisor if you want to enable remote configuration.

For collectors that can receive remote configuration, health monitoring is enabled by autogenerated self-monitoring pipelines that collect internal metrics and logs, send them to Grafana Cloud, and correlate the telemetry with the correct collector using the collector’s unique ID.

The following is an example of an Alloy remotecfg block. To learn how to fill in the remotecfg block for the Fleet Management service, refer to the set up documentation.

Alloy Copy

remotecfg {
    url = <SERVICE_URL>
    basic_auth {
        username      = <USERNAME>
        password_file = <PASSWORD_FILE>
    }

    id             = constants.hostname
    attributes     = {"cluster" = "dev", "namespace" = "otlp-dev"}
    poll_frequency = "5m"
}

The following is an example of an OpAMP Supervisor configuration file. To learn how to fill in the supervisor-config.yaml file for the Fleet Management service, refer to the set up documentation.

YAML Copy

   server:
     endpoint: "<SERVICE_URL>/v1/opamp"
     headers:
       Authorization: "<BASIC_AUTH>"

   capabilities:
     accepts_remote_config: true
     reports_remote_config: true

   agent:
     executable: "<OTELCOL_EXECUTABLE_PATH>"
     description:
       identifying_attributes:
         collector_name: "<OTEL_COLLECTOR_NAME>"
       non_identifying_attributes:
         environment: "dev"
     args: service.AllowNoPipelines

   storage:
     directory: "<STORAGE_DIRECTORY>"

Configuration pipelines

Configuration pipelines are standalone fragments of configuration that are created and assigned remotely to collectors in Fleet Management. They are composed of a unique name, the components for the collector to load and run, a configuration type, and a list of matchers that match collectors with the pipeline. The configuration type determines which collectors can receive the pipeline and which configuration syntax is used. Grafana Alloy pipelines use Alloy configuration syntax, and OpenTelemetry Collector pipelines use OpenTelemetry Collector YAML configuration.

Each configuration pipeline typically collects telemetry for different signals or from different applications and infrastructure components. You can combine them based on your observability needs to build dynamic and versatile remote configurations. Assign multiple configuration pipelines to a single collector or share one pipeline with multiple collectors in your fleet. For more details about creating pipelines, refer to Configuration pipelines.

The remote configuration assigned to a registered collector, which is made up of one or more configuration pipelines, is determined at runtime. Alloy collectors poll the GetConfig endpoint on the poll_frequency set in their remotecfg blocks. OpenTelemetry Collector deployments poll the Fleet Management OpAMP endpoint according to the default HTTP polling interval of 30 seconds. During each request, Fleet Management looks for configuration pipelines with matching attributes and returns the resulting remote configuration. You can view the remote configuration assigned to a collector in the details view for that collector. On the Inventory tab, click on the collector row to open the details drawer and then switch to the Configuration tab.

Local and remote configurations

Collectors can use local and remote configuration together, but the runtime model depends on the collector type.

Grafana Alloy: Concurrent and isolated configurations

Alloy uses a hybrid configuration model where local configuration files can coexist with dynamic remote configurations. Collectors run the local configuration and remote configuration concurrently.

The component controller is the core part of Alloy that manages configuration components at runtime. The controller is responsible for reading and validating configuration files, managing the lifecycle and reporting the health of components, as well as evaluating the arguments used to configure them.

Local and remote configurations each have their own component controller, which means components loaded by one configuration are isolated from the other. This isolation is designed to protect runtime safety, ensuring that Alloy continues working despite temporary configuration errors or network hiccups. It also expands configuration options for different use cases. Users can mix and match local and remote configuration to fit their specific needs for cost management, operational independence across teams, or regulatory compliance. Refer to Configuration deployment strategies for more information.

Because local and remote configurations are isolated, the following statements are true:

• Local and remote configurations cannot be in conflict. If a pipeline is duplicated in the local and remote configuration, it runs two times, separately and without conflict. The resulting data is written twice to Grafana Cloud. Depending on the type of data, you might see an error log about duplicate data being written.

• Local and remote configurations cannot share components or values between them. But remote configuration pipelines can share components with other remote pipelines by injecting pipeline exports.

• Any remote configuration errors do not impact local configuration. Local configuration errors can prevent Alloy from starting, so local misconfigurations could impact remote configurations.

• Any configuration error on pipeline “A” does not affect how pipeline “B” is behaving. However, if pipelines are merged in a remote configuration file and one pipeline has an error, that configuration file is discarded and Alloy looks for and loads the last known valid configuration. This situation means that an error in one pipeline might cause Alloy to receive an older version of all pipelines that make up the remote configuration.

  For example, a remote configuration contains two pipelines, “A” and “B”. Pipelines “A” and “B” are both updated before the collector next polls for remote configuration, creating v.2 of the remote configuration. The updated pipeline “A” has an error. Alloy cannot run remote configuration v.2 because of this error, so it looks for the last known valid configuration, which is remote configuration v.1, and loads it. Remote configuration v.1 does not contain the updates to pipelines “A” or “B”, even though only “A” contained the error. You can use the pipeline history feature to identify which change caused the error in pipeline “A”.

  If any pipeline in a remote configuration file has an error and there is no previous valid configuration to fall back to, Alloy discards the invalid remote configuration and relies only on its local configuration.

The following sequence diagram shows how Alloy uses separate component controllers to manage local and remote configurations:

Local configuration

One or more local configuration files are passed to the Alloy run command. If Alloy cannot load the local configuration, it fails to start.

Local configuration sets up core Alloy functionality, such as logging and tracing behavior and the HTTP server parameters. You can also form telemetry pipelines by configuring various components.

Local configuration mode is static. To apply local file changes, you must reload the configuration file by sending a SIGHUP signal or calling the /-/reload endpoint.

Remote configuration

Alloy can opt in to remote configuration, but by default, the feature is disabled. You can enable it by adding the remotecfg block to the local configuration of each Alloy instance.

If Alloy has remote configuration enabled, it periodically polls a remote server (in the case of Grafana Cloud, the Fleet Management API) for configuration updates and applies them automatically, without requiring Alloy to reload the configuration.

When you create a configuration pipeline in the Fleet Management application, you must give the pipeline a unique name to prevent conflict between pipelines. The Fleet Management service wraps each pipeline in a declare block, and the declare blocks are marshalled and merged into a single file on the server side, called the server configuration.

When Alloy polls, it receives a single complete remote configuration file to load. The loaded remote configuration is cached in a remotecfg/ directory in Alloy’s storage path. Upon successfully loading the server configuration, Alloy’s actively running remote configuration is known as the effective configuration.

When you make changes to a configuration pipeline, the server configuration changes immediately, but the collector continues to run its current effective configuration until it polls and successfully applies the update.

Alloy handles any errors in remote configuration gracefully. If the server configuration cannot be loaded, Alloy finds and falls back to the last known valid effective configuration to continue operations. If there never was a valid server configuration, none is loaded and Alloy continues to run using only its local configuration.

The remotecfg block and protocol

When the remotecfg block is defined in the local configuration, Alloy opts in to remote configuration. The block defines the URL of the Fleet Management instance, authentication credentials, the polling interval, as well as a set of local attributes that characterize the Alloy instance, such as cluster, namespace, team, or owner. In addition, the remotecfg block can configure additional authorization parameters, TLS settings, proxying behavior and more. Refer to the remotecfg documentation for the full list of configuration options.

The Grafana Alloy Remote Config API, which is used for client-server communication to manage and fetch remote configuration, is available under the Apache 2.0 license.

OpenTelemetry Collector: Merged configuration

OpenTelemetry Collector deployments can use local configuration together with remote configuration when they are managed by the OpAMP Supervisor. The Supervisor starts and manages the OpenTelemetry Collector process and passes configuration to the Collector.

Fleet Management builds remote configuration

Fleet Management sends remote configuration to the Supervisor as OpenTelemetry Collector YAML. The remote configuration is made up of the configuration pipelines that match the Collector’s attributes. Fleet Management sorts matched pipelines by name in ascending alphabetical order, then builds a single remote configuration.

Fleet Management uses the following naming and precedence rules when building the remote configuration:

• Signal-only keys under the OpenTelemetry service.pipelines block become named keys. For example, if a Fleet Management pipeline named host_metrics defines the service pipeline key metrics, Fleet Management converts it to metrics/host_metrics.
• Already-named service pipeline keys are suffixed with the Fleet Management pipeline name. For example, if a Fleet Management pipeline named mysql defines metrics/custom, Fleet Management converts it to metrics/custom-mysql.
• If pipeline names collide, Fleet Management appends a numeric suffix to keep the generated keys unique.
• Fleet Management updates any hard-coded pipeline references in connector settings to match the converted pipeline name. For example, if two merged configuration pipelines both define a service.pipelines called logs/out, Fleet Management renames them to logs/out-a and logs/out-b to avoid a clash. A connector might also send data to logs/out as a hard-coded setting in its configuration, so Fleet Management rewrites the config to point to logs/out-a or logs/out-b depending on which merged config is the intended pipeline.
• Fleet Management may also rename components. When it does, it also updates all pipeline references to use the new component name. For example, if the host_metrics Fleet Management pipeline defines an otlp receiver and Fleet Management renames that receiver to otlp/host_metrics, any receivers: [otlp] reference in any service pipeline is rewritten to receivers: [otlp/host_metrics].
• When different component definitions use the same original key, Fleet Management keeps them as separate suffixed components instead of overwriting one with another. For example, if the host_metrics and mysql pipelines both define an otlp receiver with different settings, Fleet Management keeps both receivers by naming them otlp/host_metrics and otlp/mysql and renaming their original references accordingly.
• Fleet Management can reuse identical shared component definitions across matched OpenTelemetry pipelines. If two pipelines define the exact same receiver, processor, exporter, or extension, Fleet Management emits that component definition only once and rewrites both pipelines to reference it. Connectors are excluded from this behavior, because they define routing between service pipelines. Fleet Management keeps connector definitions from separate Fleet Management pipelines distinct, even when their contents match. Component reuse can make individual pipelines safer to deploy, for example, by avoiding default port conflicts from multiple OTLP receivers.
• The service.telemetry block uses last-wins behavior. When multiple matched pipelines define the same service.telemetry keys, values from the Fleet Management pipeline name that appears last in the alphabetical list take precedence.

The Supervisor combines configurations

After Fleet Management builds the remote configuration, the Supervisor combines it with the Collector’s local configuration. In the Supervisor configuration, the agent.config_files setting defines the configuration files used to build the Collector’s effective configuration. These files can include local configuration files and special Supervisor-managed files, such as the remote configuration received from Fleet Management. To receive remote configuration, the Supervisor must have the accepts_remote_config capability enabled.

This model is different from Alloy’s isolated local and remote component controllers. For OpenTelemetry Collector deployments, the Supervisor combines local and remote configuration into a single Collector configuration. The Supervisor merges configuration key by key, recursively, using the same configuration merge behavior as the OpenTelemetry Collector. The remote configuration build rules described in the previous section ensure that pipelines don’t collide and can run in parallel.

Configuration rollout

When you make changes to or create a new configuration pipeline, Fleet Management updates the remote configuration available to the Supervisor. The Collector continues to run its current configuration until the Supervisor receives the update. When the Supervisor receives new remote configuration, it merges the remote configuration with any local configuration files, writes the resulting Collector configuration, and restarts or updates the Collector process according to the Supervisor lifecycle. The Supervisor reports the result back to Fleet Management using OpAMP remote configuration status messages, such as applying, applied, or failed. If the resulting configuration can’t be applied, the Supervisor reports the failure with an error message to Fleet Management. The Supervisor doesn’t automatically roll back to the last working remote configuration, so the Collector might remain in a failed state until Fleet Management sends a corrected configuration.

OpAMP protocol

Fleet Management exposes an OpAMP endpoint at /v1/opamp for OpenTelemetry Collector deployments. The endpoint uses OpAMP over HTTP with protobuf-encoded AgentToServer and ServerToAgent messages.

Fleet Management supports OpAMP capabilities for status reporting and remote configuration. During each OpAMP request, Fleet Management can register or update the Collector from the reported agent description, record heartbeat and remote configuration status, and return new remote configuration when the Collector supports it and the configuration has changed.

The OpAMP Agent Extension can report status to Fleet Management, but the OpAMP Supervisor is required to receive and apply remote configuration.

Service components

Fleet Management is deployed as a standard cloud-native service. The core APIs are deployed in a highly available setup and use hosted database instances for durable storage as well as an in-memory cluster for caching.

Collector and Pipeline APIs

Fleet Management exposes the Collector HTTP API and Pipeline HTTP API so you can perform CRUD operations on collectors and configuration pipelines. Calls to the APIs can come from several different sources:

• Alloy collectors polling for configuration
• Grafana Terraform provider
• Other GitOps workflows
• Users accessing the Fleet Management application in Grafana Cloud
• Users making direct calls

OpAMP endpoint

Fleet Management exposes the /v1/opamp endpoint for OpenTelemetry Collector deployments that use OpAMP. OpenTelemetry Collectors use this endpoint to poll for remote configuration and report status. When an OpenTelemetry Collector connects to Fleet Management through the OpAMP endpoint, Fleet Management uses metadata reported by the collector, such as its service name and resource attributes, to create or update the collector record shown in Inventory and keep its local attributes available for pipeline matching. You can also use the Collector HTTP API to create and manage collector records for both Alloy and OpenTelemetry Collectors, including listing, updating, disabling, or deleting collectors.

Grafana Cloud gateway

The Grafana Cloud gateway handles authentication and routing within your stack to prevent unauthorized access. All Fleet Management API and OpAMP traffic passes through this gateway.

Storage

All data used to manage your fleet, including collector attributes and configuration pipelines, is durably stored in the Fleet Management database. Telemetry collected by your fleet is stored in your designated backend, such as Grafana Cloud.