hostmetricsreceiver

Host Metrics Receiver

The Host Metrics receiver generates metrics about the host system scraped from various sources and host entity event as log. This is intended to be used when the collector is deployed as an agent.

Status
Stability	development: logs
	beta: metrics
Distributions	core, contrib, k8s
Issues
Code coverage
Code Owners	@dmitryax, @braydonk, @rogercoll

Getting Started

The collection interval, root path, the categories of metrics, and individual metrics to be scraped can be configured:

host_metrics:
  collection_interval: <duration> # default = 1m
  initial_delay: <duration> # default = 1s
  root_path: <string>
  scrapers:
    <scraper1>:
      metrics:
        <metric1>:
          enabled: true
        <metric2>:
          enabled: false
        ...
    <scraper2>:
    ...

Note: While scrapers may support multiple operating systems, individual metrics within a scraper may have different OS support. Refer to each scraper's documentation for metric-level OS compatibility.

The available scrapers are:

Notes

Several scrapers support additional configuration:

Disk

disk:
  <include|exclude>:
    devices: [ <device name>, ... ]
    match_type: <strict|regexp>

File System

filesystem:
  <include_devices|exclude_devices>:
    devices: [ <device name>, ... ]
    match_type: <strict|regexp>
  <include_fs_types|exclude_fs_types>:
    fs_types: [ <filesystem type>, ... ]
    match_type: <strict|regexp>
  <include_mount_points|exclude_mount_points>:
    mount_points: [ <mount point>, ... ]
    match_type: <strict|regexp>
  include_virtual_filesystems: <false|true>

Load

cpu_average specifies whether to divide the average load by the reported number of logical CPUs (default: false).

load:
  cpu_average: <false|true>

Network

network:
  <include|exclude>:
    interfaces: [ <interface name>, ... ]
    match_type: <strict|regexp>

Process

process:
  <include|exclude>:
    names: [ <process name>, ... ]
    match_type: <strict|regexp>
  mute_process_all_errors: <true|false>
  mute_process_name_error: <true|false>
  mute_process_exe_error: <true|false>
  mute_process_io_error: <true|false>
  mute_process_user_error: <true|false>
  mute_process_cgroup_error: <true|false>
  scrape_process_delay: <time>

The following settings are optional:

• mute_process_all_errors (default: false): mute all the errors encountered when trying to read metrics of a process. When this flag is enabled, there is no need to activate any other error suppression flags.
• mute_process_name_error (default: false): mute the error encountered when trying to read a process name the collector does not have permission to read. This flag is ignored when mute_process_all_errors is set to true as all errors are muted.
• mute_process_io_error (default: false): mute the error encountered when trying to read IO metrics of a process the collector does not have permission to read. This flag is ignored when mute_process_all_errors is set to true as all errors are muted.
• mute_process_cgroup_error (default: false): mute the error encountered when trying to read the cgroup of a process the collector does not have permission to read. This flag is ignored when mute_process_all_errors is set to true as all errors are muted.
• mute_process_exe_error (default: false): mute the error encountered when trying to read the executable path of a process the collector does not have permission to read (Linux only). This flag is ignored when mute_process_all_errors is set to true as all errors are muted.
• mute_process_user_error (default: false): mute the error encountered when trying to read a uid which doesn't exist on the system, eg. is owned by a user that only exists in a container. This flag is ignored when mute_process_all_errors is set to true as all errors are muted.

Advanced Configuration

Filtering

If you are only interested in a subset of metrics from a particular source, it is recommended you use this receiver with the Filter Processor.

Different Frequencies

If you would like to scrape some metrics at a different frequency than others, you can configure multiple host_metrics receivers with different collection_interval values. For example:

receivers:
  host_metrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:

  host_metrics/disk:
    collection_interval: 1m
    scrapers:
      disk:
      filesystem:

service:
  pipelines:
    metrics:
      receivers: [host_metrics, host_metrics/disk]

Collecting host metrics from inside a container (Linux only)

Host metrics are collected from the Linux system directories on the filesystem. You likely want to collect metrics about the host system and not the container. This is achievable by following these steps:

1. Bind mount the host filesystem

The simplest configuration is to mount the entire host filesystem when running the container. e.g. docker run -v /:/hostfs ....

You can also choose which parts of the host filesystem to mount, if you know exactly what you'll need. e.g. docker run -v /proc:/hostfs/proc.

2. Configure root_path

Configure root_path so the host_metrics receiver knows where the root filesystem is. Note: if running multiple instances of the host metrics receiver, they must all have the same root_path.

Example:

receivers:
  host_metrics:
    root_path: /hostfs

Resource attributes

Currently, the host_metrics receiver does not set any Resource attributes on the exported metrics. However, if you want to set Resource attributes, you can provide them via environment variables via the resource_detection processor. For example, you can add the following resource attributes to adhere to Resource Semantic Conventions:

export OTEL_RESOURCE_ATTRIBUTES="service.name=<the name of your service>,service.namespace=<the namespace of your service>,service.instance.id=<uuid of the instance>"

Entity Events

Entity Events as logs are experimental and might eventually be replaced by the result of the OTEP. For now, the host_metrics receiver can send the host entity event as a log records. By default, the host_metrics receiver sends periodic EntityState events every 5 minutes. You can change that by setting metadata_collection_interval. Entity Events as logs are experimental. The result of the OTEP might eventually replace that.

Using a Collector not linked to glibc (Linux-specific)

This is a living section for known adverse behaviours within the host_metrics receiver when using a Collector on Linux that is not linked to glibc (which is always the case when using CGO_ENABLED=0 and can be the case when using CGO_ENABLED=1 under specific circumstances).

Unable to resolve usernames when the system relies on NSS for user and group lookups

When a Collector is not linked to glibc, it can only resolve usernames and group names by directly parsing /etc/passwd and /etc/group. It has no awareness of the Name Service Switch (NSS) subsystem, which libc uses to delegate user and group lookups to pluggable backends including nss-systemd, nss-ldap, sssd, winbind, and more. As a result, any user or group whose record is provided exclusively by an NSS module rather than being present in /etc/passwd or /etc/group will fail to resolve, and the receiver will be unable to determine the username/owner of the given process.

This manifests most often as a failure to set the process.owner resource attribute in the process scraper. One concrete example is if a process is run as a systemd service using the DynamicUser feature. A Collector not linked to glibc will be incapable of resolving the username for the owner of that process because the DynamicUser feature leverages NSS (Name Service Switch) to resolve the username dynamically at user lookup time; a Go binary that uses the pure-Go netgo implementation is not NSS-aware and subsequently fails the user lookup.
See more in the demo repo created by host_metrics codeowner @braydonk.