lookupprocessor

Lookup Processor

The lookup processor enriches telemetry signals by performing external lookups to retrieve additional data. It evaluates an OTTL value expression to extract a lookup key, queries a lookup source, and writes the results as new attributes.

Status
Stability	development: logs, traces, metrics
Distributions	[]
Issues
Code coverage
Code Owners	@jsvd, @dehaansa, @VihasMakwana

Configuration

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

Full Configuration

Lookup Configuration

Each entry in lookups defines a lookup rule:

The key field supports any OTTL value expression, including paths across contexts and converters. The path prefix depends on the signal type:

Resource attributes use resource.attributes["..."] for all signals. A context prefix is always required; bare attributes["..."] paths are rejected at configuration time.

Examples:

• attributes["user.id"] - record attribute (works for all signals)
• log.attributes["user.id"] - log record attribute (logs only)
• span.attributes["user.id"] - span attribute (traces only)
• datapoint.attributes["host.id"] - datapoint attribute (metrics only)
• resource.attributes["service.name"] - resource attribute (all signals)
• Trim(attributes["raw.id"]) - apply a converter

Attribute Mapping

Each entry in attributes defines where to write a lookup result:

Examples

Scalar Lookup (1:1)

When the source returns a single value per key, leave the source field empty:

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

Map Lookup (1:N)

When the source returns a map of fields per key, use the source field to extract individual values:

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/user-details.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - source: name
            destination: user.name
            default: "Unknown"
          - source: email
            destination: user.email
          - source: role
            destination: user.role
            context: resource

OTTL Converter on Key

The key field supports OTTL converters for transforming the lookup key before querying the source:

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: Trim(log.attributes["raw.id"])
        attributes:
          - destination: display.name

Context

• record: Write to the signal's record-level attributes (default). This maps to log record attributes for logs, span attributes for traces, and datapoint attributes for metrics. For metrics, lookups are evaluated for each datapoint across all metric types (Gauge, Sum, Histogram, ExponentialHistogram, Summary).
• resource: Write to resource attributes.

The context field on a key sets the default for all its destination attributes. Each attribute mapping can override this with its own context field.

Lookups are evaluated per record. When writing to resource attributes, later records in the same resource may overwrite values written by earlier records.

Built-in Sources

• noop - No-operation source for testing
• yaml - Key-value mappings from YAML files
• dns - DNS lookups with caching

Caching

Sources that support external lookups (like DNS) can use the built-in LRU caching system to reduce latency and external queries. The cache uses a doubly-linked list with a hash map for O(1) lookups, insertions, and evictions.

Cache Configuration

Cache Performance

Run with go test -bench=BenchmarkCache -run=^$ ./lookupsource/

Single-threaded (Apple M4 Pro):

Concurrent (12 goroutines):

Using Cache in Custom Sources

Custom sources can use the cache by wrapping their lookup function:

func createSource(ctx context.Context, settings lookupsource.CreateSettings, cfg lookupsource.SourceConfig) (lookupsource.Source, error) {
    myCfg := cfg.(*Config)

    // Create base lookup function
    lookupFn := func(ctx context.Context, key string) (any, bool, error) {
        // Perform actual lookup
        return result, true, nil
    }

    // Wrap with cache if enabled
    if myCfg.Cache.Enabled {
        cache := lookupsource.NewCache(myCfg.Cache)
        lookupFn = lookupsource.WrapWithCache(cache, lookupFn)
    }

    return lookupsource.NewSource(lookupFn, ...), nil
}

Custom Sources

Custom lookup sources can be added using WithSources:

import (
    "github.com/open-telemetry/opentelemetry-collector-contrib/processor/lookupprocessor"
    "github.com/example/httplookup"
)

factories.Processors[lookupprocessor.Type] = lookupprocessor.NewFactoryWithOptions(
    lookupprocessor.WithSources(httplookup.NewFactory()),
)

Source contract

• Concurrency: Lookup is called concurrently from multiple goroutines. Implementations must be safe for concurrent use.
• Keys are strings: The OTTL expression result is converted to a string before calling Lookup.
• Return values: For scalar (1:1) lookups, return any single value. For map (1:N) lookups, return map[string]any. Values are written to attributes via pcommon.Value.FromRaw. Unsupported types are stringified via fmt.Sprintf.
• Errors are non-fatal: When Lookup returns an error the processor logs it at Debug level and skips the lookup. It does not fail the batch.
• Lifecycle: Start is called once before any Lookup; Shutdown is called once after all processing stops. Both are optional (pass nil to NewSource).
• Config tags: Source config structs must use mapstructure struct tags. The processor decodes source configuration from a raw map using mapstructure.

Implementing a Source

package mysource

import (
    "context"
    "errors"
    "time"

    "github.com/open-telemetry/opentelemetry-collector-contrib/processor/lookupprocessor/lookupsource"
)

type Config struct {
    Endpoint string        `mapstructure:"endpoint"`
    Timeout  time.Duration `mapstructure:"timeout"`
}

func (c *Config) Validate() error {
    if c.Endpoint == "" {
        return errors.New("endpoint is required")
    }
    return nil
}

func NewFactory() lookupsource.SourceFactory {
    return lookupsource.NewSourceFactory(
        "mysource",
        func() lookupsource.SourceConfig {
            return &Config{Timeout: 5 * time.Second}
        },
        createSource,
    )
}

func createSource(
    ctx context.Context,
    settings lookupsource.CreateSettings,
    cfg lookupsource.SourceConfig,
) (lookupsource.Source, error) {
    c := cfg.(*Config)

    return lookupsource.NewSource(
        func(ctx context.Context, key string) (any, bool, error) {
            // Perform lookup - return (value, found, error)
            return "result", true, nil
        },
        func() string { return "mysource" },
        nil, // start function (optional)
        nil, // shutdown function (optional)
    ), nil
}

Benchmarks

Run benchmarks with:

make benchmark

Processor Performance

Measures the full processing pipeline including OTTL key evaluation, source lookup, value conversion, and attribute writes. Uses noop source to isolate processor overhead from source implementation (Apple M4 Pro):

YAML Source Performance

Measures only the source lookup operation (map access), isolated from processor overhead: