Telemetry

Gather relevant insights about your application and modules with custom metrics and telemetry.

Overview

The telemetry package provides observability tooling for Cosmos SDK applications using OpenTelemetry. It offers a unified initialization point for traces, metrics, and logs via the OpenTelemetry declarative configuration API. This package:

Initializes OpenTelemetry SDK using YAML configuration files
Provides backward compatibility with Cosmos SDK’s legacy go-metrics wrapper API
Includes built-in instrumentation for host, runtime, and disk I/O metrics

Quick Start

1. Start a Local Telemetry Backend

docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

2. Create Configuration File

Create an otel.yaml file:

file_format: "1.0-rc.3"
resource:
  attributes:
    - name: service.name
      value: my-cosmos-app

tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

meter_provider:
  readers:
    - pull:
        exporter:
          prometheus/development:
            host: 0.0.0.0
            port: 9464

logger_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

extensions:
  instruments:
    host: {}
    runtime: {}
    diskio: {}
  propagators:
    - tracecontext

3. Initialize Telemetry

Option A: Environment Variable (Recommended) Set OTEL_EXPERIMENTAL_CONFIG_FILE to your config path. This initializes the SDK before any meters/tracers are created, avoiding atomic load overhead.

export OTEL_EXPERIMENTAL_CONFIG_FILE=/path/to/otel.yaml

Option B: Node Config Directory An empty otel.yaml will now be generated in ~/.<node_home>/config/. Place the desired configuration in otel.yaml. Option C: Programmatic Initialization The SDK will first attempt to initialize via env var, then using the config in the node’s home directory. You may optionally initialize telemetry yourself using the telemetry.InitializeOpenTelemetry function:

err := telemetry.InitializeOpenTelemetry("/path/to/otel.yaml")
if err != nil {
    log.Fatal(err)
}
defer telemetry.Shutdown(context.Background())

Configuration

OpenTelemetry Configuration

The package uses the OpenTelemetry declarative configuration spec. Key sections:

Section	Purpose
`resource`	Service identity and attributes
`tracer_provider`	Trace export configuration
`meter_provider`	Metrics export configuration
`logger_provider`	Log export configuration

For examples containing available options, see the OpenTelemetry configuration examples.

Extensions

The extensions section of the otel.yaml configuration file provides additional features not yet supported by the standard otelconf:

extensions:
  # Optional file-based exporters
  trace_file: "/path/to/traces.json"
  metrics_file: "/path/to/metrics.json"
  metrics_file_interval: "10s"
  logs_file: "/path/to/logs.json"

  # Custom instrumentation additions
  instruments:
    host: {}
    runtime: {}
    diskio:
      disable_virtual_device_filter: true # removes the automatic filtering of virtual disks. Operating systems such as Linux typically add virtual disks, which can add duplication to disk io data. These disks usually take the form of loopback, RAID, partitions, etc.

  # Trace context propagation
  propagators:
    - tracecontext
    - baggage
    - b3
    - jaeger

Custom Instruments

Host Instrumentation (`host`)

Reports host-level metrics using go.opentelemetry.io/contrib/instrumentation/host:

CPU usage
Memory usage
Network I/O

extensions:
  instruments:
    host: {}

Runtime Instrumentation (`runtime`)

Reports Go runtime metrics using go.opentelemetry.io/contrib/instrumentation/runtime:

Goroutine count
GC statistics
Memory allocations

extensions:
  instruments:
    runtime: {}

Disk I/O Instrumentation (`diskio`)

Reports disk I/O metrics using gopsutil:

Metric	Description
`system.disk.io`	Bytes read/written
`system.disk.operations`	Read/write operation counts
`system.disk.io_time`	Time spent on I/O operations
`system.disk.operation_time`	Time per read/write operation
`system.disk.merged`	Merged read/write operations

extensions:
  instruments:
    diskio: {}
    # Or with options:
    diskio:
      disable_virtual_device_filter: true  # Include loopback, RAID, partitions on Linux

By default, virtual devices (loopback, RAID, partitions) are filtered out on Linux to avoid double-counting I/O.

Propagators

Configure trace context propagation for distributed tracing:

Propagator	Description
`tracecontext`	W3C Trace Context (default)
`baggage`	W3C Baggage
`b3`	Zipkin B3 single header
`b3multi`	Zipkin B3 multi-header
`jaeger`	Jaeger propagation

Developer Usage

Using Meters and Tracers

After initialization, use standard OpenTelemetry APIs:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric"
)

var (
    tracer = otel.Tracer("my-package")
    meter  = otel.Meter("my-package")

    myCounter metric.Int64Counter
)

func init() {
   var err error
   myCounter, err = meter.Int64Counter("my.counter")
   if err != nil {
	   panic(err)
   }
}

func MyFunction(ctx context.Context) error {
    ctx, span := tracer.Start(ctx, "MyFunction")
    defer span.End()

    myCounter.Add(ctx, 1)

    // ... your code
    return nil
}

Shutdown

Always call Shutdown() when the application exits:

func (a *App) Close() {
    telemetry.Shutdown(ctx)
}

Legacy API (Deprecated)

The package provides backward-compatible wrappers for github.com/hashicorp/go-metrics. These are deprecated and users should migrate to OpenTelemetry APIs directly.

OpenTelemetry Bridge

Cosmos SDK v0.54.0+ provides a bridge to send existing go-metrics to the meter provider defined in your OpenTelemetry config. To bridge your metrics, set the metrics-sink in app.toml to “otel”.

###############################################################################
###                         Telemetry Configuration                         ###
###############################################################################
[telemetry]

# other fields...

metrics-sink = "otel"

Legacy Configuration

cfg := telemetry.Config{
    ServiceName:            "my-service",
    Enabled:                true,
    EnableHostname:         true,
    EnableHostnameLabel:    true,
    EnableServiceLabel:     true,
    PrometheusRetentionTime: 60,  // seconds
    GlobalLabels:           [][]string{{"chain_id", "cosmoshub-1"}},
    MetricsSink:            "otel",  // "mem", "statsd", "dogstatsd", "otel"
    StatsdAddr:             "localhost:8125",
}

m, err := telemetry.New(cfg)

Legacy Metrics Functions

All are deprecated; prefer OpenTelemetry:

// Counters
telemetry.IncrCounter(1.0, "tx", "count")
telemetry.IncrCounterWithLabels([]string{"tx", "count"}, 1.0, labels)

// Gauges
telemetry.SetGauge(42.0, "mempool", "size")
telemetry.SetGaugeWithLabels([]string{"mempool", "size"}, 42.0, labels)

// Timing
start := telemetry.Now()
// ... operation
telemetry.MeasureSince(start, "tx", "process_time")

// Module-specific helpers
telemetry.ModuleMeasureSince("bank", start, "send", "time")
telemetry.ModuleSetGauge("bank", 100.0, "balance", "total")

Metrics Sink Types

Sink	Description
`mem`	In-memory sink with SIGUSR1 dump support (default)
`statsd`	StatsD protocol
`dogstatsd`	Datadog DogStatsD
`otel`	OpenTelemetry (bridges to configured MeterProvider)

Best Practices

Use environment variable initialization for production to avoid atomic load overhead
Always call Shutdown() to ensure metrics/traces are flushed
Thread context.Context properly for correct span correlation

Viewing Telemetry Data

With Grafana LGTM running:

Open http://localhost:3000
Use the Drilldown views to explore:
- Traces: Distributed trace visualization
- Metrics: Query and dashboard metrics
- Logs: Structured log search

Cosmos SDK Metrics

The following metrics are emitted from the Cosmos SDK.

Metric	Description	Unit	Type
`tx_count`	Total number of txs processed via `DeliverTx`	tx	counter
`tx_successful`	Total number of successful txs processed via `DeliverTx`	tx	counter
`tx_failed`	Total number of failed txs processed via `DeliverTx`	tx	counter
`tx_gas_used`	The total amount of gas used by a tx	gas	gauge
`tx_gas_wanted`	The total amount of gas requested by a tx	gas	gauge
`tx_msg_send`	The total amount of tokens sent in a `MsgSend` (per denom)	token	gauge
`tx_msg_withdraw_reward`	The total amount of tokens withdrawn in a `MsgWithdrawDelegatorReward` (per denom)	token	gauge
`tx_msg_withdraw_commission`	The total amount of tokens withdrawn in a `MsgWithdrawValidatorCommission` (per denom)	token	gauge
`tx_msg_delegate`	The total amount of tokens delegated in a `MsgDelegate`	token	gauge
`tx_msg_begin_unbonding`	The total amount of tokens undelegated in a `MsgUndelegate`	token	gauge
`tx_msg_begin_begin_redelegate`	The total amount of tokens redelegated in a `MsgBeginRedelegate`	token	gauge
`tx_msg_ibc_transfer`	The total amount of tokens transferred via IBC in a `MsgTransfer` (source or sink chain)	token	gauge
`ibc_transfer_packet_receive`	The total amount of tokens received in a `FungibleTokenPacketData` (source or sink chain)	token	gauge
`new_account`	Total number of new accounts created	account	counter
`gov_proposal`	Total number of governance proposals	proposal	counter
`gov_vote`	Total number of governance votes for a proposal	vote	counter
`gov_deposit`	Total number of governance deposits for a proposal	deposit	counter
`staking_delegate`	Total number of delegations	delegation	counter
`staking_undelegate`	Total number of undelegations	undelegation	counter
`staking_redelegate`	Total number of redelegations	redelegation	counter
`ibc_transfer_send`	Total number of IBC transfers sent from a chain (source or sink)	transfer	counter
`ibc_transfer_receive`	Total number of IBC transfers received to a chain (source or sink)	transfer	counter
`ibc_client_create`	Total number of clients created	create	counter
`ibc_client_update`	Total number of client updates	update	counter
`ibc_client_upgrade`	Total number of client upgrades	upgrade	counter
`ibc_client_misbehaviour`	Total number of client misbehaviors	misbehaviour	counter
`ibc_connection_open-init`	Total number of connection `OpenInit` handshakes	handshake	counter
`ibc_connection_open-try`	Total number of connection `OpenTry` handshakes	handshake	counter
`ibc_connection_open-ack`	Total number of connection `OpenAck` handshakes	handshake	counter
`ibc_connection_open-confirm`	Total number of connection `OpenConfirm` handshakes	handshake	counter
`ibc_channel_open-init`	Total number of channel `OpenInit` handshakes	handshake	counter
`ibc_channel_open-try`	Total number of channel `OpenTry` handshakes	handshake	counter
`ibc_channel_open-ack`	Total number of channel `OpenAck` handshakes	handshake	counter
`ibc_channel_open-confirm`	Total number of channel `OpenConfirm` handshakes	handshake	counter
`ibc_channel_close-init`	Total number of channel `CloseInit` handshakes	handshake	counter
`ibc_channel_close-confirm`	Total number of channel `CloseConfirm` handshakes	handshake	counter
`tx_msg_ibc_recv_packet`	Total number of IBC packets received	packet	counter
`tx_msg_ibc_acknowledge_packet`	Total number of IBC packets acknowledged	acknowledgement	counter
`ibc_timeout_packet`	Total number of IBC timeout packets	timeout	counter
`store_iavl_get`	Duration of an IAVL `Store#Get` call	ms	summary
`store_iavl_set`	Duration of an IAVL `Store#Set` call	ms	summary
`store_iavl_has`	Duration of an IAVL `Store#Has` call	ms	summary
`store_iavl_delete`	Duration of an IAVL `Store#Delete` call	ms	summary
`store_iavl_commit`	Duration of an IAVL `Store#Commit` call	ms	summary
`store_iavl_query`	Duration of an IAVL `Store#Query` call	ms	summary

Learn

Building a Chain

Building a Module

Run a Node

How-to Guides

v0.54 Upgrade

Tooling

Reference

Security

Overview

Quick Start

1. Start a Local Telemetry Backend

2. Create Configuration File

3. Initialize Telemetry

Configuration

OpenTelemetry Configuration

Extensions

Custom Instruments

Host Instrumentation (`host`)

Runtime Instrumentation (`runtime`)

Disk I/O Instrumentation (`diskio`)

Propagators

Developer Usage

Using Meters and Tracers

Shutdown

Legacy API (Deprecated)

OpenTelemetry Bridge

Legacy Configuration

Legacy Metrics Functions

Metrics Sink Types

Best Practices

Viewing Telemetry Data

Cosmos SDK Metrics

Learn

Building a Chain

Building a Module

Run a Node

How-to Guides

v0.54 Upgrade

Tooling

Reference

Security

​Overview

​Quick Start

​1. Start a Local Telemetry Backend

​2. Create Configuration File

​3. Initialize Telemetry

​Configuration

​OpenTelemetry Configuration

​Extensions

​Custom Instruments

​Host Instrumentation (host)

​Runtime Instrumentation (runtime)

​Disk I/O Instrumentation (diskio)

​Propagators

​Developer Usage

​Using Meters and Tracers

​Shutdown

​Legacy API (Deprecated)

​OpenTelemetry Bridge

​Legacy Configuration

​Legacy Metrics Functions

​Metrics Sink Types

​Best Practices

​Viewing Telemetry Data

​Related Documentation

​Cosmos SDK Metrics

Overview

Quick Start

1. Start a Local Telemetry Backend

2. Create Configuration File

3. Initialize Telemetry

Configuration

OpenTelemetry Configuration

Extensions

Custom Instruments

Host Instrumentation (`host`)

Runtime Instrumentation (`runtime`)

Disk I/O Instrumentation (`diskio`)

Propagators

Developer Usage

Using Meters and Tracers

Shutdown

Legacy API (Deprecated)

OpenTelemetry Bridge

Legacy Configuration

Legacy Metrics Functions

Metrics Sink Types

Best Practices

Viewing Telemetry Data

Related Documentation

Cosmos SDK Metrics