celestia-node metrics

This tutorial is for running metrics for your celestia-node data availability instance. This tutorial will focus on running metrics for a light node.

This tutorial assumes you have already setup your light node by following the tutorial in the light node guide.

Running metrics flags

You can enable the celestia-node metric flags with the following command:

celestia <node-type> start --metrics.tls=<boolean> \
    --metrics --metrics.endpoint <URI> \
    --p2p.network <network> \
    --core.ip <URI> --core.port <port>

Add metrics flags to your node start command and restart your node to apply it. The metrics endpoint will gather your node's data to track your uptime.

Note that the --metrics flag enables metrics and expects an input into --metrics.endpoint.

We will go over what the endpoint will need to be in the metrics endpoint design considerations section.

Mainnet Beta

Here is an example for Mainnet Beta:

celestia <node-type> start --metrics.tls=true \
    --metrics --metrics.endpoint otel.celestia.observer \
    --core.ip <URI> --core.port <port>

Mocha testnet

Here is an example for Mocha testnet:

celestia <node-type> start --metrics.tls=true \
    --metrics --metrics.endpoint otel.mocha.celestia.observer \
    --core.ip <URI> --core.port <port> --p2p.network mocha

TLS connections

The --metrics.tls flag enables or disables a TLS connection to the OpenTelemetry Protocol metrics backend. You need to choose a boolean value (true or false) for this flag.

It's also common to set this flag to false when spinning up a local collector to check the metrics locally.

However, if the collector is hosted in the cloud as a separate entity (like in a DevOps environment), enabling TLS is a necessity for secure communication.

Here are examples of how to use it:

# To enable TLS connection
celestia <node-type> start --metrics.tls=true --metrics \
    --metrics.endpoint <URI> \
    --p2p.network <network> --core.ip <URI> --core.port <port>
 
# To disable TLS connection
celestia <node-type> start --metrics.tls=false --metrics \
    --metrics.endpoint <URI> \
    --p2p.network <network> --core.ip <URI> --core.port <port>

Key metrics reference

Node metrics come from celestia-node and dependencies such as go-header. Exact panel names on community Grafana dashboards may differ from the OpenTelemetry metric names below. Prefer the metric names from the codebase when wiring alerts.

Overview and sync health

Source: header store metrics, header sync metrics, and DAS metrics.

Metric	Description	Acceptable value
`hdr_store_head_height_gauge`	Local header store head height (subjective height of stored headers).	Near network tip for a synced node.
`hdr_store_tail_height_gauge`	Lowest header height retained in the local store.	Depends on pruning / retention; should not unexpectedly jump up.
`hdr_sync_subjective_head_gauge`	Syncer’s subjective head height (what the node treats as latest).	At or very near network tip when synced. Can look ahead of a lagging RPC/core peer.
`hdr_sync_subjective_head_lag_seconds`	Seconds between a header’s BFT timestamp and when syncer adopted it as subjective head.	Low and stable; sustained growth suggests sync or peer issues.
`das_network_head`	Most recent network head observed by the DAS’er.	Near chain tip.
`das_sampled_chain_head`	Highest height through which continuous DAS has succeeded.	Should catch up to `das_network_head` over time.
`das_sampling_head_lag`	`max(network_head - sampled_chain_head, 0)` in headers.	Prefer `0` when fully caught up; see DAS sampling head lag.
`das_busy_workers_amount`	Active parallel DAS workers (labeled by job type).	Non-zero while sampling backlog exists; idle when caught up.
`das_sampled_headers_counter`	Count of recorded sampling attempts, including failures (`failed=true`).	The `failed=false` series should increase when sampling succeeds.
`das_latest_sampled_ts`	Unix timestamp of the latest recorded sampling attempt, including failures.	Recent while sampling is active; freshness alone does not show success.

Failed attempts can keep both metrics advancing. Use the failed=false series of das_sampled_headers_counter to track successes, and das_sampled_chain_head or das_sampling_head_lag to check catch-up.

Peers and ShareX (shrex)

Source: ShareX metrics.

Metric	Description	Acceptable value
`shrex_client_total_requests`	Client ShareX requests, labeled by protocol and status.	Mix of `success`; spikes of `not_found` / errors warrant investigation.
`shrex_client_request_duration`	Client ShareX request latency (seconds).	Stable; rising p99 under load may indicate peer or network pressure.
`shrex_server_total_responses`	Bridge-node ShareX responses sent, labeled by protocol and status.	Increases on bridge nodes serving peers.
`shrex_server_request_duration`	Bridge-node ShareX handler latency (seconds).	Stable under expected peer load.

Peer count panels on operator dashboards (full vs archival DA peers) are typically derived from libp2p / discovery metrics bridged into OTEL. Aim for > 0 full peers on a light node that needs to fetch shares; bridge nodes should also retain healthy peer connectivity.

EDS store and caches

Source: EDS store metrics and EDS cache metrics.

These metrics are available only on bridge nodes. Light nodes do not have an EDS store and do not export eds_store_* or eds_cache_* metrics; their absence on a light node is expected.

Metric	Description	Acceptable value
`eds_store_put_time_histogram`	Time to put an EDS into the store (seconds).	Stable histogram; sustained elevation may mean disk pressure.
`eds_store_get_time_histogram`	Time to get an EDS from the store (seconds).	Stable; correlate with cache hit rates below.
`eds_store_has_time_histogram`	Time for store `Has` checks (seconds).	Stable.
`eds_store_put_exists_counter`	Puts skipped because the file already existed.	Can be non-zero during retries; large sustained rates may indicate duplicate work.
`eds_cache_<name>_get_counter`	Cache get events (attribute `found=true/false`).	Healthy hit ratio: most gets with `found=true`.
`eds_cache_<name>_evicted_counter`	Cache eviction events.	Expected under memory limits; correlate with miss rate.
`eds_cache_<name>_size`	Items currently in the named cache.	Within configured cache capacity.

Dashboard labels such as “BlockStore cache hit/miss” or “Recent cache hit/miss” map to these eds_cache_* counters for the corresponding cache name.

Blobs

Source: blob metrics.

Metric	Description	Acceptable value
`blob_retrieval_total`	Blob retrieval attempts (error type labeled on failure).	Successes should dominate; investigate rising `not_found` / `timeout`.
`blob_retrieval_duration_seconds`	Blob retrieval latency.	Stable for your workload.
`blob_proof_total`	Blob proof operations.	Increases when serving or verifying proofs.
`blob_proof_duration_seconds`	Blob proof latency.	Stable.

Operator tips

Synced tip: compare hdr_sync_subjective_head_gauge, hdr_store_head_height_gauge, and das_network_head. Large lasting gaps mean header sync, store, or DAS is behind.
DAS catch-up: watch das_sampling_head_lag first, then das_busy_workers_amount and sample error logs if lag does not trend to zero.
Serving load (bridge nodes): shrex server totals and store put/get latency show whether you are healthy under peer demand.
Acceptable values depend on node type (light vs bridge), network, and hardware. Use the table as a starting point and baseline your own node after a clean sync.

DAS sampling head lag

Use das_sampling_head_lag to monitor how far data availability sampling (DAS) is behind the network head known to your node. Enable metrics with the flags above and view the gauge in your metrics backend.

This metric requires a celestia-node build that includes celestia-node #5307. The change is merged but is not included in a published release as of 5 October 2026.

The gauge reports max(NetworkHead - SampledChainHead, 0) in headers. SampledChainHead is the head of the continuously sampled chain. For example, a network head of 100 and a sampled chain head of 90 produce a lag of 10. This measures a difference in height, not elapsed time.

A sustained or growing positive value means sampling remains behind the node's known network head; check the node logs for sampling errors. A value of zero means the sampled chain head has reached or exceeded that known head. It does not by itself prove the node has the latest network header.

If the metric is missing, check that your build includes the change and that metrics are enabled and reaching your collector.

Metrics endpoint design considerations

At the moment, the architecture of celestia-node metrics works as specified in the following ADR #010.

Essentially, the design considerations here will necessitate running an OpenTelemetry (OTEL) collector that connects to Celestia light node.

For an overview of OTEL, check out the guide.

The ADR and the OTEL docs will help you run your collector on the metrics endpoint. This will then allow you to process the data in the collector on a Prometheus server which can then be viewed on a Grafana dashboard.

In the future, we do want to open-source some developer toolings around this infrastructure to allow for node operators to be able to monitor their data availability nodes.

Configure celestia-node to export to multiple OTEL collectors

It is not supported to directly export metrics to multiple OTEL collectors (see the discussions here). To achieve this goal, an agent OTEL collector needs to be deployed for the node, from which the metrics can be forwarded to any other OTEL collectors. Here are the necessary steps and example configurations.

Follow the instructions here to install the OTEL collector. If you have the binary installed in /usr/local/bin/otelcol, you may consider creating a systemd service to run the collector as a background service. Here is an example of a systemd service definition in /etc/systemd/system/otelcol.service:

[Unit]
Description=OpenTelemetry Collector
After=network.target
 
[Service]
ExecStart=/usr/local/bin/otelcol --config /path/to/otelcol_config.yaml
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

The following is an example of the otelcol_config.yaml file that transforms the metrics into Prometheus metrics and reports them to the Celestia OTEL collector at the same time:

receivers:
  otlp:
    protocols:
      http: # Enable HTTP receiving
        endpoint: "127.0.0.1:4318" # the endpoint where the celestia-node will send metrics (the default value of --metrics.endpoint for celestia when --metrics is specified)
exporters:
  prometheus:
    # the node metrics will be transformed to Prometheus format and exposed on the following endpoint
    endpoint: "0.0.0.0:8889"
    namespace: "celestia"
  otlphttp:
    # report the metrics to Mocha testnet OTEL collector as an example
    # change it according to your network
    endpoint: https://otel.mocha.celestia.observer
 
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus, otlphttp]

Run the following commands to enable the OTEL collector systemd service:

sudo systemctl daemon-reload
sudo systemctl enable --now otelcol.service
 
# check the status of the service
sudo systemctl status otelcol.service

If the collector is up and running without any error, you can adjust the options for the celestia-node service:

celestia <node-type> start --metrics.tls=false \
    --metrics --metrics.endpoint localhost:4318 \
    ...

Edit this page