# celestia-node metrics

This tutorial is for running metrics for your celestia-node data
availability instance. This tutorial will focus on running metrics for a light node.

This tutorial assumes you have already setup your light node
by following the tutorial in the
[light node guide](/operate/data-availability/light-node/quickstart).

## Running metrics flags

You can enable the `celestia-node` metric flags with the following
command:

```sh
celestia <node-type> start --metrics.tls=<boolean> \
    --metrics --metrics.endpoint <URI> \
    --p2p.network <network> \
    --core.ip <URI> --core.port <port>
```

Add metrics flags to your node start command and restart your node to apply it.
The metrics endpoint will gather your node's data to track your uptime.

Note that the `--metrics` flag enables metrics and expects
an input into `--metrics.endpoint`.

We will go over what the endpoint will need to be in the
[metrics endpoint design considerations](#metrics-endpoint-design-considerations)
section.

### Mainnet Beta

Here is an example for Mainnet Beta:

```sh
celestia <node-type> start --metrics.tls=true \
    --metrics --metrics.endpoint otel.celestia.observer \
    --core.ip <URI> --core.port <port>
```

### Mocha testnet

Here is an example for Mocha testnet:

```sh
celestia <node-type> start --metrics.tls=true \
    --metrics --metrics.endpoint otel.mocha.celestia.observer \
    --core.ip <URI> --core.port <port> --p2p.network mocha
```

### TLS connections

The `--metrics.tls` flag enables or disables a TLS connection to the
OpenTelemetry Protocol metrics backend. You need to choose a boolean
value (`true` or `false`) for this flag.

It's also common to set this flag to `false` when spinning up a local
collector
to check the metrics locally.

However, if the collector is hosted in the cloud as a separate entity
(like in a DevOps environment), enabling TLS is a necessity for secure
communication.

Here are examples of how to use it:

```bash
# To enable TLS connection
celestia <node-type> start --metrics.tls=true --metrics \
    --metrics.endpoint <URI> \
    --p2p.network <network> --core.ip <URI> --core.port <port>

# To disable TLS connection
celestia <node-type> start --metrics.tls=false --metrics \
    --metrics.endpoint <URI> \
    --p2p.network <network> --core.ip <URI> --core.port <port>
```

## Key metrics reference

Node metrics come from celestia-node and dependencies such as go-header.
Exact panel names on community Grafana dashboards may differ from the
OpenTelemetry metric names below. Prefer the metric names from the codebase
when wiring alerts.

### Overview and sync health

Source: [header store metrics](https://github.com/celestiaorg/go-header/blob/main/store/metrics.go),
[header sync metrics](https://github.com/celestiaorg/go-header/blob/main/sync/metrics.go),
and [DAS metrics](https://github.com/celestiaorg/celestia-node/blob/main/das/metrics.go).

| Metric | Description | Acceptable value |
|--------|-------------|------------------|
| `hdr_store_head_height_gauge` | Local header store head height (subjective height of stored headers). | Near network tip for a synced node. |
| `hdr_store_tail_height_gauge` | Lowest header height retained in the local store. | Depends on pruning / retention; should not unexpectedly jump up. |
| `hdr_sync_subjective_head_gauge` | Syncer’s subjective head height (what the node treats as latest). | At or very near network tip when synced. Can look ahead of a lagging RPC/core peer. |
| `hdr_sync_subjective_head_lag_seconds` | Seconds between a header’s BFT timestamp and when syncer adopted it as subjective head. | Low and stable; sustained growth suggests sync or peer issues. |
| `das_network_head` | Most recent network head observed by the DAS’er. | Near chain tip. |
| `das_sampled_chain_head` | Highest height through which continuous DAS has succeeded. | Should catch up to `das_network_head` over time. |
| `das_sampling_head_lag` | `max(network_head - sampled_chain_head, 0)` in headers. | Prefer `0` when fully caught up; see [DAS sampling head lag](#das-sampling-head-lag). |
| `das_busy_workers_amount` | Active parallel DAS workers (labeled by job type). | Non-zero while sampling backlog exists; idle when caught up. |
| `das_sampled_headers_counter` | Count of recorded sampling attempts, including failures (`failed=true`). | The `failed=false` series should increase when sampling succeeds. |
| `das_latest_sampled_ts` | Unix timestamp of the latest recorded sampling attempt, including failures. | Recent while sampling is active; freshness alone does not show success. |

Failed attempts can keep both metrics advancing. Use the `failed=false` series
of `das_sampled_headers_counter` to track successes, and
`das_sampled_chain_head` or `das_sampling_head_lag` to check catch-up.

### Peers and ShareX (shrex)

Source: [ShareX metrics](https://github.com/celestiaorg/celestia-node/blob/main/share/shwap/p2p/shrex/metrics.go).

| Metric | Description | Acceptable value |
|--------|-------------|------------------|
| `shrex_client_total_requests` | Client ShareX requests, labeled by protocol and status. | Mix of `success`; spikes of `not_found` / errors warrant investigation. |
| `shrex_client_request_duration` | Client ShareX request latency (seconds). | Stable; rising p99 under load may indicate peer or network pressure. |
| `shrex_server_total_responses` | Bridge-node ShareX responses sent, labeled by protocol and status. | Increases on bridge nodes serving peers. |
| `shrex_server_request_duration` | Bridge-node ShareX handler latency (seconds). | Stable under expected peer load. |

Peer count panels on operator dashboards (full vs archival DA peers) are typically
derived from libp2p / discovery metrics bridged into OTEL. Aim for **> 0** full
peers on a light node that needs to fetch shares; bridge nodes should
also retain healthy peer connectivity.

### EDS store and caches

Source: [EDS store metrics](https://github.com/celestiaorg/celestia-node/blob/main/store/metrics.go)
and [EDS cache metrics](https://github.com/celestiaorg/celestia-node/blob/main/store/cache/metrics.go).

These metrics are available only on bridge nodes. Light nodes do not have an
EDS store and do not export `eds_store_*` or `eds_cache_*` metrics; their absence
on a light node is expected.

| Metric | Description | Acceptable value |
|--------|-------------|------------------|
| `eds_store_put_time_histogram` | Time to put an EDS into the store (seconds). | Stable histogram; sustained elevation may mean disk pressure. |
| `eds_store_get_time_histogram` | Time to get an EDS from the store (seconds). | Stable; correlate with cache hit rates below. |
| `eds_store_has_time_histogram` | Time for store `Has` checks (seconds). | Stable. |
| `eds_store_put_exists_counter` | Puts skipped because the file already existed. | Can be non-zero during retries; large sustained rates may indicate duplicate work. |
| `eds_cache_<name>_get_counter` | Cache get events (attribute `found=true/false`). | Healthy hit ratio: most gets with `found=true`. |
| `eds_cache_<name>_evicted_counter` | Cache eviction events. | Expected under memory limits; correlate with miss rate. |
| `eds_cache_<name>_size` | Items currently in the named cache. | Within configured cache capacity. |

Dashboard labels such as “BlockStore cache hit/miss” or “Recent cache hit/miss”
map to these `eds_cache_*` counters for the corresponding cache name.

### Blobs

Source: [blob metrics](https://github.com/celestiaorg/celestia-node/blob/main/blob/metrics.go).

| Metric | Description | Acceptable value |
|--------|-------------|------------------|
| `blob_retrieval_total` | Blob retrieval attempts (error type labeled on failure). | Successes should dominate; investigate rising `not_found` / `timeout`. |
| `blob_retrieval_duration_seconds` | Blob retrieval latency. | Stable for your workload. |
| `blob_proof_total` | Blob proof operations. | Increases when serving or verifying proofs. |
| `blob_proof_duration_seconds` | Blob proof latency. | Stable. |

### Operator tips

- **Synced tip:** compare `hdr_sync_subjective_head_gauge`,
  `hdr_store_head_height_gauge`, and `das_network_head`. Large lasting gaps mean
  header sync, store, or DAS is behind.
- **DAS catch-up:** watch `das_sampling_head_lag` first, then
  `das_busy_workers_amount` and sample error logs if lag does not trend to zero.
- **Serving load (bridge nodes):** shrex server totals and store put/get latency
  show whether you are healthy under peer demand.
- **Acceptable values** depend on node type (light vs bridge), network,
  and hardware. Use the table as a starting point and baseline your own node
  after a clean sync.

## DAS sampling head lag

Use `das_sampling_head_lag` to monitor how far data availability sampling (DAS)
is behind the network head known to your node. Enable metrics with the flags
above and view the gauge in your metrics backend.

> This metric requires a celestia-node build that includes
> [celestia-node #5307](https://github.com/celestiaorg/celestia-node/pull/5307).
> The change is merged but is not included in a published release as of
> 5 October 2026.

The gauge reports `max(NetworkHead - SampledChainHead, 0)` in headers.
`SampledChainHead` is the head of the continuously sampled chain. For example,
a network head of 100 and a sampled chain head of 90 produce a lag of 10.
This measures a difference in height, not elapsed time.

A sustained or growing positive value means sampling remains behind the node's
known network head; check the node logs for sampling errors. A value of zero means
the sampled chain head has reached or exceeded that known head. It does not
by itself prove the node has the latest network header.

If the metric is missing, check that your build includes the change and that
metrics are enabled and reaching your collector.

## Metrics endpoint design considerations

At the moment, the architecture of celestia-node metrics
works as specified in the following [ADR #010](https://github.com/celestiaorg/celestia-node/blob/main/docs/adr/adr-010-incentivized-testnet-monitoring.md).

Essentially, the design considerations here will necessitate
running an OpenTelemetry (OTEL) collector that connects to Celestia
light node.

For an overview of OTEL, check out [the guide](https://opentelemetry.io/docs/collector).

The ADR and the OTEL docs will help you run your collector on the metrics endpoint.
This will then allow you to process the data in the collector on a
Prometheus server which can then be viewed on a Grafana dashboard.

In the future, we do want to open-source some developer toolings around
this infrastructure to allow for node operators to be able to monitor
their data availability nodes.

### Configure celestia-node to export to multiple OTEL collectors

It is not supported to directly export metrics to multiple OTEL collectors (see the discussions [here](https://github.com/open-telemetry/opentelemetry-go/issues/3055)). To achieve this goal, an agent OTEL collector needs to be deployed for the node, from which the metrics can be forwarded to any other OTEL collectors. Here are the necessary steps and example configurations.

Follow the instructions [here](https://opentelemetry.io/docs/collector/installation/) to install the OTEL collector. If you have the binary installed in `/usr/local/bin/otelcol`, you may consider creating a systemd service to run the collector as a background service. Here is an example of a systemd service definition in `/etc/systemd/system/otelcol.service`:

```ini
[Unit]
Description=OpenTelemetry Collector
After=network.target

[Service]
ExecStart=/usr/local/bin/otelcol --config /path/to/otelcol_config.yaml
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

The following is an example of the `otelcol_config.yaml` file that transforms the metrics into Prometheus metrics and reports them to the Celestia OTEL collector at the same time:

```yaml
receivers:
  otlp:
    protocols:
      http: # Enable HTTP receiving
        endpoint: "127.0.0.1:4318" # the endpoint where the celestia-node will send metrics (the default value of --metrics.endpoint for celestia when --metrics is specified)
exporters:
  prometheus:
    # the node metrics will be transformed to Prometheus format and exposed on the following endpoint
    endpoint: "0.0.0.0:8889"
    namespace: "celestia"
  otlphttp:
    # report the metrics to Mocha testnet OTEL collector as an example
    # change it according to your network
    endpoint: https://otel.mocha.celestia.observer

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus, otlphttp]
```

Run the following commands to enable the OTEL collector systemd service:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol.service

# check the status of the service
sudo systemctl status otelcol.service
```

If the collector is up and running without any error, you can adjust the options for the `celestia-node` service:

```sh
celestia <node-type> start --metrics.tls=false \
    --metrics --metrics.endpoint localhost:4318 \
    ...
```