Monitoring from the client side
Turn on metrics
- celestia-app Go client: metrics use OpenTelemetry. Set a global
MeterProviderin your process, or passClientConfig.Meter. With neither, nothing is exported. - celestia-node: start the node with
--metrics --metrics.endpoint <metrics_endpoint>. See node metrics.
What to watch
| Signal | Where to get it | Alert when |
|---|---|---|
Escrow available_balance | fibre.QueryEscrowAccount, or GET /fibre/v1/escrow-account/<address> | Below your threshold |
| Wallet balance | state.Balance, or the bank query | Too low to pay gas |
| Upload success rate and latency | fibre.client.upload.duration (attribute success). Node: fibre_submit_duration_seconds, fibre_upload_duration_seconds | Error rate or latency rises |
| Failures per validator | fibre.client.upload_to.duration with success=false, by validator_address | Many validators fail at once |
| Download success rate and latency | fibre.client.download.duration | Error rate rises |
| Settlement | Transaction result for each tx_hash, or EventPayForFibre | Any blob unsettled after a few blocks |
| Unexpected charges | EventPaymentPromiseTimeout where escrow_signer is your address | Any occurrence |
| Core endpoint health | Your client logs: HTTP 429, timeouts, connection errors | Sustained errors |
EventPaymentPromiseTimeout means one of your promises was charged without a
normal settlement. Usually an upload failed after the promise had already
reached validators.
Your problem or the network's?
| What you see | Likely cause | What to do |
|---|---|---|
| Every upload fails with an escrow or balance error | Yours: escrow empty, or too many promises in flight | Top up escrow. Lower concurrency. |
payment promise chain ID mismatch | Yours: client and endpoint are on different networks | Fix the endpoint or chain config |
creation_timestamp ... must not be after | Yours: host clock is ahead. Or the chain has stalled. | Sync the clock. Check the status page. |
| Failures spread across most validators, all timeouts | Usually yours: outbound bandwidth or network | Check egress. Compare with a second host. |
| Failures on a few specific validators only | Those validators | Nothing, while quorum still holds |
not enough voting power and many validators unreachable | Often the network | Check the dashboards and status page. |
fibre storage budget exceeded from many validators | Network capacity | The client retries on its own. |
| HTTP 429 from the core endpoint | Endpoint provider | Back off. Use your own endpoint. Check whether the blob settled before resubmitting. |
Download returns blob not found after retention | Expected behaviour | Recover from your own copy |
Cross-check with:
- the network status page
- the Mainnet Beta and Mocha community dashboards: Valopers (registered providers and reachability) and Tensile (endpoint reachability and whether validators serve their rows)