metrics

Exposes Prometheus-compatible metrics for monitoring node performance and health.

Method: GET /metrics

Parameters: None

Returns: Prometheus-format text response with all collected metrics

Example Request:

curl -X GET http://localhost:9153/metrics

Example Response:

# HELP genlayer_node_cpu_usage_percent Current CPU usage percentage (0-100)
# TYPE genlayer_node_cpu_usage_percent gauge
genlayer_node_cpu_usage_percent{component="node"} 15.5
genlayer_node_cpu_usage_percent{component="genvm-llm"} 45.2
genlayer_node_cpu_usage_percent{component="genvm-web"} 12.8
genlayer_node_cpu_usage_percent{component="webdriver"} 5.3

# HELP genlayer_node_memory_rss_bytes Resident Set Size (RSS) memory usage in bytes
# TYPE genlayer_node_memory_rss_bytes gauge
genlayer_node_memory_rss_bytes{component="node"} 524288000
genlayer_node_memory_rss_bytes{component="genvm-llm"} 2147483648

# HELP genlayer_node_network_rx_bytes_total Total bytes received across all network interfaces
# TYPE genlayer_node_network_rx_bytes_total counter
genlayer_node_network_rx_bytes_total{component="node"} 1073741824

Available Metrics

The node collects metrics for three main components:

• Node - The main node process
• GenVM - The GenVM modules (LLM and Web)
• WebDriver - The WebDriver container

Operator Condition Metrics

• genlayer_node_operator_condition_state{condition,state}: one-hot gauge for the current catalog state. All state series are prewarmed at process start.
• genlayer_node_operator_condition_occurrences_total{condition,reason}: accepted failed observations by stable reason.
• genlayer_node_operator_condition_transitions_total{condition,from,to}: lifecycle transitions, including recovery.
• genlayer_node_operator_condition_last_success_timestamp_seconds{condition}: Unix timestamp of the last successful observation, or zero before one occurs.

condition, state, reason, from, and to come from the fixed startup catalog; upstream error text is never used as a label.

occurrences_total counts differently per condition kind, so the counter stays comparable: an observed condition (folded from duty traffic) counts every failed observation, while a polled condition counts only entry into a failing state. Counting every failed poll would scale the counter with the poll interval rather than the fault — a three-day ban at a ten-second poll would read as roughly 26,000 events.

CPU Metrics

genlayer_node_cpu_usage_percent

Current CPU usage percentage (0-100).

Type: Gauge

Labels:

• component: One of node, genvm-llm, genvm-web, webdriver

Example:

genlayer_node_cpu_usage_percent{component="node"} 15.5
genlayer_node_cpu_usage_percent{component="genvm-llm"} 45.2
genlayer_node_cpu_usage_percent{component="genvm-web"} 12.8
genlayer_node_cpu_usage_percent{component="webdriver"} 5.3

Memory Metrics

genlayer_node_memory_rss_bytes

Resident Set Size (RSS) memory usage in bytes.

Type: Gauge

Labels:

• component: One of node, genvm-llm, genvm-web, webdriver

Example:

genlayer_node_memory_rss_bytes{component="node"} 524288000
genlayer_node_memory_rss_bytes{component="genvm-llm"} 2147483648

genlayer_node_memory_vms_bytes

Virtual Memory Size (VMS) in bytes.

Type: Gauge

Labels:

• component: One of node, genvm-llm, genvm-web, webdriver

Note: Only available for node and GenVM components.

genlayer_node_memory_usage_bytes

Current memory usage in bytes (WebDriver only).

Type: Gauge

Labels:

• component: webdriver

genlayer_node_memory_limit_bytes

Memory limit in bytes (WebDriver only).

Type: Gauge

Labels:

• component: webdriver

genlayer_node_memory_percent

Memory usage percentage.

Type: Gauge

Labels:

• component: One of genvm-llm, genvm-web, node, webdriver
• node: Node address (hex format)

genlayer_node_memory_peak_bytes

Peak memory usage in bytes (GenVM components only).

Type: Gauge

Labels:

• component: One of genvm-llm, genvm-web
• node: Node address (hex format)

Network Metrics

genlayer_node_network_rx_bytes_total

Total bytes received across all network interfaces.

Type: Counter

Labels:

• component: One of node, genvm-llm, genvm-web, webdriver
• node: Node address (hex format)

Example:

genlayer_node_network_rx_bytes_total{component="node",node="0x84b6cbd007511352d3fea26834c5c39a440903c4"} 96879942
genlayer_node_network_rx_bytes_total{component="webdriver",node="0x84b6cbd007511352d3fea26834c5c39a440903c4"} 84

genlayer_node_network_tx_bytes_total

Total bytes transmitted across all network interfaces.

Type: Counter

Labels:

• component: One of node, genvm-llm, genvm-web, webdriver
• node: Node address (hex format)

Disk Metrics

genlayer_node_disk_free_bytes

Available free disk space in bytes.

Type: Gauge

Labels:

• component: node
• directory: One of genvm.db, ops.db, keystore, logs
• node: Node address (hex format)

genlayer_node_disk_total_bytes

Total disk space in bytes.

Type: Gauge

Labels:

• component: node
• directory: One of genvm.db, ops.db, keystore, logs
• node: Node address (hex format)

genlayer_node_disk_usage_bytes

Used disk space in bytes.

Type: Gauge

Labels:

• component: node
• directory: One of genvm.db, ops.db, keystore, logs
• node: Node address (hex format)

genlayer_node_disk_usage_percent

Used disk space as percentage.

Type: Gauge

Labels:

• component: node
• directory: One of genvm.db, ops.db, keystore, logs
• node: Node address (hex format)

Node Status Metrics

genlayer_node_blocks_behind

Number of blocks behind the latest block.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_latest_block

Latest block number in the network.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_processing_block

Currently processing block number.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_synced_block

Last synced block number.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_synced

Node synchronization status (1 = synced, 0 = not synced).

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_sync_halted

Whether replay has stopped at consensus.sync_halt_block (1 = halted, 0 = running). An intentional halt is not a stalled sync; the node continues serving RPC and snapshots.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_prime_blocked_insufficient_funds

Whether a mandatory validator prime is currently blocked by an unfunded operator wallet (1 = blocked, 0 = not blocked). While blocked the validator cannot pay for consensus duty transactions and is flagged idle each round — fund the operator wallet to recover.

Type: Gauge

Labels:

• node: Node address (hex format)

State pruning metrics

Contract-state pruning (node.state.enabled). All are gauges labelled with node; the _total ones count since the process started.

genlayer_node_uptime_seconds

Node uptime in seconds.

Type: Gauge

Labels:

• component: node
• node: Node address (hex format)

Transaction Metrics

genlayer_node_transactions_accepted_synced_total

Total number of accepted and synced transactions.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_transactions_activated_total

Total number of activated transactions.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_transactions_leader_proposed_total

Total number of transactions proposed as leader.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_transactions_leader_revealed_total

Total number of transactions revealed as leader.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_transactions_validator_commit_total

Total number of validator commit votes.

Type: Gauge

Labels:

• node: Node address (hex format)

genlayer_node_transactions_validator_reveal_total

Total number of validator reveal votes.

Type: Gauge

Labels:

• node: Node address (hex format)

Gas Metrics (duty transactions)

Emitted inline from the shared consensus gas estimator and transactor, so they cover every duty method (finalizeTransaction, commitVote, revealVote, proposeReceiptWithFees, topUpAndSubmitAppeal, activateTransaction, validatorPrime, epochAdvance, ...). All are true counters.

genlayer_node_gas_estimation_total

Outcome of every duty gas estimation.

Type: Counter

Labels:

• method: Contract method name
• outcome: One of success (real RPC estimate), fallback_block_limit (estimation failed; submitted at full block gas limit), fallback_cache / fallback_cache_scaled (estimation failed; reused gas from the shared calldata-length-keyed cache, possibly learned from a different method)

genlayer_node_gas_estimation_capped_total

Estimation succeeded but the safety-multiplied estimate (or per-call reserve) exceeded the block gas limit and was silently capped — the safety margin is gone while everything still works.

Type: Counter

Labels:

• method: Contract method name

genlayer_node_gas_shortfall_total

Confirmed on-chain out-of-gas: the transaction landed but ran out of gas mid-execution. Self-heals — the node retries with a larger gas limit — so a single occurrence is not actionable; a sustained rate signals systematic under-estimation (treat as a warning, not a page).

Type: Counter

Labels:

• method: Contract method name
• reason: One of typed_shortfall (InsufficientGasForInternalMessageCreation revert with exact gas requirement), out_of_gas (explicit OutOfGas revert), heuristic (undecoded revert that consumed gas above the exhaustion threshold)

genlayer_node_gas_block_limit_exhausted_total

Gas shortfall while the gas limit was already capped at the block limit: the work cannot fit in one block and retrying cannot help. Non-self-healing — any occurrence is actionable (page-worthy).

Type: Counter

Labels:

• method: Contract method name

genlayer_node_gas_spent_wei_total

Cumulative wei actually billed for gas, from each mined duty receipt (gasUsed × effectiveGasPrice). Reverted attempts are billed too and are accounted per attempt; submissions that never land cost nothing. The reverted slice is the exact cost of failed work, and sum(genlayer_node_gas_spent_wei_total) is the operator's total billed duty gas.

Type: Counter

Labels:

• method: Contract method name
• outcome: One of success, reverted
• price_source: effective (receipt's billed price) or fee_cap (submitted fee cap used as an upper bound when the RPC omits effectiveGasPrice)

Consensus Submission Metrics

genlayer_node_consensus_submit_total

Terminal outcome of every duty submission.

Type: Counter

Labels:

• method: Contract method name
• outcome: One of success (landed and executed), reverted (landed, contract rejected — gas still billed), failed (never landed: RPC/nonce/funds/authorization)

genlayer_node_consensus_revert_total

Decoded revert causes, one increment per observed revert (attempt-level).

Type: Counter

Labels:

• method: Contract method name
• error: Decoded custom error from the fee/gas allowlist (InsufficientFeesForRound, InsufficientFees, FailedFeeTransfer, FeeSettlementNotCompleted, MaxPriceExceeded, ExecutionBudgetExceeded, MessageBudgetExceeded, BudgetTooLow, RollupBudgetBelowFloor, InsufficientBondCustody, InvalidAppealBond, AppealBondTooLow, InsufficientGasForInternalMessageCreation, OutOfGas); anything else collapses to other

Transaction Read Metrics

genlayer_node_transaction_reads_total

Reads of consensus transaction data from the chain, made through the node's one transaction reader. One read can be several chain calls: a snapshot is the transaction record plus, unless the caller skips them, its idleness timestamps and phase timeouts; a fee_state read is the fee inputs of one execution at the transaction's processing block: 8 reads in parallel, plus the idleness timeouts when the transaction has no phase timeouts of its own, plus 2 reads when it has a message fee allocation tree; a latest_*_with_effect lookup pages the contract's history and makes one snapshot read for each candidate with an agreed execution, which counts separately. Use it to see which node path reads which data, and how often.

Type: Counter

Labels:

• method: One of snapshot (the transaction as the chain holds it at a block), fee_state (the fee inputs GenVM needs to execute the transaction), latest_accepted_with_effect, latest_finalized_with_effect (the newest accepted or finalized transaction of a contract that changed its state)
• flow: The node path that asked: duty (a consensus duty, including the leader's and validators' executions), sync (the sync filter and its history preload), recovery (accepted data missing from a replay batch), replay (a replayed execution and the finalized-state settle of the replay), tribunal (the judgement of an accused execution), genvm (a contract history lookup of a GenVM execution), rpc (a JSON-RPC request, including the replay of a trace that is not stored), other (no path named)
• outcome: One of ok, error

Reads the node's transaction read cache answers from memory are not counted here: this counter is the chain reads only.

genlayer_node_transaction_read_cache_total

Transaction reads answered by the node's transaction read cache. The cache keeps a transaction's snapshot at an explicit block, its fee state at a processing block, and the answers of the contract history lookup ("latest accepted / finalized transaction with an effect on contract C at block A", which every GenVM execution asks and the sync filter preloads), so the repeats of one node read the chain once: the sync filter, the accepted handler and the last-vote reveal reading an accepted transaction at the same block; a validator's duty run and its replay reading the fee state and the history of the same round. A value read at or below the chain's finalized block is kept for consensus.transaction_cache.settled_ttl; one read above it, which a reorg can replace, for unsettled_ttl (snapshots) or unsettled_long_ttl (fee states, history). See the configuration reference.

miss and bypass match the genlayer_node_transaction_reads_total reads of the same method and flow, except the snapshot reads a history lookup makes for its candidates, which do not go through the cache. A history preload counts the accepted answer and, when that answer has no transaction, the finalized answer a GenVM execution then falls back to: each miss when the preload read the chain and hit when the answer was already kept or being read. Only a preload that read the accepted answer reads the finalized one. A history lookup that excludes the latest answer itself counts miss: only the chain can give the older answer.

The share of reads the chain did not serve, per method:

sum by (method) (rate(genlayer_node_transaction_read_cache_total{outcome=~"hit|shared"}[1h]))
/ sum by (method) (rate(genlayer_node_transaction_read_cache_total[1h]))

Type: Counter

Labels:

• method: One of snapshot, fee_state, latest_accepted_with_effect, latest_finalized_with_effect
• flow: The node path that asked, as in genlayer_node_transaction_reads_total
• outcome: One of hit (served from the cache), miss (read from the chain; kept when the read succeeded), shared (served by another caller's chain read of the same data, made at the same time), bypass (a read the cache does not keep: one at the latest block)

With consensus.transaction_cache.enabled: false the cache passes every read through and none of the genlayer_node_transaction_read_cache_* metrics move.

genlayer_node_sync_preloads_queued_total

History preloads the sync filter queued: one per accepted transaction it read, to put into the transaction read cache the answers the replay's GenVM execution will ask for. consensus.transaction_cache.preload_workers preloads read the chain at a time; the others wait for a free worker, and none is dropped while the sync runs. A sync that stops, for a restart or a shutdown, drops its backlog; a restart filters the blocks it did not sync again and queues their preloads again. Each finished preload counts once in genlayer_node_transaction_read_cache_total{flow="sync",method="latest_accepted_with_effect"}, whether or not it also loaded the finalized answer, so the difference is the backlog, plus the backlogs that sync restarts dropped:

genlayer_node_sync_preloads_queued_total
- sum(genlayer_node_transaction_read_cache_total{flow="sync",method="latest_accepted_with_effect"})

A backlog that keeps growing in a catch-up means the preloads cannot keep up with the filter; more workers fill the cache sooner and add chain RPC load. With consensus.transaction_cache.enabled: false a preload does nothing and the query does not apply.

Type: Counter

genlayer_node_transaction_read_cache_purges_total

Times the transaction read cache was emptied. The cache pins its reads by block number, and a reorg can replace a block above the chain's finalized block, so the node empties it when it detects one. A steady rate means the chain reorganizes often; each purge costs the cache, not correctness.

Type: Counter

Labels:

• reason: One of canonical_reorg (the replay rolled back to the canonical chain), router_rewind (the event router rewound over a replaced block range)

genlayer_node_transaction_read_cache_evictions_total

Values the transaction read cache dropped by itself. Use it to tune the cache: many expired mean a TTL is shorter than the time between the reads it should serve; any capacity means a size limit is too small for what the node reads within the TTL. A value nobody read again that is dropped after its TTL is not counted.

Type: Counter

Labels:

• method: One of snapshot, fee_state, latest_with_effect (the history answers, accepted and finalized together)
• reason: One of expired (a read found its value past its TTL), capacity (a live value dropped to make room)

genlayer_node_transaction_read_cache_entries

Values the transaction read cache keeps now, per tier and per finality of the block they were read at. Expired values not yet dropped are included.

Type: Gauge

Labels:

• method: One of snapshot, fee_state, latest_with_effect
• finality: settled (read at or below the chain's finalized block) or unsettled (above it)

Fee Flow Metrics (v0.6 ConsensusMainWithFees)

Note: there are no dedicated appeal-bond metrics. No node runtime path submits appeals — appellants are external clients funding bonds from their own wallets. If a validator appeal duty ever ships, the method-labeled gas/submission metrics above cover it under method="topUpAndSubmitAppeal" with no code change.

genlayer_node_fees_operator_insufficient_funds_total

Duty submissions rejected because the operator account cannot cover value + gas. Complements the genlayer_node_prime_blocked_insufficient_funds gauge by leaving a trace of dips that recover between scrapes. A sustained increase means the operator wallet needs funding (page-worthy).

Type: Counter

genlayer_node_fees_settlement_events_total

Fee-settlement events decoded from this node's own mined finalize receipts. Finalize runs under a staggered lease, so each finalize is observed by exactly one node — aggregate fleet-wide with sum by (network). Use the severity label to alert on failures without enumerating event names: genlayer_node_fees_settlement_events_total{severity="alarm"} covers all three failure modes and any future one.

Type: Counter

Labels:

• event: One of UnifiedFeesDistributed (fees distributed at settlement), FeesRefunded (surplus returned to the sender), FeeSettlementFailed (finalize succeeded but settlement failed — validators unpaid until retryFeeSettlement), CommitFinalFeesFailed (tx finalized without fee settlement), ValueWithdrawalFailed (a best-effort transfer — refund/appealer payout — failed; funds stay custodied pending recovery)
• severity: alarm for the three failure events (unpaid or stuck-funds modes needing operator action) or info for the normal UnifiedFeesDistributed / FeesRefunded happy path

genlayer_node_fees_distributed_wei_total

Total wei moved at fee settlement across all recipients — committee payouts plus sender/contributor refunds plus appealer rewards — summed from UnifiedFeesDistributed amounts in this node's finalize receipts. It excludes amounts retained in staking custody (amountToStaking). This is a network "value settled" liveness signal, not per-node earnings — for a validator's own income use genlayer_node_fees_earned_wei_total.

Type: Counter

genlayer_node_fees_earned_wei_total

This node's validator earnings credited at prime, read from the ValidatorPrime event when a prime is effective. This is the accurate per-node income (unlike fees_distributed_wei_total, which is whole-committee plus refunds). Earnings settle into stake per epoch, so this series lags settlement by up to one prime interval. Pair type="fee_rewards" with genlayer_node_gas_spent_wei_total for operator net P&L (earned − gas_spent).

Type: Counter

Labels:

• type: One of fee_rewards (fee-flow income), validator_rewards, delegator_rewards (staking-reward components)

genlayer_node_fees_penalties_wei_total

This node's fee penalties deducted at prime, read from the same ValidatorPrime event. Kept separate from fees_earned_wei_total so earnings stay gross income; subtract this for net.

Type: Counter

genlayer_node_external_message_delivery_total

External messages this node's own delivery calls (flushExternalMessages, flushInternalMessages, finalizeTransaction) resolved or paused, read from their mined receipts. A paused message was not called and not charged; the node retries it with a backoff (30 s doubling to 10 min) until the chain reports it issued. A paused_* series that keeps rising for one network means a message nobody can deliver.

Type: Counter

Labels:

• outcome: delivered (ran and succeeded), call_failed (ran and failed; final), paused_gas (no room to forward its gas envelope), paused_value (its value was not yet available), cancelled_unfunded (its open-bucket envelope was never funded; value refunded), cancelled_ceiling (its gas limit is above the current ceiling; value refunded)

genlayer_node_external_message_reimbursed_wei_total

Wei the chain reimbursed this node for running external messages: gas used times the transaction's locked gas price. Pair it with gas_spent_wei_total for the net cost of delivery.

Type: Counter

Go Runtime Metrics

The endpoint also exposes standard Go runtime metrics including:

• go_gc_duration_seconds - Garbage collection duration summary
• go_goroutines - Number of active goroutines
• go_memstats_* - Go memory statistics (heap, stack, GC)
• go_threads - Number of OS threads

Process Metrics

Standard process metrics are also available:

• process_cpu_seconds_total - Total CPU time
• process_resident_memory_bytes - Resident memory size
• process_virtual_memory_bytes - Virtual memory size
• process_open_fds - Number of open file descriptors
• process_network_receive_bytes_total - Network bytes received
• process_network_transmit_bytes_total - Network bytes transmitted

Prometheus Handler Metrics

Metrics about the metrics endpoint itself:

• promhttp_metric_handler_requests_total - Total scrape requests by status code
• promhttp_metric_handler_requests_in_flight - Current scrapes being served

Configuration

Metrics collection can be configured in the node configuration file:

metrics:
  interval: "15s"        # Default interval for all collectors
  collectors:
    node:
      enabled: true      # Enable/disable node metrics
      interval: "30s"    # Optional: Override default interval
    genvm:
      enabled: true      # Enable/disable GenVM metrics
      # Uses default interval (15s)
    webdriver:
      enabled: true      # Enable/disable WebDriver metrics
      interval: "60s"    # Optional: Override default interval

Default Values

• metrics.interval: 15s (applies to all collectors by default)
• enabled: true (for each collector)
• interval: Uses metrics.interval if not specified per collector

Configuration Priority

1. Collector-specific interval (if specified)
2. Global metrics.interval (if specified)
3. System default (15s)

Troubleshooting

No Metrics for GenVM

GenVM processes are ephemeral and only run when executing smart contracts. If you don't see GenVM metrics, it's likely that no contracts are currently being executed.

WebDriver Metrics Missing

Ensure the WebDriver container is running:

docker ps | grep genlayer-node-webdriver

If not running, start it:

task docker:webdriver:up

Metrics Not Updating

Check if metrics collection is enabled in your configuration and that the node has been restarted after configuration changes.

Performance Considerations

• Collection intervals can be increased to reduce overhead further
• Each collector runs independently and won't block others if one fails