> For the complete documentation index, see [llms.txt](https://docs.gravity.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.gravity.xyz/gravity-networks/node-operation-anti-patterns.md). # Node Operation Anti-Patterns Node operation mistakes that cause sync problems on Gravity L1, with the symptoms to look for, why they happen, how to confirm them, and how to fix them. This page collects node operation mistakes that cause sync problems on Gravity L1. Each entry lists what you will see, why it happens, how to confirm it, and how to fix it. The guidance applies to PFNs, VFNs, and validators unless an entry says otherwise. ## Sharing One Network Identity Across Nodes **Anti-pattern:** running two or more nodes with the same `identity.yaml`, or pointing two nodes at the same identity secret. This usually happens when a second node is created by copying an existing one: the whole deploy directory, a Docker workspace, or the `cluster/output//` tree. `just init` reuses an identity that already exists in `cluster/output//config/`, so copying that tree also copies the identity. ### Symptoms * The node height moves in a **staircase**. It stays flat for most of an epoch, catches up about one epoch of blocks right after each epoch change (roughly every 2 hours on Mainnet), and then goes flat again. It stays about one epoch behind the chain head and never reaches it. * Another node that shares the identity also syncs slowly or stalls. * The node's connections to its upstream seeds close and reopen every few seconds. ### Why It Happens A node's network identity is the x25519 network key in `identity.yaml`. Other nodes identify it by the peer ID derived from that key (`account_address` in `identity.public.yaml`), and an upstream node keeps at most **one connection per peer ID**. When two nodes share an identity, each new connection from one node replaces the other node's existing connection. The upstream node logs this as: ``` Closing existing connection with Peer to mitigate simultaneous dial ``` Requests that were still in flight on the replaced connection fail. The two sync paths react to this differently: * **Catching up inside the current epoch** fetches the missing range in one pass and only keeps it after the whole range arrives. A dropped connection anywhere in the pass discards the work, and the next attempt starts again against a newer chain head. When the connection is replaced every few seconds, the pass never completes. * **Syncing an epoch that has already ended** persists every batch and resumes from the node's local position after an interruption. It keeps advancing, but only after each epoch change. Together these produce the staircase. ### How to Confirm Compare the identity files of every node you run. Each node must have its own identity. ```bash # Run on each host, then compare the hashes. Equal hashes mean a shared identity. sha256sum //config/identity.yaml ``` For the Docker flow, check `config/identity.yaml` in each Docker workspace instead. For `identity.source = "gcp_secret"`, check that each node references a different secret. If you cannot reach every host, check the node's consensus log for upstream connections that close and reconnect every few seconds. You can also contact the Gravity team with your node's peer ID and ask whether upstream nodes see its connection being replaced repeatedly. ### Fix Keep one node on the existing identity and give every other node a new one. For a **PFN**: 1. Stop the duplicate node. 2. Move its old identity out of the way and generate a new one in place: ```bash CONFIG_DIR=//config mv "$CONFIG_DIR/identity.yaml" "$CONFIG_DIR/identity.yaml.shared.bak" gravity_cli genesis generate-key \ --output-file "$CONFIG_DIR/identity.yaml" \ --public-output-file "$CONFIG_DIR/identity.public.yaml" ``` For the Docker flow, set `CONFIG_DIR` to that Docker workspace's `config` directory. Do not re-run the Docker guide's `just init` step for this: it reuses the identity already generated for the same node `id`. For `identity.source = "gcp_secret"`, create a separate secret for the node. 3. Start the node again. Chain data does not need to be reset. 4. Confirm that both nodes now advance during an epoch, not only right after an epoch change. If other nodes dial this PFN as a seed, give them the new `account_address` and `network_public_key` from `identity.public.yaml`. For a **VFN or validator**, do not rotate the identity on your own. Other nodes and the validator set reference it. Stop the duplicate and contact the Gravity team. ### Prevention * Create every node's identity with its own node `id` in `cluster.toml`. This includes Docker nodes: running the Docker guide's identity step again with the same `id` returns the same identity. Do not copy `cluster/output//`, a deploy directory, or a Docker workspace to bring up another node. * To warm-start a new node from an existing one, copy only the chain databases (`consensus_db`, `quorumstoreDB`, and `reth`), as described in [Copy From an Existing Node You Control](/gravity-networks/run-a-mainnet-pfn.md#option-b-copy-from-an-existing-node-you-control). * Use one identity secret per node. ## See Also * [Run a Gravity Mainnet Public Full Node](/gravity-networks/run-a-mainnet-pfn.md) * [Run a Gravity Mainnet Public Full Node With Docker](/gravity-networks/run-a-mainnet-pfn-with-docker.md) * [Run a Gravity Mainnet VFN](/gravity-networks/run-a-mainnet-vfn.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.gravity.xyz/gravity-networks/node-operation-anti-patterns.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.