Slow Starting Nodes

This section gives general recommendations if your nodes are slow to start

Nodes Are Slow to Start

Normally nodes should be up and running after a restart within 5 minutes. If your nodes always takes 10 or more minutes to startup, see if it is due to one of these reasons:

Reason 1: Database not synced

This usually happens in the following circumstances:

• Node is newly setup

• Node was shutdown for some time before starting up

• A recent upgrade requires a db sync

Solution: Wait for the database to resync. Next time you startup the node, it should not take that long. In the case of node upgrades, always read the release notes where it is usually stated if there will be a significant delay on startup

Reason 2: Hardware/Network

If you are running on hardware that is old, has limited resources (e.g. cpu, memory, hard disk, network bandwidth does not fit current recommended specifications), then your startup will be slow. It is important to monitor this closely to ensure that this does not prevent you from minting blocks. Use monitoring tools to check out utilization levels

Solution: If you see frequently high resource utilization, it is a good idea to upgrade the component that is causing problems. If only one component is causing problems (e.g. lack of memory), you may choose to upgrade only that component.

Reason 3: Ungraceful Shutdown

In some environments, it seems that the shutdown command ungracefully terminates the cardano-node even if a service has been setup to properly terminate it during a shutdown. This triggers longer pre-start checks on the next start up. (Thanks to Chris [HONEY] for contributing to this section).

Solution

You need to ensure that the node is terminating gracefully. Graceful termination requires a SIGINT to be sent to the cardano-node process. Depending on how you are running your node this can be achieved in multiple ways:

1. Directly terminating the process

If you are running the process yourself (unlikely unless testing) pressing ctrl+c is enough to send SIGINT.

If you are trying to terminate the process based on PID you can run this at the command line:

kill -2 <pid of cardano-node>

In the above example, -2 is the equivalent of -SIGINT

2. Terminating via systemctl

If you have a service setup to control the cardano-node via systemctl then you need to ensure the service has the correct signals set up, e.g. ensure these parameters are set:

[Service] 
KillSignal=SIGINT 
RestartKillSignal=SIGINT

Note: This will only work if the signals are propagated to the cardano-node process.

3. Ensuring correct signal via wrapper shell script

If you are in a situation where you are using a wrapper shell script and you cannot control the signal being passed to it you can use something like this:

#!/bin/bash -eu 
_term_int() { 
echo "Caught SIGINT signal!" 
kill -2 "$child" 
sleep 15 
} 

_term_term() { 
echo "Caught SIGTERM signal!" 
kill -2 "$child" 
sleep 15 
} 

trap _term_term SIGTERM 
trap _term_int SIGINT 

cardano-node run \ 
--config "mainnet-config.json" \ 
--database-path "db" \ 
--port "3001" \ 
--socket-path "path/to/socket/node.socket" \ 
--topology "mainnet-topology.json" & 

child=$! 
wait "$child"

This will trap both SIGTERM and SIGINT and ensure a SIGINT is passed to the cardano-node. In the above example, there is also a “sleep” command to ensure the underlying process has enough time to complete a graceful shutdown.

4. Shutting down via docker

There are various ways to achieve this, but to kill a cardano-node running with docker from the command line one can use something similar to:

```shell docker kill --signal=SIGINT cardano_container

5. Shutting down manually via systemctl

If the above steps do not work for you, an option that some people find useful is to shutdown the cardano service manually before performing the actual shutdown.

sudo systemctl stop cardano-node.service
sudo shutdown now