Being prepared for catastrophic system failures
Disaster recovery strategies for blockchains are more flexible than those of regular databases, because the content of the chain is automatically replicated to (and can be recovered from) other nodes in the network. Nonetheless, there are two good reasons to back up individual nodes.
First and most importantly, if the private keys for a node’s addresses are not being deliberately stored outside its wallet, then it is vital to back up the wallet and its keys. If the private key for an address is permanently lost, no more transactions can be performed on behalf of that address.
Second, if a node needs to recover quickly from a system failure, without the delay caused by retrieving and reindexing the blockchain’s contents, then it may make sense to back up the node’s entire state instead.
Both of these methods are described below, along with instructions on how to restore each type of backup. It should also be noted that clustering can serve as an alternative to backups, and as a workaround for that fact that nodes must be stopped when backing up their full state.
Performing a wallet backup
A node’s wallet, which includes both private keys and watch-only addresses, is stored in the
wallet.dat file in that node’s blockchain directory. By default, this directory is
~/.multichain/[chain-name]/ on Linux and Mac and
%APPDATA%\MultiChain\[chain-name]\ on Windows.
In general, the
wallet.dat file can be backed up directly. However, if a node is running, it is best to use the
backupwallet API command to create a copy of the file under an exclusive lock.
Note that while backing up a node’s wallet, it may also be helpful to back up two more things:
- The list of assets and streams to which that node is subscribed. Use the
liststreamsAPI commands, and note the
nameof any entity for which
- The username and password for accessing the node’s API along with any other fixed runtime parameters. These are stored in the
multichain.conffile in the blockchain directory.
Restoring a wallet backup
The process of recreating a node from a wallet-only backup is as follows:
- Obtain the
nodeaddressof another node in the chain using the
- On the server to be recovered, make sure that the blockchain directory no longer exists.
- Start connecting to the blockchain in the usual way, passing the previously obtained node address as a parameter to
multichainddoes not stop with a message about needing
connectpermissions, stop it after a few seconds using
multichain-cli [chain-name] stop. (This can happen in a blockchain with parameter
- Replace the
wallet.datfile in the node’s new blockchain directory with the version that had been backed up.
- If a backup was kept of the node’s
multichain.conffile, restore that file as well.
- Restart the node in the usual way:
multichaind [chain-name] -daemon. In the event that the first address in the wallet no longer has
connectpermissions, use the
handshakelocalruntime parameter to specify which address to use for handshaking.
- Watch the
getinfoto see when the node has caught up with the rest of the network. For large chains, this may take a considerable amount of time.
- If a record was kept of the subscribed assets and streams, re-subscribe using the
subscribecommand without rescanning, e.g.
subscribe '["stream1","asset1"]' false. Then
stopthe node, restart it with the
-rescanoption, and wait for rescanning to complete.
That’s it! Your restored node should be up to date with the blockchain and ready to be used again.
Full state backup and restore
As an alternative to backing up and restoring the node’s wallet and some other small pieces of information, it is possible to make a straightforwards full backup of all the files in the blockchain directory, e.g.
~/.multichain/[chain-name]/ on Linux or Mac, or
%APPDATA%\MultiChain\[chain-name]\ on Windows. Restoring from a full backup is simply a matter of putting these files back in the right place, restarting the node, and waiting for it to catch up with any blocks created after the backup was made.
There are two key advantages to backing up the full state instead of the wallet only:
- The process of backup and recovery is much simpler, based on copying an entire directory of files.
- A functioning node can be instantly recovered without needing to retrieve and reindex the entire blockchain.
However full state backups also have two key disadvantages:
- The node must be stopped before the backup is performed, to ensure consistency in the backed up files. Clustering can help here – while one node in a cluster is stopped for backup, another can continue to serve the application.
- The total amount of data backed up will be drastically larger. However it should be noted that standard compression techniques such as
zipcan reduce this size by 50-75%, depending on the pattern of activity within the chain.