Not able to recover/start a node.

Thu, 30 Nov 2017 11:31:40 +0000

Hi Team,

I 've a 2 peer multichain system and I was running few performance tests on the system by generating 20 rps and I was using the sendassetfrom api to another single address. So my node was holding only 2 wallets (watch only) one for the sender and one for the receiver. The autocombine settings was the default one, so my node was generating a lot of utxo(s).

Now eventually my node crashed and since then I 'm not able to recover my genesis node which had the admin permission. My 2nd node is up but it didn't have the admin permissions. So now I can't add other peers to my multichain since I can't grant permissions from the 2nd node.

Initially my nodes were running on m4.large instance and when it crashed, I thought of bringing the node up by upgrading the node to bigger instances, I even tried with c4.8x large system for that multichain but the node never recovered.

When I started the daemon on the big machine it basically hanged for ever. Logs see below link.

https://gist.github.com/ashish235/7ad9aaf8f0311577b6cf4d87754200ba

Then I though of decreasing the utxo from the 2nd node, so I triggered autocombine manually and brought the utxo to just 18 or 19. Then I tried again to start the primary, this time after a good 20 mins I got the message node started but my blocks never got synced to each other.

See the logs below.

https://gist.github.com/ashish235/b85691a309d1bd10761ad49f1cc49cd3

Even then there was no transactions happening on the nodes, my blockcount on the 2nd node kept on increasing and my primary node just couldn't catch it, it was always 200 blocks behind. I left the system overnight and check in the morning, the difference of the synced blocks was same and all I could see the logs on primary is this. Please see below. I got about a million at least of this invalid flag message. When I checked on both the nodes this txid didn't exist, maybe this is not txid. :)

2017-11-28 19:21:59 Tx 35b807faa046809a3708f9fcea81b87ffab12a98bcbc2a43ddd00d4c1dec7934 was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx ecb81a6495211e3435a5cee6a11c3026fc943456e3752bbccd35f07ab1afaf2a was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx dd94407d874182acdc3176d54fd2377793b8bacb4c82a2a65715fd3b914aaaf0 was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx 5c733ed7c495d2e5396b0f38d6348dc35b570ce599534d14e96425caf8e4244e was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx 1d267fa07e6b1fe67670c955c3f11c44070e66467005f85f76f01eb82fa3352d was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx 8acf5bb55a082565625c3df9ca072ec9db5b74cb5331bb1233b8d80322474839 was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx 432a0f1c48453315b3d36bfb9defc822bc28c9b58f730023630c568ae4ee47d6 was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx 10852321116e5bcd25d2a7bf3b39aec262598f4226044ba90c7025b9b565d18d was not accepted to mempool, setting INVALID flag
2017-11-28 19:21:59 Tx f3ca1390b3faf35b4fe42a35c3521c0b796e454a3a5552f74fd63eda45e3d346 was not accepted to mempool, setting INVALID flag
mchn: Sending minimal parameter set to 172.31.17.70:35518
2017-11-28 19:21:59 receive version message: /MultiChain:0.1.0.9/: version 70002, blocks=4742, us=172.31.28.155:4411, peer=20
2017-11-28 19:21:59 mchn: Connection from 13zFHBxiYKbVBdefyiDG3A8EkqDUJa8xabRQLq received on peer=20 in verack
2017-11-28 19:21:59 mchn: Sending minimal parameter set to 172.31.17.70:4411
2017-11-28 19:21:59 receive version message: /MultiChain:0.1.0.9/: version 70002, blocks=4742, us=172.31.28.155:55716, peer=21
2017-11-28 19:21:59 mchn: Connection from 13zFHBxiYKbVBdefyiDG3A8EkqDUJa8xabRQLq received on peer=20 in verackack (172.31.17.70:35518)
2017-11-28 19:21:59 mchn: Parameter set from peer=20 verified
2017-11-28 19:21:59 ResendWalletTransactions()

In between all of this one strange thing also happened which I think is worth mentioning during the heavy load testing, my multichain was once not responding so I stopped multichain and it said

[ec2-user@ip-172-31-28-155 ~]$ multichain-cli chain1 stop
{"method":"stop","params":[],"id":1,"chain_name":"chain1"}
error: no response from server

So I killed the daemon myself.

Please any pointers on how can I avoid such issues and how to get my primary node back? Let me know if I can help with any other information to help debug this.

Thanks.

Recovering nodes

Fri, 29 Jan 2016 10:41:16 +0000

I'm trying to recover a test chain that has a corrupt admin node database and can not start.

I have 3 healthy nodes running, the only admin node is corrupt and on start displays: "ERROR: Couldn't initialize asset database for network <chain>. Exiting..."

I was under the impression that this wouldn't be a problem since I have a wallet.dat and multichain.conf (see http://www.multichain.com/qa/304/rebuilding-the-block-database?show=307#a307 ). When I try this I am asked to give permission to connect on another node but there is no admin node left, so how do I recover this test node?

Recovering a blockchain

Mon, 04 Jan 2016 10:11:28 +0000

I had two instances of multichain running on the same machine, and the machine crashed. I'm now unable to start either of these instances. The error I'm getting is:

me@518781d398a5:~$ multichaind -datadir=/home/me/mc2 chain1

MultiChain Core Daemon build 1.0 alpha 14 protocol 10003

ERROR: Couldn't initialize permission database for blockchain chain1. Probably multichaind for this blockchain is already running. Exiting...

I checked (ps -x / ps -ef), and there's no multichain process on this machine. Is there anything I can do to restart the blockchain, or do I need to create it from scratch?

MultiChain Developer Q&A - Recent questions tagged recovery

Not able to recover/start a node.

Recovering nodes

Recovering a blockchain