I have the following scenario:
- 1 node that created the blockchain and a stream, which I used to create a stream and add 2M items into it, with random off-chain content (images).
- 2 nodes that join the blockchain after all the data has been imported through the master node, so on startup they begin to catch up with the master node at their own pace.
At the very beginning when the 2 new joined nodes are initially catching up, I can issue commands to any of the 3 nodes using the multichain-cli tool. For instance, I send this command to the master node and I get the reply back immediately:
multichain-cli ipregister -rpcuser=multichainrpc -rpcpassword=m4lt1ch41nRpC! getaddresses
However, at some point in time, when the volume of data being already synced up grows, the multichain-cli tool stops responding for all the nodes, and as the volume of synced data still grows, the nodes start to degrade dramatically until they just do not respond to commands.
Both 2 nodes join the blockchain through the DNS name of the master and they just subscribe to the stream to be able to find data by key, that's all.
Then I stop everything (without deleting the data) and restart first the master node. It starts to respond again to multichain-cli commands. But when I start again the first of the 2 additional nodes, I start getting this from it:
mchn: Sending minimal parameter set to 220.127.116.11:8099
receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=28632, us=18.104.22.168:50952, peer=1
Added time data, samples 2, offset +0 (+0 minutes)
mchn: Parameter set from peer=1 verified
Loading addresses from DNS seeds (could take a while)
0 addresses found from DNS seeds
dnsseed thread exit
socket sending timeout: 61s
And then in the master:
Sending minimal parameter set to 22.214.171.124:50952
receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=28627, us=126.96.36.199:8099, peer=7
Added time data, samples 3, offset +0 (+0 minutes)
mchn: Parameter set from peer=7 verified
Sending minimal parameter set to 188.8.131.52:8099
receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=28627, us=184.108.40.206:58276, peer=12
mchn: Parameter set from peer=12 verified
Clearly, the 2nd node degrades just after joining again the blockchain. If I join the 3rd node it gets even worse.