Getting different blocks on two nodes of the same chain

Question

I have setup a blockchain of 5 nodes. On each one of them I have hooked blocknotify to run the getlastblockinfo and getmempoolinfo commands. I save the output of these into a file. I compared this file for two nodes and somehow both have different blocks. Mostly the blocks are same but I spotted a few blocks that were not present on the other node. That block was not repeated or sent later to the other node, it was missing from that node completely.

It might be possible for the file to have missed it while writing, but in our case its not possible.
We have set our BCT to 15 seconds and mine-empty rounds is 0. In our case each block is created after 15 seconds. So its not possible for the file to have missed it.

Both nodes should get the same blocks otherwise one node misses some transactions and data and the whole chain is not so reliable anymore.

Please help me understand what's happening here.

Answer 1

First you should confirm that the two nodes currently have the same chain. This can be done easily by checking that the output of getlastblockinfo is the same on each. If the hash matches, so much the entire chain.

Assuming yes, then take a look at the blocks that differ between the two nodes. It is possible that there was some temporary non-consensus across the network, in which case a block might have been included in one node's chain but not in another's. This is not a cause for concern since the network automatically re-establishes consensus, unwinding a block and including another one where necessary. You can query any particular block using the getblock API with its hash – if confirmations=-1 then it's not on the active chain.