Secondary nodes not recording new blocks

Question

I have an issue where secondary nodes are receiving new blocks, but not updating their block count or storing the block, wondering if there are any thoughts on what to cover next.

Peers connections are ok and connection is confirmed with the mining node and debug log produces no errors.

Doing a -reindex=1 multichaind start rebuilds but only up to the previous 137121 blocks. Also tried removing all bocks/chainstate/permissions/entities content as well before a reindex but same behaviour.

Permissions seem ok with the secondary node having connect permission and access to all blocks

        "type" : "connect",
        "startblock" : 0,
        "endblock" : 4294967295

 

From debug.log - receiving a new block from mining node - 148711 and gets header, but node stays on 137121

mchn: RECV: inv, peer=34
received: inv (37 bytes) peer=34
got inv: block 000000712d15c34bf8ad0403bc5ef68d7138a0216eb3df63849707d11a7ea30d  new peer=34
sending: getheaders mchn: SEND: getheaders
(965 bytes) peer=34
getheaders (148711) 000000712d15c34bf8ad0403bc5ef68d7138a0216eb3df63849707d11a7ea30d to peer=34
mchn: RECV: headers, peer=34
received: headers (82 bytes) peer=34

 

multichain-cli test getinfo
{"method":"getinfo","params":[],"id":1,"chain_name":"test"}

{
    "version" : "1.0 alpha 23",
    "protocolversion" : 10005,
    "chainname" : "test",
    "description" : "TestChain",
    "protocol" : "multichain",
    "port" : 8341,
    "setupblocks" : 60,
    "nodeaddress" : "test@10.x.x.x:8341",
    "burnaddress" : "aXXXXXXXXXXXXXXXXXXXXXXXXXXXViQbzh",
    "incomingpaused" : false,
    "miningpaused" : false,
    "walletversion" : 60000,
    "balance" : 0.00000000,
    "walletdbversion" : 2,
    "reindex" : false,
    "blocks" : 137121,
    "timeoffset" : 0,
    "connections" : 6,
    "proxy" : "",
    "difficulty" : 0.00395623,
    "testnet" : false,
    "keypoololdest" : 1480382861,
    "keypoolsize" : 2,
    "paytxfee" : 0.00000000,
    "relayfee" : 0.00000000,
    "errors" : ""
}

Answer 1

This does sound like a bug. The first thing I'd suggest is upgrading to the most recent version (I see you're on alpha 23 which is now quite old) and then reindexing, then at least we can rule out some old bug.

If the problem persists in the latest version we'll ask you to stop the node showing the problems, zip up its blockchain directory, then send it to us for diagnosis. Let me know and I'll provide the email address.

Comments

Got it working - tried two things at once though - not 100% certain which one fixed it.

This time around I manually removed all existing data on the faulty node -
/blocks
/chainstate
entities.*
permissions.*
/wallet

and started again with a reindex=1 ( a basic reindex=1 only rebuilds some of the components ? ).
Also disconnected all other nodes except the miner to force a single source.

After these it resynced fine with the miner and has stayed in sync. So I have two options - faulty node in the network send bad chainstate or more likely a local corrupt chainstate or wallet freezing it at a particular block. There were no errors though indicating corruption in local files.

---

Well, glad you fixed it. In any event we recommend upgrading to the latest version. And if you see this problem again we'd appreciate an opportunity to debug it by looking at the blockchain files, if you're willing.