Nodes not mining

+1 vote
The mining in a blockchain has unexpectedly stopped, I had a blockchain running for about a couple of months, the blockchain has three nodes all with mining permissions, and mining diversity set to 0.66.

The blockchain was working fine but today I noticed excess memory consumption by the blockchain, on closer inspection I found that the mining has stopped about a month ago, and the mempool is filled with about 4000 transactions.

 

All three nodes are still connected and broadcasting new transactions to each other but none of the transactions is getting mined, the last block was mined on 24th June 2020.

 

In getinfo api call, the mining paused parameter is false.

 

This is the debug log, which also shows something different happening after 24th june 2020

2020-06-24 23:32:30 ResendWalletTransactions()
2020-06-24 23:32:39 UpdateTip:            new best=00930c22e713129df7458d6c5e20ca3de5c17ec4ff1ba0b175f489734c4f73b2  height=31160  log2_work=22.927454  tx=34372  date=2020-06-24 23:31:24 progress=1.000000  cache=0
2020-06-24 23:32:41 MultiChainMiner: Block Found - 002120d5c508d36fb47ebc1ebf5f63fa2a259288acc299834d41d2c5b0820b70, prev: 00930c22e713129df7458d6c5e20ca3de5c17ec4ff1ba0b175f489734c4f73b2, height: 31161, txs: 1
2020-06-24 23:32:41 UpdateTip:            new best=002120d5c508d36fb47ebc1ebf5f63fa2a259288acc299834d41d2c5b0820b70  height=31161  log2_work=22.9275  tx=34373  date=2020-06-24 23:32:41 progress=1.000000  cache=0
2020-06-24 23:33:03 ResendWalletTransactions()
2020-06-26 16:18:21 CommitTransaction: 1f1ab826af80026512b92be1a9c515722405e2788e5cac536ab3b94a761e3a09, vin: 1, vout: 2
2020-06-26 16:25:02 CommitTransaction: b9f51c3f27fdb2039a5f6dbb3567206dbdb5d0dce0f5416fa0a1ce2b9330b05e, vin: 1, vout: 2
2020-06-26 16:25:12 CommitTransaction: 38083a29ced1dbf2db12f18241f8ef9ebf10d04252af0020f2b8b36df7d35323, vin: 1, vout: 2
2020-06-26 16:25:22 CommitTransaction: bd189c83a98aebd1e06013951fdd69f62ff899646b8b84be84b68b28e482748e, vin: 1, vout: 2
2020-06-26 16:25:52 CommitTransaction: 425b2ca954dc1ab436e891b2daafa7217034f89183915e531eb4013f161a02a8, vin: 1, vout: 2
2020-06-26 16:26:02 CommitTransaction: 961a228bd581f965ca6060486d49b234e14b921ac37aacc8ffc897478297f281, vin: 1, vout: 2
2020-06-26 16:26:12 CommitTransaction: 639bda696c2ae49fe476c7a4ad608610659a20fdad3e035e45be897b30270896, vin: 1, vout: 2
2020-06-26 16:26:22 CommitTransaction: b9308a0699474402195b9e6a39c446c304bf8cbd0a6e070f43a688941efaba8f, vin: 1, vout: 2
2020-06-26 16:27:52 CommitTransaction: eae40ed402ae2a5e6402b49abc6798031ecab43e7d094416e299cfcb785782d8, vin: 1, vout: 2
2020-06-26 16:28:02 CommitTransaction: da7d7ab2668fda67901283335e4db1c412e60dbdcd71bba33100471197364e28, vin: 1, vout: 2
2020-06-26 16:28:12 CommitTransaction: f6335a9cc5b70201217f4d094c481cbaae79e129146a4af59211690d01c70587, vin: 1, vout: 2
2020-06-26 16:28:22 CommitTransaction: 71ddc56bb11311be177d0bdcb3b7bcdfb8c2a00df7a9eb4dc41ae8a486c4d166, vin: 1, vout: 2
2020-06-26 16:28:32 CommitTransaction: 45d3ea3644fe15aff4a7f1597af769a9e15c0bfbef77d6ea9b23627ba840ca20, vin: 1, vout: 2
2020-06-26 16:30:22 CommitTransaction:

 

.........

2020-07-05 15:32:39 socket sending timeout: 61s
2020-07-05 15:35:15 mchn: Sending minimal parameter set to 10.0.2.218:47216
2020-07-05 15:35:15 receive version message: /MultiChain:0.2.0.10/: version 70002, blocks=31161, us=11.0.2.19:6011, peer=44
2020-07-05 15:35:15 mchn: Connection from 1LWFDzTX9GHwHdpx8zBLExZbwAevCGC8s3EuVS received on peer=44 in verack
2020-07-05 15:35:15 mchn: Connection from 1LWFDzTX9GHwHdpx8zBLExZbwAevCGC8s3EuVS received on peer=44 in verackack (10.0.2.218:47216)
2020-07-05 15:35:15 mchn: Parameter set from peer=44 verified
2020-07-05 15:35:16 mchn: Synced with node 44 on block 31161 - requesting mempool
2020-07-05 15:35:16 mchn: Sending minimal parameter set to 10.0.4.238:6011
2020-07-05 15:35:16 receive version message: /MultiChain:0.2.0.10/: version 70002, blocks=31160, us=11.0.2.19:58198, peer=45
2020-07-05 15:35:16 mchn: Connection from 149kFCMKAenKXZAcENGLD7FMB5BvuXt1JhHGSb received on peer=45 in verack
2020-07-05 15:35:16 mchn: Connection from 149kFCMKAenKXZAcENGLD7FMB5BvuXt1JhHGSb received on peer=45 in verackack (10.0.4.238:6011)
2020-07-05 15:35:16 mchn: Parameter set from peer=45 verified
2020-07-05 19:31:50 CommitTransaction: 0213d891653686ee1dc9ba5153077aa73c7734abe8cd90a424a3f5ce07b71051, vin: 1, vout: 2
2020-07-05 19:32:09 CommitTransaction: e617efe2efee075b85fdcd0ce6ac641a22e19e5e4f49cb8f64c7c2542f2a95d0, vin: 1, vout: 2
2020-07-05 19:35:08 CommitTransaction: 61ee58ce6c5fd3b1e2a2a8e858709d6196c5f832df7dc090d494f8f5517d6937, vin: 1, vout: 2
2020-07-06 04:55:41 CommitTransaction: 3a02331bea3751f9ccaf3853da3d0bbd45ae066b51885bd07a2518eedb032bdd, vin: 1, vout: 2
2020-07-06 05:11:08 CommitTransaction: cb20a200e4be2321faf58ebcfddcdd6d6de4745f0ae08cb7d4581c410e56dd43, vin: 1, vout: 2
2020-07-06 05:14:39 CommitTransaction: 1b695abd3ec00e89c84434486ab8f3542b5a7f1e4985a27fd48f2f2b87497364, vin: 1, vout: 2
2020-07-06 05:26:15 CommitTransaction: 4c732e38089e3006fe91556bb439033962bd3ed9bae38bfb45f151656de95d0c, vin: 1, vout: 2
2020-07-06 05:29:38 CommitTransaction: acfd840431885ea2b2c4744dbc5380e91a50890128e164ac2a5d9175ec25f5b8, vin: 1, vout: 2
2020-07-06 05:32:59 CommitTransaction: 405f1729041503fa7a6af5d86851de1fb80d60a05040e4f5b49a0551202d7023, vin: 1, vout: 2
2020-07-06 05:50:54 CommitTransaction: 34f0f285471d4b1e0ace1b42b94df6992ab7f2130b1e7ceb7a80d9018e826097, vin: 1, vout: 2
2020-07-06 06:06:15 CommitTransaction: 36b7c44a74619a98acbd1018179b26c8aa47b66edc03e6e76bf38ff316372d9d, vin: 1, vout: 2
2020-07-06 06:06:32 CommitTransaction: 257f061dc05453f11f1ce6116d29df8336893e220b006f4d64b7b9f0bc5c97be, vin: 1, vout: 2
2020-07-06 06:19:55 CommitTransaction: 71596557511585e33d0809f3182165232e7a4facb34c8a35cdf5e437000e1f98, vin: 1, vout: 2
2020-07-06 06:21:25 CommitTransaction: 424815aa076186c4fd8ca66412371ebb37657513858b85b6314dbfd52745834d, vin: 1, vout: 2
2020-07-06 06:24:13 CommitTransaction: bb770909698bf71d99f2e02a677bd58748878b7f395f119156cd7f7088a93b3e, vin: 1, vout: 2
2020-07-06 06:27:54 CommitTransaction: 080825c10bd21852a35f21c5246ef39e2e15ab5c4e969993cb37c398b5879cf8, vin: 1, vout: 2
2020-07-06 06:45:58 CommitTransaction: 0c0837454310870bda106667b26951d9663517697f3640eab1ce526a7f6f7437, vin: 1, vout: 2
2020-07-06 06:57:58 CommitTransaction: 1fc900bbb1b451d086d245390dc105f1e63462c0c1a7d94044d95be66b370c8f, vin: 1, vout: 2
2020-07-06 06:58:09 CommitTransaction: 240a1a82dcd2ec4b224854ddbfcaba036e1992df3d8543ef409df4908ab067b6, vin: 1, vout: 2
2020-07-06 08:41:09 CommitTransaction: 433eea579db4969491960d4856c1d7938f042d9330dc6766c77ba937cb622876, vin: 1, vout: 2
2020-07-06 08:41:29 CommitTransaction: e240a48e37eb9bbac01048bb39e10df98ae2bfb8e73e194f21604c75d3f2dde1, vin: 1, vout: 2
2020-07-06 08:45:55 CommitTransaction: 4ac900ac0137b28767e0ad708ef6e335b9e1b83b3e825390eb98ed7a8e5f527e, vin: 1, vout: 2
2020-07-06 08:47:44 CommitTransaction: 7b309d628c715f631d94fec1c50c5732f3bf4423c6d9615080865624d24c559e, vin: 1, vout: 2
2020-07-06 08:48:30 CommitTransaction: 7bd43c5e4ff5152a7195ba9d0e288a566c4e80041ed31074fa5613499e231192, vin: 1, vout: 2
2020-07-06 08:49:13 CommitTransaction: 411ec0d859f8e3183e3b54782f4214e2d4e26c6ea8806247f20932f864ed4570, vin: 1, vout: 2
2020-07-06 08:49:34 CommitTransaction: d81d15d5c67b3ecf190271121491871b5226278a40bf7cf485be4907d8393d8b, vin: 1, vout: 2

 

Did not post the full log as the logs are quite a lot. The logs end with commit transaction

 

Not able to find the exact cause of the problem.

 

Thanks
asked Jul 18, 2020 by DarkBlaze
edited Jul 19, 2020 by DarkBlaze

1 Answer

0 votes

I would guess the most likely explanation is a consensus issue, that not enough of the nodes with the mine permissions are online and connected to each other, to reach the required mining diversity.

You can use listpermissions mine to see which addresses have mining permission, and then confirm that you have nodes online with these addresses in their wallets (including the private keys, showing ismine: true in listaddresses), and that they are connected to each other (see getpeerinfo).

answered Jul 19, 2020 by MultiChain
all three nodes are still connected, there are two entries in peerinfo result, but there is one address which is not present in any of the nodes, think that is causing the issue if I revoke the mine permission to that node will mining start again? Is there any way to restart the mining without clearing the mempool. Should I revoke the mining permissions of other nodes to test the working again?
How many miners are defined in listpermissions mine?
...