Mining stopped, here is what happened

+1 vote
- created first node in September 2017
- granted permission to second node
- granted permission to 3rd node

three years later

- one of the server failed (1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj)
- I detached the volume
- created a new server, attached the volume to access the multichain data
- copied the chain folder to the new hdd
- detached the old volume
- restarted the node with the same params

I wasnt able to mine. I tried various things to get this node to run properly.
I added IP address to peers, checked firewall, restarted a number of times.
I used addnode to add the node again, tried granting access to the node again.

I then added a 4th (1bWg4kH68PCkusaSDC12R2TrnqD2jzBs8WFicA) node, granted permission and tried to get it running without 1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj. I couldn't get past the other node because of mining-diversity being set to 1. However, I subsequently rolled back before this node was added and have been trying to clear the bogus 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N address.

There have been various attempts after that, thanks to Micheal's suggestions, I have rolled back further clearmempool, used bantx, revoked and still mining wont get past this.

You can see that the chain stop/starts
#11171 it stops
#11172 I managed to get it running again

You can see that the three miners have always been there since 2017

genesis
1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD
added at block 166
1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY
added at block 168
1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj

There has been one other developer with access, and I cant discount something unexpected done on the chain, but it has only been me over the last 6 months, and its been running untouched.
We have not exported the private key, we have not imported addresses between nodes. It's been a fairly simple, untouched setup.

 

To follow this up, I have a SQL database with all the wallet addresses mapped to users.

I have an audit trail and logs. The 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N has never appeared in any of this data.
 

Listminers

Node 1
[
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : "other-local-address-preferred"
    },
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11189,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11190,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : true,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : "no-mine-permissions"
    }
]

 

Node 2

[
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : null
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : false,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : null
    }
]

 

Node 3

[
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : null
    },
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : false,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : null
    }
]
asked May 23 by MaSsv
Is this a new issue, or is it covered by the other long thread?

1 Answer

+1 vote

Although I cannot account for the 4th miner, or why the fork occurred, the issue was resolved by rolling back the fork, bantx a few transactions and other commands all in a coordinated way. 
 

See the end of the long set of comments on this thread.
https://www.multichain.com/qa/5386/mining-stop-possible-bug

 

answered May 25 by MaSsv
...