Transaction missing on one node (using sendwithmetadata and listaddresstransactions)

Question

Hi MultiChain Team,

We are testing a small network with few nodes on Amazon Linux 2 to send some transactions (using sendwithmetadata), and found that sometimes a transaction is missing on one of the nodes (using listaddresstransactions). For example, node X only contains 99 transactions, while all other nodes contain 100 transactions. We have tried both v2.0.1 and v1.0.8 and found the same situation. We have also tested on Ubuntu and did not observe this situation. Just wondering if there may be any configuration that we can adjust to avoid the transaction lost?

Thank you very much!

Answer 1

That is very strange - are you able to check that the nodes that are giving answers are fully synchronized with each other in terms of the blockchain? You can check using the getblockchaininfo API on all the nodes – they should have the same getbestblockhash.

Also I'm assuming that all nodes are tracking this address using importaddress (or perhaps importprivkey), and then when you used that command, you kept the rescan parameter on its default value of true? If not, some nodes will not have checked the blockchain from the start for transactions relating to that address, and that would explain the discrepancy you see in the output from listaddresstransactions.

Comments

Thank you for the replies! Regarding the checks:

1) Yes, the "getblockchaininfo" API gives the same "getbestblockhash" (actually all info are the same, with ~2,000 blocks).

2) Yes, we used "importaddress" and kept "rescan" as its default value.

After few more tests, we found that reason of the missing transaction is actually because a node stops receiving transactions from another node. For example, node 1 created the address and other nodes imported the address, and every node write/read that address. After some time (e.g., ~2,000 blocks, which may vary in each test), node 1 can no longer see transactions sent by node 2. However, node 1 can see transactions from all other nodes, and all other nodes can see transactions from node 2. It seemed like only happen between node 1 and 2 - we reset MultiChain and rerun it, and similar situation happened again (i.e., exact node 1 stop seeing transactions from exact node 2). We are also checking if this is because of firewall/antivirus setting so that the port of node 1 is blocked for node 2.

Just wondering if there may be any other possibilities on the MultiChain side that we can do some check as well?

Thank you very much!

---

Is it possible you sent two conflicting transactions from two nodes simultaneously, using the same address? If so it is quite normal that one node's transaction will not be accepted by the blockchain, and therefore not by the rest of the network. So the transaction will only be visible to that node.

---

Just tried to setup another set of nodes, and found that everything works fine without the issue. Will continue to investigate and see where the cause of the issue may be. Thank you so much for helping!

---

Just realized that we actually used "false" for the rescan parameter, sorry about the wrong information. However with rescan=false, MultiChain seems still work perfectly on another set of nodes. Will keep finding the cause of the issue. Thank you very much!

---

We tried again setting rescan=true, as well as adding -rescan=1 to the runtime parameter of multichaind. However, after 2,000~2,500 blocks, the same situation happened again. We used -debug=mchn, -debug=mchnminor and -debug=net as well, and from the log we could not identify errors near the time that node 1 (i.e., the seed node) stopped receiving transaction from node 2 (i.e., the first node joining the network). Few findings:

1) This issue happened only between node 1 and 2 - only node 1 could not see transactions from node 2 after certain time point, and other nodes were all good.

2) Once this issue happened, even if node 1 created a new address for node 2 to import, node 1 could still no longer receive transaction from node 2 on the new address.

3) We have also tried different ways to connect the nodes, such as all nodes connected to node 1 (i.e., seed node), or chained them (i.e., node 1 <-- node 2 <-- node 3 <-- ...), and the same situation remained (node 1 stops receiving transaction from node 2). Since we have already initiated a new set of VMs, not sure why this only happened between node 1 and 2.

Could you please advice what else we may check?

---

Again, please first check if the nodes are synchronized in terms of the "getbestblockhash" field of the "getblockchaininfo" command. If they are synchronized, and both subscribed to the stream with "rescan=true", and you are not sending conflicting transactions from the same address from two nodes simultaneously, then please let us know. If the nodes are not synchronized, also check that the server times are not too far apart from each other.

---

Will do! Meanwhile, we have two quick follow-up questions:

1) The server times are apart from each other within 5 seconds. Would this be okay for MultiChain?

2) Since we are running MultiChain in a private network, just wondering if there may be any need for MultiChain to perform external IP calls (e.g., connecting to a public URL)?

---

1) 5 seconds should be fine

2) No external network access is required

---

The problem still occur in several trials, and we found that the transactions from node 2 are no longer included in the blocks (while the transactions from other nodes are still included in the blocks) after certain time point. We also confirmed that the nodes are still synchronized (with the same "getbestblockhash" value).

As for the conflicting transactions, how can we detect it since we are using amount = 0 (i.e., only storing metadata)?

Also, even if there are some conflicting transactions, will they cause a node (e.g., node 1) to never accept transactions from another node (e.g., node 2) thereafter?

---

You cannot send transactions simultaneously from two nodes using the same address - this will lead to conflicts. This is true even if the transactions are not transferring assets. See 'Approaches for writing' on this page: https://www.multichain.com/developers/clustering-high-availability/

---

Many thanks, just another update: after the situation (i.e., node 1 not receiving transactions from node 2) happened again, we tried to restart only node 1 and 2 (with -rescan=1 runtime parameter), and MultiChain seemed continue to work properly. Just wondering if this may provide additional information regarding the root cause of the issue?

---

This is expected, because the content of each node's wallet is regenerated from the blockchain, i.e. from transactions that were actually confirmed. The root cause of the issue is that you cannot send transactions simultaneously from two nodes using the same address – that's all there is to it.