Multichain performance drop on high transaction count after short period of time

Question

Hi.

I am running a performance test on MC 1.0.6 on AWS c5.4xlarge instance (single node).

Using 10 worker on the same machine (so no network latency) which are sending as fast as possible (but synchronoulys) and writing random key-values to the root stream. (key: 64Byte, value: 128byte)

 

I recognized a slow but steady drop in throughput of tx/s.

Tx/s started at 1100 and are getting lower over time. At the moment the Tx/s rate is only at 440 and that was 24 hours after I started the test.

Tx/s: https://snapshot.raintank.io/dashboard/snapshot/3RYKP94zrQvc1Gbl8MPwSE9XmI7mu7YE

#TX: https://snapshot.raintank.io/dashboard/snapshot/avBNI3h1YM391Kz3KigVKPhqeUQ1Cbzn

CPU is constantly at 20%

Memory is slowly rising: https://snapshot.raintank.io/dashboard/snapshot/eOEmIrenI3uQ00LKqvOktbk7UsmRp8HR

High disk usage  (blocks-folder is small, seems to be indexing)

https://snapshot.raintank.io/dashboard/snapshot/UP8euPezGN2A1b3K4kmPr3Dhr6FDqNqR

Is this an expected behaviour? I am quite curious how far the tx rate will drop, since we are already below our minimum expectation.

Answer 1

Your problem here is unlikely to be related to the sending transaction side, but rather to the fact that this node is subscribed to the root stream. If a node is subscribed to a stream, it is indexing that stream's content in real time, in about 10 different ways. Inevitably, inserting into an index gets slower as that index increases in size.

If you remove this subscription, you'll still see some slowdown, as the number of transactions in the node's wallet increases over time, but it should be a much weaker effect. If you wanted to, you could also remove that effect by building transactions externally so that they are not stored in the node's wallet.

Comments

I checked the volume metrics together with an aws technician. There seem to be no issues and there are no thresholds that would slow down the instance io wise.

Nevertheless I tripled the number of IOPS and used a high performance io1 volume. But this had no impact on the TX/s.
Node is still running at ~200TX/s.

---

Thanks for your reply. We would like to simulate your exact situation to see what is causing the bottleneck and excess memory usage. You can post the full details of the setup (including OS, MultiChain version, and the Go code) here, or if you prefer by private correspondence, please email to multichain dot debug at gmail.

---

Just an update on the "autosubscribe":
I also experience a TX/s drop when not subscribed to the streams I write to:
https://snapshot.raintank.io/dashboard/snapshot/w85YT3GZGmpMRuv570JbRzHGrH501oAo?orgId=2

---

Yes, we would expect some drop if the node is not subscribed, since the transactions are still being included in the wallet and indexed there. But it seems to be a weaker effect, which we'd also expect.

---

Ok so we have to be aware of this behavior since many scenarios use a blockchain to write a lot of hashes and check for existence / timestamp and publisher. It seems that we have an upper bound on the TX/s that depends on the target blockchain size (that is not always known upfront). I am also curious to see if the TX/s stays stable at some point or slowly approach 0.

Thanks for the information and your support so far.

Answer 2

Just a note for anyone else following this thread. The issue with increasing memory usage has been resolved. There's an unofficial 1.0.6.1 release of MultiChain available here (Linux only):

https://www.multichain.com/download/multichain-1.0.6.1.tar.gz

And the issue was also fixed in the official release of MultiChain 2.0 alpha 4.