This is follow up of a previous thread regarding dropped connections, that was getting lengthy, so starting a new thread with some concrete observations this time. ( https://www.multichain.com/qa/7815/multichain-benchmarking-with-nodejs-wrapper )
As mentioned previously, I said I was getting two errors predominantly, ETIMEDOUT and ECONNRESET.
For ETIMEDOUT, it turned out that the client itself was dropping requests, so they never reached the peer and the error callback fired. So I'm guessing a single node process cant handle send rates above 500tps, and this is clearly a client problem.
Now, I used to get ECONNRESET only if I send a much higher send rates. So now, I chose two clients, with send rates that both can handle (to avoid ECONNTIMEDOUT problems), and now I started getting ECONNRESET. I did a tcpdump to check again if requests are there, and they were present both on client and peer machines, so no problems there. --debug=mcapi doesnt show this though, so somewhere after receiving the request, its dropped either by Multichain or the OS (but tcpdump shows the request so I'm guessing its the former). But this definitely rules out any client side issues for this error.
You guys had initially pointed out (in the previous thread) that it could be that CPU is being used beyond its capability, so I ran htop and took some screenshots to log the CPU usage. And unsurprisingly, the reset problems were happening for high usage, particularly when one of the cores hit 60%+, and generally the process usage was 100%+
Some Screenshots of peer CPU - https://imgur.com/a/R1zC0
Is this kind of CPU usage normal? And only way to be able to handle is a better system if all parameters are kept the same?
Multichain version v1.0.2
Command: multichaind $CHAINNAME -debug=mcapi -printtoconsole -autosubscribe=streams -blocknotify="script.sh %s"
Blockchain params are default except target-block-time is set to 2 seconds
CPU is Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz (4 cores with Hyperthreading x2)