The speed of publishing items to stream

Question

Hi,
  I have 2 nodes named A and B. I create a stream named AStream on node A, then I grant receive,send authentication of Node A and write authentication of AStream to Node B.
  After that, I code a JAVA project using JSON to publish items to AStream on Node B, then Node A will receive this items from Node B.
  PS. The JAVA project make 1 JSON request contains 10,000 items, so the JAVA projcet publish 10,000 items once.
  Now I publish 10 million items to AStream, but I find that the count of items in AStream is not increase, getmempoolinfo command shows there are about 2 million items in memory pool which have not yet been confirmed.
  Q1: What is the confirm speed of AStream between 2 nodes ?  
  If the publish speed faster than confirm speed, I can make the publish speed slower.
  Q2: After I use pause incoming,mining and clearmempool command, the memory pool size is 0, but I can't publish items any more, the error message is :
  Error code: -26
  Error message: this transcation was rejected. This my be because you are sharing private keys between nodes, and another nodes has spent the funds used by this transaction.
  However, I only publish items to AStream without other operations, I don't understand the error means what, and how I could resolve this problem?
  Q3: What is the maximum number of concurrent publish items to a stream ?
  Q4: What is the maximum number of items per stream? or What is the maximum size of items per stream? could I publish items to a stream as many as I can?
  Q5: When I publish items to a stream, is the item published before will be affected? like sort all of them? update some hash value of them?
  Q6: Do you have some suggestion for me to improve the speed of publishing items to a stream? I want to publish 1 billion items to a stream in average 500 items per second, but this seems too diffcult for me now.

  Thanks for you time, hope your reply of each questions.

Answer 1

What i've learned from everything is that:

A stream is just a NoSQL key-value fashioned way of storing things into the blockchain. There isn't a maximum. So as long you have unlimited resources in terms of storage , then you can add unlimited amounts to the blockchain. Just like the bitcoin Blockchain can expand exponentionally as long there are sufficient amount of storage resources.

Q1: depends on answer of Q6

Q2: i think you need to do it in this ordering: https://www.multichain.com/qa/2311/how-do-i-revoke-a-transaction

dunno for sure

Q3: Depends on the consensus. Local or Global ordering. Tradeoff between global consistency and local consistency. I think

Q4: See above text.

Q5: No, It doesn't update the 'record' before it. If it updates it what is the point of using blockchain over a ACID strict database. What it does it appends a new record with a new timestamp and a new txHash of that transaction and bounds it to a key in the fashion of NoSQL key-value store or document store. The result is that you can review the complete revision history of that particular key based on timestamp they were first accepted on the blockchain. So if you append exactly the same record without any changes, it would still append a new record , with a new txId. It doesn't care about redundancy of records. And doesn't update them accordingly

Note: The transaction itself doesn't got a timestamp. The timestamp is given when a miner starts mining a block and inserts that transaction into the block.

Note: Make Sure that your node A and B time is correctly configured otherwise it will mess up the transaction timestamps. If your nodes are miners.

Q6: i think you could achieve it if you reduce blocktimes, however more blocks are created. So its a tradeoff. However its better to manage by the miners instead of putting a shitload of transactions into one block which could take ages to calculatecomplete. Which hits the availability of your system.

There is always a tradeoff between availability and @ eventual consistency.

Answer 2

Lots of questions here, but here is our basic recommendation.

If you want to publish multiple stream items in a single transaction, you need to be on MultiChain 2.0 alpha 1. You can then do this using createrawsendfrom. I would recommend publishing something like (say) 100 items per transaction to keep the transactions to a reasonable size.

After you send every (say) 100 transactions, check the memory pool size with getmempoolinfo. If its bytes value is more than the size of (say) two blocks then pause your publishing process and give the chain a chance to catch up and confirmed previous transactions, until the bytes value goes below that threshold again.

You can tweak all these numbers for optimal performance but these basic guidelines should work OK.