Performance Streams &. Key-Value

Question

1. What is more performant (regarding r/w access time):

a) Creating a lot of Streams with several keys

b) Creating several Streams with a lot of keys

2. Are there measurements how the stream concept performs regarding numer of streams, number of key-value pairs?

 

Greetings

Answer 1

We haven't measured this formally, but based on how the indexing is performed internally, it should not make a lot of difference either way.

Answer 2

Just a comment here on the performance writing on a single stream:

 

We wrote 3700000 key-values on the root stream with the following format:

key: random uuid

value: 100byte hex

So: 1 stream -> n keys -> 1 entry per key

When you do a liststreamkeyitems with default parameters you directly get back the result.

When you do a liststreamkeys with default parameters you the node hangs with 100% CPU and does not return even after minutes.

I saw that the default of count is MAX_INT. Maybe this is a bad idea since the function performes still great with low count values.

 

Greetings,

Alex

Comments

You make a fair point. The default count is "everything" for both liststreamkeys and liststreampublishers, and naturally this causes a hang. Consider even the amount of memory required to build the JSON response. The problem is that there is no particularly good default here (it is not like "show me the last 10 items" which makes some kind of logical sense) and in any event it is a problem to change the API at this stage. I guess the better thing to do is just give back an error if the response is going to be too large.