Forum: Building APIs & Wire Protocol

Post: 1M string limit

1M string limit
henning
May 16, 2010
Almost for curiosity only, what is the reason for the 1M limit on strings?


Thanks,
Henning
We made it up, but...
jhugg
May 16, 2010
We chose this number after much discussion, though in the end it's just a number we picked. The goal was to balance the flexibility of VoltDB with the performance and reliability.

VoltDB transactions move through a series of buffers. There are network buffers between the client and our initiator code that assigns transaction ordering and routes work. There are network buffers between sites. There is a special ordering buffer that holds work until it is safe to run at each site. There are buffers used for exporting data to other systems. Etc...


A big payload uses a bigger percentage of any fixed size buffers or it can cause the dynamic buffers to grow to larger sizes. This makes it easier for VoltDB to run out of JVM memory. It also adds latency to other messaging waiting to be sent after a large payload. It's made more complicated because VoltDB can have thousands of transactions active in the system.


So we picked 1M for now, even though VoltDB certainly prefers smaller data. Someday we hope to support larger sizes.