Forum: VoltDB Architecture

Post: Memory leak ?

Memory leak ?
gambitg
Oct 24, 2011
I have a node with total of 48G memory and no processes running. Before start of VoltDB, almost all the available memory is free(about 46G is free).


After starting VoltDB and populating with about 12G of data (snapshot size is 6G), the memory consumption jumps to about 40G.
After shut down of VoltDB memory consumption drops to about 24G.


My questions are:
1. Why is the memory consumption so high in relation to the snapshot size ? Is there a thumb-rule between size of snapshot and amount of memory ?
2. The more important question is after stop of the database, I expected almost all memory to be released. But the free memory is still far less than expected. As listed above, about 24G is used up when nothing else is running. Any thoughts on this ?


Thanks.
Re: Memory leak
gambitg
Oct 24, 2011
I meant the snapshot size is 12G and not 6G.
Hi Gambit,Snapshot size on
aweisberg
Oct 24, 2011
Hi Gambit,


Snapshot size on disk is not correlated to size in memory because the on disk representation does not include indexes, views, or indirection overhead. Also a 1 kilobyte VARBINARY takes 1536 bytes in memory (due to pooling), but only 1 kilobyte when flattened to disk.


Can you be more specific about what you are using to measure memory usage? If you are using top, what columns are you looking at? What application are you using?


Free memory reported system wide by top doesn't include cached disk pages. If you do "echo 1 > /proc/sys/vm/drop_caches" as root it will drop all the entries in the page cache and show much memory is actually free. Apparently someone made a website about this http://www.linuxatemyram.com


We have a test application that inserts and removes large amounts of data from the database that we run for weeks at time to try and catch fragmentation and memory leaks. That doesn't mean we haven't missed something, but we do know it works in the common case.


-Ariel
Re:
aweisberg
Oct 24, 2011
Hi Gambit,


Snapshot size on disk is not correlated to size in memory because the on disk representation does not include indexes, views, or indirection overhead. Also a 1 kilobyte VARBINARY takes 1536 bytes in memory (due to pooling), but only 1 kilobyte when flattened to disk.


Can you be more specific about what you are using to measure memory usage? If you are using top, what columns are you looking at? What application are you using?


Free memory reported system wide by top doesn't include cached disk pages. If you do "echo 1 > /proc/sys/vm/drop_caches" as root it will drop all the entries in the page cache and show much memory is actually free. Apparently someone made a website about this http://www.linuxatemyram.com


We have a test application that inserts and removes large amounts of data from the database that we run for weeks at time to try and catch fragmentation and memory leaks. That doesn't mean we haven't missed something, but we do know it works in the common case.


-Ariel


Thanks.
Yes, for #2, it was just a question of disk cache being reported as used. Don't think there is any memory leak.


For #1, how would I estimate how much RAM I need for a given database schema and table counts?
Hi Gambit,Tim Callaghan put
aweisberg
Oct 24, 2011
Thanks.
Yes, for #2, it was just a question of disk cache being reported as used. Don't think there is any memory leak.


For #1, how would I estimate how much RAM I need for a given database schema and table counts?


Hi Gambit,


Tim Callaghan put together this great sizing spreadsheet.


All indexes are now tree by default. Let me know if you have questions about mapping from your schema to the spreadsheet.


-Ariel
Re:spreadsheet
gambitg
Oct 24, 2011
Hi Gambit,


Tim Callaghan put together this great sizing spreadsheet.


All indexes are now tree by default. Let me know if you have questions about mapping from your schema to the spreadsheet.


-Ariel


For some reason, the link to the spreadsheet is not working.
Link fixed
ajgent
Oct 24, 2011
For some reason, the link to the spreadsheet is not working.


Hi,


I have fixed the link in Ariel's response. Please try again.


--Andrew