Announcement

Collapse
No announcement yet.

Transparent huge pages (THP) not supported

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Transparent huge pages (THP) not supported

    I am starting VoltDB in a Docker container and when I execute bin/voltdb create I am greeted with the following errors:
    ERROR: The kernel is configured to use transparent huge pages (THP). This is not supported when running VoltDB. Use the following commands to disable this feature for the current session:
    ERROR:
    ERROR: sudo bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
    ERROR: sudo bash -c "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
    ERROR:
    ERROR: To disable THP on reboot, add the preceding commands to the /etc/rc.local file.

    As this is a container, I can't modify the underlying OS parameters. So, I would like to get some more information about what this error is about, why VoltDB does not support this configuration, and subsequently, what would be the best way to handle this in the contexts of containerized VoltDB instances.

    Looking forward getting VoltDB up and running.

  • #2
    The Admin documentation has a section on it (2.3 Configure Memory Management), but it doesn't speak of the conflict this creates for containerized applications of VoltDB. If your Docker or Kubernetes cluster is configured with efficient multi-tenant memory management features of the OS, would that mean that you can't run any VoltDB instance on that cluster? That seems a bit harsh, and customer unfriendly.

    2.3. Configure Memory Management
    Because VoltDB is an in-memory database, proper memory management is vital to the effective operation of VoltDB databases. Three important aspects of memory management are:

    Swapping

    Memory Mapping (Transparent Huge Pages)

    Virtual memory

    The following sections explain how best to configure these features for optimal performance of VoltDB.

    2.3.1. Disable Swapping
    Swapping is an operating system feature that optimizes memory usage when running multiple processes by swapping processes in and out of memory. However, any contention for memory, including swapping, will have a very negative impact on VoltDB performance and functionality. You should disable swapping when using VoltDB.

    To disable swapping on Linux systems, use the swapoff command. If swapping cannot be disabled for any reason, you can reduce the likelihood of VoltDB being swapped out by setting the kernel parameter vm.swappiness to zero.

    2.3.2. Disable Transparent Huge Pages
    Transparent Huge Pages (THP) are another operating system feature that optimizes memory usage for systems with large amounts of memory. THP changes the memory mapping to use larger physical pages. This can be helpful for general-purpose computing running multiple processes. However, for memory-intensive applications such as VoltDB, THP can actually negatively impact performance.

    Therefore, it is important to disable Transparent Huge Pages on servers running VoltDB. The following commands disable THP:

    $ echo never >/sys/kernel/mm/transparent_hugepage/enabled
    $ echo never >/sys/kernel/mm/transparent_hugepage/defrag
    Or:

    $ echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
    $ echo madvise >/sys/kernel/mm/transparent_hugepage/defrag
    For RHEL systems (including CentOS), replace "transparent_hugepage" with "redhat_transparent_hugepage".

    Note, however, that these commands disable THP only while the server is running. Once the server reboots, the default setting will return. Therefore, we recommend you disable THP permanently as part of the startup process. For example, you can add the following commands to a server startup script (such as /etc/rc.local):

    #!/bin/bash
    for f in /sys/kernel/mm/*transparent_hugepage/enabled; do
    if test -f $f; then echo never > $f; fi
    done
    for f in /sys/kernel/mm/*transparent_hugepage/defrag; do
    if test -f $f; then echo never > $f; fi
    done
    THP are not enabled by default in Ubuntu prior to release 14.04 or RHEL 5.x. But they are enabled by default for recent Ubuntu releases and RHEL 6.x. To see if they are enabled on your current system, use either of the following pair of commands:

    $ cat /sys/kernel/mm/transparent_hugepage/enabled
    $ cat /sys/kernel/mm/transparent_hugepage/defrag

    $ cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
    $ cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
    If THP is disabled, the output from the preceding commands should be either “always madvise [never]” or “always [madvise] never”.

    2.3.3. Enable Virtual Memory Mapping and Overcommit
    Although swapping is bad for memory-intensive applications like VoltDB, the server does make use of virtual memory (VM) and there are settings that can help VoltDB make effective use of that memory. First, it is a good idea to enable VM overcommit. This avoids VoltDB encountering unnecessary limits when managing virtual memory. This is done on Linux by setting the system parameter vm.overcommit_memory to a value of "1".

    $ sysctl -w vm.overcommit_memory=1
    Second, for large memory systems, it is also a good idea to increase the VM memory mapping limit. So for servers with 64 Gigabytes or more of memory, the recommendation is to increase VM memory map count to 1048576. You do this on Linux with the system parameter max_map_count. For example:

    $ sysctl -w vm.max_map_count=1048576
    Remember that for both overcommit and the memory map count, the parameters are only active while the system is running and will be reset to the default on reboot. So be sure to add your new settings to the file /etc/sysctl.conf to ensure they are in effect when the system is restarted.

    Comment


    • #3
      Hi,

      You can override that check with --ignore=thp on the command line.

      Comment


      • #4
        Please note that the reason we recommend disabling Transparent Huge Page is to avoid unnecessary memory usage and negatively impacting performance. In some cases, it may cause the system to run out of memory even if the data fits perfectly in memory. You can read about the details in this blog post.

        As Ruth pointed out, you can also disable the check using the command line argument.
        Ning

        Comment


        • #5
          Originally posted by nshi View Post
          Please note that the reason we recommend disabling Transparent Huge Page is to avoid unnecessary memory usage and negatively impacting performance. In some cases, it may cause the system to run out of memory even if the data fits perfectly in memory. You can read about the details in this blog post.

          As Ruth pointed out, you can also disable the check using the command line argument.

          @Ning It is good to know that THP support is about resource management, not functionality. As a side note, all our infrastructure, which consists of Ubuntu 14 and 16 and Windows Server, all have THP enabled in their default configurations.

          Comment

          Working...
          X