Continuing the JVM blog series, this is part 3 of a JVM adventure. This post doesn't require you have the knowledge covered in my previous blogs Part 1 and Part 2, however, I still highly suggest you read them so that you will have a better understanding of materials covered here.
Background
Recently, I have been tuning JVM at work to reduce the frequency of garbage collection happening in our spring server application, and I found an interesting behavior of JVM when it is running inside the docker container environment. The problem is that when the container environment is explicitly limited with memory by providing the option -m=<memory amount> to docker run command, JVM will not recognize that memory limit of the container environment. So, why does this happen? To answer this question, we have to first understand the underlying Linux OS feature called cgroups which makes containerization possible.
Like most developers, I tend to think of a container as a light-weight virtual machine when I started using it, but in reality, it resembles an isolation mechanism that can isolate the host's resource (e.g. CPU, memory, filesystem, etc.) for one process from others. This isolation is made possible because of the OS feature cgroups mentioned above. Since cgroups is a relatively new feature for Linux, and many programs (e.g. top, free even JVM) were developed before this feature was added to Linux kernel, when those programs are executed inside the container environment, they will not realize the resource constraints put by cgroups.
Demonstration
To illustrate the problem described above, let's go through the following example.
Preparation
Make sure you have Docker & VirtualBox installed on your machine. Once they are installed, let's first create a docker machine using the following command:
docker-machine create -d virtualbox --virtualbox-memory "1024" demo
This will create a docker machine with 1GB of memory within which a docker engine is running. Once the docker machine is created, follow the description of output to connect your docker client to the docker engine inside the newly created machine.
Experiment
Now that everything is ready, let's run a free command inside an Ubuntu container, and see what result we will get.
docker run -it -m=80M --memory-swap=100M ubuntu free -h
The picture below shows the output:
As you can see, I explicitly constraint the memory to 80M (with an additional memory swap constraint of 100M), but the output of free command shows interesting result:
The total memory is 989M which is about the size of the docker machine's virtual memory.
This unawareness of resource constraints also applies to JVM as I mentioned above. Let's try it out. Run the following command to start an OpenJDK container in the docker machine created before.
docker run -it -m=100M --memory-swap=100M openjdk:8u181 java -XshowSettings:vm -version
The result is shown in the picture below:
Notice the max heap is over 200M which exceeds the 100M memory limit. As a side note: the reason why max heap is not 1GB of the memory limit given to the docker machine created is because of the JVM Ergonomics. In short, Ergonomics will give us platform-dependent default values for JVM parameters. As for max heap, it's about 1/4 of physical memory.
In the example above, the result shows it's about 1/4 of the physical memory of that docker machine.
What's the fix?
Up until now, you may wonder how we could fix this issue? In short, there are 2 ways to overcome this before official support coming out.
The first option is to explicitly specify memory limit for JVM when launching an application by supplying the parameters -Xmx which will constraint max JVM heap size. If you know what the memory limit for your application to perform better than the default value chosen, then this is the option to go for.
The second option is for using default values, to let JVM recognize the constraints, JDK 8 provides 2 parameters -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, by supplying these 2 parameters to JVM when launching an application, JVM will correctly honor the resource constraints. Let's try it out, with the following command:
docker run -it -m=100M --memory-swap=100M openjdk:8u181 java -XshowSettings:vm
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -version
Now the output shows correct parameter values as shown below:
Final Note
You may notice that I kept the JDK version used in this blog to 8u181, the reason is that I found JDK 8 versions after 8u181 fixed the problem, so developers don't really need to specify the 2 experiment parameters if you are using later JDK 8 versions, or even newer JDK (9, 10, 11 etc.)
