Tuning Minecraft for OpenJ9

We have been running the OpenJ9 JVM at Mineteria since July 2018. Our own experience with OpenJ9 has led us to continually tweak our set up and ensure that our servers are not subjected to long garbage collection pauses.

Our servers (which span long-lived lobbies, to a creative-like server with frequent chunk loading, to short-lived game servers), while likely not representative of all Minecraft servers, have been using these flags for many months and have virtually eliminated any issues with garbage collection pauses.

My research started with Aikar’s flags for G1 and moved towards how to apply a similar set of flags for OpenJ9. He can not be thanked enough for both inspiring my OpenJ9 garbage collection research and providing a well-tested set of flags to improve server performance.

After almost eleven months of live production experience, I am now sharing my research with OpenJ9 to the wider public in the hopes of promoting the use of OpenJ9 in Minecraft along with providing a set of well-tested flags for those who decide to make the switch.

These flags will remain an ongoing area of research for me - keep an eye out for updates!

Why use OpenJ9?

The biggest reason why OpenJ9 appealed to me was because it was container-friendly. This was important as the Linux kernel was reaping our Java processes. We were strapped for cash and so upgrading VM instances was a last resort option, and we did not have the capability to turn off OOM killing in our containers because our setup was based on a managed Kubernetes solution (at the time, it was Google Kubernetes Engine - we have since moved to DigitalOcean Kubernetes).

Up until then, we had been using the standard HotSpot VM. HotSpot is a very proven JVM implementation, but only in Java 10 did it become container-aware. OpenJ9 goes much further and has a flag (-Xtune:virtualized) which optimizes the JVM for lower CPU usage. Funny enough, my adventure with OpenJ9 began when I was tuning our HotSpot-based setup for better garbage collection performance with Aikar’s flags in preparation for our beta launch in July 2018.

This primary consideration helped sell us, but OpenJ9 has also helped deliver lower memory usage, which is important for Minecraft servers as they often have a lot of chunks loaded. In our particular case, we have cloud virtual servers focused on providing more CPU and varying memory needs for each server, so a set of flags that was easily tuned was crucial.

Why does it matter for me?

Not everyone uses containers, but everyone can still benefit from OpenJ9’s many unique features compared to OpenJ9:

  • Idle garbage collection: when your server is idle, OpenJ9 can perform a full garbage collection to clean out the tenured space and nursery without negatively impacting performance.
  • Shared classes and ahead-of-time compilation mean that your server can restart and ramp back up much more quickly compared to HotSpot.
  • Lower memory usage compared to HotSpot.
  • The best part is that in most cases, you can just drop in OpenJ9 and your plugins will “just work”.

Tuning the OpenJ9 VM

Now with some reasons why to use OpenJ9 out of the way, we can turn to the Minecraft-tuned garbage collection settings for OpenJ9. These settings will result in short (usually 1 to 5 millisecond) GC pauses, which is quite good. The memory settings will need to be tweaked depending on your memory usage, but I’m providing a shell script that will take care of the entire process for you.

TL;DR: Here’s a shell script that takes care of the entire process. Just drop it in your server directory as, e.g. start.sh, edit it to tweak your memory setting, chmod +x start.sh, and then ./start.sh. It will “just work” with no further set up required. For your convenience, I’ve also made it wgettable.

#!/bin/bash

#
# Properly tunes a Minecraft server to run efficiently under the
# OpenJ9 (https://www.eclipse.org/openj9) JVM.
#
# Licensed under the MIT license.
#

## BEGIN CONFIGURATION

# HEAP_SIZE: This is how much heap (in MB) you plan to allocate
#            to your server. By default, this is set to 4096MB,
#            or 4GB.
HEAP_SIZE=4096

# JAR_NAME:  The name of your server's JAR file. The default is
#            "paperclip.jar".
#
#            Side note: if you're not using Paper (http://papermc.io),
#            then you should really switch.
JAR_NAME=paperclip.jar

## END CONFIGURATION -- DON'T TOUCH ANYTHING BELOW THIS LINE!

## BEGIN SCRIPT

# Compute the nursery size.
NURSERY_MINIMUM=$(($HEAP_SIZE / 2))
NURSERY_MAXIMUM=$(($HEAP_SIZE * 4 / 5))

# Launch the server.
CMD="java -Xms${HEAP_SIZE}M -Xmx${HEAP_SIZE}M -Xmns${NURSERY_MINIMUM}M -Xmnx${NURSERY_MAXIMUM}M -Xgc:concurrentScavenge -Xgc:dnssExpectedTimeRatioMaximum=3 -Xgc:scvNoAdaptiveTenure -Xdisableexplicitgc -jar ${JAR_NAME}"
echo "launching server with command line: ${CMD}"
${CMD}

## END SCRIPT

With the default settings, your server will launch with java -Xms4096M -Xmx4096M -Xmns2048M -Xmnx3276M -Xgc:concurrentScavenge -Xgc:dnssExpectedTimeRatioMaximum=3 -Xgc:scvNoAdaptiveTenure -Xdisableexplicitgc -jar paperclip.jar.

We will break down each of these flags, but there is other important context to get out of the way.

Sizing the heap

The first and natural set of flags to turn to are our humble friends -Xms and -Xmx. You should set -Xms and -Xmx to the same value, whether you use HotSpot or OpenJ9.

In the case of OpenJ9, this is even more important because OpenJ9 tends to be quite conservative in requesting more memory from the operating system. In our experience, when the heap is insufficient to satisfy allocations, OpenJ9 will prefer to perform aggressive garbage collection first and only when it can’t free up a significant amount of memory will it request more from the operating system. This is bad because it will degrade server performance due to frequent garbage collection pauses.

Now we need to actually decide on the heap size. The best policy to take here is to allocate as much as you reasonably can, but make sure to leave enough room for the operating system and for the JVM’s overhead, otherwise the operating system may kill the server to free up memory. For instance, if you have a 4GB VPS, you should stay in the 2-3GB range so that the operating system will still run smoothly and the JVM can allocate overhead.

Selecting a GC policy

OpenJ9 features five different garbage collection policies:

  • gencon: This is the default GC policy in OpenJ9. This policy breaks the Java heap into a nursery and a tenured space. All objects are allocated initially into the nursery. If an object survives a certain number of collections while in the nursery, it will be moved into the tenured space, which is only collected when it is full.
  • balanced: This is the closest analogue that OpenJ9 has to the HotSpot G1 garbage collector. It breaks the heap into regions, each of which is individually managed and garbage-collected.
  • optthruput: This policy is similar to optavgpause but does not mark concurrently, resulting in longer pauses. Because of this, it is a poor choice for a Minecraft server.
  • optavgpause: This is a fully concurrent mark-and-sweep collector which aims to minimize the pause time. It is however mostly suited for very large heaps.
  • metronome: This is a policy available on AIX and x64 Linux. It is interesting as it places a focus on minimizing GC pause times to ensure precise response times. However, we will not need this policy for Minecraft.

The natural question to ask, when presented with all these different choices, is “which of these collectors is the best?”

To answer this question, we need to examine the object allocation behavior of the Minecraft server. Since Minecraft 1.8, the Minecraft server has steadily grown in complexity and now has a very high, but steady, object allocation rate. Furthermore, these objects tend to die young, after just a few collections. This makes the gencon or the balanced collector suitable for Minecraft.

I selected gencon as it is the default and provides good performance as long as it is tuned for the application. The following steps tune the Minecraft server for the gencon collector. I may experiment with the balanced collector in the future.

Sizing the gencon nursery

Now that we have selected our desired GC policy and sized our heap, we need to tune the behavior of the garbage collector. Since we are using gencon, the heap will be divided up into a nursery space and a tenured space. For Minecraft, we need to take maximum advantage of the nursery space because Minecraft allocates lots of very short-lived objects, and collecting it is quite cheap compared to the full GC that is required to collect the tenured space.

The first major set of flags to concern ourselves with is the size of the nursery, controlled by -Xmns to set the initial size and -Xmnx for the maximum size. By default, -Xmns is 25% of -Xms and -Xmnx is 25% of -Xmx. As I said earlier, you will be setting -Xms and -Xmx to the same value. This means that by default, the nursery will never grow. Since Minecraft tends to allocate lots of very short-lived objects, this will result in very frequent garbage collections and will tenure garbage objects more often, degrading server performance.

Instead, you should set -Xmns to 50% of your -Xmx and -Xmnx to 80% of your -Xmx. This allows OpenJ9 to expand the nursery when needed and is a more realistic nursery size for virtually every Minecraft server.

Tweaking gencon behavior

Finally, we have a few flags that improve the behavior of the gencon policy for Minecraft.

Our first flag to look at is -Xgc:concurrentScavenge. This flag enables the gencon collector’s pauseless mode. We exchange a slight throughput reduction (usually 8%) for short pause times of 1-5 milliseconds.

The second flag is -Xgc:dnssExpectedTimeRatioMaximum=3. This reduces the amount of time spent collecting the nursery.

Our third flag, -Xgc:scvNoAdaptiveTenure makes sure that objects are not promoted to the tenured heap too quickly. This is, again, because the object allocation rate of the Minecraft server is very high, which means that objects will typically get tenured more quickly.

The final flag, -Xdisableexplicitgc, protects us from plugins that think they’re being smart by asking the garbage collector to do a collection when one is not needed, and may actually hurt server performance later on by bringing a full GC cycle closer.

Other settings to change

If you are on a VPS or some sort of cloud environment, you should add in the -Xtune:virtualized flag. This tunes OpenJ9 to reduce CPU and memory usage.

My script does not configure OpenJ9’s ahead-of-time compiler or the shared classes functionality. This can improve startup times, but we can’t fully exploit this feature due to the unique peculiarities of our setup.