Disclaimer #
Mineteria is long gone. It stopped using OpenJ9 in April 2020. I have limited interest in Minecraft nowadays. I'm keeping this post up since lots of links to it exist on the internet. Follow the advice in this post at your own risk.
Andrew Steinborn, October 16, 2022
Original introduction #
Mineteria ran the OpenJ9 JVM from July 2018 to April 2020. Our own experience with OpenJ9 has led us to continually tweak our set up and ensure that our servers are not subjected to long garbage collection pauses.
Our servers (which span long-lived lobbies, to a creative-like server with frequent chunk loading, to short-lived game servers), while likely not representative of all Minecraft servers, have been using these flags for many months and have virtually eliminated any issues with garbage collection pauses.
My research started with Aikar's flags for G1 and moved towards how to apply a similar set of flags for OpenJ9. He can not be thanked enough for both inspiring my OpenJ9 garbage collection research and providing a well-tested set of flags to improve server performance.
Why use OpenJ9? #
The biggest reason why OpenJ9 appealed to me was because it was container-friendly. This was important as the Linux kernel was reaping our Java processes. We were strapped for cash and so upgrading VM instances was a last resort option, and we did not have the capability to turn off OOM killing in our containers because our setup was based on a managed Kubernetes solution (at the time, it was Google Kubernetes Engine - we have since moved to DigitalOcean Kubernetes).
Up until then, we had been using the standard HotSpot VM. HotSpot is a very proven JVM implementation, but only in Java 10 did it become container-aware. OpenJ9 goes much further and has a flag (-Xtune:virtualized
) which optimizes the JVM for lower CPU usage. Funny enough, my adventure with OpenJ9 began when I was tuning our HotSpot-based setup for better garbage collection performance with Aikar's flags in preparation for our beta launch in July 2018.
This primary consideration helped sell us, but OpenJ9 has also helped deliver lower memory usage, which is important for Minecraft servers as they often have a lot of chunks loaded. In our particular case, we have cloud virtual servers focused on providing more CPU and varying memory needs for each server, so a set of flags that was easily tuned was crucial.
Why does it matter for me? #
Not everyone uses containers, but everyone can still benefit from OpenJ9's many unique features compared to OpenJ9:
- Idle garbage collection: when your server is idle, OpenJ9 can perform a full garbage collection to clean out the tenured space and nursery without negatively impacting performance.
- Shared classes and ahead-of-time compilation mean that your server can restart and ramp back up much more quickly compared to HotSpot.
- Lower memory usage compared to HotSpot.
- The best part is that in most cases, you can just drop in OpenJ9 and your plugins will "just work".
Tuning the OpenJ9 VM #
Now with some reasons why to use OpenJ9 out of the way, we can turn to the Minecraft-tuned garbage collection settings for OpenJ9. These settings will result in short (usually 1 to 5 millisecond) GC pauses, which is quite good. The memory settings will need to be tweaked depending on your memory usage, but I'm providing a shell script that will take care of the entire process for you.
TL;DR: Here's a shell script that takes care of the entire process. Just drop it in your server directory as, e.g. start.sh
, edit it to tweak your memory setting, chmod +x start.sh
, and then ./start.sh
. It will "just work" with no further set up required. For your convenience, I've also made it wgettable.
#!/bin/bash
#
# Properly tunes a Minecraft server to run efficiently under the
# OpenJ9 (https://www.eclipse.org/openj9) JVM.
#
# Licensed under the MIT license.
#
## BEGIN CONFIGURATION
# HEAP_SIZE: This is how much heap (in MB) you plan to allocate
# to your server. By default, this is set to 4096MB,
# or 4GB.
HEAP_SIZE=4096
# JAR_NAME: The name of your server's JAR file. The default is
# "paperclip.jar".
#
# Side note: if you're not using Paper (http://papermc.io),
# then you should really switch.
JAR_NAME=paperclip.jar
## END CONFIGURATION -- DON'T TOUCH ANYTHING BELOW THIS LINE!
## BEGIN SCRIPT
# Compute the nursery size.
NURSERY_MINIMUM=$(($HEAP_SIZE / 4))
NURSERY_MAXIMUM=$(($HEAP_SIZE * 2 / 5))
# Launch the server.
CMD="java -Xms${HEAP_SIZE}M -Xmx${HEAP_SIZE}M -Xmns${NURSERY_MINIMUM}M -Xmnx${NURSERY_MAXIMUM}M -Xgcpolicy:balanced -Xdisableexplicitgc -jar ${JAR_NAME}"
echo "launching server with command line: ${CMD}"
${CMD}
## END SCRIPT
With the default settings, your server will launch with java -Xms4096M -Xmx4096M -Xmns1024M -Xmnx1638M -Xgcpolicy:balanced -Xdisableexplicitgc -jar paperclip.jar
.
We will break down each of these flags, but there is other important context to get out of the way.
Sizing the heap #
The first and natural set of flags to turn to are our humble friends -Xms
and -Xmx
. You should set -Xms
and -Xmx
to the same value, whether you use HotSpot or OpenJ9.
In the case of OpenJ9, this is even more important because OpenJ9 tends to be quite conservative in requesting more memory from the operating system. In our experience, when the heap is insufficient to satisfy allocations, OpenJ9 will prefer to perform aggressive garbage collection first and only when it can't free up a significant amount of memory will it request more from the operating system. This is bad because it will degrade server performance due to frequent garbage collection pauses.
Now we need to actually decide on the heap size. The best policy to take here is to allocate as much as you reasonably can, but make sure to leave enough room for the operating system and for the JVM's overhead, otherwise the operating system may kill the server to free up memory. For instance, if you have a 4GB VPS, you should stay in the 2-3GB range so that the operating system will still run smoothly and the JVM can allocate overhead.
Selecting a GC policy #
OpenJ9 features five different garbage collection policies:
gencon
: This is the default GC policy in OpenJ9. This policy breaks the Java heap into a nursery and a tenured space. All objects are allocated initially into the nursery. If an object survives a certain number of collections while in the nursery, it will be moved into the tenured space, which is only collected when it is full.balanced
: This is the closest analogue that OpenJ9 has to the HotSpot G1 garbage collector. It breaks the heap into regions, each of which is individually managed and garbage-collected.optthruput
: This policy is similar tooptavgpause
but does not mark concurrently, resulting in longer pauses. Because of this, it is a poor choice for a Minecraft server.optavgpause
: This is a fully concurrent mark-and-sweep collector which aims to minimize the pause time. It is however mostly suited for very large heaps.metronome
: This is a policy available on AIX and x64 Linux. It is interesting as it places a focus on minimizing GC pause times to ensure precise response times. However, we will not need this policy for Minecraft.
For Minecraft 1.12.2 and below, the right answer was the default gencon
policy. However, with 1.13 and above, gencon
proves to be a bad fit for servers due to the severely increasedstrain on the garbage collector subsystem. The balanced
policy is the next-best.
Sizing the nursery #
Now that we have selected our desired GC policy and sized our heap, we need to tune the behavior of the garbage collector.
The size of the nursery is key to improving Minecraft performance. By default, -Xmns
is 25% of -Xms
and -Xmnx
is 25% of -Xmx
. Since you will be setting -Xms
and -Xmx
to the same value. This means that by default, the nursery will never grow. Since Minecraft tends to allocate lots of very short-lived objects, this will result in very frequent garbage collections and will tenure garbage objects more often, degrading server performance.
Instead, you should set -Xmns
to 25% of your -Xmx
and -Xmnx
to 40% of your -Xmx
. This allows OpenJ9 to expand the nursery when needed and is a more realistic nursery size for a Minecraft server.
Disabling explicit GC #
The other flag we provide is -Xdisableexplicitgc
, which protects us from plugins that think they're being smart by asking the garbage collector to do a collection when one is not needed, and may actually hurt server performance later on by bringing a full GC cycle closer.
Other settings to change #
If you are on a VPS or some sort of cloud environment, you should add in the -Xtune:virtualized
flag. This tunes OpenJ9 to reduce CPU and memory usage.
My script does not configure OpenJ9's ahead-of-time compiler or the shared classes functionality. This can improve startup times, but we can't fully exploit this feature due to the unique peculiarities of our setup.