Andrew Steinborn

I have spent a lot of time in the time in the Minecraft server scene. I've been highly visible, in both my good and bad moments. But perhaps my most substantial and consequential project in that scene, specifically the Velocity Minecraft server proxy, is the most interesting story to tell.

Velocity is notable today for being the proxy in front of a number of notable servers. Want to play on a network of completely user generated content servers? Powered by Velocity. Even the infamous 2b2t server uses Velocity, having moved from Waterfall (which Velocity replaces).

This is the first in a series of posts. Here, I will discuss the earliest days of Velocity, along with the first "trick" it used to improve performance over its contemporaries.

What exactly is a Minecraft server proxy? #

Throughout this series, I am going to assume the reader:

• has a technical background
• has some understanding of what Minecraft is, but has never played it

Therefore I need to define some terminology. Let's start from the beginning:

• A Minecraft server is the server-side component of Minecraft. It allows players to interact with a world hosted on a remote machine instead of on their own computer. Popular Minecraft server software options include the server provided by Mojang themselves ("vanilla"), Spigot, and Paper. Modding frameworks such as Fabric and NeoForge also permit custom modifications to the Minecraft server. We'll call this a server, except where we need to disambiguate.
• A Minecraft server proxy sits between the Minecraft player's client and a group of servers (typically referred to as a "network"). The proxy accepts connections from players and routes them to a given Minecraft server on the backend. We'll call this just a proxy from now on, except where we need to disambiguate.

From this broad viewpoint, a Minecraft server proxy is no different from a typical reverse proxy, such as NGINX or HAProxy. Just as a reverse proxy manages and routes traffic to web servers, a Minecraft server proxy offers similar benefits: you don't have to expose each server to the Internet and it enables load balancing. As an example, take Hypixel, by and far the largest Minecraft server (at one point, it had over 200,000 concurrent users). A single Minecraft server simply cannot handle 200,000 players on at once, and so Hypixel uses a web of proxies and a lot of backend Minecraft servers to host their games on.

Here's an example topology of a Minecraft server network (with some thanks to Claude for drawing up the graph):

Minecraft server proxies, though, have a few unique features that make them different from your regular every day reverse proxy:

• The proxy can force the client to move to a completely different Minecraft server. We'll dig into this later. But the end result: imagine a bastion server that was completely invisible: you use SSH to log into the bastion, and it directs you to a server that you want by default, then you initiate a command that can seamlessly log you into a different SSH server behind the proxy, without ever closing the original SSH session.
• Most proxies support plugins, which extend the functionality of the proxy. You get the full power of the runtime and language the proxy uses - for Velocity, that means "any JVM language you want" - Kotlin being a popular choice in the community, but you could also use Scala, Clojure, or just about any language that can compile to the JVM, as long as it can interoperate enough with Java that Velocity can load it. Plugins can be as simple or as complex as you want: as simple as a plugin to manage server punishments like bans to plugins that add protocol translation support (everything from permitting newer versions of Minecraft to connect to servers running older versions of Minecraft, to even adding support for Minecraft: Bedrock Edition to Minecraft: Java Edition servers.)

The two giants in the Minecraft server proxy scene are BungeeCord and the subject of this series, Velocity.

First, the story of my first Minecraft server #

The month is August 2013. I had just recently started high school. I learn of a Minecraft server network in need of developers (this is another fascinating topic I'll be sure to devote some time to eventually), and it looks like an interesting opportunity.

It was a more innocent time. Here, I'll even provide you some period-accurate videos of the three minigames that were on offer at the time.

First, here's MinerWare, a Minecraft clone of Mario Party:

Next is Ender, a Minecraft clone of the 2012 video game Slender:

Finally, there was Mineshooter, a clone of Call of Duty in Minecraft. Unfortunately, I had trouble finding a good contemporary video of the "old" Mineshooter (since there was a later rework), but I did find this video (voiceover in Russian, game text in English) which should give you a good idea of what the gameplay was like. There was also a separate server called GLDesert with a custom map, but this was detached from the rest of The Chunk in early 2014 in order to exclusively focus on minigames.

In those days, you had only one real option for having a proxy, and it was BungeeCord. (There was also LilyPad, but it required more involved set up and eventually fell by the wayside.) The Chunk used BungeeCord, and it had some special unique needs. In late September 2013, I began developing plugins for BungeeCord, moving away briefly from my previous focus on the minigames side of the house.

It didn't take me very long to make my first BungeeCord plugins: AdvancedBungeeAnnouncer and RedisBungee. The former made global announcements across the network more consistent, the latter synchronized the player count and player list across all the proxies. Since it was and to some degree still is a common practice to add load balancing to the proxies as well, Minecraft server network owners would run multiple proxies on different machines, much like you would scale up an application vertically by launching more than one instance of it on a different machine. The trouble was that because each proxy was independent, they would only report the player count of those players connected to a specific proxy. This was the problem that RedisBungee solved: it put all of this data into Redis and did its best to "fake" making a series of independent proxies look like one large proxy.

The Chunk wound up being a fairly successful network. It peaked at about 1,500 to 1,600 players online during early 2014, particularly the period between February 2014 and May 2014, but then went into decline. The network would eventually be absorbed into a different Minecraft network in early 2016 - I would do some development work for that network before leaving.

After that, I would wonder around quite a bit, taking on roles with different Minecraft servers and companies (some quite large, some quite small). I did work ranging from standard plugin development to custom modifications of Minecraft server software (mostly Paper). Perhaps most notable was the time I worked for Tebex (back when it was called Buycraft), where I rewrote their Minecraft server plugin and built their second ever integration with Minecraft: Bedrock Edition. I had a brief detour into making Minecraft server lists using Django, Celery, and PostgreSQL, and this is where I honed some of my web development skills (along with learning how to work with Python web frameworks). I enrolled in college. Eventually, I took on some work for Mineteria, which is where we'll pick up the story.

Well, Andrew, how involved were you in the "proxy scene"? #

Quite a long time, I suppose.

What's often lost in the discussion, and has been a seemingly rich vein of confusion over the years, is the role I played in the Waterfall project. I founded this project in January 2016 as an effort to lower the bar to adding contributions to BungeeCord. It used the same patching system as Spigot and Paper, and included a number of patches to improve BungeeCord's behavior, add new features, and included some performance improvements. People came and went, but I was the only real constant until I handed the project over to the Paper team in September 2018 to focus more on Velocity.

The sad reality with Waterfall is that we were basically kind of stuck with the lowest common denominator. electronicboy would always relay the tale of trying to add improved scoreboard support to Waterfall (instead of a very raw API that BungeeCord exposed at the time by allowing plugins to just send packets if they wanted, a practice very quickly banished from the Velocity plugin API on day one) and dealing with broken plugins as a result, effectively killing any desire to make Waterfall a better BungeeCord in any way that mattered. In other words, a Minecraft manifestation of Hyrum's law:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

After all of that, there was increased conviction to build an alternative proxy instead. This evolved into Velocity.

You said you'd discuss another Minecraft network, too? #

In May 2018, after doing some initial web development work for Mineteria, I was enlisted to work on the actual Minecraft server. The server had a go-live date in July 2018, so I was thrown onto the project fairly late - development had previously began in early 2017. Like The Chunk and many Minecraft networks before it, it also used BungeeCord.

The server was pretty unique as it used Kubernetes as its native control plane. It was amongst the earliest examples of this, and so we'd run into a lot of rough edges. Kubernetes on bare metal was not as mature back then, so our production set up wound up being run on Google Kubernetes Engine.

Those of you in the audience familiar with Minecraft servers would immediately know that something is wrong here. Other people in the audience may be puzzled: isn't Google Cloud a pretty standard commodity cloud provider? They offer all the services you'd expect, just like any other cloud provider.

Swiping left on the cloud #

Proxies have two primary characteristics:

• They need to do a lot of I/O-intensive work, which means lots of system calls. This was all after the impact of Spectre/Meltdown, which meant that everyone likely took massive performance penalties, which were further magnified by running inside of a virtual machine.
• Part of the proxy's job is compressing packets destined for players, and this process is very resource-intensive. Worse, we cannot just serve pre-compressed assets like a web server might do - we have to compress packets on the fly.

This became apparent when we launched the server on July 1st. The servers did collapse under their weight and we had to add additional resources to keep things stable. We realized that for the proxy, we would need to run absurdly large instances (or scale out the proxy), in essence throwing a lot of money at the problem. That's what we did, for a while at least. But we knew that the persistently high CPU usage was not sustainable (we were seeing well over one or two cores used by a small number of players!) and so we sought a third option.

It was also in July 2018 when I was on the PaperMC Discord complaining about BungeeCord, primarily around its development process (a cathedral) and its almost fanatical pursuit of backwards compatibility. At that time, I was experimenting with a fork of BungeeCord I had created, LibertyCord, seeking to make more radical changes. At the time, the owners of Mineteria were skeptical of Velocity, but I pressed on anyway. With encouragement from electronicboy, kashike, and even Aikar, I began work on Velocity. Progress did eventually get to the point where Mineteria made the move to Velocity starting in November 2018, and I could actually start "eating my own dogfood".

Velocity is born #

The first commit to Velocity was on July 24th, 2018. This initial version of Velocity didn't have much functionality. In fact, it was just barely capable of responding to server list ping packets from the Minecraft client. Still, it was pointing in the right direction, and we just needed to get Velocity relaying packets across the network.

By July 27th, I had something that you could call a "proof of concept": a player could connect to a Velocity server, be connected successfully to the proxy, which would open a connection to a specific server (localhost:25565), and be able to switch to a different server (localhost:25566). Furthermore, we had implemented the first "trick" Velocity used to gain an edge over BungeeCord and LilyPad. But before we do, we'll need to discuss a few more concepts.

Who are you calling, me? #

A typical video game has entities contained within the game world. An entity is simply a object that exists in a given world.

In a single-player game, we might be able to store direct memory references to other entities in the world. Suppose we were making a very simple tower defense game where we have turrets and enemies and we need to tick them (update them) on a periodic basis. We might refer to the entities in the world in code as follows:

  private List<Turret> _turrets;
  private List<Enemy> _enemies;

But what if we are playing over the network? It would make no sense (and be a security risk) to pass around memory pointers. Instead, what we can do is assign an ID to each entity in the world, and refer to the entity by this ID alone when referring to a specific entity over the network.

So far, so good, right?

Here's the catch: we can move the player to a different server, but we can't tell the Minecraft server to use a specific entity ID. Furthermore, if we just send packets referring to the connected player with a different entity ID than the one the client believes it has, then all sorts of interesting issues can occur on both the client and server sides. To add further complications, the tricks that proxies do are technically not supported by Mojang, but are only tolerated because proxies are very popular in the server community!

That said, we have two options:

• The proxy can rewrite every packet that contains an entity ID. Should it find that it needs to rewrite a packet, it will copy all the fields in the packet over. If it finds the server-side entity ID for the player in a packet bound for the player, it will rewrite it to include the entity ID that the client believes it uses. It also does this in the reverse direction for packets originating from the player towards the server they are connected to.
• Or, you can try to make the client use a different entity ID, and dispense with the need to rewrite packets entirely.

BungeeCord and LilyPad opted to rewrite packets. Velocity chose the path of telling the client to use a different entity ID.

So, how do we fool the client? #

First, let's understand what Velocity is sending the client. First, here is how Velocity told the client to change entity IDs and load the new world:

    public void handleBackendJoinGame(JoinGame joinGame) {
        if (!spawned) {
            spawned = true;
            currentDimension = joinGame.getDimension();
            player.getConnection().write(joinGame);
        } else {
            player.getConnection().write(joinGame);
            int tempDim = joinGame.getDimension() == 0 ? -1 : 0;
            player.getConnection().write(new Respawn(tempDim, joinGame.getDifficulty(), joinGame.getGamemode(), joinGame.getLevelType()));
            player.getConnection().write(new Respawn(joinGame.getDimension(), joinGame.getDifficulty(), joinGame.getGamemode(), joinGame.getLevelType()));
            currentDimension = joinGame.getDimension();
        }
    }

It should first be noted that this continues to be the base for Velocity's current implementation, which is more complex (notably, we can now omit one of the Respawn packets starting in Minecraft 1.16), but the fundamental logic remains the same.

What does this code do? First, we need to explain a few Minecraft packets and how the client responds to them:

• The JoinGame packet is sent by the server when it indicates that the server has spawned the player into the server's world. It is sent very early after the player's connection to the server enters the PLAY state (in other words, when the player has just logged into the server).
• The Respawn packet it sent by the server when a player is moved to another dimension (i.e. going from the Overworld to the Nether), or if the player is respawning after dying.

Now we can explain what Velocity is doing:

• If this is the first server the player has connected to during this session, indicate as such and send the packet plain. There's nothing for us to do.
• Otherwise:
  • We send the original, unmodified JoinGame packet from the new server to the client.
  • The client drops its current state of the world, and crucially, sets its entity ID to the one we sent.
  • We send a Respawn packet, set to a different dimension. At the time, the Minecraft client did not behave correctly when we respawn the client into the current dimension immediately after sending JoinGame.
  • We send another Respawn packet to bring the player into the correct dimension.
• The server switch is done, and packets can stop flowing from the old server (if any) and start flowing from the new server.

This solves a couple of issues:

• It makes the codebase considerably simpler. Instead of maintaining complex, non-trivial packet rewriting logic, we could focus on other important attributes such as extensibility and performance of the overall networking pipeline. It also means that the code is less buggy, since we know that the client can update the entity ID it uses, so we don't need to worry about missing a packet we were supposed to rewrite.
• It made updating Velocity to work with newer versions of Minecraft much easier. Some versions of Minecraft were supported simply by telling Velocity that it needs to recognize a new protocol version number.
• It worked with every version of Minecraft that Velocity supports, from Minecraft 1.7.2 to 1.21 - nearly 11 years of compatibility.
• It causes less problems with mods. Entity ID rewriters don't know how to rewrite mod packets containing entity IDs, so these mods would simply break on a proxy that uses them.
• Perhaps most importantly, since we don't need to examine these packets, the proxy does essentially no processing on the packets except for uncompressing the packet from the server if needed, determining that no action is necessary, and forwarding it onto the client (compressing and encrypting the packet as needed).

Of course, this wouldn't be the only trick Velocity employed in the service of performance, but it was amongst the first.

That's all for this installment #

With this post, I hope I've began to lay down the foundations for how Velocity came to be. I've introduced you to some key concepts along with three of the formative moments in my Minecraft journey. I've also given you an introduction to the earliest days of Velocity.

In the next installment, we'll discuss some more of the early days of Velocity:

• The plugin API.
• The lead up to, first the deployment of Velocity on Mineteria, and then Velocity 1.0.0.

Stay tuned for more. There's a lot of stories for me to tell, as these were some of my formative years as a software engineer. (Trust me, going from Minecraft modding to fintech is a very unexpected turn of events!)