Andrew Steinborn

☄️ Precipices

2025-03-05T00:00:00Z

This is a work of fiction. Names, characters, businesses, places, events, locales, and incidents are either the products of the author’s imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.

In particular, Caspoke and Wedge are fictional companies. Any resemblance to real companies is purely coincidental.

On the Precipice of Success

Samuel strutted into the French bakery, the cold, bitter air outside biting at him. The aroma of freshly ground coffee beans and buttery pastries welcomed him in, a little haven in a world that was up in flames. He placed his order: a potato confit garlic quiche, a danish, and a latte with oat milk. While he waited for his order, Samuel glanced outside the window. An elevated train rumbled overhead, its steel frame casting a shadow over the street below, as it ferried people between the boroughs, a steady constant in the swirling chaos.

Now with his food, Samuel scrolled through his phone while eating. His employer was on the verge of going public. Ten years ago, buying pizza for his family was a stretch. Now, he stood at the precipice of becoming a millionaire, even after taxes. Company DSCLSRE channels buzzed with dreams of early retirement, luxury cars, high-end vacations, mortgage payoffs, and kids' college funds. Samuel, ever practical, had no debt, no significant other, and no major expenses beyond rent. His plan was modest: sell some stock, indulge in a nice vacation to Paris, and let the rest ride.

Technically, Samuel was supposed to be out of office, but who in the startup world truly unplugged, especially with life-changing money in the pipeline? The boundary between work and rest had long since blurred. He justified his compulsion—he had earned the right to hover.

After scrolling around DSCLSRE, he opened Facegrind, his sole link to home. Baby and truck pictures from rural Georgia filled his feed. Not many opportunities back home. "They could never imagine being a millionaire like me," he thought. It was a poignant reminder of how far he'd come.

Then there was Oblivious. He hesitated but opened the app. Layoff gossip flooded the tech scene. Comments ranged from bitter to panicked to resigned. Older workers remarked that this was just 2008 all over again. Others shrugged it off—2008 was so long ago!

A post caught his eye: "Got laid off. Severance won’t last. Any leads?" The user was from his company. Samuel hoped it was a mistake. He didn’t check. Sometimes ignorance was a blessing.

Samuel locked his phone and wandered the neighborhood. The park, the sketchy-yet-reliable bodega—all staples of his routine. More kids played in the park these days, their parents conspicuously whiter and bearing pricy strollers. Remittance shops specializing in transfers to Latin America were substituted by artisanal coffee shops serving expensive pour-overs to the keyboard warrior class. Samuel, with his quarter-million-dollar salary, was part of that shift. Whatever. Rent was still acceptable—for now.

Jules lived in a dilapidated, dimly-lit basement apartment somewhere deep in Brooklyn. The persistent mold made his eyes itch daily, and rats scurried about, disrupting his sleep and fueling his nightmares. Despite the poor condition, the rent remained steep, the price of desperation after a rushed search for housing after his girlfriend broke up with him.

Rent was just one layer of Jules' problems. Jules chased his dreams of being a software engineer, attending the University of Michigan at Ann Arbor, where he did well—at the cost of his now omnipresent student loan debt. Opening up the student loan payment portal was a constant dread.

Jules' last job at a big tech company had ended in a layoff. Caspoke hadn't been his first choice, but given the job market back in 2022, he had to take it. Caspoke promised him the world—"home automation that works", unlimited opportunity, quick career progression, and a slightly higher salary.

His manager, Samuel, was supportive, engaging, and genuinely interested in Jules’ growth. Samuel's guiding principle was that success was based on luck, but that you could improve your odds. He encouraged Jules to be an active contributor to Caspoke's engineering culture. To that end, Jules hosted a bi-weekly engineering meeting where other engineers would introduce projects they'd been working on. Last meeting, he presented on build time improvements to the company's continuous integration process.

He was at the precipice of being elevated—last week, Samuel said that he'd compile a promotion packet for him for next quarter.

On the Precipice of Being Undone

A series of incredibly dumb events was about to unfold—events beyond Samuel's and Jules' control, yet destined to reshape their lives. For the worse.

First, the company's IPO was delayed. The markets were spooked, and leadership decided to hold off. Understandable—markets were known for their fickleness. That paled compared to what came next.

Blitcoin cratered.

Hard.

The stock market crashed. Samuel had never touched cryptocurrency, thinking it immaterial to his life. But now? Everyone's hands are in an empty cookie jar.

Team,

We regret to inform you that due to unforeseen developments, our IPO process has been indefinitely delayed. We know that many of you are deeply disappointed by this development. We share your frustrations as well.

Over the next few weeks, we will be working to optimize liquidity for as many of our employees as possible through a tender round. We have had promising conversations with our investors, as well as with a prominent sovereign wealth fund in the Middle East. As soon as we have more news, we will share it with you all.

Rest assured that no layoffs are planned.

This news does not change our return-to-office policy, as part of our office rationalization strategy.

In good spirits,
Bryce Peters
CEO, Caspoke

— #announcements, Caspoke DSCLSRE

Plans for Paris were shelved for good. Samuel figured the tender offer was a mirage.

Of course, the universe was conspiring against me, he thought. and iff God exists, He seeks a joke at our expense.

On the precipice of success, reality delivered its shove.

Samuel's manager, Kevin, scheduled an urgent one-on-one with him. His brain filled with chaos and fear.

The call started, and to Samuel's surprise, the call was much warmer than expected.

"Samuel, I know this is sudden, but let's start with the good news."

"Okay."

"Management has decided to reward you for your hard work."

More work? Worthless stock options? No more office lunch?

"Ok, what's the reward?"

"Well, first, some news that you might not like."

"Don't sugar-coat it."

"You're getting an involuntary promotion. Effective immediately, you're the head for consumer product engineering. They chose you because you're great at untangling messes."

"Ah, such a 'reward'! I'm soooo flattered!"

"Exactly, you get two jobs for the price of one!"

"So generous when most people would kill to have just one!"

"In this economy? Certainly."

"Anyway, was this 'promotion' the only 'reward' they were talking about?"

Jules' stomach twisted as the invite showed up in his inbox. "Important News." He knew what was coming.

He was joined by 300 of his co-workers on the Velocity meeting. Bryce, Caspoke's CEO, started sharing his screen - not the usual company slides, but a dense dashboard of metrics: user engagement, retention cohorts, NPS scores.

"The data tells a story," Bryce began, his voice carrying an enthusiasm for data over people, "Our north star metrics have deviated from the mean. NPS is trending downwards in all cohorts, conversion rates have declined despite continued A/B testing, and returns have spiked to a critical level."

He switched to another dashboard. Red arrows everywhere.

"We have had to optimize the company for the long haul. We ran the numbers. The most optimal strategy is a 20% reduction in our workforce, extending our runway whilst preserving operational capacity."

Jules texted Samuel. "Layoff."

"Unfortunately, this is the end of your time here at Caspoke. To minimize security risk, your access to Caspoke systems will be revoked immediately. Two months severance, one month additional COBRA. We believe a single, standard severance package will help reduce our cash burn."

He switched to yet another dashboard, a projection of the company's recovery after the layoffs.

"This is the only way forward."

Jules' company laptop went dark.

This VanillaBook is locked.
To unlock this computer, please contact your system administrator.

"No, but it's the most certain thing you're getting."

"Damn, I was so close to being a millionaire."

Samuel's phone buzzed. It was a message from Jules. He said he'd been laid off.

Kevin continued as if nothing had happened, "I used to say that when I was younger and dumber. Now I know better."

Samuel checked DSCLSRE. Jules' account had been deactivated. Oh dear.

"Uhh, I just heard Jules got laid off?"

Kevin paused. "Oh, shit."

On the Precipice of Being Ruined

"Why didn't I hear of this earlier?"

"You don't stare at dashboards all day?"

"I have work to do!"

"Hey, Bryce treats his models like they're his kids!"

"What are we, a data visualization company?"

"Actually, someone hacked our device to run Doom. Runs better than our software."

"Brutally honest. Coffee?"

"Thanks... rent's becoming a problem for me, though. Eight weeks severance is nothing."

"Wish I could do more. Do you want to grab a coffee?"

"I'd appreciate that."

"What time?"

"3 o'clock?"

"Meet me at Elena's Coffee."

"Can do."

Elena's was a small, cozy coffee shop not far from the Caspoke office. The thought of $9 lattes was foreign to Jules, and yet it was bustling, even in this economy. Samuel ordered a latte, Jules chose a drip coffee. Jules declined Samuel's offer to pay.

"Time for the MacDonough's coffee life," Samuel said, pulling out his metal credit card.

Jules thought of his battered coffee maker he’d snagged off Denisetable and his giant bag of pre-ground Boldrin's.

"Liked the coffee machine at work... well, used to."

"That thing brews despair."

"Guess I've got different standards." He's never suffered a day in his life.

"Wish I could help you more."

"You’re simply not in my shoes."

"When I was a teenager, my family was on the brink of homelessness—"

"I have to pick between rent or my student loans!"

"Jules, I didn't mean—"

"I graduated from Michigan! You went to some state school in Georgia!"

"The layoff didn’t discriminate by alma mater—"

"Now I have to ask whether I'm paying my student loans or my rent!"

"I got lucky. I know."

"Wish I had your luck."

"The universe can be a cruel mistress—"

"Like you're being right now!"

Jules stormed out of the cafe. Guilt sat with Samuel with the bridge ablaze.

Jules hunched over his laptop. Not the company-provided VanillaBook he had to return, but a laptop he bought back in college. The poor thing wheezed as the machine used every ounce of its power trying to bring up his budgeting spreadsheet. Once it opened up, Jules could finally start taking stock of his situation.

Okay, rent is $2,000 per month. Student loan payment, $1,300 per month. $500 per month for groceries. $100 per month for entertainment. Utilities are $200 per month. Internet is $70 per month, cell service is $35 per month, thank goodness for bargain-basement plans. Health expenses, around $100 per month. Transportation, $100 per month.

Every keystroke lagged the computer, it too reflecting upon his grim reality. No more entertainment. $200 per month for rice and beans and trips to the food bank.

He had five months of runway after the severance package and tapping his savings. Deferring student loan payments bought two more. After that? Beg his ex to take him in, or live with his parents in Florida. He hoped the job market would improve before he arrived at Morton's fork.

Samuel's aloofness weighed heavily on Jules as he made his calculations. Samuel was annoyed about skipping $9 lattes, while Jules was living precariously.

He finished the edits to his budget, closed the laptop, and cried in his bed. Today was a gut punch. Tomorrow, the countdown begins.

On the Precipice of Being Lost

Samuel walked out of the office. Winter had melted into spring, but the blossoms on the trees did little to lift his spirits. As he strolled down Broadway toward Union Square, his exchange with Jules replayed in his head on an endless loop. Each repetition tightened the knot in his chest. Jules' biting remark about Samuel attending "some random state school in Georgia" echoed louder each time. Samuel had fought hard for every inch of progress. Georgia Southern wasn’t Michigan, but it had been his stepping stone. Rejection from Georgia Tech had been a wake-up call, but Georgia Southern gave him a second chance. He took it and thrived. Yet, Jules’ words reduced his effort to nothing.

Samuel’s pace quickened as if he could outrun the thoughts, but the hurt lingered as he descended into the labyrinth beneath Union Square. As he searched for the L train, Samuel's thoughts turned to yesterday. Had Jules known that he had almost come to the precipice of being homeless? That his family had almost lost everything? That without the government's meager assistance, they would have been on the streets? It was so visceral, so real! Yet Jules looked at where Samuel was now, and considered himself to be the one who had it rough.

Samuel didn't have to wait long for the train. He boarded the train and took a seat by the window. The train was crowded, but Samuel was lost in his thoughts. The harsh, fluorescent lights of the subway beamed down on him, casting a harsh light on his face. He stared at his phone, longing for something, but uncertain of what he was looking for. The train rumbled on after First Avenue, dipping under the East River.

As the train sped under the river, Samuel finally lost it all. He began to cry uncontrollably, dark thoughts entering his mind. I'm a failure. I'm an overprivileged tech bro and I know it. He was consoled by a stranger, who offered him a tissue. He wiped his face with it.

"I appreciate it. But I had such a difficult conversation with someone I had to lay off..."

"It's going be OK."

"I can't! I can't connect on a human level!"

"Relax. We all make mistakes."

The anger in Jules' voice played again, and Samuel cried yet again.

"I'm sorry... I'm just so lost."

"We all have moments like this. Listen... what's your name?"

"Samuel."

"Mike. I'm not sure what you're going through, but I'd like to help you out as much as I can."

Samuel wiped his eyes. "I appreciate it, Mike. I'm just... who are you? Can you help me?"

The train now prepared to ascend into Brooklyn.

"I'm a philosophy professor at Columbia. We seek to have the world make sense."

"Making sense of the world? How?"

"There's always meaning. You just have to find it."

"I don't know if I can do it."

"Philosophy is for everyone. We have non-traditional student programs, too."

"I appreciate it, Mike. I'll think about it."

Mike left a card with his office hours and contact information. "See me whenever you feel like you're ready. Until then—"

The train's automated announcement system interrupted the moment. "This is Bedford Avenue." A different voice continued, proudly exclaiming that the station was accessible by those with disabilities and where the elevator was located.

He had many more stops to go before he could get home. That meant plenty of time to dwell, in a city where dwelling was a luxury few could afford.

Samuel walked back to his apartment after getting off at his stop. This had been the heaviest day of his life. Samuel rubbed salt in the wound of someone who'd just been laid off.

His bed awaited him. He embraced it, with tears in his eyes. Jules left his cell phone number on the company DSCLSRE. Samuel saved it. He would try to text Jules when he was in a better state of mind, try to make amends.

He fell asleep, his mind still racing.

Jules gasped for air as he woke up from the worst nightmare he'd ever had.

His ex demanded he move out immediately—worse, his apartment hunt was turning up nothing, and he hadn't found a job in six months. He was running on fumes and out of friends willing to let him couchsurf, so he bought a flight back home departing the following week. As he was making his way to Newark Airport on a nearly empty commuter train, a mentally ill man was saying that the planes were being manipulated by a shadowy force that he couldn't quite coherently identify. Midway through the flight to Florida, the plane suffered from a malfunction, crashing onto land somewhere in coastal North Carolina. Now here he was, the only survivor, and there was no calvary coming to the rescue. Fear struck deep into his heart, knowing that this was the end.

Before he could finish filling in his fate, he woke up from his slumber. It was 5 AM. At this moment, he would probably be brushing his teeth, getting dressed, and preparing breakfast. Well, he still had eggs and bacon in the refrigerator, the one that always seemed to be on its last legs. He feared the day that a power outage would finally send the refrigerator to its fate—discarded like his ambitions.

He turned on the stove. Oh good, I remembered to pay the gas bill. He cracked the eggs, scrambled them, and let them cook. Next was the bacon, the smell deeply pleasing to Jules as it sizzled. Once it was ready, he decanted the grease for later disposal.

Today, he was eating like a king. Tomorrow? Who knew.

Open to work.

Now, who wants to hire him?

The poor laptop screamed as it was called upon to open his resume. Each keystroke caused the computer to pause, contemplating Jules' next move.

Caspoke, May 2022 - April 2023
Software Engineer

Contributed substantially to a project to improve the efficiency of the data pipeline for Caspoke's consumer products.

Rewrote a service responsible for ingesting data from IoT devices to support higher throughput, reducing cloud costs by 24%.

Identified and resolved customer issues that were escalated to engineers.

Jules knew that these accomplishments meant little aside from showing he was potentially qualified. He couldn't show his work off or articulate what kind of societal value it had, aside from it trickling down into a bonus—oh, wait, they cancelled bonuses for the 2022 review cycle! What dumb luck!

The universe can be a cruel mistress. Samuel's words stung.

His machine crashed with a blue screen of death. Jules rebooted the computer, to no avail—the operating system didn't come back to life. Now I'm really up shit creek without a paddle.

He chose to buy a new computer, since he had been meaning to do so anyway. Not being able to recover the old computer wouldn't be catastrophic, since he kept a copy of his resume and budgeting spreadsheet in the cloud, along with using a password manager.

As he navigated the maze sitting below Barclays Center, a homeless person came up to him and asked him for a dollar. Avoiding them, Jules continued to exit the complex and walked to his destination, a gleaming glass-encased monument to tech, the Orchid Store in Downtown Brooklyn. If he hoped to get a job anywhere, nevermind the rough and tumble tech industry, he'd need to look the part.

The store had a flatiron shape. Jules thought the exterior reminded him of a stapler. A brightly-lit representation of an orchid flower in bloom was prominently centered between the top of the door and the roof. The glass made the store's layout apparent: clean, warmly lit, and neatly organized into a grid of tables with Orchid's latest and greatest devices on display.

He went inside. A beautiful, calming floral note wafted in the air while quiet, smooth muzak played in the background. The enormity of the store outside vastly diminished inside. His eyes were drawn to Orchid's most affordable laptop, the VanillaBook Bud, but also its upscale line for the aspirational, the VanillaBook Bloom. As he was looking, a "specialist" walked up to him and asked how he could help.

"I'm looking for a new computer," Jules meekly replied, "I'm a software engineer."

The specialist was chipper. "The base VanillaBook Bloom has a 8-core CPU as a minimum. How old is your current laptop?"

"It's from 2016."

"The P1 Turbo chip will be perfect for—"

"I just need a computer!"

"Let me check what we have in stock... how about a 15-inch Bloom with 12 cores, 48GB memory, and 2TB SSD?"

"I'm on a budget."

"We also have a 15-inch Bloom with 10 cores, 32GB memory, and 1TB of storage."

"Close enough."

"That's $2,399.00. With tax, $2,612.51."

That's two and a quarter months of rent. Or one month of runway.

Jules was resigned to his fate. He pulled out his black-matte metal credit card, a relic of better times, and paid for the laptop. To his relief, the transaction was approved. The specialist handed him the laptop.

"Here you go! Do you have any questions or need any more help?"

"I'm good."

Jules went back to his apartment and unboxed the computer. The Orchid experience continued with its clean, minimalist packaging. The wrapper protecting the computer had a subtle yet beautiful vanilla scent. He took the laptop out of its protective wrapper and opened the lid. A reassuring chime played and the orchid flower bud lit up the display, as the new electronic and vanilla smells became one ornate ensemble. As the computer progressed through its boot-up sequence, the beautiful bud blossomed into a brilliant bloom.

For just a moment, Jules could forget about his credit card statement and the mold.

On the Precipice of Being (Un)hired

Samuel found himself in a conference room across from Kevin, the head of engineering, the CTO, and Bryce, the CEO. The monitor displayed an endless loop of the company logo, the cheerful animation now a relic from a more optimistic era.

"Samuel," Bryce began, "As you know, our products resonate with millions of people, through the power of the internet of things. The data doesn't lie—you're our most important individual contributor, and ought to be compensated accordingly."

Kevin shot Samuel a sympathetic look. "Let's talk numbers."

The CTO leaned forward. "We've proposed a substantial equity package that reflects our confidence in—"

"Equity?" Samuel cut in. "Who in their right mind wants to cut us a check? Our latest launch was a disaster!"

Before Samuel could speak, Bryce cut in. "But do you know about the expected value—"

"Expected value? It's zero, because our products don't fucking work!" Samuel exclaimed.

Bryce ignored Samuel, continuing, "of your time here? You don't even know about all the future—"

Samuel stood up, "I quit!"

"Please listen to me—"

"Bryce," the CTO cut in sharply, "Can you build out that TPS reporting model you've been talking about?"

Kevin nodded emphatically. After a moment of confused silence, Bryce left.

"Samuel, please sit down. We can now discuss this properly," Kevin said.

Samuel sat down and took a deep breath. The CTO cleared his throat. "We can explore a cash bonus as well, though I want to emphasize our belief in the company's long-term—"

Samuel interrupted him. "I'm fighting a war with one arm tied behind my back. Half my team is gone. I've been spending nights and weekends executing a turnaround of the consumer division. You must appropriately compensate me."

"Samuel's right," Kevin interjected, earning a sharp look from the CTO. "We've asked him to take on an enormous, thankless responsibility. The consumer division needs a complete overhaul."

"What number did you have in mind?" the CTO said.

Samuel stated his figure. The CTO visibly winced.

"We'll need to discuss this internally," the CTO said.

"My best and only offer," Samuel said. Kevin gave him a subtle nod of approval.

A short moment of silence. "We'll get back to you," the CTO said.

Samuel stood up. "Thank you. I expect an answer in the next two weeks."

As Samuel walked out of the conference room, he knew this was a gutsy move that was likely to backfire. Gone was the man from rural Georgia happy that he had a big boy tech job at all, uncritically swallowing all the promises and lies. Now this Samuel was much harsher in the wake of his broken dreams, politeness be damned. Either he would get what he asked for—in cash, not equity—or he would walk away from the company that had once been his dream job. The monitor's looping company logo caught his eye one last time as he closed the door, its cheerful animation now seeming more like a tombstone for his former self than a symbol of future prosperity.

If there was one thing the VanillaBook Bloom was good at, it was being fast. Jules didn't need all the power he had on tap, but it didn't matter as long as he could pursue Graphdegree connections, read reviews on Woodknock, tap Oblivious for tips and referrals, and join Velocity calls with hot startups backed by ZZCombinator. It didn't hurt that the webcam was better compared to his old computer, although that meant he'd position himself towards the door instead of the moldy walls. More dignified than taking calls at the coffee shop.

Seeking Referrals!

I was affected by the recent layoff at Caspoke. Can someone please refer me for a position at Facegrind, Hell Gate Trading, or Wedge? Thank you so much!

TC: 🥜 (was $334k)
YOE: 3

— verne1997 (ex-Caspoke), Oblivious "Jobs and Referrals" section

There were some helpful leads posted in the thread. Not many of them worked out in the end.

He continued to apply everywhere.

Thank you for your interest in DSCLSRE. While your experience is a strong fit, we have moved forward with another candidate whose alignment with our core values better reflects our dynamic culture of transparency and innovation. Please check our careers page for future opportunities that match your passion for disclosure.

Jules, I wanted to thank you for your time interviewing with our team. Unfortunately, you will not be moving forward with the process at LongChain. We wish you the best of luck in your future opportunities.

Our team has reviewed your job application for the Senior Software Engineer (Trade Execution) position. Unfortunately, we have decided to move forward with other, more promising candidates. We cannot provide any feedback on your application. Thank you for considering Hell Gate Trading!

Hi Jules—thanks for applying, great resume. Currently we're looking for a head of engineering, I'll let you know when we've found them!

Some companies ghosted him after promising interviews. Others wasted his time with endless interviews.

The due date on his credit card statement grew closer, like a bomb's fuse slowly burning away, ready to ignite with interest.

Jules, we're pleased to extend you an offer as a Senior Software Engineer at BlockKit!

The salary was lower than what Caspoke paid him, but a pay cut trumps unemployment. He fired off an email to the recruiter, saying he'd accept the offer.

Hi Jules, thanks for your response. Unfortunately, due to market conditions, we will need to rescind your offer...

Jules was dejected. The only things that kept him going were spite and the prospect of being forced back to Florida, networking at a sad "Tech Professionals of Central Florida" event over stale Publix sweet tea, begging his parents to take him out to Waffle House instead of going back to rice and beans, and maybe having to settle for taking contract work on legacy applications built upon equally legacy tech stacks.

Then, finally, a breakthrough.

Hi Jules, do you have some time for a call today?

Jules' heart jumped. Was he going to be rejected again?

At five o'clock, his phone buzzed.

"Hi, is this Jules?"

"Yes. Is this Sylvia with Wedge?"

"Yes, it's Sylvia. We have excellent news to share with you."

Jules relaxed a bit.

"We are pleased to extend you an offer as a Software Engineer."

Jules cried. Finally? Only a day before the credit card statement became due?!?

Sylvia chimed in, "Are you OK, Jules?"

"Yes. I've been searching for so long."

"It's been a tough market, that's what I've been hearing from candidates. We have no shortage of applications for the limited number of positions we do have."

"Thanks, Sylvia. So what's the offer?"

"$150,000 base. Substantial equity. I'll send you a copy of our benefits guide. Plus, a product that works."

"You've heard about Caspoke?"

"Their smart remote makes a great e-book reader."

Jules chuckled. I guess those metrics made a twisted amount of sense. "It sure does have a great use case!"

"Better for reading Agatha Christie novels than turning on the lights. Still, a solid use case!"

The next two months were a blur. Samuel held an all-hands to introduce himself to the team. Then came the real challenge—discovering all the skeletons in the closet.

Samuel formulated a plan to turn things around. For the first time in months, the team felt like they had direction. They axed half-baked features in favor of prioritizing the core customer experience. A tactical team of engineers was laser-focused on shrinking cloud spend. A skunkworks team was hard at work reworking essential components that barely functioned. Engineers ran on pizza, soda, and a medically unadvisable amount of caffeine.

The product performance indicators started trending positive: cloud cost basis down 10.2%, NPS delta +3 points week-over-week, and latency metrics improved 15% after a key service was rewritten by Samuel's skunkworks team. Bryce proudly shared these dashboards in the all-hands, but the whispers grew louder—someone was gathering data points for his inevitable exit.

But now the day of reckoning came. The CTO called him back into the conference room. The company logo still played on the monitor, but the mood was different. The CTO's mood were more subdued, more respectful. They had made their decision. Samuel sat down, his heart pounding.

"Samuel," the CTO began, "we've decided to approve your bonus."

A bit of the old, snarky Samuel returned. "Oh, that's good. Just wanted to make sure you knew my value."

The CTO chuckled. "We're going to wire it directly to your bank account."

Samuel raised an eyebrow. "That'll make dealing with the taxman harder, but I'll take it."

The CTO smiled. "One more change. After today, you will now report to Kevin, who will be our new interim CTO."

Samuel shook hands with the CTO, "Glad I could help. Best of luck on your future endeavors."

With that, the CTO left with nothing more to say. Samuel had stood up for himself and he'd won. The next day, as promised, a wire transfer hit his bank account. With that, his bank account was now north of $100,000.

But Samuel didn't care about the cash. He cared about the principle: did leadership treat him as the valuable employee they claimed he was? He made a, well, somewhat unreasonable demand, and they caved.

He had won. But victory tasted like MacDonough's coffee.

On the Precipice of Security

Looking at his bank account that evening, Samuel couldn't feel the rush he'd expected. The six figures staring back at him from his phone screen felt less like victory and more like combat pay.

Jules's desk was empty, and it stayed that way for months. The desk still bore the signs that it had been once his - a faded spot where a photo frame had blocked the sun, and an unplugged monitor cable dangling towards the floor. Every morning, the empty desk reminded Samuel of his "win."

He still went to that same French bakery, ordered the same quiche and danish, watched the same elevated train rumble overhead. But something had shifted. The comfort these rituals once provided had hollowed out. His coffee, despite its perfect roast and precise preparation, tasted bitter.

Samuel stared at his metal credit card, heavy in his pocket. He had the means to go to Paris. But the money felt like a fortress he had to maintain, knowing his own position was more precarious than ever. The equity that was supposed to make him a millionaire still sat there, digital numbers in a digital account, worth less than the electricity used to display them.

Bryce was now gone. Officially, he was leaving to "pursue other projects," but everyone knew the truth—an investor who poured $120 million of fresh capital into Caspoke had insisted on Bryce’s departure, sweetened by a golden parachute. Was it that mythical Middle Eastern sovereign wealth fund? For all Samuel knew, this investor would've done better incinerating the money.

The CTO didn’t last much longer. His handling of Caspoke’s latest botched launch and ballooning cloud bills sealed his fate. In his place came Kevin, first as interim CTO, and later the permanent CTO, with Samuel reporting directly to him.

Changes were swift. Instead of swooning over dashboards, data was now balanced with customer feedback. The return-to-office policy was scrapped. A large group of underperforming employees, whom Bryce believed could become effective employees with additional training and mentoring—a program that was bleeding cash and ineffective—were laid off, in a dignified fashion. Finally, to much employee delight, the office lunch program, based on a complex employee engagement and performance model that was hard to understand but easy to game, was replaced by a straightforward daily stipend.

Samuel had always been the company’s go-to fixer. His turnaround of the consumer product had saved the company—at least for now. But Samuel felt no relief. If anything, he was standing on a different kind of precipice—personal, emotional, existential. Materially, things were better than ever. Inside, he was unraveling. The sleepless nights, the constant high-stakes decisions—each one chipped away at him, until he finally broke.

He looked up Mike on Columbia's website, found his office hours, and decided to pay him a visit. A long journey to the campus on the 1 train followed.

Mike's office was small, but cozy, with a bookshelf filled with old, dusty tomes. Mike looked up from his desk, a smile on his face.

"Do you remember me?" Samuel asked.

Mike squinted. "I'm sorry, I see a lot of students. Can you remind me?"

"I'm Samuel. We met on the L train a few months ago. You offered me a tissue. Told me about philosophy."

"Ah, Samuel! I remember now. How have you been?"

"Been better. Been worse."

"What's going on?"

"I'm still that same person. Richer, but still broken. I've won, at an enormous cost. Therapy isn't helping."

"Meaningless suffering is a curse. Meaningful suffering is a choice. That's what philosophy explores."

"I'm done with tech. I'm ready to explore."

"I'll get you in touch with the right people in admissions."

"Thank you, Mike. I appreciate it."

Returning now to the office, Samuel drafted an email to his boss. "Kevin, I've decided to enroll in Columbia's philosophy program. I've been feeling lost, and I think this is the right path for me. I am requesting a leave of absence for the next year to focus on the program full-time, and if this is not possible, I will have to resign. I hope you understand."

He hit send, and felt a weight lift off his shoulders. He was still on the precipice, but now he was moving forward. He was still lost, but now he was searching for answers.

This led to a three-way conference between him, Jeff, the new CEO, and Kevin, his manager and the CTO.

"Samuel, we understand your decision," Jeff said. "But we don't want you to leave."

"I'm sorry, Jeff," Samuel said. "I need the space."

Kevin spoke up. "We can offer you a sabbatical. You can take a year off, and we'll hold your position for you."

Samuel considered the offer. "I appreciate it. I'll take it."

Kevin nodded. "We'll get the paperwork started. We'll miss you, Samuel."

With that, Samuel spent a few weeks preparing for his sabbatical. He wasn't going to be paid during his leave, but he had enough to live in New York for two years without any income. He had enough to pay his rent, to buy food, to live comfortably. He had enough to find himself.

He walked out of the office, the weight of the world off his shoulders. He was on the precipice of moving forward.

"Jules? Mind if I sit down?"

Jules looked up from his coffee and his VanillaBook Bloom. They were at a small coffee shop close to Washington Square Park. "Samuel! Of course, sit down. Coffee's on me, if you want."

"Hope the new gig is treating you well. I'll pay for my own coffee."

"It's great. Commute hits different since it's downtown."

"Moved out of the basement?"

"Indeed! I'm in Long Island City now. Have a new girlfriend too."

"Excellent! I'm on sabbatical, beginning my philosophy study at Columbia. I had to find myself."

"Good to hear. I hope you find what you're looking for."

"I had everything I wanted, besides myself."

Jules nodded as he pointed at his laptop. "Materialism can only take you so far," he said with a knowing smile, as to remind Samuel that he was the catalyst for Jules' unplanned purchase.

Jules never stood at the precipice of being a millionaire. Yet, he was ruined, lost, and gambled it all. Now he was at the precipice of material security.

Samuel was no longer at the precipice of being a millionaire. He'd been ruined, he'd been lost, he too gambled it all. Now he was now at the precipice of security in self-actualization.

Perhaps security is the most important precipice of all.

🌀 Being nonlinear: A transition from modding

2025-01-06T00:00:00Z

I've written quite a bit about Minecraft on this blog. For better or worse, it has defined a large chunk of my life. But there inevitably had to be a time where I'd have to make the transition to a "real" job. Making this adjustment was painful but necessary.

I'm writing this blog post based on my experiences with Minecraft, particularly its server modding scene, however I have taken great pains to make this applicable to any game. Your situation will be unique, but I hope this post provides you with useful advice.

What I find in general is that modding can be excellent at providing a good foundation for your "hard skills" (the act of writing code and designing things), it does make it dangerously easy to not develop the "soft skills" that are so often needed in the workplace. I wish I'd found a post like this before heading down the professional career path, and so I'm immortalizing some of the hard lessons I had to learn.

You add value!

As a mod developer, you're in a category of people known as non-traditional candidates. This is a diverse category: career changers, parents returning to the workforce, and self-taught programmers. These people all bring their own unique talents and perspectives to the table.

Specifically, what do you bring to the table?

You can self-direct your work. You can accomplish tasks without being prompted by others, and you have a bias towards action.
You have practical experience. In some cases, you learned how to build software by yourself, showing that you are agile and can pick up new skills easily.
You are a creative problem-solver. Mods can improve upon existing systems, or find creative ways to overcome game engine limitations.
You can interact with communities. You have likely worked with other developers and supported your users in some way.

If you choose the gaming industry, then you additionally bring:

A deep understanding of the player experience, because your mods build upon a base to add more depth, or correct perceived poor experiences.
Familiarity with game systems and mechanics, especially if your mods add entirely new game modes to an existing game.
Experience balancing technical constraints with user enjoyment, because you'll find creative ways to work around limitations of game engines.

Simply put, your experience with modding need not be a liability. In recent years, more companies have begun to understand that diversity can strengthen teams. You are in a prime position to take advantage of this!

Where do you want to work?

The most "natural" path to advance is to transition from modding to game development. After all, modding entails understanding a lot of industry-specific knowledge. You could easily take the experience you have and apply it to making your own games.

The downside of this is that the gaming industry does not provide a good working environment:

Work-life balance is poor. "Crunch" is a well-documented phenomenon, working additional hours beyond what a developer is paid for, often resulting in lower-quality work and health issues.
The industry is unstable: layoffs and studio closures are common.
Compensation is often lower. While just one data point (and only an approximation at most), Glassdoor cites a median salary of $106,000 for game developers, versus $162,000 for software engineers in the United States. That's more than a 50% difference!

Another path that is available is to translate your experience to other fields outside of the gaming industry. This takes more time and preparation, since you'll likely need to develop a new set of skills to navigate that transition.

I chose to transition into a non-gaming industry for a few reasons:

I had a desire to understand how the "real world" works. What's more real-world than software that other people depend on for their jobs? My work at Ramp touches not only the lives of accountants but also regular everyday people who need to travel for business.
I had previous experience with databases, networking, web development, and tackling scalability challenges. At various points in my time in the Minecraft scene, I handled some combination of all of the above, so a transition to working at a scaling startup made a lot of sense to me.

Have a solid foundation

This is the only section I'm going to devote to hard skills, because if you are an experienced modder, then you have a foundation to build off. The challenge is in developing skills that are industry-relevant and in demand. The good news is that the modding scene actually offers opportunities for this, if you know where to look.

I'll take Minecraft as an example. Disregarding Bedrock Edition, the Java Edition gives away a lot in the title: it's written in Java, and you are able to mod the game using Java (or, in fact, any JVM language you desire). As it turns out, Java is a rather common language used in actual software engineering jobs! You already have the fundamentals you need to learn, say, Spring Framework (a very common Java framework often used for web applications), so you may want to take some time to develop a web application using Spring. In addition, you can leverage this to learn web development using HTML, CSS, and JavaScript.

Even if the modding language of choice isn't commonly listed on a job posting, it can still be applicable to learning another related language. For instance, if you understand Lua, then you already conceptually understand the core of JavaScript, as both languages are multi-paradigm and have prototypical object systems. Also, Lua has had some very large scale real-world applications - for many years, Cloudflare's HTTP reverse proxy logic was partially built with Lua!

You can also work on projects related to the game of your choice in a language that is different from the one that the game is programmed in. At various points in my time in the Minecraft scene, I also made use of PHP (with Laravel), Python (with Django and Flask), Go, JavaScript/TypeScript, and even Rust. One network I worked on (Mineteria) was built on top of Kubernetes, so I also gained experience with that as well.

Now, this comes with some obvious caveats:

Code standards that might fly for modding may not work in a professional setting. Often, the priority in a game modding setting is just to "get the thing working", which means that you will inevitably have code that is not clean or maintainable by others, or have poor documentation. These things will not fly in a professional setting. As Guido van Rossum, the creator of Python, would put it, code is more often read than written.
Techniques useful for game development aren't broadly applicable to enterprise applications. For instance, if you work in a garbage-collected language, then you have probably optimized your code to reduce the amount of garbage objects generated and value low pause times. Enterprise applications are often less sensitive to pause time and can afford to generate more garbage. Some of these techniques are even considered bad form in some contexts! The upshot is that in some applications that require low latency, such as high-frequency trading (where shaving off nanoseconds really matters), this can be a competitive advantage.
You will necessarily have to get out of your comfort zone. The types of applications you're going to work on in a professional context are different, so you will be best served by gaining experience outside of modding. That could be as simple as building a simple web-based tool to help simulate some key game mechanic.

Prepare to have lowered expectations

Sadly, the median software engineer will be stuck at a large corporation focused primarily on internal CRUD (create-read-update-delete) applications that are obscure but also important enough to necessitate hiring someone to take care of it. This work may not be necessarily creative in nature. In some cases, you're doing this simply to "pay your dues" and set yourself up for greater things. Other employers may not offer a realistic path to advance.

I can give you an actual example from my own life: from 2018 to 2021, I was working on Mineteria. During this time, I built key infrastructure for them (working across many different programming languages), and created my own Minecraft server proxy, Velocity, which was later adopted by Mineteria and countless other Minecraft servers. After leaving Mineteria behind and eventually landing at Ramp, the very first project I was assigned was "create a system to link a personal bank account" - quite the step down from "build out an efficient proxy that speaks the Minecraft protocol"!

The skills I got from my time were still helpful: thanks to previous work, I was familiar with Python, ORMs (although I had to make the transition from working with the Django ORM to SQLAlchemy), Celery (from a previous Minecraft server list project I'd worked on), and PostgreSQL (Ramp's backend database of choice).

Much later on, having the skills needed to read a profile meant that when we had a particularly bad memory leak in one of our applications, I was able to pinpoint it and fix it easily. Over time, too, I established enough of a reputation and track record that I was entrusted to architect and execute upon much larger, more complex projects.

So how can you still be happy while working a day job that might be less exciting? Well, you could still take up modding, but on a smaller scale. Even now, I still occasionally contribute to Velocity or update my existing Minecraft mods for newer versions of the game. You could also take up new hobbies to get your creative energies flowing. For me, due to burnout and having less interest in Minecraft in general, I found that redirecting my creative energies towards writing helped me a lot. If your employer has put you in a position where your job is effectively a dead-end with no realistic path to advance, then you should shake things up and find a new job that can make you happier!

Being in the workplace

Your company's internal chat is not going to be a Discord voice chat (if it is, I would probably run far away!)

The top priority for anyone transitioning from a modding community to the workplace is to studiously separate your gaming life from your professional life. Gaming communities, particularly competitive ones, can foster toxic behavior that is almost normalized in that setting but would be unacceptable in a professional setting, where inclusivity, constructive feedback, and respect for diverse perspectives are a must.

There are many, many excellent online resources that can dive into way more detail about how to be effective in the workplace than this blog post could ever hope to accomplish. I will describe some broad categories of professional expectations below, how these could be treated in the context of a modding setting (potentially with an example with my own work), and provide resources for you to learn how to act professionally.

Project management

You may or may not be practicing project management in your work, but it is essential in the workplace, not just to keep yourself organized and make sure you can advance a project, but also to provide a bird's eye view of the status of a project to other stakeholders, outside of updates you provide in writing or by hand.

There are a number of approaches on how to learn project management and apply them to modding. You could go all out and take a free course on how to do project management on a site like Coursera, but often, having a working knowledge of Agile can be sufficient, as many companies utilize Agile methodologies. I won't go into detail about Agile, but consider understanding the Agile Manifesto and putting it into practice yourself. Users of your mods are customers!

You can also leverage tools such as Trello or Notion to be more organized. These tools can help you:

Break down large features into smaller tasks
Track bugs and feature requests
Manage release schedules
Document design decisions

These tools and techniques can make your modding work more efficient and also provide practical, hands-on experience with professional project management practices.

Professional writing

Some people in the modding scene may write documentation, but it's often not of the highest quality. Being able to express your ideas with clarity and specificity is important in a professional context, even if you're not going to be a software engineer.

I cannot hope to recreate existing guidance on professional writing, so I will instead refer you to Google's technical writing courses which comprehensively covers this topic. While it is geared towards software engineers and engineering-adjacent roles, there is useful advice in it for everyone. Once you have the foundation, you can begin by providing detailed documentation of how what your mods do and how they work.

Many projects have poor documentation, and one especially low-hanging fruit way to contribute is to improve the documentation. If there's something that trips up newcomers and is difficult to explain, take the time to put yourself in a newbie's shoes and write a guide to help them navigate the troubled waters.

From personal experience, improving my writing skills was one of the most important factors in allowing me to progress at Ramp. Previously, I had never written down important things about my software, or ever committed software design down to paper. Having to learn this habit was tough, as prior to Ramp, I only did this during my college studies! Since then, I rediscovered my strong command of writing and embraced it.

Interacting with coworkers

Professional relationships require respect and courtesy. While gaming communities often have casual or playful communication styles, workplace interactions require more care. A good rule of thumb: if your 13-year-old cousin would take offense at what you're about to say, it's probably not appropriate for work. Focus on building positive relationships with coworkers through clear, respectful communication. Even when you disagree with others, it's important to handle those gracefully and work towards an acceptable resolution - perhaps one party in the conversation simply lacks context on why the decision makes sense, for instance.

In addition, your coworkers may come from different backgrounds, and you should be sensitive and respectful of those differences. Those different backgrounds bring valuable perspectives to the table, and you'll be expected to work effectively with people of different backgrounds.

From my personal experience, this isn't difficult to do, but your old habits could take some time to shake off. Take advantage of free resources on workplace etiquette and communication. Many universities and online platforms offer professional skills courses. Once I found my bearings, though, I found it easy to establish good rapport with my coworkers, enough to the point where I am something of a "minor celebrity" at Ramp!

Having respect for workplace norms

In most gaming contexts, there is little etiquette expected. This is absolutely not the case when you are working somewhere else!

Every workplace has its own culture and expectations. This includes:

Meeting etiquette (being on time, coming prepared, actively listening)
Communication channels (when to use email vs chat vs scheduling a meeting)
Work hours and availability expectations
Dress code and office behavior standards
Using professional language and tone

The nice thing that you can practice doing at least some of these things while still being involved in the modding scene:

Run a structured community meeting, if applicable
Set up "office hours" to talk with anyone who needs help
Utilize professional communication tools such as GitHub or Slack instead of Discord
Have a consistent schedule for updates and communication
Writing professional documentation and release notes

There are many free resources online that can help you understand these societal norms. Consider taking a professional skills course such as one on Coursera and researching typical professional dress codes.

Actually getting a job

So far, everything that's been discussed so far has focused pn what you are expected to do in the workplace. But how do you actually get a job with a non-traditional background?

I will skip over the essentials of how to apply for a job, as this topic has been done to death by many others. Instead, I want to focus specifically on how people of non-traditional backgrounds can begin the process.

Generally speaking, you have three options:

Ignoring your modding experience, going to college, getting internships, and then a full-time job after graduating. This works best if you're young.
Make connections with others who can vouch for your ability.
Being lucky enough to find an employer that independently values your experience, or is willing to take a chance on you.

I'm not going to cover the first option, simply because this is a well-trodden path. Instead, I'll talk about 2 and offer some advice that can help for 3. This isn't going to be a full guide on how to find a job, perform well in an interview, and understanding offers and whether or not to accept them, as this topic has been done to death by many others.

Building a resume

There are many resources on how to format a resume, so I won't belabor the point here. However, translating your modding experience to software engineering can be difficult: perhaps poor documentation and record-keeping, working with particularly niche frameworks and languages, and the fact that a life of modding video games is a significant divergence from a typical software engineer's resume which would typically include a number of internships or student jobs prior to obtaining their first initial junior-level position.

Feel free to use my own resume, in particular my experiences with Tebex and Mineteria, as inspiration. Some key advice:

Try to generalize your experiences. For instance, I took on a number of roles at Mineteria, but I mostly honed in on web development, experience with Java, and work with Kubernetes, as these are the skills most relevant to opportunities outside of the modding scene. If you don't have any direct experience with something an employer wants, discuss analogous, equivalent experiences in language a typical employer would understand.
Focus on concrete accomplishments. I developed a Minecraft server proxy called Velocity. However, on a resume, instead of noting that I "developed a Minecraft server proxy", I would emphasize that I "built an extensible, high-performance reverse proxy capable of supporting X Minecraft players at a time." This highlights that Velocity is a proxy that just happens to speak the Minecraft protocol, which is a precise high-level description of what Velocity is.
Emphasize the underlying accomplishments behind your work. Did you fix a particularly bad performance issue in a game you like? Did you use a profiler to examine why it was so bad? Congratulations, you have the unique skill of being able to diagnose and fix performance problems in applications.
Be ready to explain your unusual background. I was very upfront about my experience being primarily with Minecraft when I interviewed at all my prospective employers! One employer even had someone on the call who quizzed me about certain things related to Minecraft and I provided the best answers I could. They actually extended me an offer, but I declined because Ramp offered me more money.

Networking

Other developers you work with can be a valuable resource, especially if they are also going down this road. Some people who create mods may already have full-time jobs and can potentially serve as potential job leads by referring you. For instance, through working on a Minecraft component library, I was acquainted with a modder who was also a product manager at Cloudflare, and he referred me for a position working on Cloudflare Workers! This person was willing to vouch for my abilities and show how they apply to a context outside of Minecraft.

You can also pursue more traditional means of networking. Find local user groups for a programming language or technology that interests you. Many people who work professionally have a presence on LinkedIn. Connect with them and get inquisitive about their work!

Ultimately, your goal is to establish connections that help you now and in the future. This is all the more reason to be respectful of others and to embrace diversity.

Finding employers willing to take a bet on you

This is much harder, since you'll need to start from the very bottom. For most positions, this is equivalent to applying to a job where it's not obvious that you meet the requirements for the job. Therefore, you'll need to start by looking at employers who may be more forgiving to candidates with less experience.

You may need to start with local employers, who will typically focus on basic CRUD applications along with taking on junior maintenance work, and typically pay less. You could also consider contract positions. While they may be less complex than modding projects, these roles provide crucial professional experience and help you build a track record. They're also opportunities to learn how businesses operate, which is valuable knowledge that's hard to get from modding alone.

Humble beginnings mark the career of a lot of software engineers. What matters is using these early positions to learn professional practices, build a network, and demonstrate reliability.

Tying it all together: my experience

Getting my job

My last job search was in October 2021. Of the roughly 20 companies I applied to (through recruiters, referrals, or job applications), the most successful leads involved referrals from people who I knew from the Minecraft scene that successfully made the transition to other employment. Of those, the furthest I got was with Ramp, for a few reasons:

I was referred by a former Ramp employee whose referrals were generally quite successful there. I worked with this person on many projects from 2012 to 2016. This baseline level of trust between Ramp and this former employee, along with keeping this connection alive, was necessary just to get me in the door.
Ramp likes to find "mispriced" talent: my background was unusual enough that curiosity and the desire to hire someone who could bring something unique to the table led them to seriously consider an obviously neurodivergent Minecraft modder with unusual expertise. This is a bet that has paid off for them: I am often the "Minecraft whiz" being discussed.
I had strong computer science fundamentals from not only a largely completed college education, but also my prior experience with Minecraft. Ramp's interview process is a practical, and I was able to ace it with ease. I interviewed quite well and made a good impression upon the hiring team.

I chose to accept the Ramp offer as I believed the growth story (in terms of valuation and my career) was robust and that it would provide the best chance of success at establishing myself as credible with future employers.

My first year at Ramp

At Ramp, I joined at a level that you might call "Software Engineer II" (internally, we call it IC4, being loosely based on Meta's similar ranking system). This reflected that I likely could deliver complete features on my own, unblock myself, and write production-quality code, and overall exhibit sound software engineering fundamentals. I entered this level having extensive "hard skills" experience outside of a formal context, and so progressing at Ramp required me to develop a number of soft skills that I had only partially begun to develop.

The projects I worked on at Ramp in my first year were representative of a newly-hired mid-level engineer at a rapidly-scaling fintech startup:

Integrate personal bank account linking.
Finish work on integrating a new bank partner for our payments platform.
Add the necessary support for representing wires completing more quickly than the days-long timeframes we already supported (ACH and printed checks).
The biggest project of all: successfully integrate support for international payments through a third-party provider, Wise.

The first three projects primarily dwelled upon my "hard skills": creating DB models, testing software, and getting code reviewed and approved by other engineers. The project scope and requirements were clearly defined, and the work primarily focused on hard skills and being an individual contributor at Ramp. I spent most of my time writing code compared to engaging in inter-team or cross-team collaboration.

The Wise integration was the most complex project of all, because the requirements were poorly scoped and I was expected to take on most of the work of designing the implementation. As part of that, I had to collaborate with other teams at Ramp (product and legal), with an external team (a solutions team in New York and an operations team located in Estonia, so quite the extreme time difference!), write clear documentation for engineers to review, and clearly and timely communicate with other stakeholders. Along the way, I learned a lot about the U.S. financial system, the complexities of integrating with a non-trivial API, and solve challenges that I knew other teams would face. In particular, the need to automatically extract KYC data to provide to other third-party providers prompted me to build a reusable solution that has since been leveraged by other teams. The successful delivery of this complex project was a significant consideration in my promotion to senior software engineer (IC5) in March 2023.

Conclusion

Making the transition from game modding to professional software engineering won't be easy. You will need to learn new skills, adjust your expectations, and learn to navigate workplace dynamics. However, the fundamental skills you develop while modding - problem solving, coding, working with complex systems - can serve as an excellent foundation for a professional career.

My journey from Minecraft modding to fintech wasn't conventional, but it worked because I leveraged my existing technical skills while actively developing the professional skills I needed. The key is to be patient with yourself during the transition while remaining committed to growth and learning.

If you're currently in the modding scene and considering making the transition, start preparing by:

Building a solid foundation in widely-used programming languages and tools
Practicing professional communication
Writing good, high-quality documentation
Developing project management skills
Building a network that can help you bridge the gap
Being open to starting from humble beginnings

Most importantly, don't view your modding experience as something to hide or minimize - it's a unique part of your journey that provides a unique perspective. The right employer will recognize and value that experience, even if it takes time to find them.

The path forward might not be straightforward, but with dedication and the right approach, you can successfully transition from modding to a professional career in software engineering. Take me as an inspiring example of what you can accomplish!

📚 A 2024 dispatch

2025-01-01T00:00:00Z

In classic Steinborn fashion, I forgot about my 2023 dispatch until August 2024(!), and it wasn't necessarily a sufficient explanation of what had actually gone on. So it's time for me to digest all of 2024, and also try to be way more detailed about what went on, but I wound up starting on this post on New Years' Eve. Why is that, you might ask?

Logistical. I simply hadn't made it a priority then. I didn't even know people read this blog until relatively recently. For the longest time, it felt like I was simply shouting into the void.

Now I get to dive into 2024, and let's just say... it was a decidedly mixed bag for me. A lot of unambigously good things happened, but there were also quite a few low points that dragged me down. But let's waste no more time.

Sorrow

This year has been marked by one rather consistent theme: sorrow. With no way to control what's going on.

The year began with me losing two family members: my grandmother died on January 30th, 2024, and my mother died on February 11th, 2024. Just now, into New Years Eve when I wrote this, I looked at the photos I took at the funeral: cold, embalmed, what once was is now no more. Thus, my first two trips of the year entailed me flying (at the last minute) from NYC to Atlanta to attend first my grandmother's and my mother's funerals.

My grandmother's fate was explicitly chosen by her. Her kidneys had failed, and instead of dialysis, she chose to let nature take its course. I don't blame her for taking that course—having approached beyond 80, a thoroughly empty nest, and a progressive decline in her ability to do things, I wasn't shocked to see her take this path.

My mother, on the other hand, had been in very poor health for years. There were many health scares, some sending her to the hospital. The episode that would finally claim her life started in late October 2023. She wound up at Grady for several months before being released to a rehabilitation facility, where she seemed to improve before her heart gave out through cardiac arrest. I still remember how much time I had with her: five minutes at the rehab facility, before they needed to do something with her and I had to go back to the airport (conveniently, this facility was located just south of Hartsfield-Jackson).

After my mother died, this left me in a pickle: my brothers relied on her Social Security disability, while I was trying to live my own life in New York. This caused some tension. The good news is that my younger brother did finally manage to find a job and keep it down, so I'm not as worried about them any more, and they're no longer having to rely on me to support their day-to-day needs, but there were a lot of very uncomfortable conversations with them and other family members in the wake of it all, and that's something I wish not to repeat ever again.

All of this has been painful. It's part of the reason why I simply haven't worked on very many personal projects. (That said, I did manage to redo this website using UnoCSS and learned an entirely new way of CSS, so there is that.)

Writing

Sometimes I consider myself a storyteller who just happens to be a software engineer.

It was late in the year when I first started being serious about writing again. Prior to this, I had written a poem about grieving, before I gave my website a six month hiatus due to other events that were going on at the time (primarily work-related). As a result, my site hasn't taken off (on the orange site or anywhere else, in fact).

I started being much more serious about writing starting in August. In fact, my most prolific topic was the two website redesigns I did, one in August (a fairly minor one, mostly adding dark mode to an HTML5 UP template) and one in October (basically a total redesign). The latter is not my best work, as I documented it quickly after doing the redesign and GitHub Copilot filled in some of the text with some rather awkward language. (Pro-tip: code completion LLMs produce rather boring English sentences. Prefer using a fully-fledged chatbot instead. I've been mostly using Claude.)

I also wrote my second longform technical article, The Velocity Chronicles: Part 0, discussing quite a bit of my background in the Minecraft scene, and the motivation behind me starting the Velocity project in the first place. I've gotten some positive feedback on it, and I hope to get Part 1 out in the new year, primarily covering events that happened in late 2018 and 2019.

I also started work on two pieces of fiction in earnest: a novel and a short story called "Precipices." I started on the novel in September and while I've made substantial progress, it's not at a point where I'd be happy with publishing the work. "Precipices", which I started on over the holiday break, is much closer to complete (it will probably be out in January 2025).

Overall, a much more successful year in terms of writing things! I hope to continue doing more writing with a mix of fiction and technical writing, in the hopes of establishing a bigger "brand" for myself. I'm obviously not looking to be some sort of "thought leader" or the like, I just want to keep my brain active. I'm also hoping to write my 2025 dispatch as the year goes on instead of doing it all at one big spurt at the end of the year, just to reduce the cognitive load on myself.

I do also feel as if my focus on writing at work has also greatly contributed to sharpening my writing skills overall. That said, if I had two things to focus on, it would be "sharpening my creative writing skills" and "actually remembering to write things."

Ramp, Ramp, Ramp!

First, let's talk about the good parts: 2024 was actually a banner year for me at Ramp! That year saw a few major accomplishments, which I hope to blog about either on the Ramp engineering blog or here (if they're not particularly Ramp-specific).

My impact is recognized. Also, I sport a funky, graffiti-inspired credit card-themed tie.

So far, Ramp has been the longest gig I've ever worked, and that's including the Minecraft server space (the longest was previously my time at Mineteria/Voldex? That was March 2018 to January 2021, so close to three years). Having this level of stability and longevity is previously unexplored territory for me. It has also been, arguably, the most interesting and impactful gig I've ever had, albeit less exciting in some ways (only in Minecraft would you be gutsy enough to write your own proxy at small scale and get away with it).

I joined Ramp in November 2021, at what I consider to be the "Goldilocks zone": early enough to continue to see the company scale, and yet late enough that almost all the risk was removed. (I also lucked out in joining at a time when the job market was unusually good for software engineers.) Ever since that point, Ramp has progressed from being a small, ~200 person startup, to having over 1,000 employees and penetrating the enterprise market. With this move upscale comes significant new challenges that we'll have to solve, and a lot of my work in 2024 was principally to prepare our issuing platform for this next stage of growth. We took a long, hard look at the issuing platform that got us to a $7.65 billion valuation at the time, and considered it unfit for purpose as we continued to scale. The responsibility for making it scale for future growth has largely fallen upon me. No pressure!

This doesn't look like much, but it was actually a pretty significant milestone!

Our team changed a lot over the year. Some of the new folks we hired were people I mentored was responsible for mentoring (yes, we're hiring... come work with me!)

I also got to work on some interesting side quests. I'm hoping to write about those in the future (whether here or on the Ramp engineering blog), but a lot of those have the end effect of making the Ramp platform more reliable, important as we begin to be depended upon by larger customers.

The personal tragedy of 2024 did spill over into my work. There were times where I almost wanted to give up. One such point came in early April, where I almost sought to leave my present team, but after carefully thinking it over and discussing it, cooler heads prevailed and I decided to stay and continue down my current path. It's been quite successful, all things considered.

So, what does 2025 have in store for me? Ramp has only captured a very small fraction of the TAM of U.S. corporate spending, and an even lower percentage of global corporate spending. My job now is to keep the momentum going! That means evolving our issuing platform, helping build up a world-class financial technology primitives team providing leverage to the rest of Ramp, and supporting our move towards new markets for Ramp. It's going to be an exciting time to be at Ramp!

Travel

I only made two major trips in 2024, aside from funerals and a emergency trip related to estate matters. The first trip was to Miami, to work out of the Ramp office there for a week, and the second trip took me to a place I'd never been before: Washington, D.C. Both trips were prompted by the above family deaths, the former being a welcome change of scenry, the latter being a much-needed break from work and the family situation I wound up in.

Looking ahead to 2025, I'm planning to keep my travel schedule light. While my trips to Miami and DC had their bright spots, 2024 reminded me just how draining travel can be – especially the multiple last minute flights to Atlanta for funerals, which took both an emotional and financial toll. The upside? Amtrak is very good and can make things less painful, at least if your destination is on the Northeast Corridor. Between those difficult journeys and the general hassles of travel (choosing transportation, booking hotels, navigating airports, and planning itineraries), I've appreciated staying put. There's so much of New York I haven't seen yet!

Miami

I was in Miami in late March for a work trip, which I got permission to take in the aftermath of my mother dying. Because this was for work, I wasn't able to spend too much time doing other things, but I did at least visit the Wynwood Walls and the Museum of Graffiti, along with going to a gun range (which brought out a decidedly different side of me, closer to my origins than before).

That said, there was a significant work-related milestone accomplished while I was here in the course of reworking our issuing platform...

Why is this Chicago "L" train lounging in the Miami heat?!?

Getting a lesson in graffiti typography

Some cool skateboards

Washington, D.C.

I decided to go on this trip in April 2024 for Memorial Day week. I needed this: my last "break" was in late October 2023 and I stayed in the city after a particularly rough on-call cycle and "quality cycle" that sapped me. I had never been to D.C. before, not even for school, but surprisingly, there were a lot of connections that Ramp and even my team had to the D.C. area! It was where my aunt went to school, and I also knew someone from the Minecraft scene that lived in the area.

Arrival was smooth thanks to taking the Amtrak Northeast Regional train from New York Penn Station, which drops you off at Union Station, not far from the Capitol and with convenient access to the Red Line of the Washington Metro. You can bet I enjoyed riding on perhaps America's greatest Great Society metro system...

I got to see the obligatory things: the Capitol complex, the White House, various random government buildings, museums on the National Mall, the Metro, Arlington National Cemetery, Old Town Alexandria, and more.

Perhaps most interesting of all, I spent time in the chaos known as Tysons and the area around the Dulles Access Road (accessed via the Silver Line, of course) to meet up with an old buddy of mine from the Mineteria days. Tysons is a real mess of a place: highways, wide stroads and other auto-centric land uses, office towers, shopping malls, and even Washington Metro stations sprouting in a place that was once largely forest and farmland. On the particular Friday I went there, it was a well-appointed ghost town. But hey, now I can say I've been there!

The trip, unfortunately, did end with me contracting COVID-19. Totally worth it, though!

Here's a few photos from that trip.

An Orange Line 7000-series train bound for New Carrollton has stopped at Farragut West. The Metro is known for its iconic "waffle" brutalist station design, especially in downtown D.C. and the stops closest to downtown.

Of course I went to the usual attractions and centers of power. Only do it once, though.

Nice, almost bucolic Georgetown, the end of the Baltimore and Ohio Canal.

I took a lot of photos of the Washington Monument but this was my favorite.

Tysons is a secondary business district within the Washington metropolitan area, and it's a weird place.

Joey invited me in for pour-overs. I didn't lack for caffeination on this trip for sure!

Prediabetes, eating disorders, and weight loss

First, a bit of context, since I haven't talked about this before.

I'd always been on the larger side. My parents would say my stomach was like a black hole since I would go back for seconds or even thirds. I hadn't known that I was actually planting the seeds of larger health problems down the line. That said, when I moved up to New York, the fact I was walking around everywhere caused me to begin losing at least some weight and be more fit.

In 2022, I went for my first check-up in a long time, and it was determined that I had prediabetes. In late 2023, I was prescribed semaglutide (specifically Wegovy), the blockbuster anti-obesity medication that costs a lot of money and is very popular in America for obvious reasons. (In fact, the U.S. obesity rate has stabilized and is likely to decline thanks to these medications). I had difficulty sourcing the medication in 2023, so I only started benefitting from it in 2024. In the meantime, I was prescribed metformin to help with blood sugar control.

I think the results really do speak for themselves:

I am less likely to act on food-related urges, being better able to control them.
I lost a lot of weight, going from 320 pounds at the end of 2023 to 250 pounds at the end of 2024, due to a combination of exercise, eating less food, and improving my diet. People in real life have noticed the significant weight loss.
Hemoglobin A1c, an average of blood sugar levels in the body, went from being in the prediabetes range (peaking at 5.9% in September 2023) to being in the non-diabetic range (5.6% in December 2024). My prediabetes is now well-managed by medication.
My blood pressure has also gone down.

I feel like a completely different person now, and I am hoping to build upon this progress in 2025. However, it has been hard to derive all the benefits due to all the other things that erupted in my personal life.

Finally, a callout to an old colleague...

Pablo Meier was also late in writing his 2023 in review post, but there is one memorable photo from it:

The offer to cross the Brooklyn Bridge (or any bridge of your choice) is still valid! 😀

🛠️ Fixing a broken Fedora 38 to 40 upgrade

2024-11-02T00:00:00Z

I recently upgraded my gaming computer from Fedora 38 to 40 and it didn't go as planned. I went through a bunch of pain to attempt to fix my partially-completed upgrade, but luckily I was able to complete the upgrade. Here, I took some notes of the process I used to rescue my system.

More regular programming will resume eventually. I have another Fedora-related bugbear to write about (related to Asahi Linux)... but that's for another day.

Disclaimer

This is a very specific set of steps that I took to fix my system. I can't guarantee that it will work for you, but it might be worth a shot if you're in a similar situation. Your mileage may vary. If you're in doubt, I would ask for help on the Fedora forums.

Waiting for an upgrade to complete

First, I booted the system and ran dnf upgrade to see if there were any packages that needed to be updated. There were (the machine hadn't been powered on since February), and I installed those and rebooted. So far, so good.

Now, we need to begin the actual upgrade process using the GNOME Software application, as recommended in the Fedora documentation. Everything was downloaded successfully and I was able to reboot the machine. The upgrade started but did not progress much beyond 52% completion. I let the upgrade sit there for 45 minutes. After about 10 minutes of waiting, I decided to power off the machine and reboot, since the upgrade appeared stuck with no output or progress.

Big mistake.

The reboot

After rebooting, the system was in a bad state. The system booted with a Fedora 38 kernel. I could not log into the desktop. When I switched to a shell with Ctrl+Alt+F2, the system claimed to be on Fedora Linux 40. I was able to log in as my user on the shell successfully, which did tell me that there was a chance to recover the system, or at the very least, backup my data before a reinstall.

So we have a system in a partially upgraded state: some chunk of the system is from Fedora 38, some other chunk is from Fedora 40. Let's try to fix it.

Fixing the partially upgraded system

My first step was to try the obvious thing, and run dnf distro-sync --releasever=40. This command should sync the system to the Fedora 40 release. Unfortunately, this command failed with a deceptively simple error:

Traceback (most recent call last):
  File "/usr/bin/dnf", line 57, in <module>
    from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'

Oh dear. Our package manager is broken. Are we screwed?

A Python scavenger hunt

I guessed that in the intervening period, the system Python version had been upgraded. (It turns out that Fedora 39 upgraded to Python 3.12, up from Python 3.11 in Fedora 38. But it was late into the night and looking over changelogs was the last thing I wanted to do.) I opened a Python shell and confirmed that the system Python version was indeed 3.12.

Now we just need to make sure the dnf module actually existed somewhere, otherwise we would need to manually re-install dnf via rpm before we made any progress. While there's a more elegant, Pythonic way of doing it, I examined /usr/lib/python3.12/site-packages and confirmed that the dnf module was absent, and then tried /usr/lib/python3.11/site-packages, and discovered that the module existed. I didn't know this at the time, but we now know that dnf was not yet upgraded at the time I rebooted the system. From there, I made my first leap of faith: I patched /usr/bin/dnf to invoke python3.11 instead of python.

Now we can run dnf again. This time, the command runs successfully. Our system is far from fixed, but we now have a working package manager, which we will need to fix the rest of the system.

Fooling the system into thinking it's Fedora 40

Now that dnf works, we can attempt to fix the system. That is where I found a really helpful forum post from a user experiencing a rather similar issue to me, except I had the dnf hoop to jump through.

First, we need to reinstall the fedora-release package using the "correct" version. This package contains the release information for the system, and since it's probably not correct, it's important for what we need to do in order to proceed. We can do this with the following command:

dnf --releasever=40 reinstall fedora-release-\*

Now, dnf should believe that we are "supposed" to be on Fedora 40. I'm still using --releasever=40 to ensure that we're installing the Fedora 40 packages, but it should not be necessary once you run this command.

Note that you will need internet access for this to work. Luckily, the system was connected via Ethernet, so I didn't have to worry about configuring that myself, presumably as NetworkManager was working correctly. (If you are on Wi-Fi, you might have to configure the network manually.)

The strategy

The system has duplicate packages (incompletely-removed Fedora 38 packages that were upgraded to Fedora 40, that dnf hasn't cleaned up), and packages that simply haven't been upgraded to Fedora 40. Our goal is to remove all the duplicate Fedora 38 packages, finish up the upgrade using dnf distro-sync, and then reboot to get to a hopefully-working Fedora 40 system.

Removing duplicate packages

First, let's remove all the duplicate packages. We can do this with the following command:

dnf remove --duplicates --releasever=40

In theory, this command should remove all the Fedora 38 packages and install the Fedora 40 equivalents. However, it didn't work for me, initially due to multi-arch conflicts. At this point, I wanted to prioritize safety over speed, so I opted to resolve each conflict by hand to ensure that I knew what I did, instead of specifying any dnf options to go "nuclear" (like permitting broken dependencies).

I was concerned about the large number of packages that I needed to download, but since dnf was installing Fedora 40 packages in exchange, it was probably safe to proceed.
After the download, dnf's transaction test failed with a large volume of conflicts caused by multi-arch packages (this was an x86_64 system with i686 packages installed). I elected to remove all the i686 packages from the system using dnf remove *.i686 and temporarily losing the ability to run 32-bit applications on the system. We can reinstall those later.
Next, I had conflicts related to the KDE Framework 5 to 6 upgrade. I removed all the kf5-* packages from the system in order to proceed, since I don't use KDE on this machine.
There was a conflict caused by the mozjs78 (this being the SpiderMonkey engine version bundled with Firefox 78) being missing in Fedora 38. I removed the mozjs78 package from the system. Nothing depended on it in my case.
Finally, there was a conflict caused by the cups-filters package being partially split with a -driverless package in Fedora 40. I removed the cups-filters package from the system. I don't print on this machine, so it was reasonably safe to remove.

After all these steps, I was able to run the dnf remove --duplicates --releasever=40 command successfully. This command removed all the duplicate Fedora 38 packages from the system, and installed the Fedora 40 equivalents.

The distro-sync and solving the mystery of my long upgrade

Now that we have removed all the duplicate packages, we can run the dnf distro-sync --releasever=40 command. This command will upgrade all the packages that have yet to be upgraded to Fedora 40. This command ran successfully for me, and I was able to reboot the system.

But it was taking quite a while for the upgrade to complete. In particular, it was stuck on running a script for smartmontools-selinux. After a few minutes, I open up another shell and run htop.

Turns out, 100% of a CPU core is being pegged, running restorecon. That means the system is relabeling the entire filesystem. Luckily, patience was key, and after another 30 minutes, the relabeling process completed and the rest of the upgrade proceeded smoothly. I did wonder whether or not this was why the upgrade had taken so long, since there was essentially no feedback for a lengthy period of time.

Once the command completed, I rebooted the system, hoping for a working system.

Almost there

The reboot was successful, however the dracut initramfs let me know that the system was still not fully upgraded, letting me know that Fedora 38 was end-of-life. However, I had a fully working system at this point. I still wanted to fix this issue, though.

I had to find the latest kernel-core version (ref) and reinstall it with dnf reinstall kernel-core. After this, I rebooted the system and saw that the new kernel was present and ready to be booted into.

My nightmare is over. I have a fully-upgraded Fedora 40 system.

So, what did I learn?

Don't interrupt a system upgrade. This is the most important lesson. Interrupting an upgrade guarantees a hosed system, which might be difficult to recover.
I was lucky that someone had a similar issue to mine, but the fact that I had to rely on a random internet forum post to fix my system was unnerving. However, I understood what the commands were doing and was willing to accept the risks.
Linux can be resilient. The system actually booted to multi-user and I had working internet access, making the system recoverable.
But it is also not anti-fragile. It is laughably easy to screw up a Linux system and put it in a broken or unbootable state. I was lucky that I was able to recover the system. If things had shaken out differently, I would had to resort to attempting to fix the system via a chroot (there was also a Ubuntu install on a separate partition on the system) or reinstalling the system from scratch via a live USB.

🎨 Sprucing up my website for 2024, part 2

2024-10-22T00:00:00Z

Well, it has only been... 2 months since I last did this exercise. But I've been thinking about this for a while, even as I was redoing my previous design. This time around, I decided to just go ahead and do it and see where the cards fell, and I was very pleased with the result I came up with.

So why now?

A lot of it has to do with a few factors:

I am moving away from technical writing, which has not been a major focus for this blog as of late (not withstanding that Ramp enjoys most of the fruits of that labor) in favor of creative writing.
The old design was simply not suitable for long-form story telling, which is a topic I have gotten progressively more interested in. (As in, I am looking to post novels on this website.)
The performance, readability, and accessibility of the old site could be much better.
The old design worked best being image-heavy, but I simply don't provide images for every post, nor do I want to. (I also don't want to just use DALL-E or look up stock images for the sake of having images.) In some cases, images are just a flat-out tall order. One post in the works is a novel, and I don't have the resources or desire to commission an artist to draw things for it, especially when the universe is still, to some degree, fluid.
Finally, I just felt tired of fighting my extensively hacked-up template, and wanted a fresh start.

So what's new?

I rebuilt the entire website's style with UnoCSS. This proved to be surprisingly less painful than expected. With only a few hours of effort, I got a sketch of a design I was happy with, and with a few more hours of work, I integrated it into my existing Eleventy setup.

Some of the highlights of the new design:

New font stack

I am now using Inter as the primary font for the website. It's a very readable font, and will provide consistency across platforms. I may be a Mac user, but that doesn't apply to everyone who reads this blog. It also seems to work well enough with my creative writing, but we will see if I need to change things around.

I am also embracing serif fonts, and I went with the Source Serif 4 font for this. It will prove to be a good font for my longest-form creative writing in particular, where I often have characters writing letters as part of the story. Previously, I used an italicized version of Source Sans Pro for this purpose, a font I was not attached to but was the only one I felt working with using the original template, as that's the font it used.

Finally, I swapped out Source Code Pro for JetBrains Mono. Turthfully, JetBrains Mono has been a font I have been using since it came out in 2019, and I have carried it over to every code editor I use. It looks great on the code snippets on this website.

New color scheme

Out with the messy mix of colors. We're going simple: hues of blacks and whites, with a corresponding dark/light theme to switch. This time, light mode wasn't an afterthought - I consciously considered it in the design.

I otherwise kept this simple. I was certainly inspired by brutalist design, and I am mostly happy with the choices I made. My biggest hope is that the site is more legible and readable, which is important as I take the blog more decisively in that direction.

Improved code syntax highlighting

I swapped out Eleventy's official syntax highlighting plugin for Shiki via its support for markdown-it. There were a few hiccups getting this working, mostly boiling down to the fact that Eleventy only supports "after the fact" asynchronous configuration, and it didn't seem that Eleventy was consistently applying the Shiki plugin to the markdown-it integration after initializing, but it seems to mostly work.

The most notable difference is that the theme will change in light mode to an appropriate light theme. Enjoy this little delight!

Get dropped off on the blog

...since you are not here to read a short blurb about me, that's what the about page is for (or my LinkedIn).

You get a list of eight posts to read, and you can go back in the archives as far back as you please. I am not going to waste your time.

A touch of animation

I added a few animations to the site, mostly to make the site feel more alive. There's a nice "slide-in" effect that I use mostly for post listing and the header to each post.

What's next?

You'll see me continue to publish technical content here, but I'm going to have an increased focus on creative writing as that is where my passion and interest is right now. I have quite a few stories in the works, and I am looking forward to sharing them with you.

There will probably be more changes coming to this site, but I think the "big bang" is finally over with. I do have some next steps ahead:

I need to queue up some more content, both creative and technical in nature. Keep an eye out...
Eleventy v3 came out.

Update, late on October 22, 2024

The Eleventy v3 upgrade was very trivial. The upgrade helper plugin indicated I had no issues to fix. On top of that, the ability to export an asynchronous configuration function was helpful, since Shiki now consistently initializes correctly (previously, it was a bit of a crapshoot).

I am pretty happy with my choice of static site generator, even if was a bit of a pain to configure at times. I had to migrate this site from Eleventy's first v0.x releases to v2 as I tried to do more complex things with it, and I'm pleased that the v2 to v3 upgrade was so trivial.

✍️ The Velocity Chronicles, Part 0

2024-08-31T00:00:00Z

I have spent a lot of time in the time in the Minecraft server scene. I've been highly visible, in both my good and bad moments. But perhaps my most substantial and consequential project in that scene, specifically the Velocity Minecraft server proxy, is the most interesting story to tell.

Velocity is notable today for being the proxy in front of a number of notable servers. Want to play on a network of completely user generated content servers? Powered by Velocity. Even the infamous 2b2t server uses Velocity, having moved from Waterfall (which Velocity replaces).

This is the first in a series of posts. Here, I will discuss the earliest days of Velocity, along with the first "trick" it used to improve performance over its contemporaries.

What exactly is a Minecraft server proxy?

Throughout this series, I am going to assume the reader:

has a technical background
has some understanding of what Minecraft is, but has never played it

Therefore I need to define some terminology. Let's start from the beginning:

A Minecraft server is the server-side component of Minecraft. It allows players to interact with a world hosted on a remote machine instead of on their own computer. Popular Minecraft server software options include the server provided by Mojang themselves ("vanilla"), Spigot, and Paper. Modding frameworks such as Fabric and NeoForge also permit custom modifications to the Minecraft server. We'll call this a server, except where we need to disambiguate.
A Minecraft server proxy sits between the Minecraft player's client and a group of servers (typically referred to as a "network"). The proxy accepts connections from players and routes them to a given Minecraft server on the backend. We'll call this just a proxy from now on, except where we need to disambiguate.

From this broad viewpoint, a Minecraft server proxy is no different from a typical reverse proxy, such as NGINX or HAProxy. Just as a reverse proxy manages and routes traffic to web servers, a Minecraft server proxy offers similar benefits: you don't have to expose each server to the Internet and it enables load balancing. As an example, take Hypixel, by and far the largest Minecraft server (at one point, it had over 200,000 concurrent users). A single Minecraft server simply cannot handle 200,000 players on at once, and so Hypixel uses a web of proxies and a lot of backend Minecraft servers to host their games on.

Here's an example topology of a Minecraft server network (with some thanks to Claude for drawing up the graph):

Minecraft server proxies, though, have a few unique features that make them different from your regular every day reverse proxy:

The proxy can force the client to move to a completely different Minecraft server. We'll dig into this later. But the end result: imagine a bastion server that was completely invisible: you use SSH to log into the bastion, and it directs you to a server that you want by default, then you initiate a command that can seamlessly log you into a different SSH server behind the proxy, without ever closing the original SSH session.
Most proxies support plugins, which extend the functionality of the proxy. You get the full power of the runtime and language the proxy uses - for Velocity, that means "any JVM language you want" - Kotlin being a popular choice in the community, but you could also use Scala, Clojure, or just about any language that can compile to the JVM, as long as it can interoperate enough with Java that Velocity can load it. Plugins can be as simple or as complex as you want: as simple as a plugin to manage server punishments like bans to plugins that add protocol translation support (everything from permitting newer versions of Minecraft to connect to servers running older versions of Minecraft, to even adding support for Minecraft: Bedrock Edition to Minecraft: Java Edition servers.)

The two giants in the Minecraft server proxy scene are BungeeCord and the subject of this series, Velocity.

First, the story of my first Minecraft server

The month is August 2013. I had just recently started high school. I learn of a Minecraft server network in need of developers (this is another fascinating topic I'll be sure to devote some time to eventually), and it looks like an interesting opportunity.

It was a more innocent time. Here, I'll even provide you some period-accurate videos of the three minigames that were on offer at the time.

First, here's MinerWare, a Minecraft clone of Mario Party:

Next is Ender, a Minecraft clone of the 2012 video game Slender:

Finally, there was Mineshooter, a clone of Call of Duty in Minecraft. Unfortunately, I had trouble finding a good contemporary video of the "old" Mineshooter (since there was a later rework), but I did find this video (voiceover in Russian, game text in English) which should give you a good idea of what the gameplay was like. There was also a separate server called GLDesert with a custom map, but this was detached from the rest of The Chunk in early 2014 in order to exclusively focus on minigames.

In those days, you had only one real option for having a proxy, and it was BungeeCord. (There was also LilyPad, but it required more involved set up and eventually fell by the wayside.) The Chunk used BungeeCord, and it had some special unique needs. In late September 2013, I began developing plugins for BungeeCord, moving away briefly from my previous focus on the minigames side of the house.

It didn't take me very long to make my first BungeeCord plugins: AdvancedBungeeAnnouncer and RedisBungee. The former made global announcements across the network more consistent, the latter synchronized the player count and player list across all the proxies. Since it was and to some degree still is a common practice to add load balancing to the proxies as well, Minecraft server network owners would run multiple proxies on different machines, much like you would scale up an application vertically by launching more than one instance of it on a different machine. The trouble was that because each proxy was independent, they would only report the player count of those players connected to a specific proxy. This was the problem that RedisBungee solved: it put all of this data into Redis and did its best to "fake" making a series of independent proxies look like one large proxy.

The Chunk wound up being a fairly successful network. It peaked at about 1,500 to 1,600 players online during early 2014, particularly the period between February 2014 and May 2014, but then went into decline. The network would eventually be absorbed into a different Minecraft network in early 2016 - I would do some development work for that network before leaving.

After that, I would wonder around quite a bit, taking on roles with different Minecraft servers and companies (some quite large, some quite small). I did work ranging from standard plugin development to custom modifications of Minecraft server software (mostly Paper). Perhaps most notable was the time I worked for Tebex (back when it was called Buycraft), where I rewrote their Minecraft server plugin and built their second ever integration with Minecraft: Bedrock Edition. I had a brief detour into making Minecraft server lists using Django, Celery, and PostgreSQL, and this is where I honed some of my web development skills (along with learning how to work with Python web frameworks). I enrolled in college. Eventually, I took on some work for Mineteria, which is where we'll pick up the story.

Well, Andrew, how involved were you in the "proxy scene"?

Quite a long time, I suppose.

What's often lost in the discussion, and has been a seemingly rich vein of confusion over the years, is the role I played in the Waterfall project. I founded this project in January 2016 as an effort to lower the bar to adding contributions to BungeeCord. It used the same patching system as Spigot and Paper, and included a number of patches to improve BungeeCord's behavior, add new features, and included some performance improvements. People came and went, but I was the only real constant until I handed the project over to the Paper team in September 2018 to focus more on Velocity.

The sad reality with Waterfall is that we were basically kind of stuck with the lowest common denominator. electronicboy would always relay the tale of trying to add improved scoreboard support to Waterfall (instead of a very raw API that BungeeCord exposed at the time by allowing plugins to just send packets if they wanted, a practice very quickly banished from the Velocity plugin API on day one) and dealing with broken plugins as a result, effectively killing any desire to make Waterfall a better BungeeCord in any way that mattered. In other words, a Minecraft manifestation of Hyrum's law:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

After all of that, there was increased conviction to build an alternative proxy instead. This evolved into Velocity.

You said you'd discuss another Minecraft network, too?

In May 2018, after doing some initial web development work for Mineteria, I was enlisted to work on the actual Minecraft server. The server had a go-live date in July 2018, so I was thrown onto the project fairly late - development had previously began in early 2017. Like The Chunk and many Minecraft networks before it, it also used BungeeCord.

The server was pretty unique as it used Kubernetes as its native control plane. It was amongst the earliest examples of this, and so we'd run into a lot of rough edges. Kubernetes on bare metal was not as mature back then, so our production set up wound up being run on Google Kubernetes Engine.

Those of you in the audience familiar with Minecraft servers would immediately know that something is wrong here. Other people in the audience may be puzzled: isn't Google Cloud a pretty standard commodity cloud provider? They offer all the services you'd expect, just like any other cloud provider.

Swiping left on the cloud

Proxies have two primary characteristics:

They need to do a lot of I/O-intensive work, which means lots of system calls. This was all after the impact of Spectre/Meltdown, which meant that everyone likely took massive performance penalties, which were further magnified by running inside of a virtual machine.
Part of the proxy's job is compressing packets destined for players, and this process is very resource-intensive. Worse, we cannot just serve pre-compressed assets like a web server might do - we have to compress packets on the fly.

This became apparent when we launched the server on July 1st. The servers did collapse under their weight and we had to add additional resources to keep things stable. We realized that for the proxy, we would need to run absurdly large instances (or scale out the proxy), in essence throwing a lot of money at the problem. That's what we did, for a while at least. But we knew that the persistently high CPU usage was not sustainable (we were seeing well over one or two cores used by a small number of players!) and so we sought a third option.

It was also in July 2018 when I was on the PaperMC Discord complaining about BungeeCord, primarily around its development process (a cathedral) and its almost fanatical pursuit of backwards compatibility. At that time, I was experimenting with a fork of BungeeCord I had created, LibertyCord, seeking to make more radical changes. At the time, the owners of Mineteria were skeptical of Velocity, but I pressed on anyway. With encouragement from electronicboy, kashike, and even Aikar, I began work on Velocity. Progress did eventually get to the point where Mineteria made the move to Velocity starting in November 2018, and I could actually start "eating my own dogfood".

Velocity is born

The first commit to Velocity was on July 24th, 2018. This initial version of Velocity didn't have much functionality. In fact, it was just barely capable of responding to server list ping packets from the Minecraft client. Still, it was pointing in the right direction, and we just needed to get Velocity relaying packets across the network.

By July 27th, I had something that you could call a "proof of concept": a player could connect to a Velocity server, be connected successfully to the proxy, which would open a connection to a specific server (localhost:25565), and be able to switch to a different server (localhost:25566). Furthermore, we had implemented the first "trick" Velocity used to gain an edge over BungeeCord and LilyPad. But before we do, we'll need to discuss a few more concepts.

Who are you calling, me?

A typical video game has entities contained within the game world. An entity is simply a object that exists in a given world.

In a single-player game, we might be able to store direct memory references to other entities in the world. Suppose we were making a very simple tower defense game where we have turrets and enemies and we need to tick them (update them) on a periodic basis. We might refer to the entities in the world in code as follows:

  private List<Turret> _turrets;
  private List<Enemy> _enemies;

But what if we are playing over the network? It would make no sense (and be a security risk) to pass around memory pointers. Instead, what we can do is assign an ID to each entity in the world, and refer to the entity by this ID alone when referring to a specific entity over the network.

So far, so good, right?

Here's the catch: we can move the player to a different server, but we can't tell the Minecraft server to use a specific entity ID. Furthermore, if we just send packets referring to the connected player with a different entity ID than the one the client believes it has, then all sorts of interesting issues can occur on both the client and server sides. To add further complications, the tricks that proxies do are technically not supported by Mojang, but are only tolerated because proxies are very popular in the server community!

That said, we have two options:

The proxy can rewrite every packet that contains an entity ID. Should it find that it needs to rewrite a packet, it will copy all the fields in the packet over. If it finds the server-side entity ID for the player in a packet bound for the player, it will rewrite it to include the entity ID that the client believes it uses. It also does this in the reverse direction for packets originating from the player towards the server they are connected to.
Or, you can try to make the client use a different entity ID, and dispense with the need to rewrite packets entirely.

BungeeCord and LilyPad opted to rewrite packets. Velocity chose the path of telling the client to use a different entity ID.

So, how do we fool the client?

First, let's understand what Velocity is sending the client. First, here is how Velocity told the client to change entity IDs and load the new world:

    public void handleBackendJoinGame(JoinGame joinGame) {
        if (!spawned) {
            spawned = true;
            currentDimension = joinGame.getDimension();
            player.getConnection().write(joinGame);
        } else {
            player.getConnection().write(joinGame);
            int tempDim = joinGame.getDimension() == 0 ? -1 : 0;
            player.getConnection().write(new Respawn(tempDim, joinGame.getDifficulty(), joinGame.getGamemode(), joinGame.getLevelType()));
            player.getConnection().write(new Respawn(joinGame.getDimension(), joinGame.getDifficulty(), joinGame.getGamemode(), joinGame.getLevelType()));
            currentDimension = joinGame.getDimension();
        }
    }

It should first be noted that this continues to be the base for Velocity's current implementation, which is more complex (notably, we can now omit one of the Respawn packets starting in Minecraft 1.16), but the fundamental logic remains the same.

What does this code do? First, we need to explain a few Minecraft packets and how the client responds to them:

The JoinGame packet is sent by the server when it indicates that the server has spawned the player into the server's world. It is sent very early after the player's connection to the server enters the PLAY state (in other words, when the player has just logged into the server).
The Respawn packet it sent by the server when a player is moved to another dimension (i.e. going from the Overworld to the Nether), or if the player is respawning after dying.

Now we can explain what Velocity is doing:

If this is the first server the player has connected to during this session, indicate as such and send the packet plain. There's nothing for us to do.
Otherwise:
- We send the original, unmodified JoinGame packet from the new server to the client.
- The client drops its current state of the world, and crucially, sets its entity ID to the one we sent.
- We send a Respawn packet, set to a different dimension. At the time, the Minecraft client did not behave correctly when we respawn the client into the current dimension immediately after sending JoinGame.
- We send another Respawn packet to bring the player into the correct dimension.
The server switch is done, and packets can stop flowing from the old server (if any) and start flowing from the new server.

This solves a couple of issues:

It makes the codebase considerably simpler. Instead of maintaining complex, non-trivial packet rewriting logic, we could focus on other important attributes such as extensibility and performance of the overall networking pipeline. It also means that the code is less buggy, since we know that the client can update the entity ID it uses, so we don't need to worry about missing a packet we were supposed to rewrite.
It made updating Velocity to work with newer versions of Minecraft much easier. Some versions of Minecraft were supported simply by telling Velocity that it needs to recognize a new protocol version number.
It worked with every version of Minecraft that Velocity supports, from Minecraft 1.7.2 to 1.21 - nearly 11 years of compatibility.
It causes less problems with mods. Entity ID rewriters don't know how to rewrite mod packets containing entity IDs, so these mods would simply break on a proxy that uses them.
Perhaps most importantly, since we don't need to examine these packets, the proxy does essentially no processing on the packets except for uncompressing the packet from the server if needed, determining that no action is necessary, and forwarding it onto the client (compressing and encrypting the packet as needed).

Of course, this wouldn't be the only trick Velocity employed in the service of performance, but it was amongst the first.

That's all for this installment

With this post, I hope I've began to lay down the foundations for how Velocity came to be. I've introduced you to some key concepts along with three of the formative moments in my Minecraft journey. I've also given you an introduction to the earliest days of Velocity.

In the next installment, we'll discuss some more of the early days of Velocity:

The plugin API.
The lead up to, first the deployment of Velocity on Mineteria, and then Velocity 1.0.0.

Stay tuned for more. There's a lot of stories for me to tell, as these were some of my formative years as a software engineer. (Trust me, going from Minecraft modding to fintech is a very unexpected turn of events!)

🎨 Sprucing up my website for 2024

2024-08-24T00:00:00Z

Well, it sure has been a while since I last updated this website back in the far away time of... October 2022. Naturally this means I now need to go in and actually give it a refresh, especially as I am trying to make this website a little bit more than a blog that gets an article roughly semi-annually, and start exploring doing more creative writing along with deeper technical reads.

First steps first

I did give brief contemplation to a complete overhaul of the website. I would continue to use Eleventy as it's a reasonably fast site generator, and I've already circulated through WordPress, Ghost, Jekyll, Hugo, and now Eleventy in the past and didn't want to go explore new site generators (or, gasp, go write my own).

In fact, my original plan was to go rework this entire website with the hot new stuff, Tailwind, and retire my very boring and excessively modified HTML5 UP template that was a wonderful base to work with in 2014 but downright not even worthy of a yawn in 2024. But I didn't have the luxury of time to wrap myself around Tailwind, which is a very much a fundamentally different way to view CSS. I do see the value in it, though, and I will probably adopt it for future projects (if I'm able to get that far, that is).

So I decided just to press on with updating my existing theme. To be honest, I was simply not happy with the customizations I made. There was some stuff that was just outright broken or difficult to see, and of course, the website very conspiciously lacked support for dark mode.

Tweaking the theme

The first thing you'll notice is that there's a lot less spacing all over the place, and the header is much slimmer than it once was. I've decided to bite the bullet and adopt a widescreen view by default to improve readability and allow me to write more information-dense pages.

The second thing you'll notice: this website now supports dark mode. There is a nice transition from the light theme to the dark theme, and there's a button to change the theme as you please. When you do, the background will alternate: in light mode, you get the classic shot of the Brooklyn Bridge that has been there since late 2022, and in dark mode, you get a black-and-white version of the Miami skyline from Wynwood, from a work trip I took earlier this year:

The third thing you'll notice is that a lot of little tiny nits (especially with regard to responsiveness) were fixed. Finally, you'll notice that the post headers now look a lot more "normal" rather than being a giant <h1> with the date and tags underneath.

Other than that, I didn't fundamentally alter the base HTML5 UP template. It still looks largely the same, with the exception of removing a bunch of the 2014-era JavaScript that the template came bundled with (it even came with jQuery, and it is mildly amusing that jQuery is still under active development). In fact, the JavaScript I added for the theme switcher is the only JavaScript I've since added, and this site should work just fine even with JavaScript disabled (but then you're getting the light theme). I didn't even change the font: it's still the venerable Source Sans Pro (although I did look at changing it, every alternative I looked at just had a different vibe I didn't like.)

So, what's next for you?

In the short term, I hope to have some more content here and make this blog more active than it is right now. I've got a few things kicking around: toy projects using whatever I'm interested in learning, some creative writing I've been working on intermittently (mostly short stories), potentially some stories from Ramp that aren't quite worthy of a post on our engineering blog, and some other assorted topics as they present themselves to me.

Until then, here is a 2024 re-imagination of Bertolt Brecht's 1935 poem "Questions From a Worker Who Reads", and a snippet from a story I am currently working on.

[...] Earth was a far, far cry from what it had once been. Once the cradle of human civilization, it now sat abandoned, a museum in all but name. Officially, it was called “neutral ground”, but everyone who had an interest in coming here was here to study the history of the planet or its former inhabitants. Oh, what a far cry from Wista, home to thriving metropolises and the National Research University of the Arctarian System, where she was a history professor specializing in 20th and 21st-century computing technology. Her maps indicated that she was in the vicinity of what was once Dulles International Airport, one of the airports serving what was the United States’s capital city of Washington, D.C. She immediately sized up the location. The layout of the buildings and the sprawling and curvy road network were immediately recognizable as typical of the suburban development of the United States in that time frame. In this part of the country, too, humanity built up large numbers of what were called “data centers” here. Dr. Mahone looked at her watch. The compute power it had probably dwarfed all the compute power this sad building had once hosted. How much has changed in 28 centuries! [...]

📚 A 2023 dispatch

2024-08-12T00:00:00Z

I would've written about 2023 some time ago, but quite a few developments happened in my life... and so, I held off for a bit. But now I am back to tell the story of 2023. It was probably one of the most stressful years in my life.

Dis is one half.

January 1st, 2023 comes and my first order of business was to move apartments! My old building was not particularly well-maintained, looked ugly on the outside, had no air conditioning, had lead service pipes, and for the last month I lived there, had no internet, forcing me to either go out somewhere where there was Wi-Fi or tether to my cellphone. This was not great considering I had signed up to be on-call for the holiday. A wonderful first apartment for you to have in the city, huh?

So starting in January, I went out and started apartment hunting. I narrowed it down to four potential options, applied for all of them, and was accepted for all of them as well. I ultimately chose an apartment located in a recently-built mixed-use apartment complex that would place me one stop closer to Manhattan.

Then I had to go pack my things. Not fun. At least in New York City, everyone moves so often that the moving market is extremely competitive. At least it takes the sting out of the high rent.

Some time later, I move into my new place. It's nice. I have my own fiber connection with all the works, even IPv6. There's AC that works. The building is maintained pretty well. The super is generally on top of things. I could stay here.

I don't have much time to settle down. Offsite in Miami with the Ramp Bill Pay team. This is where I had a severe burnout episode crop up. I survive the offsite and immediately take a week off work, mostly to visit museums in New York. I enjoyed the experience. But when I returned, I had to decide whether or not I was going to leave the company I'd joined more than a year ago, keep my current position, or try to make the situation on my current team work for me.

I tried the second option for a bit. It was comforting, I had no real qualms with the work, and I had reason to believe I could become an effective tech lead within payments in due time.

Silicon Valley Bank collapsed. I had to go jump into a war room meeting on a Friday to attempt to keep the spread of potential issues confined. In that one conference room inside a WeWork, I wrote scripts that would cancel payments to SVB and delay internal settlement of payments already initiated from SVB. We did not even know what the federal response would be. All of this was, thankfully, not necessary. The federal government de-facto nationalized Silicon Valley Bank and Signature Bank and took further actions to forestall a larger crisis.

But for a time, it was rejuvenating, so I worked on launching support for Ramp's freshly-acquired money transmission licenses within our payments platform and making other improvements. I got promoted from merely "software engineer" to senior software engineer! For a while, it goes great. I'm still having the conversations about moving teams, though, but it's at a slightly lower priority while I delivered for my existing team.

I talked with Patrick Anderson, head of Ramp's core engineering team at the time, a few times as I evaluated my options. He saw some opportunities for me to either move to our product security engineering function or move into spend management engineering (the meat and potatoes of Ramp). I rejected product security since while I had aptitude for it, I did not feel like the work would be sufficiently challenging for me. After further discussions with relevant tech leads, I was leaning towards the closest thing I had for my team's equivalent on the spend management side of the house: spend platform.

Then First Republic Bank collapsed. Then I was dropped several last-minute requirements for our money transmission launch. At that point, I pull the trigger and decide to leave teams. Over the following 6 months, effectively all of Ramp's original Bill Pay and Payments team had dissolved, and was replaced by an entirely new team.

Thus we get to June 2023, and day 1 on my new team.

My Spend Platform era

It begins innocently enough. I'm introduced to my teammates and a new manager. I had already established a rapport with the tech lead, who was already aware that I could bring valuable skills to the table. With my entry, the team counted five backend engineers, zero designers, and one product manager.

If I left it at that, that would've been a generously happy ending to a stressful first year of the year, right?

I wish it got better, but it doesn't.

Within the next three months, a lot would happen:

Engineers were forced to pause most of their work and focus on tech debt for a single sprint.
Our tech lead goes on paternity leave.
One of teammates would be fired.
My manager would be shown the door, and our tech lead, who doesn't like managerial work, is forced to take on managerial work.
My mother and my grandmother's health takes a turn for the worse. (Both would die in early 2024.)

The end of the year proved even more stressful. I had to pay for a last-minute flight to Atlanta and hotel for Thanksgiving week since my mother was in the hospital. I'd also volunteered to be on-call for the end-of-year holidays as well.

I would also be able to focus on a couple of important projects, but ultimately I would say that the time I spent that year was a bit of a wash. I didn't have too much trouble onboarding, I just had a bunch of unfortunate circumstances arise that made the year much less successful.

To be honest, I'm lucky I made it out of the other side without wanting to quit.

That's all for now

The story of 2024 is yet to be written, but it has been proving to be a mix of both a dumpster fire and one of the better years I've had thus far. I'll see you in 2025...

🕯️ Going back home, a tapestry of time

2024-02-21T00:00:00Z

A sojourn looms, heavy and stark
Homeward bound, today I shall embark,
Life does not pause for the mourner under this dome
But all I seek is home.

Security cleared, what a tiresome charade
Scanners and conveyors, a pat-down, what a bother.
With each climb of the escalator,
Towards the gate I go, burdened by time.

Chaos reigns, the cacophony unfolds
Amidst the crowd, solace a rare sight.
At least my coffee is quite bold,
My only champion is the sun’s light.

A whispered prayer, for safety’s embrace,
To one funeral I head, another I cannot face.
In the sky, between lands, my fears are sown,
Stuck in this tube, my thoughts are my own.

The ground greets me, a bittersweet peace,
Nearly home, but the ache won’t cease.
What cost is home, what price for this flight?
I seek solace, but it is far out of sight.

Past this bridge, its corridors long
Towards where my memories belong.
My life and my home torn,
Mourning also the aggrieved and forlorn.

The roads whisper tales of what was once a place,
Before the relentless march of what counted as progress,
Did they mourn, as I do, for what was erased?
Communities displaced, lives regressed.

A man, his power, and his dream,
Who oversaw the landscapes being reshaped.
Did he weep as the night beamed?
These places, we could never have replaced.

In a town forgotten by time, I arrive,
The locomotives whistled and spewed their exhaust,
I go to the chapel, at my own high cost,
For we seek solace in the stories that survive.

A confluence of pasts, a tapestry of ties,
A clear sky, a cold day, but our shared sorrow still lies.
In this gathering, our grief finds a voice,
In remembering, together, we somewhat rejoice.

Journey’s end, and I can only weep,
Closure elusive, in the town time forgot,
Never mind in the city that doesn’t sleep.
Life does not pause for the mourner in the gates.

Context

This poem is a reflection on two deaths in my family: my grandmother and my mother. In both cases, the funerals took place in rural railroad towns in the South, specifically those on the route between Atlanta and Birmingham.

I flew out of LaGuardia Airport in New York City and transited both Hartsfield-Jackson Atlanta International Airport both times.

I miss you, Bonnie and Natalie. At least you join Rex and Karl. Love from the city that never sleeps, where you are all proud that I've made something of myself.

🐦🔥 Misadventures with Phoenix Framework

2023-12-28T00:00:00Z

Back in 2020, before the pandemic really hit, I started working on two projects built on top of Cloudflare Workers. One of those projects was the Mineteria Store, which jammed together server-side React running on top of an environment that server-side React didn't exactly run on at the time mixed with traditional client-side rendering, and a service to render Minecraft avatars called Crafthead, now maintained by Nodecraft. (Funny enough, the Mineteria Store is, to my knowledge, the only deployment of Elixir that was remotely close to the Minecraft server scene. What a time that was.)

But I have a lot of holiday downtime and I wasn't exactly doing much of anything. Plus, at Ramp, I am now part of the same team that works on our authorizer service (something legendary in its own right: written by Pablo Meier, who may not be with Ramp any more, but it is the subject of this classic blog post, and the authorizer service continues to faithfully handle millions of authorizations per day with the power of Elixir).

So let's do the natural thing and rewrite Crafthead in Elixir...

...but not quite!

Crafthead is actually divided into two components:

The actual application, written in fairly standard TypeScript aside from some extensions to the runtime done by Cloudflare. It is basically a Service Worker running in a non-browser environment.
A Rust library responsible for rendering images. There is a shim API that handles the JS-to-Rust translation, and this is ultimately compiled into WebAssembly.

The main reason for this was simple: I knew more TypeScript/JavaScript than I did Rust, and I wasn't excited about the prospect of writing a Worker entirely in Rust. (Also worth noting that you couldn't have even written a Worker entirely in a language that compiled to WebAssembly until Cloudflare added support for Liftoff tiering, since on a cold boot, the WebAssembly would have to go through TurboFan compilation, a compiler not known for being fast.)

Along the way, Crafthead makes use of Cloudflare Workers KV as its primary datastore. This is essentially a Memcached-like API with some extra bells and whistles we won't care about. We can substitute it for Memcached, Redis, a relational database, or an ETS table.

So what's our strategy?

It's not really hard to deduce a strategy from here:

The actual application, written in Elixir, and built on top of the Phoenix Framework. Phoenix provides essentially everything including the kitchen sink.
We will need a cache: we'll use Nebulex to handle this.
Req reminds me a lot of python-requests and is easy to add (it's built on top of Finch and Mint), so we will use this to call the Mojang API.
A Rust library responsible for rendering images, using Rustler to handle the Erlang-to-Rust translation, and compiled as a NIF.

Fun! We have not even written a single line of code and we are already playing with fire by introducing NIFs into the mix. We will have to take care of that later, but first we need to get a Phoenix project going:

mix phx.new crafthead --no-html --no-assets --no-ecto --no-mailer

This creates the crafthead project with an extremely barebones configuration. The only HTML files we'll serve are static assets anyway, and Ecto will not be necessary at the moment, since caching can be done locally using an ETS table.

Now we can launch the project and start writing some code! First, let's install Nebulex:

defp deps do
  [
    {:nebulex, "~> 2.5"},
    {:decorator, "~> 1.4"},
    # ...
  ]
end

Run mix deps.get, mix nbx.gen.cache -c Crafthead.Cache, and then add Crafthead.Cache as instructed, and then we have our cache!

Writing a Mojang API client

Right! Let's go ahead and build a minimal Mojang API client. The API is pretty well-documented but we only need a handful of things:

Map a username to a UUID
Obtain the profile and skin of a player by their UUID

That's where Req comes into the picture. Let's install it now:

defp deps do
  [
    {:req, "~> 0.4.0"},
    # ...
  ]
end

Run mix deps.get and now we have a richly-featured HTTP client at our fingertips. Now we can write our Mojang API client:

defmodule Crafthead.Clients.Mojang do
  @username_to_uuid_mapping_url "https://api.mojang.com/users/profiles/minecraft/"
  @uuid_to_profile_url "https://sessionserver.mojang.com/session/minecraft/profile/"

  def username_to_uuid(username) do
    url = @username_to_uuid_mapping_url <> username <> "?unsigned=false"
    with {:ok, resp} <- Req.get(url) do
      case resp.status do
        200 -> {:ok, resp.body["id"]}
        404 -> {:error, :not_found}
        429 -> {:error, :too_many_requests}
        500 -> {:error, :internal_server_error}
        503 -> {:error, :service_unavailable}
        _ -> {:error, :unknown}
      end
    end
  end

  def uuid_to_profile(uuid) do
    url = @uuid_to_profile_url <> uuid
    with {:ok, resp} <- Req.get(url) do
      case resp.status do
        200 -> {:ok, resp.body}
        204 -> {:error, :not_found}
        404 -> {:error, :not_found}
        429 -> {:error, :too_many_requests}
        500 -> {:error, :internal_server_error}
        503 -> {:error, :service_unavailable}
        _ -> {:error, :unknown}
      end
    end
  end
end

There is a little bit more to it than this, because the actual skin is in the properties object and requires some
unwrapping, so let's build that out:

defmodule Crafthead.Profile.Minecraft do
  @doc """
  Implements a raw representation of a Minecraft profile, from the Mojang API. Includes a convenience function to
  obtain the skin information for a player.
  """
  @permitted_texture_types %{"SKIN" => :skin, "CAPE" => :cape}

  defstruct [:id, :name, :properties]

  def get_skin_info(profile) do
    profile
    |> find_texture_property()
    |> decode_texture_property!()
    |> get_texture_urls()
  end

  defp find_texture_property(profile) do
    Enum.find(profile.properties, fn property ->
      property["name"] == "textures"
    end)
  end

  defp decode_texture_property!(property) when is_map(property) do
    property["value"]
    |> Base.decode64!()
    |> Jason.decode!()
  end

  defp decode_texture_property!(property) when is_nil(property), do: nil

  defp get_texture_urls(textures_property) when is_map(textures_property) do
    @permitted_texture_types
    |> Enum.filter(fn {texture_type, _} ->
      Map.has_key?(textures_property["textures"], texture_type)
    end)
    |> Enum.map(fn {texture_type, texture_type_atom} ->
      {texture_type_atom, textures_property["textures"][texture_type]["url"]}
    end)
    |> Map.new()
  end

  defp get_texture_urls(textures_property) when is_nil(textures_property), do: nil
end

We elide a bunch of error handling here since all profiles come from the same trusted source, namely the Mojang servers.

While we are at it, it's also a good idea to write a unit test for this class, since it does something non-trivial:

defmodule Crafthead.Profile.MinecraftTest do
  use ExUnit.Case

  describe "get_skin_info/1" do
    test "returns skin information" do
      profile = %Crafthead.Profile.Minecraft{
        :id => "652a2bc4e8cd405db7b698156ee2dc09",
        :name => "tuxed",
        :properties => [
          %{
            "name" => "textures",
            "value" => "ewogICJ0aW1lc3RhbXAiIDogMTcwMzgyNzIzMTg0OCwKICAicHJvZmlsZUlkIiA6ICI2NTJhMmJjNGU4Y2Q0MDVkYjdiNjk4MTU2ZWUyZGMwOSIsCiAgInByb2ZpbGVOYW1lIiA6ICJ0dXhlZCIsCiAgInRleHR1cmVzIiA6IHsKICAgICJTS0lOIiA6IHsKICAgICAgInVybCIgOiAiaHR0cDovL3RleHR1cmVzLm1pbmVjcmFmdC5uZXQvdGV4dHVyZS9kOWM1MjcwMzUwOWM3MDcwNDczMGIyZDg4MmIzNjdjMTEyMWM2NGYzZDZmNmE5ZDEyNmU0N2IxZmU2NTY4MGI5IgogICAgfQogIH0KfQ=="
          }
        ]
      }

      expected_result = %{
        skin: "http://textures.minecraft.net/texture/d9c52703509c70704730b2d882b367c1121c64f3d6f6a9d126e47b1fe65680b9"
      }

      assert Crafthead.Profile.Minecraft.get_skin_info(profile) == expected_result
    end

    test "returns cape information" do
      profile = %Crafthead.Profile.Minecraft{
        :id => "1ccef50bf6ae4542b1a9d434384a5b25",
        :name => "Jeff",
        :properties => [
          %{
            "name" => "textures",
            "value" => "ewogICJ0aW1lc3RhbXAiIDogMTcwMzgyOTE5NjM0NCwKICAicHJvZmlsZUlkIiA6ICIxY2NlZjUwYmY2YWU0NTQyYjFhOWQ0MzQzODRhNWIyNSIsCiAgInByb2ZpbGVOYW1lIiA6ICJKZWZmIiwKICAidGV4dHVyZXMiIDogewogICAgIlNLSU4iIDogewogICAgICAidXJsIiA6ICJodHRwOi8vdGV4dHVyZXMubWluZWNyYWZ0Lm5ldC90ZXh0dXJlL2Y3ZjdhNWFmYmY5NDZkYWUxOWZhMzhjMjIxZTMxMDM0NTE5YjE5OWViMWZkZTkwZTI0OTUxNzJlOWQ3ZDZhIgogICAgfSwKICAgICJDQVBFIiA6IHsKICAgICAgInVybCIgOiAiaHR0cDovL3RleHR1cmVzLm1pbmVjcmFmdC5uZXQvdGV4dHVyZS85NTNjYWM4Yjc3OWZlNDEzODNlNjc1ZWUyYjg2MDcxYTcxNjU4ZjIxODBmNTZmYmNlOGFhMzE1ZWE3MGUyZWQ2IgogICAgfQogIH0KfQ=="
          }
        ]
      }

      expected_result = %{
        skin: "http://textures.minecraft.net/texture/f7f7a5afbf946dae19fa38c221e31034519b199eb1fde90e2495172e9d7d6a",
        cape: "http://textures.minecraft.net/texture/953cac8b779fe41383e675ee2b86071a71658f2180f56fbce8aa315ea70e2ed6",
      }

      assert Crafthead.Profile.Minecraft.get_skin_info(profile) == expected_result
    end

  end
end

Building our first endpoints

At this point, we are able to fetch a skin from the Mojang API, which is awesome. We can now create our first endpoints. Specifically, we will implement the /profile/<username|UUID> and /skin/<username|UUID> endpoints.

First, we need to be able to tell if we are working with a username or a UUID. It will also normalize the entity provided (such as lowercasing the username or removing dashes in UUIDs) so that equivalent entities will map to the same cached value, which will be important later on. The function is simple, so a doctest can suffice to document and serve as the entire test suite:

defmodule Crafthead.Util.Request do
  @mojang_uuid_regex ~r/^[0-9a-f]{32}$/
  @regular_uuid_regex ~r/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/

  @doc ~S"""
  Given the Minecraft profile `entity`, determine if it refers to a UUID or a username.

  ## Examples

    iex> Crafthead.Util.Request.what_entity("1ccef50bf6ae4542b1a9d434384a5b25")
    {:uuid, "1ccef50bf6ae4542b1a9d434384a5b25"}
    iex> Crafthead.Util.Request.what_entity("4525ca19-5825-4774-bb90-a004de1460c3")
    {:uuid, "4525ca1958254774bb90a004de1460c3"}
    iex> Crafthead.Util.Request.what_entity("tuxed")
    {:username, "tuxed"}
    iex> Crafthead.Util.Request.what_entity("LadyAgnes")
    {:username, "ladyagnes"}

  """
  def what_entity(entity) do
    cond do
      Regex.match?(@mojang_uuid_regex, entity) -> {:uuid, entity}
      Regex.match?(@regular_uuid_regex, entity) -> {:uuid, String.replace(entity, "-", "")}
      true -> {:username, String.downcase(entity)}
    end
  end
end

Now we can implement the first controller to fetch profiles:

defmodule CraftheadWeb.ProfileController do
  alias Crafthead.Clients.Mojang
  alias Crafthead.Util.Request

  use CraftheadWeb, :controller

  def show(conn, %{"entity" => entity}) do
    result = entity
    |> Request.what_entity()
    |> get_profile()

    case result do
      {:ok, profile} ->
        conn
        |> put_status(:ok)
        |> put_resp_header("cache-control", "public, max-age=86400")
        |> json(profile)
      {:error, :not_found} ->
        conn
        |> put_status(:not_found)
        |> put_view(json: CraftheadWeb.ErrorJSON)
        |> render(:"404")
      {:error, _} ->
        conn
        |> put_status(:internal_server_error)
        |> put_view(json: CraftheadWeb.ErrorJSON)
        |> render(:"500")
    end
  end

  defp get_profile({:uuid, entity}) do
    Mojang.uuid_to_profile(entity)
  end

  defp get_profile({:username, entity}) do
    with {:ok, uuid} <- Mojang.username_to_uuid(entity) do
      get_profile({:uuid, uuid})
    end
  end
end

Add this to the router:

scope "/", CraftheadWeb do
  pipe_through :api
  get "/profile/:entity", ProfileController, :show
  get "/ping", PingController, :show
end

Now if you go to http://localhost:4000/profile/tuxed, you'll see my Minecraft profile. Our first endpoint is implemented! We should definitely stop and clean this up a bit, though - we're going to carry a lot of duplicated code around if we don't define a fallback controller. I will elide that, but assume it exists for any other controllers we implement.

Next up, we're going to implement endpoints for fetching skins. This is pretty easy:

defmodule CraftheadWeb.SkinController do
  alias Crafthead.Clients.Mojang
  alias Crafthead.Profile.Minecraft
  alias Crafthead.Util.Request

  use CraftheadWeb, :controller

  action_fallback CraftheadWeb.FallbackControlle

  def show(conn, %{"entity" => raw_entity}) do
    entity = raw_entity |> Request.what_entity()

    with {:ok, profile} <- get_profile(entity),
        texture_info <- Minecraft.get_skin_info(profile),
         {:ok, skin} <- fetch_skin(texture_info) do
      conn
      |> put_resp_header("cache-control", "public, max-age=86400")
      |> put_resp_header("content-type", "image/png")
      |> send_resp(200, skin)
    end
  end

  defp get_profile({:uuid, entity}) do
    Mojang.uuid_to_profile(entity)
  end

  defp get_profile({:username, entity}) do
    with {:ok, uuid} <- Mojang.username_to_uuid(entity) do
      get_profile({:uuid, uuid})
    end
  end

  defp fetch_skin(%{:skin => skin_url}) do
    with {:ok, resp} <- Req.get(skin_url) do
      case resp.status do
        200 -> {:ok, resp.body}
        404 -> {:error, :not_found}
        429 -> {:error, :too_many_requests}
        500 -> {:error, :internal_server_error}
        503 -> {:error, :service_unavailable}
        _ -> {:error, :unknown}
      end
    end
  end
end

This concludes a very basic implementation of an skin and profile-fetching service for Minecraft written in Elixir, but we are far from done yet. We need to add in a few more creature comforts.

Caching and rate-limiting

Since the Mojang API has rate limits, we can't simply request data from it every time we need it, so we are expected to cache the data. Luckily, that's why we pulled in Nebulex earlier. Let's modify our Mojang API client:

defmodule Crafthead.Clients.Mojang do
  alias Crafthead.Cache

  use Nebulex.Caching

  @username_ttl :timer.hours(1)
  @profile_ttl :timer.hours(24)

  # ...same code...

  @decorate cacheable(cache: Cache, key: {:username_to_uuid, username}, opts: [ttl: @username_ttl])
  def username_to_uuid(username) do
    # ...same code...
  end

  @decorate cacheable(cache: Cache, key: {:uuid_to_profile, uuid}, opts: [ttl: @profile_ttl])
  def uuid_to_profile(uuid) do
    # ...same code...
  end
end

Now we have a cache around the Mojang API - one hour for usernames, and one day for profiles. We can also easily ensure skin textures are also cached so we don't need to fetch them from Mojang every time too.

Time to bring in the Rustler!

You have sat here for a while, now it's time for the fun part: adding a NIF to this project. Let's install Rustler first:

defp deps do
  [
    {:rustler, "~> 0.30.0", runtime: false}
    # ...
  ]
end

Run mix deps.get and then create the Rustler module:

# mix rustler.new     
This is the name of the Elixir module the NIF module will be registered to.
Module name > Crafthead.Renderer
This is the name used for the generated Rust crate. The default is most likely fine.
Library name (crafthead_renderer) >
* creating native/crafthead_renderer/.cargo/config.toml
* creating native/crafthead_renderer/README.md
* creating native/crafthead_renderer/Cargo.toml
* creating native/crafthead_renderer/src/lib.rs
* creating native/crafthead_renderer/.gitignore
Ready to go! See [...]/crafthead/native/crafthead_renderer/README.md for further instructions.

The first thing I did was copy the dependencies from the original Crafthead project into the new project's Cargo.toml:

[package]
name = "crafthead_renderer"
version = "0.1.0"
authors = []
edition = "2021"

[lib]
name = "crafthead_renderer"
path = "src/lib.rs"
crate-type = ["cdylib"]

[dependencies]
rustler = "0.30.0"
imageproc = { version = "0.22.0", default-features = false }

[dependencies.image]
# Make `image` more lightweight. We don't need every image format under the sun,
# just PNG.
version = "0.23.14"
default-features = false
features = ["png"]

Following that, I copied all the remaining files into the new Rust project as well, and made a copy of the old src/lib.rs for reference. Now we will set about moving the API from Wasm to Rustler. We'll start with simple cases and progressively move upwards. First, consider this function which serves as the gateway into the WebAssembly version:

#[wasm_bindgen]
pub fn get_rendered_image(
	skin_image: Uint8Array,
	size: u32,
	what: String,
	armored: bool,
	slim: bool,
) -> Result<Uint8Array, JsValue> {

Consider an idiomatic Elixir interface:

@type render_type() :: :avatar | :helm | :cube | :body | :bust | :cape
@type skin_type() :: :classic | :slim

@spec render_image(binary(), render_type(), [armored: boolean(), skin: skin_type(), size: integer()]) :: binary()
def render_image(image, what, options \\ []) do
    # ...
end

We're not going to be using this interface exactly, but it helped give me a good mental model for converting the existing code into Rustler. I wound up with this:

mod skin;
mod utils;

use std::io::Cursor;

use image::DynamicImage;
use rustler::{Binary, Env, Error, NifResult, NifStruct, NifUnitEnum, OwnedBinary};

use skin::{BodyPart, Layer, MinecraftSkin, SkinModel};

mod atoms {
    rustler::atoms! {
        // Error types.
        invalid_image,
        unable_to_render
    }
}

#[derive(Copy, Clone, NifUnitEnum)]
enum RenderType {
    Avatar,
    Helm,
    Cube,
    Body,
    Bust,
    Cape,
}

#[derive(NifStruct)]
#[module = "Crafthead.Renderer.RenderOptions"]
struct RenderOptions {
    pub render_type: RenderType,
    pub size: u32,
    pub armored: bool,
    pub model: SkinModel,
}

impl RenderType {
    // ...elided...
}

#[rustler::nif]
fn render_image<'a>(
    env: Env<'a>,
    skin_image: Binary<'a>,
    options: RenderOptions,
) -> NifResult<Binary<'a>> {
    let image_copy = skin_image.as_slice();

    let skin_result = image::load_from_memory_with_format(&image_copy, image::ImageFormat::Png);
    match skin_result {
        Ok(skin_img) => {
            let skin = MinecraftSkin::new(skin_img);
            let rendered = options.render_type.render(&skin, &options);

            // We can't predict the size of the output, so we can't just write directly into an OwnedBinary.
            let mut result = Vec::with_capacity(1024);
            let mut cursor = Cursor::new(&mut result);
            return match rendered.write_to(&mut cursor, image::ImageFormat::Png) {
                Ok(()) => {
                    let mut binary = OwnedBinary::new(result.len()).unwrap();
                    binary.as_mut_slice().copy_from_slice(&result[..]);
                    Ok(Binary::from_owned(binary, env))
                }
                Err(_err) => Err(Error::Term(Box::new(atoms::unable_to_render()))),
            };
        }
        Err(_err) => Err(Error::Term(Box::new(atoms::invalid_image()))),
    }
}

rustler::init!("Elixir.Crafthead.Renderer", [render_image]);

The upshot is that we do not need to copy the original skin twice, unlike the Wasm version. Because of the neater FFI provided by Rustler, the render_image function reads a lot more naturally. The only weird parts are that we need to copy the output twice and there is a lifetime now attached to the render_image call bound to the NIF call, but it's not really a big deal.

Now to load the NIF and expose the renderer:

defmodule Crafthead.Renderer.RenderOptions do
  defstruct render_type: nil,
            size: nil,
            armored: false,
            model: nil
end

defmodule Crafthead.Renderer do
  use Rustler, otp_app: :crafthead, crate: "crafthead_renderer"

  def render_image(_skin_image, _options), do: error()

  defp error, do: :erlang.nif_error(:nif_not_loaded)
end

It works!

iex(5)> render = Crafthead.Renderer.render_image(alex, %Crafthead.Renderer.RenderOptions{render_type: :avatar, size: 300, model: :slim})
<<137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 1, 44, 0,
  0, 1, 44, 8, 6, 0, 0, 0, 121, 125, 142, 117, 0, 0, 14, 15, 73, 68, 65, 84,
  120, 156, 237, 212, 177, 206, 118, 233, 28, ...>>

A very dirty NIF

Here is the problem with this NIF: although it is quite fast, it's not fast enough to be kept on the same scheduler as everything else. Quoting the Erlang documentation:

As mentioned in the warning text at the beginning of this manual page, it is of vital importance that a native function returns relatively fast. It is difficult to give an exact maximum amount of time that a native function is allowed to work, but usually a well-behaving native function is to return to its caller within 1 millisecond.

The easiest way around this is to just mark the render_image as a dirty NIF. While it is slow, it is not catastrophically slow, so the warning attached to the Erlang documentation for dirty NIFs is not really applicable. We can specify this is a dirty NIF:

#[rustler::nif(schedule = "DirtyCpu")]
fn render_image<'a>(env: Env<'a>, skin_image: Binary<'a>, options: RenderOptions) -> NifResult<Binary<'a>> {

We're done! From here, we can implement all the other endpoints. Want to give it a spin? I have pushed the code for your enjoyment. It still needs cleanup and isn't feature-complete yet, but it will be soon enough...

Did you enjoy it?

It was better than falling asleep and being like a zombie the following day. Can't complain. Even if I had to fight with the borrow checker.

🌳 A Tree Grows in Times Square

2023-10-13T00:00:00Z

Times Square in 1978 was the dictionary definition of urban decay. Neon lights that once belonged to posh theaters now advertised peep shows and adult entertainment. Street corners were populated by sleazy motels, adult stores, and dodgy nightclubs and bars. The area was populated only with the lowest of American strata, those who had no other place to go. Below the cacophony of cars on the streets seeking to get the hell out as quickly as possible, the subway whirred, with its signs of neglect perhaps worse than what was present above ground. Outside of Times Square, the city burned. Riots, burglaries, and arson had consumed large blocks of Brooklyn and the Bronx. Nobody wanted to be next.

And yet, a tree had taken root on a vacant lot. This tree was Ailanthus, known as “tree of heaven”, or more colloquially, “ghetto palm.” It was a typical weed of urban environments. Few bothered to pay it much mind; the city was burning in those times, and who cared about some weed in the seediest part of Midtown Manhattan, anyway?

The tree grew. Nobody paid much heed.

At first, it was all normal. Drug dealers would use it as a convenient background for their illicit transactions. The same happened with sleazy, unlicensed taxi drivers, and the men and women of loose morals offering their “services”.

The tree kept growing.

A strange sort of magnetism started to form around the tree, with strange occurrences happening within the vicinity of the tree.

A man named Eddie, who had frequented the area for years, claimed the tree was speaking to him, filling his nights with nightmares and his days with an inescapable dread. "It's like a shadow's whisper, man," he'd say. His erratic behavior drew the attention of the authorities, and he was committed to a mental institution. Everyone dismissed Eddie as another lost soul swallowed by the city. But something in his eyes said otherwise.

Jolie, a stripper who would often perform at Texas Sized, an early “breastaurant”, would often comment about how the tree would impact the patrons who came in. Everything from dark thoughts, people staring at the tree intently as if it had something to say, people sleeping underneath the tree for more than 18 hours on end, people dying immediately after injecting drugs next to the tree, she had seen everything. But nobody wanted to pay heed to this New Jerseyan who was just trying to scrape enough money together to go to college. She was just looking for attention, right?

The tree kept growing, developing a robust canopy, shading the vacant lot it occupied.

A major conglomerate, famous for a certain mouse mascot, sought to move in. Redevelop Times Square in its own image. A sanitized playground ideal for the average American family. A homage to commercialism. A celebration of American capitalism, rising out of chaos.

Gentrification.

In a corporate office in suburban Los Angeles, the tree was ever present in the thoughts of the executives and managers looking to execute upon their grand vision. Huddled around a drafting table, the tree was circled right there, on that lot, on a map where a grand celebration of capitalism was to be built. The company had already purchased the lot as part of the redevelopment project, ample loans and tax credits had made sure of that. It was time to decide whether or not to tear down the tree.

The tree kept growing.

Tension continued to boil, back in the office. The tree had rapidly become a serious sticking point in the plan to transform Times Square. The lead architect spoke up. “This tree does not fit. It is nothing more than an out of control weed, a plant of inconsequential value. We are not building Central Park. It’s time to kill it off, and keep it dead.”

A junior member of the project, associated with public relations, argued in favor of preserving the tree. "It's become a part of the local landscape, a monument in its own right. Can't we build around it? Incorporate it into the design?" she suggested, sketching a quick rendering that framed the tree as the centerpiece of a small park within the commercial empire.

As the debate raged on, each side armed with its own set of reasons, the decision loomed large, growing in significance much like the tree itself had done amidst the decay of Times Square.

The tree kept growing.

Miguel was walking on the street where the vacant lot stood. He was part of a Russian nesting doll of subcontractors hired by the major conglomerate in preparation for the megaproject that was unfolding. The blistering heat of July in New York City was coming down on him. He opened a cold beer he had bought from a nearby bodega. As he took a sip, the taste of beer was replaced by a rather most unfamiliar taste, a sweet but unexpected taste. He spat out his beer.

Katherine, an auditor on the project’s accounting and finance team, was staying in a close family friend’s home in the Upper East Side. She hailed a taxi to the site to examine the property for herself. She had vaguely heard of the tree, but her interest was whether or not the company paid a fair price for the lot. As the cab approached the lot, the driver started to mumble. She grew worried as the cab came to a sudden halt, feet from the lot, with the engine still running, the doors locked, and the driver frozen in place.

Meanwhile, the city continued to hum along. People would walk next to the vacant lot. Many times, nothing notable would happen. Other times, something spectacular and unusual would happen. The decay was slowing, but it was still present everywhere.

The tree kept growing.

Robert was a manager at a major banking firm that had extended loans to the conglomerate for the project. He was also a closeted gay man in a marriage that was quickly falling apart. He found solace in visiting gay spaces in Greenwich Village and Times Square and visiting with someone who had very quickly became a spouse in all but formality. He was captivated by the tree. It was almost like it called out for him.

After a bitter and messy divorce, he would buy a dilapidated apartment building across from the tree and convert it into his new home, and he and his new spouse, Jamie, would move in. Jamie didn’t understand why Robert wanted to move here when his life was so well in Connecticut, but he went along with it because he loved Robert.

On many occasions, Robert would call out sick from work and spend hours looking at the tree. He admired it, respected it, worshiped it like a god. It annoyed the ever-living hell out of Jamie.

A drug dealer who frequented Times Square who once made his sales in the shadow of the tree would report strange things about the tree. That it was making the drugs more potent. That the tree was speaking to people. That the tree was providing good fortune. He was arrested by police, and they moved onto the next common criminal.

The tree kept growing.

The decision had been made to tear down the tree. It was a nuisance.

The tree kept growing.

Reactions to the impending demise of the tree were as mixed as the clientele that roamed Times Square. It was as though the tree had woven itself into the fabric of people’s lives, more tightly for some than for others. Rumors and whispers spread like wildfire—everyone from Miguel the contractor to Robert, who lived across from the tree, felt a sense of impending doom or at least change.

Miguel, still puzzled by the bizarre taste of his beer, looked at the tree one last time before leaving the site. He wondered what else the tree might have influenced that he wasn't aware of. Katherine, shaken by her cab incident, started to research not just the lot’s financial aspects but also the strange occurrences tied to it. She couldn’t shake the feeling that this was about more than just real estate, that this was not something that an audit focused on the numbers would ever be able to solve.

Robert felt a knot tighten in his stomach. For him, the tree had become a sanctuary, a constant amid the chaos of his personal life. Jamie could see that something was bothering Robert profoundly, but he could not understand what the tree had to do with it. All the better to kill off a weed, he thought.

The clientele of Times Square continued to exchange their stories about the tree. The supernatural. The average. The merely bizarre.

Word had spread about the tree and it had become something of a minor tourist attraction in its own right.

Days before the planned removal, Robert found himself standing in front of the tree, as if to say a final goodbye. He felt a wave of sadness wash over him, only broken by Jamie's voice letting him know dinner was ready.

Protests and petitions to save the tree sprang up, but the wheels of progress, lubricated by money and corporate power, seemed impossible to halt. Even the drug dealer, from his prison cell, muttered something about the foolishness of removing a source of good fortune.

The night before the tree was scheduled for removal, strange occurrences peaked. Reports of eeriness, of whispering winds, and unexplained phenomena flooded the local police lines. Those who had never believed in the tree's 'powers' were starting to doubt themselves.

And despite all the efforts to cut it down, despite being labeled a nuisance, despite the chainsaws that were readied and aimed at its base—

The tree kept growing.

And it devoured and disabled the equipment that was meant to bring its demise about.

The tree kept growing.

And it infiltrated the minds of those who sought to tear down the tree, clouding them in dark thoughts.

The tree kept growing.

Back in California, an emergency meeting was convened. What had once been dismissed as a nuisance was now an object of both fascination and fear, leading to an atmosphere of uncertainty and chaos. The project was at risk of becoming not just a financial burden but a public relations nightmare.

The tree kept growing.

The lead architect, who had once dismissed the tree as an "out of control weed," dwelled in a mix of anger and angst. The public relations junior member, who had once advocated for the tree's preservation, couldn't help but feel a sense of vindication mixed with newfound respect for the mysterious tree.

Meanwhile, a new wave of stories about the tree swept through the city, intensifying the mythos surrounding it. Its branches now stretched high and wide, a towering spectacle that seemed to defy not just corporate will but nature itself.

The tree kept growing.

But the unstoppable will of corporate America demanded the death of this tree.

The tree kept growing.

No expense was spared, and the conglomerate sought to hire just about every expert out there. Biologists and chemists at the country’s top universities. Religious figures of every stripe. Voodoo practitioners. Even a handful of charlatans managed to get a slice of the pie. They set up shop in an old hotel near the lot, and converted it into a modern lab, all with the singular goal of killing the tree.

The tree kept growing.

Nothing they did was any good.

The tree kept growing.

Experimental herbicides served only to nourish the tree, and give the scientists both figurative and literal headaches. One would later develop cancer.

The tree kept growing.

Physical treatments all suspiciously failed, with equipment malfunctions, bizarre accidents, and even deaths.

The tree kept growing.

Shamans tried to curse the tree, but would be suspiciously and conveniently attacked with dark thoughts every time.

The tree kept growing.

The conglomerate even tried to hire nuclear scientists in an effort to poison or destroy the tree with the use of radioactivity. Operating a nuclear laboratory in the middle of New York City was a hard sell, especially with the memory of Three Mile Island so fresh, and so the plan was nixed.

The tree kept growing.

Dread became the order of the day back in the California boardroom. The project to kill the tree was quickly becoming a financial strain and a public relations disaster for the conglomerate. Shareholders grew restless, activists became more vocal, and even the most hardened executives started to question the wisdom of their endeavor. Slowly, dread turned into resignation. One faction considered the tree an insidious force that was to be ignored entirely, in hopes of having the rest of the project succeed. Others argued for respecting the tree and incorporating it into the plans.

The tree kept growing.

And as the grand theaters of Times Square were restored, the seedy adult businesses shuttered, and family-oriented businesses moved in, the tree sat there, on the same vacant lot, observing the world around it as it changed. Times Square would morph into a wonderland of commercialism, the much lauded celebration of American capitalism that was envisioned, buzzing with more activity than it had gotten in over 50 years with tourists arriving from all around the world, some just to gawk at the tree.

Urban legends would circulate about the tree, with its stories told in constantly evolving permutations. It simply blended into the urban and cultural fabric of New York. Some would continue to ascribe certain experiences with the tree.

Some would remember the tree as an eternal reminder of the “old” New York, before it gentrified and became sanitized, boring, and uninspired. Others considered it a reminder that nature was always the last remaining check on man’s hubris. Others considered the tree to be part of a grand, sprawling conspiracy. Still others ascribed religious meaning to it, whether it was the Anti-Christ or a deity to be respected in its own right.

It witnessed the ebb and flow of humanity around it, the changing seasons of a city that never sleeps, the neon lights and LEDs advertising America, and the cyclical nature of decay and rebirth. It absorbed the vibrations of a thousand stories, from Robert's newfound love and liberation to the unexplained phenomena that defied scientific scrutiny.

The tree kept growing. And it would have the last laugh.

Epilogue

The redevelopment of Times Square wound up being a smashing success for the conglomerate, but it came at a high price. The conglomerate was on the brink of declaring bankruptcy, but it was rescued by its creditors at the last minute. But it would be forever remembered in New York as “that company that tried to cut down the tree.”

The lead architect was disgraced. Combined with a severe gambling problem, he was a desolate man. The last time anyone heard from him, he was living underneath a freeway overpass in Los Angeles.

Katherine left the conglomerate and started advocating for protecting the tree as a historical landmark.

Robert and Jamie would later move out, being offered a handsome sum to convert their home into a multi-story hotel. They gladly accepted, and they went into retirement in Florida together. It was hard on Robert, but they needed to enjoy the golden years somehow.

Jolie returns to Times Square after college as an office worker working at an advertising company.

Some would say that the tree had a strange magnetism about it. They were believed.

The tree kept growing.

📚 A 2022 dispatch

2022-10-16T00:00:00Z

Well... a lot of big things happened this year. But to understand the context, we also need to understand what went down last year, which was significant in its own way.

A short recap of 2021

In 2021, I...

👾 began to say my final goodbyes to the Minecraft scene. I am still minimally involved, but I'm not going out of my way to support it.
👾 ...and started what would be a 8-month stint at Branch, which at the time was working on a "virtual office" platform, before pivoting to UGC, and then finally to producing "web3 video games" in October (along with a strong recommendation to move to Seattle, a move I was not yet ready to make), whereupon I left them.
🎓 ...dropped out of college. It felt increasingly meaningless to me and I could no longer tolerate being bullied.
💳 ...after I left Branch in October, I immediately looking for another job. Through the absolute 🐐 Yunyu Lin (thank you so much!) through my days in the early days of the Minecraft server scene, I received an offer to work as a software engineer at Ramp and started work in November 2021.

That's where we'll leave it for 2021 - a big year, and while I can't enumerate everything that went down that year, getting a job at Ramp was absolutely the highlight of 2021. This is a job that is very difficult to leave and for good reason! Keep this in mind, for one event in 2022 ties to that.

The start of 2022

The start of 2022 was pretty uneventful. I had already substantially onboarded to Ramp in late 2021 (the first code that was entirely my work was pushed just before the company holiday break). Then I decided to visit the New York office in late March 2022 - I'd never been and I wanted to go, since up to this point I hadn't left the Southeastern United States.

It was great - perhaps I fell in love too quickly, because I wanted to come back for real. I already had thoughts about moving due to the last job, since I simply was tired of living in the Atlanta area and wanted a fresh start, a chance to stake out on my own. I found that opportunity in New York.

🗽 Welcome to New York

After (too quickly) finding an apartment, I arrived back in New York on April 17, 2022. The next day, I moved into my very first apartment in NYC.

I took a week off from work to settle in and check out my neighborhood some. Ever since I moved to New York, I started to really grow into myself. The shock was immense, but I was able to adjust very well after a few weeks - granted, I had some experience from my first trip. Furthermore, my mental and physical health improved with the move: I was no longer dreading about staying home all the time, because there was so much I could do in New York, whether it's to try something I'd never done before, or enjoy working with coworkers, or just admire the beauty of the city... I wouldn't run out of options.

🌴 Ramp Builders 2022

In June, the entire product, engineering, and design teams had a chance to sync in-person at our first major internal conference, Ramp Builders, held in Miami that year.

The conference packed a lot into 3 days. It was certainly a major milestone. We spoke with some Miami-area customers, had discussions, thought about next steps, and began aligning on long-term projects we needed to ship.

📄 Finally, Redoing My Website?

It took a while... but I redid my website. The previous design, while it worked, looked somewhat ugly, and I wanted to revamp things and brush off the frontend skills that I clearly have been rusting for some time... so I went back to the past, dusted off a template from the venerable HTML5 UP, converted the template to Nunjucks for use with Eleventy, and extensively customized it for my own needs, and topped it off with a photo of the beautiful (but tourist-choked, to my great chagrin) Brooklyn Bridge:

Click the above image to expand. I might start posting some photography here! Taken with my iPhone XR on October 15, 2022 at 5:48PM, a bit before sunset.

That's All For Now

I am sure there is a lot more ahead - this has been a year of extremes, largely for the worse. But I can take some comfort knowing that I am finding myself and have become independent. The road ahead will be long and with many twists and turns, but at least I am walking down it now.

How fast can you write a VarInt?

2021-05-08T00:00:00Z

I often like to start my day by keeping myself abreast of interesting things in the world of computer programming. Often this means a visit to Hacker News or Lobsters. I somehow managed to come across Richard Startin's article about why he doesn't use Google Protocol Buffers for telemetry. Near the bottom, he has an aside about VarInts and shares a few tricks to produce them more efficiently.

Why was this relevant to me? That is because the Minecraft: Java Edition protocol uses VarInts extensively. But what exactly are we looking at here? Let's dive a bit deeper.

What even is a VarInt?

A Protocol Buffers VarInt is a 1 to 10 byte sequence that encodes a specific number (usually up to 32-bit for up to 5 byte VarInts, 64-bit for up to 10 byte VarInts). The process for encoding a VarInt is simple:

Check if the value is less than 0x80. (This is usually done with a simple AND and comparison, i.e. (value & ~0x7FL) == 0.) If so, write the value out to the next position in the buffer and return.
Grab the lowest 7 bits of the value being written (value & 0x7F) and set the 8th bit ((value & 0x7F | 0x80)). Write this out into the next position in the buffer.
Logically shift the value to the right by 7 bits and repeat from step 1.

Here is the naive implementation (sourced from the Google Protocol Buffers library's CodedInputStream):

while (true) {
  if ((value & ~0x7FL) == 0) {
    buffer[position++] = (byte) value;
    return;
  } else {
    buffer[position++] = (byte) (((int) value & 0x7F) | 0x80);
    value >>>= 7;
  }
}

For the purposes of demonstration we will focus on the 32-bit VarInts only - the techniques shown here can be extended to 64-bit VarLongs (at a not-insignificant cost to your sanity).

Now, for the main star of the show - the old VarInt writing implementation from Velocity:

while (true) {
  if ((value & 0xFFFFFF80) == 0) {
    buf.writeByte(value);
    return;
  }

  buf.writeByte(value & 0x7F | 0x80);
  value >>>= 7;
}

This is simply a version of the above method, but adapted to write to a Netty ByteBuf. With the necessary background taken care of, let's have some fun.

Disclaimer

I am by no means an expert on how the JIT operates, but I am very interested in fun and unique ways to solve problems!

The testing system is my home PC, featuring an AMD Ryzen 9 3900X processor with 32GB of DDR4-3600 memory, running Fedora 34 and Java 11 as packaged in Fedora. We use the flag -Dio.netty.leakDetection.level=DISABLED as otherwise 1% of buffers will be selected for leak detection instrumentation, which would introduce unfavorable noise in the results.

The benchmarks in question allocate a direct buffer 5 bytes long, write a VarInt into it, and then clear it. This is repeated for a pseudo-random sequence of 2,048 numbers in order to defeat the JVM's tendency to assume that a certain branch would always be taken.

Try 1: Richard Startin's Method

Let's try using Richard Startin's tricks. Here's the above method, but using his method:

int continuationBytes = (31 - Integer.numberOfLeadingZeros(value)) / 7;
for (int i = 0; i < continuationBytes; ++i) {
  buf.writeByte(((byte) ((value & 0x7F) | 0x80)));
  value >>>= 7;
}
buf.writeByte((byte) value);

What do we get?

Benchmark                                      Mode  Cnt      Score      Error  Units
VarIntWriterBenchmark.oldVelocityVarintWrite  thrpt    5  97618.250 ± 1682.211  ops/s
VarIntWriterBenchmark.startinVarintWrite      thrpt    5  73264.856 ±  664.202  ops/s

This is actually worse than we had before! But why?

Getting to the bottom of it

Is it because of the Integer.numberOfLeadingZeros() call? The answer is probably a resounding no - after all, Integer.numberOfLeadingZeros() is an intrinsic on at least x86_64 (the only architecture where we are running these benchmarks) and Arm. Indeed, my Zen 2 processor can process 3 LZCNT instructions every cycle, so in theory this should deliver performance improvements.

In isolation, we can devise a small benchmark to see if this educated guess was right:

  @Benchmark
  public int lzcnt() {
    int sum = 0;
    for (int number : numbers) {
      sum += (31 - Integer.numberOfLeadingZeros(number)) / 7;
    }
    return sum;
  }

  @Benchmark
  public int byHand() {
    int sum = 0;
    for (int value : numbers) {
      while (true) {
        if ((value & 0xFFFFFF80) == 0) {
          sum++;
          break;
        }

        sum++;
        value >>>= 7;
      }
    }
    return sum;
  }

The results:

Benchmark                      Mode  Cnt       Score       Error  Units
VarIntWriterBenchmark.byHand  thrpt    5  323063.055 ±  1759.055  ops/s
VarIntWriterBenchmark.lzcnt   thrpt    5  570563.618 ± 17726.994  ops/s

At least in isolation, this doesn't seem to be the case. But as we'll see later, this benchmark doesn't really tell us anything meaningful. Perhaps this change makes the JIT compiler more trigger-happy in a way that tanks our performance, thanks to the more traditional loop style we now employ. I'm not sure.

Try 2: Unrolling the Loop

Okay, let's go back a step. Remember that the loop we are running could run 1 to 5 times, depending on how large the VarInt will be. We also know that the number of bytes will increase with each 7 bits in the number. Given that, what if we simply unrolled the loop, accounting for all of the cases?

Here's the code:

  if (value < 0) {
    buf.writeByte(value & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte(value >>> 7);
  } else if (value < 128) {
    buf.writeByte(value);
  } else if (value < 16384) {
    buf.writeByte(value & 0x7F | 0x80);
    buf.writeByte(value >>> 7);
  } else if (value < 2097152) {
    buf.writeByte(value & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte(value >>> 7);
  } else if (value < 268435456) {
    buf.writeByte(value & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte(value >>> 7);
  } else {
    buf.writeByte(value & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte((value >>>= 7) & 0x7F | 0x80);
    buf.writeByte(value >>> 7);
  }

What do we get now?

Benchmark                                      Mode  Cnt      Score      Error  Units
VarIntWriterBenchmark.oldVelocityVarintWrite  thrpt    5  95870.960 ± 4328.322  ops/s
VarIntWriterBenchmark.startinVarintWrite      thrpt    5  73097.880 ± 1116.550  ops/s
VarIntWriterBenchmark.unrolledVarintWrite     thrpt    5  95840.321 ± 2882.220  ops/s

There's no change, but perhaps we could have expected that from naively unrolling the loop and just figuring out when certain byte lengths would be written. We can, of course, do better than that.

Try 3: Smartly Unrolling the Loop

Let's take a look at that unrolled code again. It's clearly not optimal, since it repeats a lot of the same code, including multiple calls to ByteBuf#writeByte(). Can we do better than this? In fact, we can:

  if ((value & 0xFFFFFF80) == 0) {
    buf.writeByte(value);
  } else {
    buf.writeByte(value & 0x7F | 0x80);
    value >>>= 7;
    if ((value & 0xFFFFFF80) == 0) {
      buf.writeByte(value);
    } else {
      buf.writeByte(value & 0x7F | 0x80);
      value >>>= 7;
      if ((value & 0xFFFFFF80) == 0) {
        buf.writeByte(value);
      } else {
        buf.writeByte(value & 0x7F | 0x80);
        value >>>= 7;
        if ((value & 0xFFFFFF80) == 0) {
          buf.writeByte(value);
        } else {
          buf.writeByte(value & 0x7F | 0x80);
          value >>>= 7;
          buf.writeByte(value);
        }
      }
    }
  }

This looks pretty gnarly already, but let's give it a shot:

Benchmark                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.oldVelocityVarintWrite    thrpt    5   95113.915 ±  182.119  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite  thrpt    5  124865.620 ±  550.267  ops/s
VarIntWriterBenchmark.unrolledVarintWrite       thrpt    5   93953.405 ± 1530.158  ops/s

That's very impressive - a nearly 30% performance improvement from improving the structure of the code. Can we improve this some more?

Try 4: The `numberOfLeadingZeros` Mystery Solved?

Let's try to use Integer.numberOfLeadingZeros() to optimize this function. Perhaps it will give us a bit more performance.

  int continuationBytes = (31 - Integer.numberOfLeadingZeros(value)) / 7;
  if (continuationBytes == 0) {
    buf.writeByte(value);
  } else {
    buf.writeByte(value & 0x7F | 0x80);
    value >>>= 7;
    if (continuationBytes == 1) {
      buf.writeByte(value);
    } else {
      buf.writeByte(value & 0x7F | 0x80);
      value >>>= 7;
      if (continuationBytes == 2) {
        buf.writeByte(value);
      } else {
        buf.writeByte(value & 0x7F | 0x80);
        value >>>= 7;
        if (continuationBytes == 3) {
          buf.writeByte(value);
        } else {
          buf.writeByte(value & 0x7F | 0x80);
          value >>>= 7;
          buf.writeByte(value);
        }
      }
    }
  }

I can't wait for all the performance improvements!

Benchmark                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.smartUnrolledVarintWrite  thrpt    5  110664.096 ± 1460.675  ops/s

That's not good at all. We just regressed by more than 10%.

Try 5: Remove the Data Dependency

That was a bust. Is there something else we can do? Well, there is a data dependency. We are bit-shifting to the right by 7 bits each time then seeing if we need to continue writing bytes or not. It's possible to get a little bit more parallelism out of the CPU by knowing how much we'll bit-shift by:

  if ((value & (0xFFFFFFFF << 7)) == 0) {
    buf.writeByte(value);
  } else {
    buf.writeByte(value & 0x7F | 0x80);
    if ((value & (0xFFFFFFFF << 14)) == 0) {
      buf.writeByte(value >>> 7);
    } else {
      buf.writeByte((value >>> 7) & 0x7F | 0x80);
      if ((value & (0xFFFFFFFF << 21)) == 0) {
        buf.writeByte(value >>> 14);
      } else {
        buf.writeByte((value >>> 14) & 0x7F | 0x80);
        if ((value & (0xFFFFFFFF << 28)) == 0) {
          buf.writeByte(value >>> 21);
        } else {
          buf.writeByte((value >>> 21) & 0x7F | 0x80);
          buf.writeByte(value >>> 28);
        }
      }
    }
  }

How does this attempt fare?

Benchmark                                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.oldVelocityVarintWrite                    thrpt    5   94480.674 ± 1389.432  ops/s
VarIntWriterBenchmark.smartNoDataDependencyUnrolledVarintWrite  thrpt    5  129428.934 ± 7405.911  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite                  thrpt    5  126697.664 ± 5033.494  ops/s

It looks like the two approaches are in fact similar in performance, with the newer version performing perhaps a tad better, but the results are too noisy to know for sure.

Try 6: Training Ourselves in the Dark Arts

Already we found a ~30% speedup from unrolling the loop carefully, but can we do better than that still? Yes, we can. As most performance-oriented engineers know, doing things in bulk can be a very profitable optimization. Up to this point, we've been running 1 to 5 calls to write individual bytes. It turns out that if we get a little bit creative, we can either write the entirety of the bytes up front (1, 2, or 4 bytes worth) or combine one larger writing operation with writing one byte (2 + 1 for 3 bytes, 4 + 1 for 5 bytes – VarLong writing is left as an exercise to the reader). Luckily, we don't need to handle the 2 + 1 case specially as Netty ByteBuf provides a writeMedium() method. In order to do this, we will instead construct the VarInt as a number, and then write it to the buffer. Let's give it a try:

  if ((value & (0xFFFFFFFF << 7)) == 0) {
    buf.writeByte(value);
  } else if ((value & (0xFFFFFFFF << 14)) == 0) {
    int w = (value & 0x7F | 0x80) << 8 | (value >>> 7);
    buf.writeShort(w);
  } else if ((value & (0xFFFFFFFF << 21)) == 0) {
    int w = (value & 0x7F | 0x80) << 16 | ((value >>> 7) & 0x7F | 0x80) << 8 | (value >>> 14);
    buf.writeMedium(w);
  } else if ((value & (0xFFFFFFFF << 28)) == 0) {
    int w = (value & 0x7F | 0x80) << 24 | (((value >>> 7) & 0x7F | 0x80) << 16)
            | ((value >>> 14) & 0x7F | 0x80) << 8 | (value >>> 21);
    buf.writeInt(w);
  } else {
    int w = (value & 0x7F | 0x80) << 24 | ((value >>> 7) & 0x7F | 0x80) << 16
            | ((value >>> 14) & 0x7F | 0x80) << 8 | ((value >>> 21) & 0x7F | 0x80);
    buf.writeInt(w);
    buf.writeByte(value >>> 28);
  }

The end result is this:

Benchmark                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.blendedVarintWrite        thrpt    5  175968.604 ± 5195.657  ops/s
VarIntWriterBenchmark.oldVelocityVarintWrite    thrpt    5   94270.376 ± 2796.116  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite  thrpt    5  124524.690 ± 2929.593  ops/s

This is an enormous improvement, to say the least. From our baseline we got a nearly 1.87x speedup! But there is a reason why this is suboptimal: the above code is definitely difficult to read and understand, and it is extremely fragile – extending it to 10-byte VarLongs would be a nightmare!

Bonus: Exact byte length encoding

Quoting Startin:

When I went back and read it again I was surprised to find that embedded messages are length-prefixed, but the length prefix is varint encoded, which means you don’t know how many bytes you need for the length until you’ve done the serialisation, and it’s recursive.

Startin then argues that was one reason why he considered Protocol Buffers to be bad for reporting telemetry from the edge. But for those who are stuck with VarInts like I am, you can in fact write VarInts in a sequential fashion, and even with performance slightly above the traditional approach to writing VarInts. Specifically, it is possible for us encode any number that can be encoded as a VarInt in exactly 5 bytes:

  int w = (value & 0x7F | 0x80) << 24 | ((value >>> 7) & 0x7F | 0x80) << 16
      | ((value >>> 14) & 0x7F | 0x80) << 8 | ((value >>> 21) & 0x7F | 0x80);
  buf.writeInt(w);
  buf.writeByte(value >>> 28);

Therefore, if you are willing to inflate the size of your messages, you can write the message out sequentially. Even better, you can even improve upon this result by encoding only the maximum amount you need. In the Minecraft protocol, compressed messages are always 21-bit or less, so we can encode the length of a compressed packet in just three VarInt bytes and save ourselves a memory copy. Very neat!

But performance-wise, is it worth doing that?

Benchmark                                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.blendedVarintWrite                        thrpt    5  179321.732 ± 7610.742  ops/s
VarIntWriterBenchmark.lucky5VarintWrite                         thrpt    5  113540.019 ± 6062.895  ops/s
VarIntWriterBenchmark.oldVelocityVarintWrite                    thrpt    5   95748.224 ± 2531.663  ops/s

If you have a need for this, then yes, it even beats the traditional loop. But despite appearances, this is not only not the fastest way to write a VarInt, it also completely defeats the point of using a VarInt. But it's a neat trick that could come in handy under certain circumstances.

But this is just one processor - I need more than that!

I ran the tests on a DigitalOcean CPU-optimized droplet with 4GB of memory and two dedicated CPU threads of an Intel Xeon E5-2650v4. The OS is Ubuntu 20.04 but we keep using Java 11.

Here's the results:

Benchmark                                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.blendedVarintWrite                        thrpt    5  103837.167 ±  629.934  ops/s
VarIntWriterBenchmark.oldVelocityVarintWrite                    thrpt    5   54538.692 ± 1312.263  ops/s
VarIntWriterBenchmark.smartNoDataDependencyUnrolledVarintWrite  thrpt    5   76576.717 ± 1051.005  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite                  thrpt    5   76325.907 ±  855.393  ops/s
VarIntWriterBenchmark.startinVarintWrite                        thrpt    5   46513.811 ±  706.059  ops/s
VarIntWriterBenchmark.unrolledVarintWrite                       thrpt    5   56751.155 ±  803.914  ops/s

These results comport with the rest of the article. As for Arm, here are the results from my M1 MacBook Pro:

Benchmark                                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.blendedVarintWrite                        thrpt    5  237790.486 ±  571.134  ops/s
VarIntWriterBenchmark.oldVelocityVarintWrite                    thrpt    5  165370.092 ± 1975.949  ops/s
VarIntWriterBenchmark.smartNoDataDependencyUnrolledVarintWrite  thrpt    5  188349.773 ± 2062.149  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite                  thrpt    5  187717.560 ±  755.758  ops/s
VarIntWriterBenchmark.startinVarintWrite                        thrpt    5  115166.912 ±  611.680  ops/s
VarIntWriterBenchmark.unrolledVarintWrite                       thrpt    5  183905.054 ±  766.400  ops/s

Okay, so the Intel processor is behind, probably better explained by the fact it's clocked lower and is an older chip from the Broadwell era. But the M1 mops the floor in this particular benchmark, beating out the Ryzen 9 3900X and the Intel Xeon E5-2650v4. Why?

Atomic Power

When we call a write function on a Netty ByteBuf there are several checks that have to be performed. These are typically performed in AbstractByteBuf#ensureWritable0(). Specifically:

It checks that the written bytes after writing minWritableBytes would not exceed the buffer's current capacity.
It ensures that the buffer's reference count is not zero.
If the check in (1) passed, we are good to go (this is considered the "fast path").
Otherwise, if the new size of the buffer would exceed the buffer's current capacity (but not its maximum capacity), it will expand the buffer.

Since our buffer size is always 5, the first check will always succeed, and since the JIT knows that the branch is always taken, that just means we're left with the reference count check as our last source of slowdown. In Netty, reference counts are always accessed and modified atomically. We know that uncontended atomic access is fast on M1. Does this explain the performance difference? Let's run the tests on our Ryzen processor again but with the flag -Dio.netty.buffer.checkAccessible=false:

Benchmark                                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.blendedVarintWrite                        thrpt    5  158655.053 ± 3702.731  ops/s
VarIntWriterBenchmark.oldVelocityVarintWrite                    thrpt    5  101722.002 ±  331.125  ops/s
VarIntWriterBenchmark.smartNoDataDependencyUnrolledVarintWrite  thrpt    5  131313.391 ±  392.404  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite                  thrpt    5  146280.007 ± 1271.907  ops/s
VarIntWriterBenchmark.startinVarintWrite                        thrpt    5   80280.574 ±  636.335  ops/s
VarIntWriterBenchmark.unrolledVarintWrite                       thrpt    5  101123.052 ±  915.786  ops/s

That's certainly very different. blendedVarintWrite actually lost a bit over 10% of its throughput, but the other byte-by-byte approaches gained at least by 8%. What happens if we run it on the M1?

Benchmark                                                        Mode  Cnt       Score      Error  Units
VarIntWriterBenchmark.blendedVarintWrite                        thrpt    5  237697.049 ±  304.124  ops/s
VarIntWriterBenchmark.oldVelocityVarintWrite                    thrpt    5  166456.475 ±  384.274  ops/s
VarIntWriterBenchmark.smartNoDataDependencyUnrolledVarintWrite  thrpt    5  190258.830 ± 1163.047  ops/s
VarIntWriterBenchmark.smartUnrolledVarintWrite                  thrpt    5  189399.626 ±  186.078  ops/s
VarIntWriterBenchmark.startinVarintWrite                        thrpt    5  116002.657 ±  364.835  ops/s
VarIntWriterBenchmark.unrolledVarintWrite                       thrpt    5  184978.894 ± 3182.543  ops/s

This essentially didn't change. So we can conclude that blendedVarintWrite gets most of its performance through amortizing API calling costs in the Netty API.

Conclusions

It is very possible to write VarInts very quickly, on the order of close to 500 million per second on commodity consumer hardware with only a single thread.

Here's all the various methods by their time per operation:

Method	Intel Broadwell	AMD Zen 2	Apple M1
Baseline	8.95 ns/op	5.1 ns/op	2.93 ns/op
Startin	10.5 ns/op	6.68 ns/op	4.24 ns/op
Smart Unrolling	6.4 ns/op	3.92 ns/op	2.6 ns/op
Dark Arts (#6)	4.7 ns/op	2.78 ns/op	2.05 ns/op

We have sped up the writing speed on the x86 platforms by more than 80%, whereas the M1 received a more modest but still substantial 43% performance boost.

I invite critiques on the GitHub repository where I have published the JMH tests.

Licensing

You are free to use any of the code snippets under the terms of the MIT license.

Catching up on things

2021-05-08T00:00:00Z

It's been a long time coming, but I have bitten the bullet. My blog, once a fairly stodgy thing I hadn't updated in over 2 years has received a total redesign with a focus on extremely fast load times. My goal is function over form and I think I have delivered that. Of course, I did make some concessions to keep the format largely sane. But while my blog sat empty, many things have changed in my life.

The new blog

My previous site was built on Hugo. Now I wave goodbye to Hugo and say hello to Eleventy. Eleventy is just as fast and much more flexible for my needs. I found Hugo to be somewhat limiting even as some of its features like image resizing were rather nice.

How the coronavirus pandemic has affected me

I've been touched by the pandemic myself, even having a scare back in October. I have managed to avoid catching the virus, however I have family that have managed to catch the virus. Thankfully, they are fine and we haven't had to attend any funerals. Yet.

The ongoing (as of this writing, of course) coronavirus pandemic has aroused a new sense out of myself. It has led me to some truly unique opportunities but it has shown my limitations and some of the past choices I've made were not optimal. I'm working to correct that this year.

A summary of my 2020

I didn't stop working during 2020, of course - I made sure to keep myself busy with projects.

Block By Blockwest

I had the opportunity to participate in the Block by Blockwest event that was originally going to occur on April 25th, 2020, but was postponed to May 16th, 2020. This was largely because of higher than anticipated demand. In the span of 3 weeks, a new team (including me) managed to get Block by Blockwest up and running on a more scalable setup and the event finally succeded in holding a maximum of approximately 1,700 concurrent players. Perhaps my most important contribution would be the rapid deployment of the Velocity Minecraft proxy at scale.

Velocity

For this, you could consider the announcements on the Velocity forum to be my de facto blog for all of 2020, but 2020 saw the release of the Velocity 1.1.x series, which included a slew of new features and a sustained effort at optimizing the proxy as much as possible. The optimization work paid off handsomely, since larger Minecraft networks including LemonCloud, The Archon, and several others switched to Velocity. In particular, networks that have switched to Velocity from BungeeCord derivatives have reported that CPU utilization often goes down drastically.

Fabric

I also started dabbling in Fabric modding! My first ever Fabric mod was CrossStitch but I also developed and released Krypton and LazyDFU which have seen some popularity. It's only a matter of time to see how much Minecraft will continue to capture my interest given recent developments.

Where I am now

Of course, that was last year. Now it is 2021. We've already seen quite a bit happen, especially with America seemingly filrting with yet another far-right authoritarian disaster thanks to the Capitol insurrection, egged on by the now-former president of the United States himself! How fun.

Branch

The most exciting news I have to share this year is that since February I am now employed by Branch. Of course, this was somewhat assisted by the fact Dayton has worked with me before on another project of his, Statscraft (later purchased by Tebex). I am mostly taking this opportunity to begin shifting away from Minecraft into other interests.

Branch is in an industry that could have never been viable without the COVID-19 pandemic: virtual, spatially-oriented offices. Work-from-home is destined to outlast the COVID-19 pandemic and I'm proud to be a part of that trend. As a company founded by gamers trying to stake a middle grouhd between Minecraft but for enterprise and Zoom but not boring, Branch is quite unique. I am looking forward to working with Branch!

Slowing down my Minecraft involvement

I'm still committed to most of my Minecraft projects but I'm winding down my involvement on some of them.

NuVotifier has been effectively in maintenance mode for a while. This will continue to be the case. It has matured and doesn't need many changes. Bug fixes and support for newer versions of Minecraft are the main priority at this point.
SuperbVote is also largely mature. There are some bugs that need to be fixed, but it's mature. Therefore it is now in maintenance mode only, with only bugs being fixed and features only considered when sent as a pull request.

The following projects will continue:

Velocity will continue to evolve and be developed, but it was always my intention to turn over control of the project to other members of the community. There is now a second active developer on the project that cares deeply about Velocity and we both cooperate. However, right now the project still relies almost completely on me.
LazyDFU, CrossStitch, and Krypton will continue to be developed. They are relatively new, but LazyDFU can be considered somewhat stable now (only needing minor adjustments for future Minecraft updates).

College

I have been ambivalent about college since late 2019. That ambivalence has now snowballed into an active distaste for it. The reason is simple: I haven't been intellectually stimulated enough by my classes.

Therefore I have decided to put college on hold for the coming year, working on Branch and some personal projects. The first few days of not having class have been extremely liberating to me. I've been able to focus on things that really stimulate me - see for instance my recent work on writing VarInts very quickly. It's been exciting to be liberated from college work. I want to keep this up and have a good balance of work and leisure for once.

As for the blog...

This blog will get periodic updates. I can not and will not promise when things will get posted, though I have provided an RSS feed in case you want to subscribe. I promise to write about interesting things.

I also decided to change up the hosting situation. My previous blog was hosted with Cloudflare Workers Sites. Workers is a great product, but I did not wind up using its full functionality. I decided to use another recently-introduced Cloudflare solution, Cloudflare Pages. With the fast build times of Eleventy, I think it's a good match!

Tuning Minecraft for OpenJ9

2019-05-25T00:00:00Z

Disclaimer

Mineteria is long gone. It stopped using OpenJ9 in April 2020. I have limited interest in Minecraft nowadays. I'm keeping this post up since lots of links to it exist on the internet. Follow the advice in this post at your own risk.

Andrew Steinborn, October 16, 2022

Original introduction

Mineteria ran the OpenJ9 JVM from July 2018 to April 2020. Our own experience with OpenJ9 has led us to continually tweak our set up and ensure that our servers are not subjected to long garbage collection pauses.

Our servers (which span long-lived lobbies, to a creative-like server with frequent chunk loading, to short-lived game servers), while likely not representative of all Minecraft servers, have been using these flags for many months and have virtually eliminated any issues with garbage collection pauses.

My research started with Aikar's flags for G1 and moved towards how to apply a similar set of flags for OpenJ9. He can not be thanked enough for both inspiring my OpenJ9 garbage collection research and providing a well-tested set of flags to improve server performance.

Why use OpenJ9?

The biggest reason why OpenJ9 appealed to me was because it was container-friendly. This was important as the Linux kernel was reaping our Java processes. We were strapped for cash and so upgrading VM instances was a last resort option, and we did not have the capability to turn off OOM killing in our containers because our setup was based on a managed Kubernetes solution (at the time, it was Google Kubernetes Engine - we have since moved to DigitalOcean Kubernetes).

Up until then, we had been using the standard HotSpot VM. HotSpot is a very proven JVM implementation, but only in Java 10 did it become container-aware. OpenJ9 goes much further and has a flag (-Xtune:virtualized) which optimizes the JVM for lower CPU usage. Funny enough, my adventure with OpenJ9 began when I was tuning our HotSpot-based setup for better garbage collection performance with Aikar's flags in preparation for our beta launch in July 2018.

This primary consideration helped sell us, but OpenJ9 has also helped deliver lower memory usage, which is important for Minecraft servers as they often have a lot of chunks loaded. In our particular case, we have cloud virtual servers focused on providing more CPU and varying memory needs for each server, so a set of flags that was easily tuned was crucial.

Why does it matter for me?

Not everyone uses containers, but everyone can still benefit from OpenJ9's many unique features compared to OpenJ9:

Idle garbage collection: when your server is idle, OpenJ9 can perform a full garbage collection to clean out the tenured space and nursery without negatively impacting performance.
Shared classes and ahead-of-time compilation mean that your server can restart and ramp back up much more quickly compared to HotSpot.
Lower memory usage compared to HotSpot.
The best part is that in most cases, you can just drop in OpenJ9 and your plugins will "just work".

Tuning the OpenJ9 VM

Now with some reasons why to use OpenJ9 out of the way, we can turn to the Minecraft-tuned garbage collection settings for OpenJ9. These settings will result in short (usually 1 to 5 millisecond) GC pauses, which is quite good. The memory settings will need to be tweaked depending on your memory usage, but I'm providing a shell script that will take care of the entire process for you.

TL;DR: Here's a shell script that takes care of the entire process. Just drop it in your server directory as, e.g. start.sh, edit it to tweak your memory setting, chmod +x start.sh, and then ./start.sh. It will "just work" with no further set up required. For your convenience, I've also made it wgettable.

#!/bin/bash

#
# Properly tunes a Minecraft server to run efficiently under the
# OpenJ9 (https://www.eclipse.org/openj9) JVM.
#
# Licensed under the MIT license.
#

## BEGIN CONFIGURATION

# HEAP_SIZE: This is how much heap (in MB) you plan to allocate
#            to your server. By default, this is set to 4096MB,
#            or 4GB.
HEAP_SIZE=4096

# JAR_NAME:  The name of your server's JAR file. The default is
#            "paperclip.jar".
#
#            Side note: if you're not using Paper (http://papermc.io),
#            then you should really switch.
JAR_NAME=paperclip.jar

## END CONFIGURATION -- DON'T TOUCH ANYTHING BELOW THIS LINE!

## BEGIN SCRIPT

# Compute the nursery size.
NURSERY_MINIMUM=$(($HEAP_SIZE / 4))
NURSERY_MAXIMUM=$(($HEAP_SIZE * 2 / 5))

# Launch the server.
CMD="java -Xms${HEAP_SIZE}M -Xmx${HEAP_SIZE}M -Xmns${NURSERY_MINIMUM}M -Xmnx${NURSERY_MAXIMUM}M -Xgcpolicy:balanced -Xdisableexplicitgc -jar ${JAR_NAME}"
echo "launching server with command line: ${CMD}"
${CMD}

## END SCRIPT

With the default settings, your server will launch with java -Xms4096M -Xmx4096M -Xmns1024M -Xmnx1638M -Xgcpolicy:balanced -Xdisableexplicitgc -jar paperclip.jar.

We will break down each of these flags, but there is other important context to get out of the way.

Sizing the heap

The first and natural set of flags to turn to are our humble friends -Xms and -Xmx. You should set -Xms and -Xmx to the same value, whether you use HotSpot or OpenJ9.

In the case of OpenJ9, this is even more important because OpenJ9 tends to be quite conservative in requesting more memory from the operating system. In our experience, when the heap is insufficient to satisfy allocations, OpenJ9 will prefer to perform aggressive garbage collection first and only when it can't free up a significant amount of memory will it request more from the operating system. This is bad because it will degrade server performance due to frequent garbage collection pauses.

Now we need to actually decide on the heap size. The best policy to take here is to allocate as much as you reasonably can, but make sure to leave enough room for the operating system and for the JVM's overhead, otherwise the operating system may kill the server to free up memory. For instance, if you have a 4GB VPS, you should stay in the 2-3GB range so that the operating system will still run smoothly and the JVM can allocate overhead.

Selecting a GC policy

OpenJ9 features five different garbage collection policies:

gencon: This is the default GC policy in OpenJ9. This policy breaks the Java heap into a nursery and a tenured space. All objects are allocated initially into the nursery. If an object survives a certain number of collections while in the nursery, it will be moved into the tenured space, which is only collected when it is full.
balanced: This is the closest analogue that OpenJ9 has to the HotSpot G1 garbage collector. It breaks the heap into regions, each of which is individually managed and garbage-collected.
optthruput: This policy is similar to optavgpause but does not mark concurrently, resulting in longer pauses. Because of this, it is a poor choice for a Minecraft server.
optavgpause: This is a fully concurrent mark-and-sweep collector which aims to minimize the pause time. It is however mostly suited for very large heaps.
metronome: This is a policy available on AIX and x64 Linux. It is interesting as it places a focus on minimizing GC pause times to ensure precise response times. However, we will not need this policy for Minecraft.

For Minecraft 1.12.2 and below, the right answer was the default gencon policy. However, with 1.13 and above, gencon proves to be a bad fit for servers due to the severely increasedstrain on the garbage collector subsystem. The balanced policy is the next-best.

Sizing the nursery

Now that we have selected our desired GC policy and sized our heap, we need to tune the behavior of the garbage collector.

The size of the nursery is key to improving Minecraft performance. By default, -Xmns is 25% of -Xms and -Xmnx is 25% of -Xmx. Since you will be setting -Xms and -Xmx to the same value. This means that by default, the nursery will never grow. Since Minecraft tends to allocate lots of very short-lived objects, this will result in very frequent garbage collections and will tenure garbage objects more often, degrading server performance.

Instead, you should set -Xmns to 25% of your -Xmx and -Xmnx to 40% of your -Xmx. This allows OpenJ9 to expand the nursery when needed and is a more realistic nursery size for a Minecraft server.

Disabling explicit GC

The other flag we provide is -Xdisableexplicitgc, which protects us from plugins that think they're being smart by asking the garbage collector to do a collection when one is not needed, and may actually hurt server performance later on by bringing a full GC cycle closer.

Other settings to change

If you are on a VPS or some sort of cloud environment, you should add in the -Xtune:virtualized flag. This tunes OpenJ9 to reduce CPU and memory usage.

My script does not configure OpenJ9's ahead-of-time compiler or the shared classes functionality. This can improve startup times, but we can't fully exploit this feature due to the unique peculiarities of our setup.

Andrew Steinborn

☄️ Precipices

On the Precipice of Success #

On the Precipice of Being Undone #

On the Precipice of Being Ruined #

On the Precipice of Being Lost #

On the Precipice of Being (Un)hired #

On the Precipice of Security #

🌀 Being nonlinear: A transition from modding

You add value! #

Where do you want to work? #

Have a solid foundation #

Prepare to have lowered expectations #

Being in the workplace #

Project management #

Professional writing #

Interacting with coworkers #

Having respect for workplace norms #

Actually getting a job #

Building a resume #

Networking #

Finding employers willing to take a bet on you #

Tying it all together: my experience #

Getting my job #

My first year at Ramp #

Conclusion #

📚 A 2024 dispatch

Sorrow #

Writing #

Ramp, Ramp, Ramp! #

Travel #

Miami #

Washington, D.C. #

Prediabetes, eating disorders, and weight loss #

Finally, a callout to an old colleague... #

🛠️ Fixing a broken Fedora 38 to 40 upgrade

Disclaimer #

Waiting for an upgrade to complete #

The reboot #

Fixing the partially upgraded system #

A Python scavenger hunt #

Fooling the system into thinking it's Fedora 40 #

The strategy #

Removing duplicate packages #

The distro-sync and solving the mystery of my long upgrade #

Almost there #

So, what did I learn? #

🎨 Sprucing up my website for 2024, part 2

So why now? #

So what's new? #

New font stack #

New color scheme #

Improved code syntax highlighting #

Get dropped off on the blog #

A touch of animation #

What's next? #

Update, late on October 22, 2024 #

✍️ The Velocity Chronicles, Part 0

What exactly is a Minecraft server proxy? #

First, the story of my first Minecraft server #

Well, Andrew, how involved were you in the "proxy scene"? #

You said you'd discuss another Minecraft network, too? #

Swiping left on the cloud #

Velocity is born #

Who are you calling, me? #

So, how do we fool the client? #

That's all for this installment #

🎨 Sprucing up my website for 2024

First steps first #

Tweaking the theme #

So, what's next for you? #

📚 A 2023 dispatch

Dis is one half. #

My Spend Platform era #

That's all for now #

🕯️ Going back home, a tapestry of time

Context #

🐦🔥 Misadventures with Phoenix Framework

...but not quite! #

So what's our strategy? #

On the Precipice of Success

On the Precipice of Being Undone

On the Precipice of Being Ruined

On the Precipice of Being Lost

On the Precipice of Being (Un)hired

On the Precipice of Security

You add value!

Where do you want to work?

Have a solid foundation

Prepare to have lowered expectations

Being in the workplace

Project management

Professional writing

Interacting with coworkers

Having respect for workplace norms

Actually getting a job

Building a resume

Networking

Finding employers willing to take a bet on you

Tying it all together: my experience

Getting my job

My first year at Ramp

Conclusion

Sorrow

Writing

Ramp, Ramp, Ramp!

Travel

Miami

Washington, D.C.

Prediabetes, eating disorders, and weight loss

Finally, a callout to an old colleague...

Disclaimer

Waiting for an upgrade to complete

The reboot

Fixing the partially upgraded system

A Python scavenger hunt

Fooling the system into thinking it's Fedora 40

The strategy

Removing duplicate packages

The distro-sync and solving the mystery of my long upgrade

Almost there

So, what did I learn?

So why now?

So what's new?

New font stack

New color scheme

Improved code syntax highlighting

Get dropped off on the blog

A touch of animation

What's next?

Update, late on October 22, 2024

What exactly is a Minecraft server proxy?

First, the story of my first Minecraft server

Well, Andrew, how involved were you in the "proxy scene"?

You said you'd discuss another Minecraft network, too?

Swiping left on the cloud

Velocity is born

Who are you calling, me?

So, how do we fool the client?

That's all for this installment

First steps first

Tweaking the theme

So, what's next for you?

Dis is one half.

My Spend Platform era

That's all for now

Context

...but not quite!

So what's our strategy?

Writing a Mojang API client

Building our first endpoints

Caching and rate-limiting

Time to bring in the Rustler!

A very dirty NIF

Did you enjoy it?

Epilogue

A short recap of 2021

The start of 2022

🗽 Welcome to New York

🌴 Ramp Builders 2022