The Problem We Were Actually Solving I was the lead systems engineer on a project to build a treasure hunt engine, a highly concurrent and real-time system that had to handle thousands of players simultaneously. The engine was initially built using Elixir, a language I had grown fond of due to its concurrency model and ease of development. However, as our server grew and the number of concurrent players increased, we started to notice significant performance issues. Our latency numbers were rising, and our allocator was running out of memory. I spent countless hours profiling the system, using tools like erlang-vm-profiler and fprof, to understand where the bottlenecks were. The profiler output showed that our Elixir code was spending a significant amount of time in garbage collection, which was causing our latency to spike. The allocation counts were staggering, with over 10 million allocations per second, most of which were temporary and short-lived.…