Welcome to my first edition of War Stories, where I partner with famous builders in the iOS space to share real-life stories from the frontlines as they scale their projects.
Today I’m working with Pedro Piñera Buendia, the creator of Tuist, which is designed to fix the problems shared by iOS teams as they scale Xcode projects. I have not been paid to endorse Tuist, but I do really like it.
I’m still working out the format here, so please be patient whenever I accidentally switch between first and third person. It’s part narrative, part interview, all swashbuckling fun.
To catch all my War Stories, be sure to subscribe:
Without further ado, here’s The Great Connection Pool Meltdown.
Hard Things
Like many readers of Jacob’s Tech Tavern, the Tuist guys are iOS developers turned founders. Web infra at scale abruptly switches from an interview buzzword to cold reality. This is one of those Hard Things™ that we are often shielded from when painting JSON on the client-side.
One of Tuist’s core features is caching. It builds xcframeworks from each module and caches them remotely, which dramatically cuts cold build time for Xcode projects, both locally and on CI (Tuist newly integrates with Xcode 26 Compilation Caching to make it even faster).
To power a cache, you need a server. The Tuist caching server is coded in Elixir, which compiles down to Erlang bytecode and runs on the Erlang runtime. It’s famous for being great at building highly parallel, real-time services.
Tuist’s code is actually open-source if you’re interested.
The HTTP cache server processes traffic from organisations with hundreds of devs, with bursts of traffic every time a project is freshly generated. This can rack up ~100k requests per hour. Given an average xcframework module can be 5-10 MB, adds up to (approximately) a metric f*ckton of traffic.
This was all well and good, but strange errors began 6 months ago.
The iOS Dev’s Blind Spot
We are coddled by the Apple ecosystem. URLSession does a lot under the hood like HTTP/2 multiplexing, interfacing with system daemons that manage the radio, and maintaining an internal TCP connection pool.
This stuff will make a pretty good article actually, I’ll add it to my list.
Building on the server means you must keep TCP connections warm using a connection pool, or your users will suffer cold starts, leading to slow response times.
A caching server is a high-throughput environment, but when you’re linked to CI builds, the access pattern is actually very spiky. Every organisation onboarded to Tuist adds thousands of sporadic requests to the server as devs sprint to generate projects and merge PRs.
This leads to unpredictable surges in traffic on the server. These could queue up thousands of operations waiting to be executed, from DB queries to Postgres (fetching cache metadata), to external GET requests to Amazon S3 storage (where the xcframeworks are stored).
Each millisecond of latency counts on a cache server. You can optimise your Elixir code as much as you like, but there’s no getting around “fixed costs” like disk I/O on the database or the network latency when fetching from external storage.
If you can serve these requests in time, great.
If not, bad things can happen.
The sh*t hits the fan
The early warning signs were subtle at first.
Daily connection drop warnings were flagged from the server, because they sat in the queue too long. Users weren’t complaining, since the client would just retry failed connections, but it was annoying to begin every workday like this:
The team began to talk to their cloud provider (a popular wrapper who will remain nameless) about potential network issues in their caching server instance, but they would mostly push back and blame Pedro’s config.
The cloud provider doth protest too much, methinks.
One day, things got real bad. Contagion. The network issue spread out across the database provider and their S3 storage, proving it had nothing to do with app configuration.
All of a sudden, the server crashed.
It was a particularly high-traffic spike that toppled the service. The Tuist guys spun up the server again. It went down again.
The cloud provider’s network issue was failing to process many of the S3 requests that fetched cached xcframeworks. The failing requests sat around in the queue, waiting to retry. This hogged connection pool slots, wasted CPU cycles, and rapidly filled up application memory until the server crashed again.
The CLI client kept retrying with exponential backoff, so the server immediately got hit with thousands of requests each time it recovered.
At best, users didn’t notice much, because the Tuist client fell back to locally-cached modules. At worst, this incident manifested as time-outs on cache fetches, with the client URLSessionConfiguration set up to kill sessions after 5 minutes. This blocked CI builds and newly-onboarding developers who needed to fetch from the cache.
For what it’s worth, I’ve used Tuist at work and their team responds (and usually unblocks us) very fast on Slack anytime there’s a problem.
Firefighting mode.
Well, mostly.
Pedro was in a taxi to the hospital for surgery* as this all went down, but the 3-strong engineering team coordinated the response over Slack.
*Sick days are one of the casualties of becoming a founder 🥲