Subscribe to Jacob’s Tech Tavern for free to get ludicrously in-depth articles on iOS, Swift, tech, & indie projects in your inbox every week.
Sometimes, your app crashes.
Sh*t happens.
Usually this is annoying, but straightforward to fix. Maybe you messed up the lifetime of a reference, or perhaps there was some subtle race condition you didn’t consider.
But there’s a class of crash far more insidious.
Crashes which originate deep in the kernel memory management system, where the debugger refuses to narc on the culprit.
This crash trace yields sparse detail:
It’s the main thread, and something concurrency related.
EXC_BAD_ACCESSsuggests we’ve tried to access deallocated memory.
Lady Luck is smiling on us: mercifully, the crash is 100% reproducible. Unfortunately, it lies on a core user flow and slipped through to production.
We need to pull out our advanced tools. In a pinch, there are a couple of techniques to help pinpoint the source of the crash:
Git Bisect to narrow down through time.
Manually Symbolicating Crash Logs to narrow down across space.
Today’s blog applies these advanced approaches to solve a real-life mystery crash.
Git Bisect
When you work in a fast-paced team shipping dozens of commits every day, bugs might take time to spot. When they are finally noticed, the offending code change might be dozens or even hundreds of commits in the past.
Identifying the guilty commit, where the bug or crash first appears, dramatically simplifies our job. It narrows down the possible error surface to a few hundred lines of code.
git bisect is a tool — really, a wizard — that helps locate the problematic commit using a binary search algorithm.
Running Git Bisect
You can run git bisect from the command line, and it happily displays the commands available:
% git bisect
usage: git bisect [help|start|bad|good|new|old|terms|skip|next|reset|visualize|view|replay|log|run]To get it running, just use git bisect start:
status: waiting for both good and bad commitsTo begin the binary search, you need to pass SHA hashes for known good & bad git commits. For example, you might know the latest code on the main branch is bad, but the code from last week’s release 1.2.0 is good. Let’s pass git the hash of this latest known bad commit:
git bisect bad a4e0c6b1f3a9d7e2f5c0a8d3b9e4f1c7b5a2d6e0
status: waiting for good commit(s), bad commit knownWe can then pass in the hash for the latest known good commit:
git bisect good f1c4b9a0e7f3d2a5c6e1b8f9d4a3c7b5e0a6f2d88
Bisecting: 592 revisions left to test after this (roughly 9 steps)
[b8d1f5a3e0c6d7f2a4b9e3f0a1d8c5e7b6a2f9d3] DEV-1230 | Social feed added (#1202)I called this a wizard because git bisect performs the heavy lifting for you. It checks out the branch mid-way between the current most-recent-good & oldest-bad commits.
You can build & run the code, then perform the repro steps to check for the bug or crash. Inform git each time by using plain git bisect good or git bisect bad commands.
% git bisect good
Bisecting: 296 revisions left to test after this (roughly 8 steps)
[1a2d4f65b3e6c9f8d7a0b1c2e3f4a6d5c8b7a0e3] DEV-1020 Add friends feature (#1020)As git automatically checks out our code, we can intuitively begin to narrow down the possible root cause of the crash.
% git bisect bad
Bisecting: 148 revisions left to test after this (roughly 7 steps)
[a4e0c6b1f3a9d7e2f5c0a8d3b9e4f1c7b5a2d6e0] DEV-1102 Delete account (#1090)If your code is hosted on Github, you might begin to look at the actual code diffs between the last good and first badcommits by going to github.com/your-org/your-repo/compare.
This webpage illustrates every line of code changed between the known good and bad commits. This line count rapidly decreases as you narrow down the search.
Eventually, you get to a final point, the culprit, where git illustrates some of the detail on the offending commit itself:
git bisect bad
f5a7c2d0e8b9f1a6d3c5e4b7a9d2f0b8e3c1f9a4 is the first bad commit
commit f5a7c2d0e8b9f1a6d3c5e4b7a9d2f0b8e3c1f9a4
Author: Jacob Bartlett
Date: Mon Nov 19 21:12:10 2024 +0000
DEV-1099 Introduce concurrency in logout (#1076)
.../Core/Auth/Logout.swift | 240 ++++++++++++++++-------
1 file changed, 222 insertions(+), 18 deletions(-)This technique has taken us far: instead of our entire codebase, we now only have 240 lines of code to investigate in solving our insidious crash.
Words of Wisdom
Outside of the wizard itself, there are a few important things to understand about git bisect when you apply it yourself.
First and foremost, this is an extremely tedious elimination process. Depending on your build system, the initial commit range, and the complexity of your app, this can take hours. It’s fantastic to have in your toolkit, but only worth pulling out when there’s no clear solution.
My colleague once told me, you need to keep your eye on the ball with git bisect. The wizard does so much heavy lifting for you, it’s easy to spend the whole day on autopilot. That’s a mistake — if you log a single wrong good/badcommit, your results are worthless. Clean your builds, test, and then re-test!
Git Bisect Run
If you’re dealing with a build error; or an issue that you can detect via test scrips; there’s an advanced approach called git bisect run. This was used to automatically detect errors in the Linux source code:
git bisect start linux-next/master v2.6.26-rc8
git bisect run make kernel/fork.ogit bisect start defined known bad and good commits; then git bisect run automatically performs the bisection.
It checks out the next mid-point commit to try, then runs the script or command. If this returns exit code 0, it logs good; otherwise, it’s bad. Then it automatically checks out the next commit and narrows down the culprit automatically.
If you’re desperate for more
git bisectcontent, you can look at the source code for git bisect right here on Github.
We’ve bisected in time, and dramatically narrowed down the troublesome code surface area.
Next, we can narrow down the error in space by manually symbolicating crash logs.
Symbolicating Crash Logs
This technique is a subtle one, and frankly you’re unlikely to apply it unless you’re in real trouble — mystery crashes are one of a handful of use cases. I’ve encountered a few scenarios where manual crash log symbolication can help in a tight spot:
The crash is only in release builds, so you can’t reproduce when plugged into the debugger.
Testing internal CI builds, where Xcode crash reports aren’t available.
Crashes on an app extension or app clip that doesn’t have access to your crash reporting SDK.
Something went wrong with your dSYM upload pipeline.
To manually symbolicate crash logs, you need just two things:
The
.ipsfile containing the crash report, stored on the device that crashed.The same build’s corresponding
.appfile or.dSYMfile. You’ll find this in your CI artefacts, or Xcode archives.
Exporting Crash Reports
Let’s get that crash report first.
For iOS 18 users, you can go into:
iOS Settings
Privacy & Security
Analytics & Improvements
Analytics Data
Select the timestamped
.ipslog for the crash you need
Once you export the log to your MacBook, the real fun begins. Any random crash log will look something like this:
Parsing this elaborate crash log file might seem overwhelming at first, but there’s actually just a few simple steps you need to convert this into something useful.