Who does not love iOS? It’s a great operating system. However I can tell you about a type of person that has a love/hate relationship with iOS: engineers who have to debug crashes on iOS devices. iOS makes debugging crashes trickier than most environments which in turn makes the job of tools like Sentry that much harder. In this blog post we want to give you a bit of insight into how Sentry deals with iOS crashes and what is necessary for you to have an enjoyable iOS crash reporting experience.

Crashing in the First Place

The first part that goes into a crash is to generate a report we can actually send to Sentry. For this to work you need something that can generate you a basic backtrace at the moment a crash happens. There are two popular libraries for iOS that can do that. One is KSCrash, the other is PLCrashReporter. Those two libraries hook into different parts of the OS to respond to errors and to extract a backtrace. This in itself is already a very complex undertaking and we’re glad that others have done this task for us.

There are many different situations that can cause crashes and each of them has different characteristics. I don’t want to get too much into detail here but it’s important to understand that not all crashes will result in the same quality of reporting. An extreme case for iOS are C++ exceptions which will not create proper backtraces on iOS because of how the exception system works.

When we manage to report a stacktrace and some important data on crashing we persist that temporarily on the device. Next time you start the application we send that information to the server. The stacktrace is the most interesting part and this also is the first complexity that spills over to the server side.

To give you a bit of an idea what this looks like, here is an example stacktrace after you extract it:

CrashLibiOS                     0x100077c4c
CrashProbeiOS                   0x200050220
UIKit                           0x31d104d30
UIKit                           0x31d104cb0
UIKit                           0x31d0ef128
UIKit                           0x31d10459c
UIKit                           0x31d68f628
UIKit                           0x31d68b6c0
UIKit                           0x31d68b1e0
UIKit                           0x31d68a49c
UIKit                           0x31d0ff30c
UIKit                           0x31d0cfda0
UIKit                           0x31d8b975c
UIKit                           0x31d8b3130
CoreFoundation                  0x3111ffb5c
CoreFoundation                  0x3111ff4a4
CoreFoundation                  0x3111fd0a4
CoreFoundation                  0x31112b2b8
GraphicsServices                0x314690198
UIKit                           0x31d13a7fc
UIKit                           0x31d135534
CrashProbeiOS                   0x20004f2a4
libdyld.dylib                   0x30f0f65b8

So the first step would be to find some names for those addresses. This process is often called “symbolizing” or “symbolicating”. We can already see where the addresses are located because the device sends us a list of loaded images (object files) and where they are loaded into memory. To find the names we need to look at symbol tables.

Stacktraces on iOS

So as you can see stacktraces are fairly incomplete. While we can easily find out what frameworks the addresses are contained in, it’s unlikely that you will be able to find the function names for them on the device. There are two cases you have to keep apart here. One case is where the symbols are in fact missing, the other one is where symbols are marked as redacted.

Missing symbols are typically what you have in release builds for your own applications. In release builds most of the symbols you encounter are not actually on the device so if we were to try to locate the function names on the device we will not succeed. Instead they are stored in what is commonly referred to as a “dsym file”. Technically a dsym file is a macho file just like an executable but it only contains the symbol table and debug information. So while they could be in the same file, they usually are not. When I said that “most” symbols are not on the device, this refers to the fact that some symbols need to be in the file. This is because most applications on iOS are written in Objective-C. This is relevant because Objective-C implements methods through a mechanism that is based on the idea of sending messages from object to object. These messages are referred to as “selectors” and they are essentially the name of the method.

PLCrashReporter and some other tools are often attempting to find such symbols even if the normal symbols are not on the device, however for the bulk of the symbols you need to do this on the server.

The second case of missing function names we need to concern ourselves with is a weirder one: redacted symbols.

Redacted Symbols

Redacted symbols are symbols that are indeed available on the device but tools like KSCrash or PLCrashReporter cannot access. When iOS loads system libraries it removes symbols so that when one attempts to read the symbol by parsing the framework one will only come across a symbol with the name <redacted> .This is most likely done to save some memory or for security reasons. Because all system frames will have the same constant string as a symbol there is a lot that does not have to be loaded into memory.

The downside is that we are not able to tell you which function in UIKit caused your crash. When you hook your phone up to Xcode you can see such symbols though. So how does that work? The answer is a bit bizarre and requires some understanding of what happens when iOS loads the system libraries.

When iOS redacts symbols it stores a copy of the original symbol on the file system in a cache file that is not accessible for non rooted devices. The file is named dyld_shared_cache_arm64 for arm64 etc. From the file name you can see that this is considered a cache file. This means the file is updated as redacted symbols are added to it. Apple built this system to primarily support the flow where you debug your own device. If you run your own app and you hook it up to the debugger all the frames that you are interested in will have their redacted symbols added to the cache file. When you connect the phone to Xcode, Xcode will go in and “prepare the device for development” and that will essentially download the cache file and run it through a process where dummy debug symbols are built for it. It will in fact create a folder structure below ~/Library/Developer/Xcode/iOS DeviceSupport for your version of iOS and put new macho files in there with symbols recovered from the cache file.

Now you can guess what the problem with this is: if you have never seen a symbol it won’t be in the cache file. This is particularly noticeable if you are working with “legacy” architectures. For instance if you hook up an arm64 device with Xcode it will be able to extract some armv7 symbols but it will most likely not find all. Your chances are most likely higher if you are running a lot of 32bit apps to populate the cache, but you might as well just hook it up with an older device instead. Whenever you add a device to Xcode it will merge together the symbols it extracts.

This shows one of the core issues that come up with symbolizing on iOS: you need to collect as many of these debug symbols as possible.

Symbolicating App and System

Sentry is using two separate systems for resolving functions. For customer debug symbols we are using our own LLVM based symbolication library for Python. We fetch debug symbols from our S3 backed asset storage and then symbolicate based on the symbols we have on our device. This scales quite well to the workload caused by apps. These are typically large symbol files but there are not that many per app.

On the other hand dealing with symbols from the system libraries is a different story. There are thousands of symbol files and because the cache might be incomplete we actually want to be quite fuzzy over them. As example for this fuzziness is that we might be dealing with incomplete debug symbols for system libraries from one SDK. In that case we want to try a few older versions as well in case we find matches there.

To achieve this goal we wrote a separate system we call the sentry symbol server and it is a simple HTTP service written in Rust that takes a batch request of addresses to symbolicate and then responds with the function names if it finds them. It uses a custom file format that can be memory mapped in. We then use a separate build process to create these mmap’ed files and put them to S3. In regular intervals the server checks back with S3 and fetches new memory maps if necessary.

The Final Result

After symbolication our boring crash report from before looks more like this:

CrashLibiOS        -[CRLCrashNULL crash] (CRLCrashNULL.m:37)
CrashProbeiOS      -[CRLDetailViewController doCrash] (CRLDetailViewController.m:53)
UIKit              -[UIApplication sendAction:to:from:forEvent:]
UIKit              -[UIControl sendAction:to:forEvent:]
UIKit              -[UIControl _sendActionsForEvents:withEvent:]
UIKit              -[UIControl touchesEnded:withEvent:]
UIKit              __UIGestureEnvironmentSortAndSendDelayedTouches
UIKit              __UIGestureEnvironmentUpdate
UIKit              -[UIGestureEnvironment _deliverEvent:toGestureRecognizers:usingBlock:]
UIKit              -[UIGestureEnvironment _updateGesturesForEvent:window:]
UIKit              -[UIWindow sendEvent:]
UIKit              -[UIApplication sendEvent:]
UIKit              ___dispatchPreprocessedEventFromEventQueue
UIKit              ___handleEventQueue
CoreFoundation     ___CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
CoreFoundation     ___CFRunLoopDoSources0
CoreFoundation     ___CFRunLoopRun
CoreFoundation     _CFRunLoopRunSpecific
GraphicsServices   _GSEventRunModal
UIKit              -[UIApplication _run]
UIKit              _UIApplicationMain
CrashProbeiOS      main (main.m:16)
libdyld.dylib      _start

And this allows us then to render the iOS crash report in a more presentable way. Because we know which symbols are from your app and which ones are from the system we can by default hide frames you likely don’t care about:

In an Ideal World

In an ideal world Apple would provide a web service that does what our symbol server does. You give it the UUID of the image you want to symbolicate and the address in it, and you get back a response of the symbol that is at that address. At present the process of collecting all the symbols from different SDK versions is slow, requires a lot of manual labour and is not even guaranteed to always succeed.

Future Plans

Sadly we are limited to providing system symbol resolving on our cloud hosted version. There are some concerns about the redistribution of system symbol files which is why we currently cannot offer this service for on-prem customers.

If you are interested for support for system symbol symbolication for on-prem installations leave your feedback in the forums. We are playing with the idea of making our symbol server a public API in case there is demand for it.

If this article was of interest of you let us know. We might do a followup where we explain our heuristics and the technical challenges on doing server side symbolication.