1. Countly
  2. Knowledge Base
  3. Hidden Section
  4. User Guides: Countly v20.11

Crash Symbolication

Follow
Last modified:

Crash Symbolication lets you symbolicate/de-obfuscate crash reports and convert them into human readable format, helping you pinpoint where in your code a crash is originating from.

Availability

The Symbolication plugin is available only in Countly Enterprise.

Understanding Crash Symbolication

What is "Symbolication"?

When releasing your application, you might be skipping crucial steps such as debugging information from the final binary format, obfuscating your source code, or minifying it by removing unnecessary characters. Doing all this can help make your application harder to reverse engineer, but you may then receive error stack traces that have obfuscated information in them, making it impossible to track down the source of your crash.

Depending on the platform, your stack trace may be full of memory addresses or transformed function names. Symbolication is the process of converting them into human readable, class/method names, file names, and line numbers.

You can watch a tutorial about how symbolication works below.

Symbolicating iOS, Android, and JavaScript Crashes

Save your symbol or mapping files!

For the obfuscation and minification processes to be reversible, a symbol or mapping file is also produced while building your application. It is imperative for you to keep and archive those files as the symbolication process is only possible with them. You will also need to archive symbol files for every version you would like to symbolicate.

Currently, Countly supports symbolication for DexGuard and ProGuard in Android, Apple-provided tools in iOS mobile applications, and source maps in JavaScript web applications. Find examples for the input and output for each one below.

Android Symbolication Samples

Android Symbolication input sample Android Symbolication output sample 
java.lang.IllegalStateException: Could not execute method for android:onClick
at android.view.View$DeclaredOnClickListener.onClick(View.java:4725)
at android.view.View.performClick(View.java:5637)
at android.view.View$PerformClick.run(View.java:22429)
at android.os.Handler.handleCallback(Handler.java:751)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6121)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at android.view.View$DeclaredOnClickListener.onClick(View.java:4720)
... 9 more
Caused by: java.lang.Exception: Exception at the end of the call
at ly.count.android.demo.a.b(SourceFile:29)
at ly.count.android.demo.a.a(SourceFile:21)
at ly.count.android.demo.ActivityExampleCrashReporting.c(SourceFile:98)
at ly.count.android.demo.ActivityExampleCrashReporting.b(SourceFile:94)
at ly.count.android.demo.ActivityExampleCrashReporting.a(SourceFile:90)
at ly.count.android.demo.ActivityExampleCrashReporting.onClickCrashReporting10(SourceFile:82)
... 11 more

 

iOS Symbolication Samples

iOS Symbolication input sample iOS Symbolication output sample 
libsystem_platform.dylib            0x00000001826c130c _sigtramp + 36
mahya                               0x000000010006e174 mahya + 156020
mahya                               0x000000010006d060 mahya + 151648
mahya                               0x000000010006ad34 mahya + 142644
UIKit                               0x0000000189925d74  + 184
UIKit                               0x00000001898eedb8  + 128
UIKit                               0x00000001898ff2b0  + 88
UIKit                               0x000000018a1c0ff8  + 272
UIKit                               0x0000000189b5e138  + 280
UIKit                               0x0000000189b5d96c  + 1064
UIKit                               0x0000000189b5d4c4  + 60
UIKit                               0x000000018a1c0ff8  + 272
UIKit                               0x0000000189b5d400  + 436
UIKit                               0x00000001898eea40  + 672
UIKit                               0x00000001898ee364  + 1780
UIKit                               0x00000001898ec954  + 228
UIKit                               0x00000001898eab04  + 3152
UIKit                               0x0000000189b74750  + 228
UIKit                               0x000000018975ea50  + 384
UIKit                               0x0000000189b74400  + 344
UIKit                               0x0000000189768fd4  + 2480
UIKit                               0x000000018976436c  + 3192
UIKit                               0x0000000189734f80  + 340
UIKit                               0x0000000189f2ea20  + 2400
UIKit                               0x0000000189f2917c  + 4268
UIKit                               0x0000000189f295a8  + 148
CoreFoundation                      0x00000001835b142c  + 24
CoreFoundation                      0x00000001835b0d9c  + 540
CoreFoundation                      0x00000001835ae9a8  + 744
CoreFoundation                      0x00000001834deda4 CFRunLoopRunSpecific + 424
GraphicsServices                    0x0000000184f49074 GSEventRunModal + 100
UIKit                               0x0000000189799c9c UIApplicationMain + 208
mahya                               0x000000010005fe14 mahya + 97812
libdyld.dylib                       0x00000001824ed59c  + 4

 

JavaScript Symbolication Samples

JS Symbolication input sample JS Symbolication output sample 
ReferenceError: undefined_function is not defined
    at r (file:///home/atak/Work/Countly/sample-app/dist/main.js:2:140)
    at HTMLButtonElement.document.getElementById.onclick (file:///home/atak/Work/Countly/sample-app/dist/main.js:2:521)

Configuring the Server for Symbolication

The first step for using symbolication is installing and enabling the Symbolication plugin.

To do so, in the main Countly Dashboard, go to Management > Plugins and enable the Crash Symbolication toggle.

After that, you will find Manage Symbols in the Improve section of your Countly Dashboard, under the Crashes category.

Configuring iOS Symbolication

Setting up a connection to the Countly symbolication server is only needed for symbolicating iOS stack traces. You can skip it if you will not be using Symbolication for iOS applications.

For iOS applications, the next step is creating the connection to the Countly symbolication server. To do so, you will need a symbolication API key provided by Countly. The Countly symbolication server address is https://symbolication.count.ly. When you get your API key, you will need to set it in the server configuration screen in Management > Settings, under the Crashes section.

After you have made sure that those fields contain valid information, click the Test Connection button to make sure everything is working correctly. If your provided server URL or API key is wrong or your Countly server cannot reach the symbolication server, you may diagnose it with this button.

If everything is working correctly, you will see the alert messages in the image below.

You are now done setting up your Countly instance to use iOS Symbolication.

Preparing Your App for Symbolication

This section will guide you through the Android, iOS, and JavaScript symbolication processes.

Android

Android's official tool for code shrinking and obfuscation is called ProGuard. A detailed description of its usage can be found here. There is also a paid tool with additional features called DexGuard. Both ProGuard and DexGuard can be used for Countly crash symbolication. Currently, we do not support any other Android obfuscation libraries.

If you are using Android Studio for development, the mapping files will not be produced when you make instant runs. For them to appear, you will either need to generate a signed APK or choose the Build APK option.

After the build is complete, the symbol file called mapping.txt can be found under <module-name>/build/outputs/mapping/release/ or <module-name>/build/outputs/mapping/debug/, depending on how you initiate the build process.

ProGuard Rules

You have the option of adding some rules to ProGuard (or DexGuard) and modifying how it runs. These rules should be added in the proguard-rules.pro file.

If you are deciding to use ProGuard, you have to keep in mind that it will rename the function and class names. Therefore, it will break the code that is using reflection if you don't take steps to stop this from happening. You may find more information on the ProGuard manual.

At the very least, you should include these lines in your ProGuard rule file:

-keep class org.openudid.** { *; }
-renamesourcefileattribute SourceFile
-keepattributes SourceFile,LineNumberTable

The first one is needed to prevent openudid from breaking, and Countly uses it for generating user IDs. The second and third rules are needed to add source file and line number information to your Android stack traces.

The ProGuard rule file should be named proguard-rules.pro, and it is usually located in the root of your module. More information here.

iOS

The symbol file is a dSYM file for iOS.

  • A dSYM file is Apple's standard Mach-O file which contains debug symbols for a given build.
  • It includes debug symbols for all architectures used for the build (e.g. armv7, arm64).
  • Its size may vary depending on the original source code and libraries used in the project.
  • It is a file-like folder structure, and actual dSYM data is in the binary file AppName.app.dSYM/Contents/Resources/DWARF/AppName.

dSYM Location

First of all, for each executable target (main app, extensions, and Cocoa Touch Frameworks) in the project, a separate dSYM file is generated when the project is built. The dSYM location depends on build settings. By default, it is defined by $DWARF_DSYM_FOLDER_PATH Xcode Environment Variable.

An example location is: ~/Library/Developer/Xcode/DerivedData/AppName-abcdef0123456789/Build/Products/Release-iphoneos/AppName.app.dSYM

In addition, if you use the Product > Archive option in Xcode to create the .xcarchive of your app, you may find a dSYM already created inside the .xcarchive. By default, its location is: ~/Library/Developer/Xcode/Archives/YYYY-MM-DD/AppName DD-MM-YYYY, HH.mm.xcarchive/dSYMs

Automatic dSYM Upload

For automatic dSYM, please see the Countly iOS SDK documentation here.

JavaScript

In order to symbolicate errors, we need a source map file. We will try to lay out how you can produce a source map file for builds that use webpack, you can also consult your build tool's documentation on how to generate one for your builds.

So, for this demonstration, we will use the sample frontend application that is available in the Countly web SDK repository. It is a simple project with just one source file src/index.js that imports a third party dependency, the Countly SDK, and binds a couple of buttons to throw errors.

import Countly from "countly-sdk-web"

Countly.init({
  app_key: "YOUR_APP_KEY",
  app_version: "1.0",
  url: "https://try.count.ly",
  debug: true
});

//track sessions automatically
Countly.track_sessions();

//track pageviews automatically
Countly.track_pageview();

//track any clicks to webpages automatically
Countly.track_clicks();

//track link clicks automatically
Countly.track_links();

//track form submissions automatically
Countly.track_forms();

//track javascript errors
Countly.track_errors();

//let's cause some errors
function cause_error(){
  undefined_function();
}

window.onload = function() {
  document.getElementById("handled_error").onclick = function handled_error(){
    Countly.add_log('Pressed handled button'); 
    try {
      cause_error();
    } catch(err){
      Countly.log_error(err)
    }
  };

  document.getElementById("unhandled_error").onclick = function unhandled_error(){
    Countly.add_log('Pressed unhandled button'); 
    cause_error();
  };
}

We also have a webpack configuration that takes this source file, resolves its import, and minifies the resulting file, creating a single minified file dist/main.js. Also note that we've set the devtool option to hidden-source-map which means webpack will generate a source map file but will not reference it in the main.js file. This is typically ideal for production environments where you don't need to inspect the underlying code in the browser. You might want to go with source-map in development environments, it will reference the source map file in main.js and ease debugging in the browser.

const path = require('path');
const webpack = require('webpack');
const TerserPlugin = require('terser-webpack-plugin');

module.exports = {
  mode: 'development',
  plugins: [new webpack.ProgressPlugin()],
  devtool: "hidden-source-map",

  module: {
    rules: [{
      test: /\.(js|jsx)$/,
      include: [path.resolve(__dirname, 'src')],
      loader: 'babel-loader'
    }]
  },

  optimization: {
    minimize: true,
    minimizer: [new TerserPlugin()],
  },

  output: {
    devtoolModuleFilenameTemplate: '[resource-path]',
    devtoolFallbackModuleFilenameTemplate: '[absolute-resource-path]'
  }
};

Here is a screenshot of how you would run the webpack with npx and use the webpack package that would be installed along with your project.

1612759824.png

Then, you can see that webpack produced themain.js file along with its source map file main.js.map. Now, we just need to upload that source map file to our Countly instance.

1612759876.png

Using Symbolication in Countly

This section will guide you when uploading a symbol file and then symbolicating your stack trace for Android and iOS.

After enabling the Symbolication plugin, you might have noticed that additional choices have been added to your Crashes submenu items:

  • Overview takes you to the a crash general overview.
  • Manage Symbols takes you to a screen where you can manage all the symbol files uploaded for this application.
  • Symbolication logs is similar to the request logs, where a short history of your symbolication requests is kept and therefore, this view is only populated if there are problems with the symbolication processes.

Uploading the Symbol File

Now that the connection to the symbolication server has been established and the necessary symbol files are procured, we can get to the symbolication of specific stack traces.

Only one symbol file upload required

You should only upload one symbol file per application version. Multiple symbol file uploads are not supported.

To upload a symbol file, open the Manage Symbols section. At first it will look empty, as shown below:

There, you will have to click on the Add Symbolication File button, which will open a drawer with the fields described below. Note that you will need to do this for every version of your app you would like to symbolicate.

Select Type: In the Type selection, you need to choose the platform for the symbol file you are uploading.

Build ID: For Android, the Build ID field is the version name. For iOS, the Build UUID field is your app's Build UUIDs. For JavaScript, this must match the app_version you use when initializing the Countly SDK in your web application.

You can get Build UUIDs using the dwarfdump --uuid command on the dSYM file, and you may specify more than one comma-separated Build UUID for each architecture.

Note that the Android Build ID and iOS Build UUID specified here must match your app's corresponding  Build ID or Build UUID.

Example:

dwarfdump --uuid CountlyTestApp-iOS.app.dSYM
UUID: AD4F5647-B2A0-3D85-8DF4-78D4DFD83296 (armv7) CountlyTestApp-iOS.app.dSYM/Contents/Resources/DWARF/CountlyTestApp-iOS
UUID: 70B0DE66-D337-3952-914D-A1E542667E20 (arm64) CountlyTestApp-iOS.app.dSYM/Contents/Resources/DWARF/CountlyTestApp-iOS

File Upload: While choosing a Symbolication/Mapping File upon uploading the page for iOS, do not forget to zip the dSYM file before uploading. Otherwise, you will not be able to upload a dSYM file in a folder structure.

Notes: Free text optional field.

When everything is filled out, simply click Upload file.

Again, keep in mind you should do this process for every version or build that you would like to symbolicate.

Symbolicating a Crash

After some time has passed, you will start to see new exceptions in your Crashes Overview dashboard. If you have uploaded the correct symbols for those versions, then you are ready to move forward with the symbolication process.

To symbolicate a crash, you first have to go to its detailed view by clicking on the crash or on the View button.

Screenshot_2020-12-07_at_18.32.48.png

As shown below, the detailed crash view will be divided in two parts: the top contains the crash group stack trace, and the bottom contains entries for each specific crash and its circumstances.

When clicking on a specific crash entry, it opens up additional information which contains a Symbolicate toggle button, assuming it's possible to symbolicate the crash (e.g. symbol file is uploaded for corresponding version). At the top of the crash group section, you may also see the Symbolicate button.

When you click on the Symbolicate button, you should see the Symbolication in progress message. When that process is complete, you should be able to see the symbolicated stack trace.

The current symbolication implementation has some technical limitations, and the top crash group will show a Symbolicate button only after a crash from a newer app version is received. Therefore, if you would like to symbolicate crashes from a version you had before you enabled Crash Symbolication, you will have to scroll down to a specific crash entry at the bottom of the detailed crash view.

Troubleshooting

Check Symbolication Logs

You might come across a situation when trying to symbolicate a crash where you will see something similar to the image below:

This means that there was a problem while trying to symbolicate your crash. To get more information on the problem, either click on the red text or open the Symbolication Logs section of the main Crashes menu.

After the issue which caused the failure has been fixed, you may rerun that symbolication task.

Was this article helpful?
0 out of 0 found this helpful

Looking for help?