Bytecode transformations: The Android Gradle Plugin

This is the first part of a blog post series about bytecode transformations on Android. In this part we'll cover different approaches to bytecode manipulation in Java as well as how to make it work with Android and the Android Gradle plugin. In the next two parts we'll dive into the actual bytecode, bytecode instructions and how we can modify the bytecode and inject our own instructions, using Room as an example. In the last part we'll see how we can test our transformations and how it can influence Gradle build speed.

Detecting the slow spots in your app without having to write a single line of code is an intriguing idea, but is also not that easy to implement. At Sentry we wanted to provide the capability for Android users to automatically measure the execution time of database queries because oftentimes they can become a hidden bottleneck for app performance. Room is a widely-adopted ORM solution built by Google and is a go-to library for persistence for the majority of the Android developers, so it was an obvious choice for us to start with.

If we want database operations to show up in Sentry, we need to wrap them into special objects called Spans. In short, Spans are application events that have a start and end time and some metadata like operation name, description, etc. For example, this is how the performance breakdown looks for a popular open-source app tivi:

On the screenshot above, there's a parent span ui.screen.interaction that contains multiple child spans, for instance, the network request span with a http.client operation, duration 225ms and a request url as a description. If you want to learn more about performance monitoring @ Sentry, check this post out.

Back to the topic, this all means we need to find a way to inject our code before and after Room executes its queries to measure their execution time. There's a QueryCallback available in the Room API, but it's invoked only before a query gets executed, so we couldn't really utilize it.

Options

Since there's no way to know when a SQL query starts and finishes at runtime, we started looking into compile-time solutions. There are a few well-known options in the JVM world available for bytecode weaving:

• AspectJ: an AOP framework, which allows extending methods and plugging into their execution from outside of the target codebase.

• ASM: a bytecode manipulation framework, which allows dealing with bytecode directly. For example, it's used by R8 and D8 on Android for optimizing and dexing the bytecode.

• Other higher-level abstractions like Javassist: are all based on ASM, but have a nicer and easier-to-understand APIs to deal with.

While it would be logical to pick something higher-level, considering we had no expertise in any of those, we've decided to look into how we could marry those with the Android Gradle plugin (AGP), as we are aiming to transform Android apps and need to support things like differe build types, flavours and so on. A quick search revealed that we could go with:

1. Gradle's TransformAction: a plain Gradle API for transforming outputs. This is used, for example, for dexing, jetifying, and many other things that the Android Gradle plugin does.

2. AGP's Transform: an old API from AGP that gives a list of inputs to be transformed, depending on the options provided. It also handles full/incremental builds automatically. Now deprecated in favor of the new API.

3. AGP's transformClassesWith: the new API from AGP that allows registering an ASM ClassVisitor for visiting bytecode instructions and instrumenting .class files. It utilizes the aforementioned TransformAction to transform dependencies and provides a Gradle task that handles full/incremental builds automatically. Available from AGP version 4.2.0 and above.

The first option would require us to manually hook into the AGP process and deal with its artifacts, so we decided to look into options 2 and 3 and compare them, as they come directly from the vendor.

Previously, in the old AGP versions (pre-4.2.0), if one would like to instrument compiled classes, they would need to register their own Transform, traverse the input files and perform instrumentation for each of those files using ClassWriter from ASM. For each such Transform AGP would register a new Gradle task, so if you happen to have 10 transforms instrumenting your application, you would end up with 10 additional Gradle tasks doing almost the same thing - iterating over the changed files, reading the bytecode, applying their own transformations and writing the bytecode back.

This is horrible for the build speed and most of that can be commonized up until the point of actually instrumenting the bytecode.

The new transformClassesWith API tackles exactly that by providing a single API for registering ClassVisitors and abstracting away file iteration and reading/writing the bytecode. It collects all visitors in a single list and then, for each file, runs all of them in order of registering, meaning there's just a single Gradle task running for all transformations.

We've decided to go with ASM + transformClassesWith pack, deliberately supporting only the new versions of AGP.

Note, that if you want to support bytecode transformations in lower AGP versions (below 4.2.0) you still need to use the old Transform API. However, you can perform an AGP version check at runtime and choose either a new or an old API depending on it. An example can be seen in the Hilt Gradle plugin.

Using new transform APIs

Registering AsmClassVisitorFactory

As this post is not about how to create Gradle plugins, I will skip the setup part, but in a nutshell, we have to implement the Plugin interface from Gradle and override a single method called apply, which is called when the Gradle plugin is applied to a project:

class SentryPlugin : Plugin<Project> {

	override fun apply(project: Project) {
		...
	}
}

After that we have to listen when the Android Gradle plugin is applied to the project and retrieve the new AndroidComponentsExtension like this:

project.pluginManager.withPlugin("com.android.application") {
	val androidComponentsExtension =
    project.extensions.getByType(AndroidComponentsExtension::class.java)
	...
}

The extension has a special onVariants method that configures the build variants:

androidComponentsExtension.onVariants { variant ->
	...
}

Finally we can register our custom AsmClassVisitorFactory for the variant through transformClassesWith:

variant.transformClassesWith(
  SpanAddingClassVisitorFactory::class.java,
  InstrumentationScope.ALL
) { params ->
  if (extension.tracingInstrumentation.forceInstrumentDependencies.get()) {
    params.invalidate.setDisallowChanges(System.currentTimeMillis())
  }
  params.debug.setDisallowChanges(
    extension.tracingInstrumentation.debug.get()
  )
  params.tmpDir.set(tmpDir)
}

transformClassesWith accepts 3 parameters:

• ClassVisitorFactory: a factory, which provides a ClassVisitor implementation and defines whether this visitor is interested in instrumenting a given class

• InstrumentationScope: either ALL or PROJECT. Defines whether the instrumentation applies only for project files or for project files and their dependencies (e.g. jars). In our case, we were interested in instrumenting all Room queries, regardless of their origin, so we set it to ALL

If you're using InsrumentationScope.ALL, beware that Gradle will cache the transformed artifacts across builds as long as the InstrumentationParameters do not change. This may come as a surprise while developing, as some of the classes coming from the dependencies might not show up for instrumentation. We found it useful to have a boolean parameter, which would invalidate the transform caches by simply setting System.currentTimeMillis and allow us to always receive all classes for instrumentation.

• Configuration function to be applied before passing the necessary parameters for the ClassVisitorFactory

The InstrumentationParameters are the way to pass information from the plugin to the ClassVisitorFactory. They are being used as Gradle inputs, this means they contribute to the up-to-date checks of the task and should be properly annotated. For example, here we are setting a debug boolean as well as a tmpDir to use this information later and stream debug output of instrumentation into a file under the tmpDir.

Implementing AsmClassVisitorFactory

For the ClassVisitorFactory it's necessary to implement 2 methods:

• createClassVisitor which provides a custom ClassVisitor from ASM that does the actual visiting of bytecode instructions and transformation

• isInstrumentable which defines whether a given class is applicable for instrumentation or not

It is also necessary to specify an implementation of InstrumentationParameters as a type for AsmVisitorFactory or use InstrumentationParameters.None in case there are no params.

abstract class SpanAddingClassVisitorFactory :
    AsmClassVisitorFactory<SpanAddingClassVisitorFactory.SpanAddingParameters> {

    interface SpanAddingParameters : InstrumentationParameters {

        @get:Input
        @get:Optional
        val invalidate: Property<Long>

        @get:Input
        val debug: Property<Boolean>

        @get:Internal
        val tmpDir: Property<File>
    }

    override fun createClassVisitor(
        classContext: ClassContext,
        nextClassVisitor: ClassVisitor
    ): ClassVisitor =
        TODO("If we return true from the isInstrumentable below, we should return a ClassVisitor that will inject our code for measuring the execution time")

    override fun isInstrumentable(classData: ClassData): Boolean =
        TODO("Determine if we are interested in instrumenting the given ClassData. For us it would mean a class annotated with @Dao")
}

Inside the isInstrumentable method, we determine whether we are interested in instrumenting the given ClassData and later return our custom ClassVisitor from the createClassVisitor method in case we are. Note, however, that it's always a good practice to fall back to nextClassVisitor in case there's no ClassVisitor for the given class, otherwise the Gradle build will fail.

Last, let's look at the ClassData structure:

interface ClassData {
    /**
     * Fully qualified name of the class.
     */
    val className: String

    /**
     * List of the annotations the class has.
     */
    val classAnnotations: List<String>

    /**
     * List of all the interfaces that this class or a superclass of this class implements.
     */
    val interfaces: List<String>

    /**
     * List of all the super classes that this class or a super class of this class extends.
     */
    val superClasses: List<String>
}

It may seem to have everything to help us determine whether a class is suitable for instrumentation or not, but there's a setback which we'll cover in the following post.

Using the new AGP transform APIs with ASM looks like an obvious choice for bytecode manipulation for Android as it affects the build speed almost unnoticeable (we'll cover that later),  handles full/incremental builds on its own, and offers a great API surface via ASM at the same time.

In the next part, we'll talk about Room internals, how we collected the methods for instrumentation, and what tools are available out there for dealing with ASM.

The code is available in the sentry-android-grade-plugin repo, specifically:

• Registering a ClassVisitorFactory

• ClassVisitorFactory

By the way, if you are already using the Sentry Android Gradle plugin, give this new Room instrumentation a try in version 3.0.0-beta.1 , we would appreciate your feedback via GitHub issues. If not, it's time to start using Sentry — request a demo and try it out for free.