Too many methods in main-dex?

Notes:
1. The content of this post is relevant only if your minSdk < 21. On Lollipop, no ClassLoader patching required.
2. This solution worked for me well, but it might not work for you in the same way. So please use it on your own risk. Nevertheless, I would be happy to know of your problems - so please post in comments!


Right after the release of multidex support we removed our custom solutions (for 64k methods problem) in favor of multidex support library. Gradle for Android plugin v0.14.0+ simplified the process even more - it implemented all the manual tasks (I wrote about some of them in my previous post) that we had to do previously.

Lately we encountered another issue: we have reached the 64k methods limit in main dex as well. It comes in form of the following error when we're building our project:

UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dex.DexException: Too many classes in --main-dex-list, main dex capacity exceeded
  at com.android.dx.command.dexer.Main.processAllFiles(Main.java:494)
  at com.android.dx.command.dexer.Main.runMultiDex(Main.java:332)
  at com.android.dx.command.dexer.Main.run(Main.java:243)
  at com.android.dx.command.dexer.Main.main(Main.java:214)
  at com.android.dx.command.Main.main(Main.java:106)

You can say: man, something is wrong with your design if you're having such a problem!
Well, you are quite right, and we're thinking hard on how to split various features of our application into separate apps. But for short-term, we had to find a solution for this specific issue.


So what is main dex, and which classes it must include?

On application start, the default ClassLoader has a single entry in its path - classes.dex file. It is also called the main dex. To support more than one dex file, the multidex support library implemented runtime patching of ClassLoader's path. This code should run as soon as you have the application context (the perfect place for it is in Application#attachBaseContext method).

Therefore, the Application class should be definitely included in the main dex, since it should already be present when you patching the ClassLoader.

Any other classes must be included? Yes. There's a thing called Dalvik verifier that has complex rules for determining inappropriate bytecode. For example, before loading the Application class, VM verifier performs its checks and if it finds a field whose type it can't resolve (yet), it will not allow to run instruction that accessing this field, even if at that point of time we already patched the ClassLoader and the class could be resolved.

Therefore, as a rule of thumb, all the Application's direct reference hierarchy should be included in main dex as well.


Which classes android-gradle-plugin adds to main dex?

The collect[debug/release]MultiDexComponents task generates all the root classes, and then the create[debug/release]MainDexClassList task calculates entire reference hierarchy for each root class.

Which classes are considered as roots? Well, the task runs through every single component in the application's manifest file and adds every application, service, receiver, provider, instrumentation to the roots list. Additionally, BackupAgent and all the annotation classes added as roots as well.


Our problem & solution

The result for us was that according to these (plugin-defined) rules, large portion of our application code had to be placed into main dex. And this eventually triggered the main dex capacity exceeded build error.

In attempt to understand why, according to plugin configuration, all non-Application components are required to be included in main dex, I posted a question in adt-dev group, but received no answer yet.

To overcome the aforementioned build error, we removed all the activities from the roots list. The build passed successfully. We performed many tests - so far, this change didn't introduced any runtime errors.

In order to remove the activity roots we just patched the plugin's CreateManifestKeepList task (contributed by Ariel Cabib):


I also published a sample project that demonstrates the issue (and contains the plugin patching). Check it out here:
https://github.com/alipov/main-dex-too-many-methods

---

[Update - 4/8/2016]:

Version 2.0.0 of Android Gradle plugin broke the patch code. There were a few changes in CreateManifestKeepList task - mainly the task was converted from using Groovy to Java. The KEEP_SPECS static member that I was patching previously was converted to a ImmutableMap, which throws an UnsupportedOperationException when calling its remove method:

private static final Map KEEP_SPECS = ImmutableMap.builder()
...
.put("activity", DEFAULT_KEEP_SPEC)
...
.build();

I had to update the patch, and did it in the following way:


I've also updated the sample project - added a fix on a separate branch (plugin-v2):
https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2

---

[Update 2 - 7/1/2016]:

Version 2.2.0-alpha4 of Android Gradle plugin (with build-tools v24) fixes it by generating minimal multidex keep list. I've updated the sample project - now the build succeeds without any custom logic:
https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2.2.0-alpha4