Saturday, December 6, 2014

Too many methods in main-dex?

Notes:
1. The content of this post is relevant only if your minSdk < 21. On Lollipop, no ClassLoader patching required.
2. This solution worked for me well, but it might not work for you in the same way. So please use it on your own risk. Nevertheless, I would be happy to know of your problems - so please post in comments!


Right after the release of multidex support we removed our custom solutions (for 64k methods problem) in favor of multidex support library. Gradle for Android plugin v0.14.0+ simplified the process even more - it implemented all the manual tasks (I wrote about some of them in my previous post) that we had to do previously.

Lately we encountered another issue: we have reached the 64k methods limit in main dex as well. It comes in form of the following error when we're building our project:
UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dex.DexException: Too many classes in --main-dex-list, main dex capacity exceeded
  at com.android.dx.command.dexer.Main.processAllFiles(Main.java:494)
  at com.android.dx.command.dexer.Main.runMultiDex(Main.java:332)
  at com.android.dx.command.dexer.Main.run(Main.java:243)
  at com.android.dx.command.dexer.Main.main(Main.java:214)
  at com.android.dx.command.Main.main(Main.java:106)
You can say: man, something is wrong with your design if you're having such a problem!
Well, you are quite right, and we're thinking hard on how to split various features of our application into separate apps. But for short-term, we had to find a solution for this specific issue.


So what is main dex, and which classes it must include?

On application start, the default ClassLoader has a single entry in its path - classes.dex file. It is also called the main dex. To support more than one dex file, the multidex support library implemented runtime patching of ClassLoader's path. This code should run as soon as you have the application context (the perfect place for it is in Application#attachBaseContext method).

Therefore, the Application class should be definitely included in the main dex, since it should already be present when you patching the ClassLoader.

Any other classes must be included? Yes. There's a thing called Dalvik verifier that has complex rules for determining inappropriate bytecode. For example, before loading the Application class, VM verifier performs its checks and if it finds a field whose type it can't resolve (yet), it will not allow to run instruction that accessing this field, even if at that point of time we already patched the ClassLoader and the class could be resolved.

Therefore, as a rule of thumb, all the Application's direct reference hierarchy should be included in main dex as well.


Which classes android-gradle-plugin adds to main dex?

The collect[debug/release]MultiDexComponents task generates all the root classes, and then the create[debug/release]MainDexClassList task calculates entire reference hierarchy for each root class.

Which classes are considered as roots? Well, the task runs through every single component in the application's manifest file and adds every application, service, receiver, provider, instrumentation to the roots list. Additionally, BackupAgent and all the annotation classes added as roots as well.


Our problem & solution

The result for us was that according to these (plugin-defined) rules, large portion of our application code had to be placed into main dex. And this eventually triggered the main dex capacity exceeded build error.

In attempt to understand why, according to plugin configuration, all non-Application components are required to be included in main dex, I posted a question in adt-dev group, but received no answer yet.

To overcome the aforementioned build error, we removed all the activities from the roots list. The build passed successfully. We performed many tests - so far, this change didn't introduced any runtime errors.

In order to remove the activity roots we just patched the plugin's CreateManifestKeepList task (contributed by Ariel Cabib):


I also published a sample project that demonstrates the issue (and contains the plugin patching). Check it out here:
https://github.com/alipov/main-dex-too-many-methods




[Update - 4/8/2016]:

Version 2.0.0 of Android Gradle plugin broke the patch code. There were a few changes in CreateManifestKeepList task - mainly the task was converted from using Groovy to Java. The KEEP_SPECS static member that I was patching previously was converted to a ImmutableMap, which throws an UnsupportedOperationException when calling its remove method:
private static final Map KEEP_SPECS = ImmutableMap.builder()
...
.put("activity", DEFAULT_KEEP_SPEC)
...
.build();

I had to update the patch, and did it in the following way:


I've also updated the sample project - added a fix on a separate branch (plugin-v2):
https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2



[Update 2 - 7/1/2016]:

Version 2.2.0-alpha4 of Android Gradle plugin (with build-tools v24) fixes it by generating minimal multidex keep list. I've updated the sample project - now the build succeeds without any custom logic:
https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2.2.0-alpha4

33 comments:

  1. you can create the --main-dex-list file by yourself,and then use the --minimal-main-dex

    like this:
    afterEvaluate {

    tasks.matching {
    it.name.startsWith('dex')
    }.each { dx ->
    if (dx.additionalParameters == null) {
    dx.additionalParameters = []
    }
    dx.additionalParameters += '--multi-dex'
    dx.additionalParameters += "--main-dex-list=$projectDir/multidex.keep".toString()
    dx.additionalParameters += "--minimal-main-dex"
    }
    }

    ReplyDelete
    Replies
    1. I'm familiar with that option, but it won't help in this scenario. Specifying --minimal-main-dex tells dx to put only classes that are listed in --main-dex-list file into the main dex. But our problem is that the main-dex-list file contains too many classes!

      Delete
    2. we can control the classes in the main-dex-list file

      Delete
    3. Before dex you can define a task to generate the main-dex-list file

      Delete
    4. How do you decide which classes to put in main-dex-list file and which not? From your responses I understand that you're not using built-in gradle-android-plugin functionality, but rather manually configuring it. This post talks about what happen when you DO use the plugin multidex functionality. As I mentioned in post, the plugin adds all the components to the main dex (the reason is not clear for me), and I'm showing here how you can patch plugin's task to NOT include activities.

      Delete
  2. we also use the --minimal-main-dex to limit the size of main dex

    ReplyDelete
  3. look this
    https://plus.google.com/116801968491272085809/posts/Ds7xUQipfGf

    ReplyDelete
    Replies
    1. Yes, seems like you're having the same issue as I had to deal with.

      Delete
  4. Hi Alex, I want to build multi-dex apk by maven, but I found no related useful demo for me. Could you show how to build multi-dex apk in a project? not just how to configure the pom.xml file, also how to modify the source code. I googled all the internet but found no complete solution. Thanks!!

    ReplyDelete
    Replies
    1. You had better ask for example on maven-plugin's discussion forum: https://groups.google.com/forum/#!forum/maven-android-developers

      Delete
  5. Hi Alex,

    I recently enabled multidex in my project using the default multidex support in android studio. Since then we are experiencing run time exceptions on lots of preLolipop devices. In the logcat it only says A/libc(28851): Fatal signal 11 (SIGSEGV), thread 29117 (pool-16-thread-). I posted questions on SO but haven't received any answers yet. I guess its the same problem as your's.

    ReplyDelete
  6. Hi Alex,

    I have to add my model classes to main dex to make Sugar ORM works properly.

    How could I do that?

    Thanks.

    ReplyDelete
    Replies
    1. @Rodrigo What exact problem do you have (if you have stack trace, please post it on pastebin and post the link here)?

      Delete
  7. Hi, Alex.

    I used a your solution to avoid too many methods in main-dex issue.

    but. today I have updated with Android Plugin for Gradle, Revision 2.0.0 http://developer.android.com/intl/ko/tools/revisions/gradle-
    plugin.html#

    I see this error message after build

    A problem occurred evaluating project ':App'.
    > java.lang.UnsupportedOperationException (no error message)

    at line below
    if (keepSpecsMap.remove("activity") != null) {

    do you know how to solve it?

    ReplyDelete
    Replies
    1. Update the patch. Check it out - and let me know if it works for you.

      Delete
    2. Thanks! your patch is very useful for me.
      but, I have a small issue. so I fixed it. please check my pull request :)

      https://github.com/alipov/main-dex-too-many-methods/pull/2

      Delete
  8. @alex, I am not sure to understand : does your patch still work with android gradle plugin 2.0 ? It looks like not. But why don't you replace the whole map with a new one. You should be able to do that with reflection, just remove the final modified on the field. It should work, you can then replace the field/s map by the map you want..

    ReplyDelete
    Replies
    1. As I wrote above, the updated patch works with 2.0 (you can verify it by running my example project).
      Regarding your suggestion - it is possible, though you'll still have to iterate over existing map in order to copy all the entries (excluding 'activity') into your new map. You can patch in either way - pick the one that you most comfortable with. If you still think that you have a more elegant solution - please create a pull request at https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2

      Delete
  9. Trying the solution mentioned in blog returns the following error.

    > Can not set final [Lcom.google.common.collect.ImmutableMapEntry; field com.google.common.collect.RegularImmutableMap.table to java.util.LinkedHashMap


    Are you aware of a solution for that?

    ReplyDelete
    Replies
    1. Which version are you trying to implement? Original, update (plugin-v2), or update 2 (plugin-v2.2.0-alpha4)?

      Delete
    2. Tools team fixed it in 2.2.0-alpha4 (no custom patching required anymore). Is there any reason not to upgrade your plugin?

      Delete
    3. There is no strong reason. But we are stuck with old tools for now.

      Delete
    4. Which exact (plugin) version do you use? In my plugin-v2 branch, I'm using 2.0.0. Have you tried to clone it and then build it in your environment? If true, did it work?

      Delete
  10. I have used the patch for 2.0.0 and it worked. However, after updating gradle to 2.3.0 and build tools to 25.0.2, I can no longer use it. The new gradle version does not solve the 'main dex capacity exceeded' for me.

    ReplyDelete
    Replies
    1. Have you seen the second update in the bottom of the post?
      I wrote there "Version 2.2.0-alpha4 of Android Gradle plugin (with build-tools v24) fixes it by generating minimal multidex keep list."

      Delete
  11. I'm still seeing the "too many classes in --main-dex-list" exception when building with build tools v25 and gradle plugin v2.3.3... seems like the patch is no longer working? Do you have any insight into how to debug this?

    ReplyDelete
    Replies
    1. I couldn't reproduce your error. I've updated my sample project to v2.3.3, see https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2.3.3 - try to build it by yourself. Can you provide a sample project that illustrates the unexpected behavior?

      Delete
  12. I'm also able to reproduce issue even after v2.2, for instance in v2.3.3 and in 3.0.1. That surprises me is that where is no more "Too many classes in --main-dex-list" in 3.0.1, it just goes out of id boundraries. I'm not sure what causing this exactly, but the only major difference with your sample I see so far is that I'm using Kotlin in my project. Any hints on how can I work it around for v 3.0.1 are appreciated.

    ReplyDelete
    Replies
    1. I'd love to look on your issue. Is there any chance you can provide a sample project (as a github repo) where it reproduces?

      Delete
  13. Unfortunately I'm not able to factor out a sample that easlity. But I have quite the place to reproduce the problem - in fact I have two branches, one compiles 100% ok, the second one succeeds randomly but very seldom.

    ReplyDelete
    Replies
    1. My advice will be (assuming you understand why this too-many-methods-in-main-dex issue happening - as this post aims to explain): [1] see which classes are kept in main-dex (look for maindexlist.txt) and ensure you really need them to be there; [2] try to disable keeping classes with runtime annotations in main-dex as per: https://issuetracker.google.com/issues/37097145#comment33 (if the build will start passing, decide which classes should be kept, and then add them manually).

      Delete
    2. Option [2] helped me a lot, at least it compiles now and all really required classes are still in main-dex. Unfortunately my maindexlist.txt still contains tons of stuff which not really that required to be in main dex file, so I'll have to combine it with option [1] once.

      cat build/intermediates/multi-dex/debug/maindexlist.txt | wc -l
      5551

      Delete