Saturday, October 4, 2014

Multi-dex to rescue from the infamous 65536 methods limit

You landed on this page because you've probably received the following stack trace when you tried to build your Android project:
UNEXPECTED TOP-LEVEL EXCEPTION:
java.lang.IllegalArgumentException: method ID not in [0, 0xffff]: 65536
at com.android.dx.merge.DexMerger$6.updateIndex(DexMerger.java:501)
at com.android.dx.merge.DexMerger$IdMerger.mergeSorted(DexMerger.java:282)
at com.android.dx.merge.DexMerger.mergeMethodIds(DexMerger.java:490)
at com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:167)
at com.android.dx.merge.DexMerger.merge(DexMerger.java:188)
at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:439)
at com.android.dx.command.dexer.Main.runMonoDex(Main.java:287)
at com.android.dx.command.dexer.Main.run(Main.java:230)
at com.android.dx.command.dexer.Main.main(Main.java:199)
at com.android.dx.command.Main.main(Main.java:103)
or this stack trace if you're using dx version 1.8 (shipped with SDK build tools 19.0 and above):
trouble writing output: Too many method references: [num of methods]; max is 65536.
You may try using --multi-dex option.
If this is the first time you see this message, then you'll be surprised to know that Dalvik bytecode has a fundamental limitation which allows to invoke a maximum of 2^16 (65536) methods (per dex file). One glance on a list of StackOverflow questions on this topic is enough to realize how painful this limitation for us.

There is a range of possible workarounds for this issue. The basic ones include stripping third-party libraries to a desired minimum, or using ProGuard even in the debug build. Sometimes I even found myself removing getters and setters just to make the build pass. Another, more complicated, but less error prone solution is to split the project into several projects (each of which produces a dex file with less than 65536 methods), build them separately (there's a catch with resources - each project should be built with all the resources), and then load the secondary dex file and invoke its API via reflection.

This issue was well known to the runtime team at Android - according to Anwar Ghuloum "that's the most asked question we get in the Google I/O". Recently, while I've listened to one of the Android Developers Backstage podcasts which hosted Anwar, this question came up as well. And Anwar said that they built a mechanism to support multiple dex files in the framework itself starting from Android L. Great news!

This was in episode 11 (starting from 28:00 minute). Here's a short transcript of a relevant discussion:
Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way? 
Anwar: So we've talked about raving dex, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.."... 
... So what do we do in order to address that? We will do the dex bytecode [change], but I think there's sort of building block that we needed first - what we're calling multi-dex. And the idea is.. here's something people were doing - they were breaking up their [dex] files into multiple dex files (each of which exceeds the 64k limit), the main classes in dex file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary dex files. So this is kind of hack, but a necessary one for them. 
What we're doing in L is in runtime we will have a native support for multi-dex. All your dex files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-dex it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that feesy in the future to do the dex bytecode change, and then something that partitions it and runs on the existing [devices]..
[Update - 10/31/2014] Gradle plugin v0.14.0 for Android adds support for Multi-Dex. Everything that is written below is now done automatically. All you need to do is to add this line in your build.gradle file:
android {
   defaultConfig {
      ...
      multiDexEnabled = true
}

For more information, see the official description.

Let's examine the solution. The puzzle consists of three pieces that we would like to understand:
  1. How to build a project, so in the end the output APK will contain several (multi) dex files?
  2. How to consume multi-dex APK on devices with API >= Android L?
  3. How to consume multi-dex APK on devices with API < Android L?

First piece of puzzle - how to produce APK with multiple dex files inside?
Well, the answer is pretty straightforward. This commit introduced the --multi-dex option in dx, which allows to generate several dex files. The current (1.8) version includes the following options:
--multi-dex allows to generate several dex files if needed. This option is exclusive with --incremental, causes --num-threads to be ignored and only supports folder or archive output. 
--main-dex-list= is a list of class file names, classes defined by those class files are put in classes.dex.
--minimal-main-dex only classes selected by --main-dex-list are to be put in the main dex.
To enable multi-dex, you need to edit your application's build.gradle script.


I'm adding --multi-dex option to each task whose name starts with dex (dexDebug/dexRelease). This is the place where dx is being invoked. Additionally, I'm specifying --main-dex-list parameter to which I will refer a bit later.

After unzipping the output APK file, you can clearly see that it contains two dex files:
-rwx------+ 1 Administrators None    1760     Oct  4 20:32 AndroidManifest.xml
drwx------+ 1 Administrators None       0       Oct  4 20:54 META-INF
-rwx------+ 1 Administrators None 8204912  Oct  4 20:32 classes.dex
-rwx------+ 1 Administrators None  720048   Oct  4 20:32 classes2.dex

drwx------+ 1 Administrators None       0       Oct  4 20:54 com
-rwx------+ 1 Administrators None  146498  Oct  4 20:32 font_metrics.properties
drwx------+ 1 Administrators None       0      Oct  4 20:54 org
drwx------+ 1 Administrators None       0      Oct  4 20:54 res
-rwx------+ 1 Administrators None  264448  Oct  4 20:05 resources.arsc

So the dx already supports creating multiple dex files. What about consuming them?

As Anwar said in podcast, Android L will natively support multiple dex files. That's it.
What about devices with API lower than Android L? The support library code (supports API 4+) that Anwar talks about was already pushed about a year ago. However, it was not shipped in the latest support library (revision 20), but it is anticipated to be included in next version which will be released side by side with Android L.

What should we do until then?
Just grab support library's multidex files, create the android.support.multidex package in your application, and paste the files there. Otherwise you can use this github repository, which wrapped the multidex files as a gradle library that you can add as a dependency.
[Update - 10/17/2014] As anticipated, revision 21 of support library ships with multidex supportYou can find the android-support-multidex.jar in /sdk/extras/android/support/multidex/library/libs folder.

Last step requires to call the multidex library code. If you're subclassing the Application class, then you'll have to either derive from MultiDexApplication (instead of Application)
public class MyApplication extends MultiDexApplication { .. }
or override attachBaseContext method and call MultiDex.install().
protected void attachBaseContext(Context base) {
super.attachBaseContext(base);
MultiDex.install(this);
}
Otherwise (if your application does not have custom Application implementation), declare MultiDexApplication as application implementation in your AndroidManifest.xml.
<application
    android:name="android.support.multidex.MultiDexApplication"
    .. >
..
</application>
The support library's multidex code patches the ClassLoader's path/pathList field to point on two (or more) dex files, instead of default classes.dex. This also means that when the application is starting, the support library's multidex code should be already present in the main dex file (instead of within one of the secondary dex files). This is the purpose of --main-dex-list parameter, which specifies a file that contains class names I want to be put in the main dex file:


Note that if you have a custom Application class, it must be present in the main dex as well (just add it to your main-dex-list file).
[Update - 10/24/2014] The main-dex-list file can be generated. See my post for more details.

Some additional observations (discussed in this github page as well):
  • com.android.dex.DexException thrown during dexDebug task with "Library dex files are not supported in multi-dex mode" error. Dependencies pre-dexing is used to speed up the incremental builds. This option is enabled by default, and disabling it will cause no harm. So I disabled it (see build.gradle above).
  • My Application-derived class made many initializations in onCreate callback method. The ClassLoader patch takes place in Application#attachBaseContext callback method, which is being called before Application#onCreate. This means that all the classes from secondary dex files should be known to ClassLoader at this point. However, for some reason, accessing a class, that is packaged in a secondary dex file, during Application#onCreate throws a ClassNotFoundException. The workaround for me was to move all the onCreate logic into a separate method in a separate class, and then call it from Application#onCreate. [Update - 10/24/2014] This is not relevant anymore, since you can generate the main-dex-list file.

36 comments:

  1. Hi Alex, how to enable multi dex option if I'm using eclipse IDE?

    ReplyDelete
    Replies
    1. I have not tried to build with Ant. You probably will need to customize your dex task in build.xml, so it will accept the additional parameters.

      Delete
    2. Hi Alex, been triend with mod dx exec to exec java $javaOpts -jar "$jarpath" --multi-dex "$@" , but no luck for this. Do you know which part of build.xml can mod the parameters of dex? Thanks!

      Delete
    3. Take a look at DexDexecTask - https://android.googlesource.com/platform/tools/base/+log/master/legacy/ant-tasks/src/main/java/com/android/ant/DexExecTask.java
      It looks like ant-tasks is no longer maintained (the project resides under 'legacy' folder). So you can edit the task, but you'll have to rebuild the project locally..

      Delete
    4. Another (possible) solution is to use the maven-android-plugin. They have multi-dex support. See my answer here: http://stackoverflow.com/a/26556209/1233652

      Delete
  2. Hi Alex,

    This seems not works well for me, i still getting following errors :
    [2014-10-20 18:21:08 - Dex Loader] Unable to execute dex: method ID not in [0, 0xffff]: 65536
    [2014-10-20 18:21:08 - xxx.xxx.xxx] Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0xffff]: 65536

    Already extend my application class to MultiDexApplication with support v4 r21 and android 5.0 sdk as well.

    any solution can advice for me? Thanks!

    ReplyDelete
    Replies
    1. He'll need to apply the build.gradle changes to build with --multi-dex

      Delete
    2. Sorry, i found out the reason,that's i'm using eclipse IDE, any solution for this? i has lot project bind with eclipse, it kind of hard for me convert into android studio :(

      Delete
    3. See my answer here: http://stackoverflow.com/a/26556209/1233652

      Delete
  3. Why didn't they add the library as a local repository too? Having to copypasterino the jar file is bad practice.

    ReplyDelete
    Replies
    1. They probably don't want to "advertise" this solution too much. Another example is that there's no mention/reference to multidex in Support Library's revision (21) history page. http://developer.android.com/tools/support-library/index.html

      Delete
  4. "However, for some reason, accessing a class, that is packaged in a secondary dex file, during Application#onCreate throws a ClassNotFoundException. "

    I've found that accessing any class from any method in the Application that is in the secondary dex file throws the ClassNotFoundException.

    ReplyDelete
    Replies
    1. I've just updated the post. You can generate the main-dex-list file. Please let me know if it solves your issue!

      Delete
    2. https://plus.google.com/104023661970539138053/posts/YTMf8ADTcFg

      Delete
  5. how to control the size of each dex,anyone can tell me ?

    ReplyDelete
    Replies
    1. Can I ask what for do you need that?

      Delete
    2. I am getting the following error installing my app that's been setup for multidex on any device running a pre-ICS version of android. I suspect this is due to the dex file sizes being beyond what they can handle. Is there any capability available to limit the dex file size?

      11-17 16:18:02.648 684-684/? E/dalvikvm﹕ LinearAlloc exceeded capacity (5242880), last=988
      11-17 16:18:02.648 684-684/? E/dalvikvm﹕ VM aborting
      11-17 16:18:02.648 117-117/? W/installd﹕ DexInv: --- END '/data/app/com.my.app-1.apk' --- status=0x000b, process failed
      11-17 16:18:02.648 117-117/? E/installd﹕ dexopt failed on '/data/dalvik-cache/data@app@com.my.app-1.apk@classes.dex' res = 11

      Delete
    3. Which gradle plugin version are you using? v0.14.2 calls dx with the "--set-max-idx-number=60000" parameter, which attempts to solve this issue.

      Delete
  6. I am getting an error Gradle DSL method not found: 'multiDexEnabled()' my Android SDK Build-Tools are updated and I am using 21.1.1 . Anndroid Support Repository and Library are also update. What am I missing here? Help please.

    ReplyDelete
    Replies
    1. It might be better to post your question on StackOverflow (don't forget to attach your build.gradle script).

      Delete
  7. Hi, I am facing same issue that gc exceeds limit while signing my android application build. I added mutlidex jar and renamed application to MultiDexApplication but still it didnot work.Am i missing something ? I have just started to work on big projects.In this application I have integrated Google drives, Box and Dropbox cloud services. Please help me .
    Thanks.

    ReplyDelete
    Replies
    1. What exact error are you receiving? Is it "java.lang.OutOfMemoryError: Java heap size". If true, you can tweak the maximum value by configuring dexOptions.javaMaxHeapSize property. See this answer for more details: http://stackoverflow.com/a/23688727/1233652

      Delete
    2. It's not OutOfMemoryError error . I have developed android application in eclipse Juno in which I'm using many libraries such as for Box,Dropbox and Google drive. I have removed some unwanted libraries from project. I'm able to debug it and test whole application without any error, but when i try to sign it for releasing on play store it takes too much time and gives error "GC overhead limit exceeded".
      I tried to fix with available solution but it didnot work for me .Is there any simple solution to solve this issue?
      Thanks .

      Delete
  8. Hi. After getting thru the dexDebug error by using multiDexEnabled, incremental true and javaMaxHeapSize I get java.lang.NoClassDefFoundError where my user defined classes are not being found when I ran my app. Any advice on this please?

    ReplyDelete
    Replies
    1. Start from checking your main-dex-list file that is generated by Android plugin (/build/intermediates/multi-dex/{debug/release}/maindexlist.txt). Does your user-defined class (the one that is not found) appear there?

      Delete
  9. Hi,
    Has there been any update in getting the multidex support library working with ant builds / eclipse?

    Thanks

    ReplyDelete
    Replies
    1. @Mallipeddi - Have you tried to use the android-maven-plugin? They seem to support multidex - http://simpligility.github.io/android-maven-plugin/dex-mojo.html

      Delete
  10. Hi Alex, I enabled multidex but when run app in device , I get crash with error "java.lang.NoClassDefFoundError"

    ReplyDelete
    Replies
    1. It's a pretty common error when starting to configuring multi-dex. Post the exact stacktrace (at pastebin.com for example), and I'll try to assist.

      Delete
  11. i done same as yours but ,no class found exception came

    ReplyDelete
  12. You get the "No Class found Exception" cos it does not support API 19. Am looking myself for a workaround.

    ReplyDelete
    Replies
    1. multi-dex is actually supported on pre-Lollipop devices (with some limitations). If you have a specific issue, post a question on StackOverflow and write down its URL here.

      Delete
  13. Hi i get this error while i'm generating signed apk

    Error:Execution failed for task ':app:transformClassesWithDexForRelease'.
    > com.android.build.api.transform.TransformException:
    com.android.ide.common.process.ProcessException:
    java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException:
    Error while executing java process with main class com.android.dx.command.Main with arguments {--dex --num-threads=4 --multi-dex --main-dex-list
    D:\codecanyon-21201902-coloring-book\new\Project_FingerColoringAndroid_MainFile\app\build\intermediates\multi-dex\release\maindexlist.txt --output
    D:\codecanyon-21201902-coloring-book\new\Project_FingerColoringAndroid_MainFile\app\build\intermediates\transforms\dex\release\0 --min-sdk-version 14
    D:\codecanyon-21201902-coloring-book\new\Project_FingerColoringAndroid_MainFile\app\build\intermediates\transforms\jarMerging\release\0.jar}

    ReplyDelete
  14. hi .. i did these steps but still i have this error :

    Error:Execution failed for task ':app:transformDexArchiveWithExternalLibsDexMergerForDebug'.
    > com.android.builder.dexing.DexArchiveMergerException: Unable to merge dex

    what should i do ? :-(

    ReplyDelete
    Replies
    1. There are many potential causes for this error, please browse Stack Overflow for similar questions (i.e. this one: https://stackoverflow.com/q/46267621), or create a new question with your specific details.

      Delete