Multi-dex to rescue from the infamous 65536 methods limit

You landed on this page because you've probably received the following stack trace when you tried to build your Android project:

UNEXPECTED TOP-LEVEL EXCEPTION:
java.lang.IllegalArgumentException: method ID not in [0, 0xffff]: 65536
at com.android.dx.merge.DexMerger$6.updateIndex(DexMerger.java:501)
at com.android.dx.merge.DexMerger$IdMerger.mergeSorted(DexMerger.java:282)
at com.android.dx.merge.DexMerger.mergeMethodIds(DexMerger.java:490)
at com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:167)
at com.android.dx.merge.DexMerger.merge(DexMerger.java:188)
at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:439)
at com.android.dx.command.dexer.Main.runMonoDex(Main.java:287)
at com.android.dx.command.dexer.Main.run(Main.java:230)
at com.android.dx.command.dexer.Main.main(Main.java:199)
at com.android.dx.command.Main.main(Main.java:103)

or this stack trace if you're using dx version 1.8 (shipped with SDK build tools 19.0 and above):

trouble writing output: Too many method references: [num of methods]; max is 65536.
You may try using --multi-dex option.

If this is the first time you see this message, then you'll be surprised to know that Dalvik bytecode has a fundamental limitation which allows to invoke a maximum of 2^16 (65536) methods (per dex file). One glance on a list of StackOverflow questions on this topic is enough to realize how painful this limitation for us.

There is a range of possible workarounds for this issue. The basic ones include stripping third-party libraries to a desired minimum, or using ProGuard even in the debug build. Sometimes I even found myself removing getters and setters just to make the build pass. Another, more complicated, but less error prone solution is to split the project into several projects (each of which produces a dex file with less than 65536 methods), build them separately (there's a catch with resources - each project should be built with all the resources), and then load the secondary dex file and invoke its API via reflection.

This issue was well known to the runtime team at Android - according to Anwar Ghuloum "that's the most asked question we get in the Google I/O". Recently, while I've listened to one of the Android Developers Backstage podcasts which hosted Anwar, this question came up as well. And Anwar said that they built a mechanism to support multiple dex files in the framework itself starting from Android L. Great news!

This was in episode 11 (starting from 28:00 minute). Here's a short transcript of a relevant discussion:

Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way? 

Anwar: So we've talked about raving dex, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.."... 

... So what do we do in order to address that? We will do the dex bytecode [change], but I think there's sort of building block that we needed first - what we're calling multi-dex. And the idea is.. here's something people were doing - they were breaking up their [dex] files into multiple dex files (each of which exceeds the 64k limit), the main classes in dex file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary dex files. So this is kind of hack, but a necessary one for them. 

What we're doing in L is in runtime we will have a native support for multi-dex. All your dex files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-dex it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that feesy in the future to do the dex bytecode change, and then something that partitions it and runs on the existing [devices]..

[Update - 10/31/2014] Gradle plugin v0.14.0 for Android adds support for Multi-Dex. Everything that is written below is now done automatically. All you need to do is to add this line in your build.gradle file:
android {
   defaultConfig {
      ...
      multiDexEnabled = true
}

For more information, see the official description.

Let's examine the solution. The puzzle consists of three pieces that we would like to understand:

1. How to build a project, so in the end the output APK will contain several (multi) dex files?
2. How to consume multi-dex APK on devices with API >= Android L?
3. How to consume multi-dex APK on devices with API < Android L?

First piece of puzzle - how to produce APK with multiple dex files inside?
Well, the answer is pretty straightforward. This commit introduced the --multi-dex option in dx, which allows to generate several dex files. The current (1.8) version includes the following options:

--multi-dex allows to generate several dex files if needed. This option is exclusive with --incremental, causes --num-threads to be ignored and only supports folder or archive output. 

--main-dex-list= is a list of class file names, classes defined by those class files are put in classes.dex.

--minimal-main-dex only classes selected by --main-dex-list are to be put in the main dex.

To enable multi-dex, you need to edit your application's build.gradle script.


I'm adding --multi-dex option to each task whose name starts with dex (dexDebug/dexRelease). This is the place where dx is being invoked. Additionally, I'm specifying --main-dex-list parameter to which I will refer a bit later.

After unzipping the output APK file, you can clearly see that it contains two dex files:

-rwx------+ 1 Administrators None    1760     Oct  4 20:32 AndroidManifest.xml
drwx------+ 1 Administrators None       0       Oct  4 20:54 META-INF
-rwx------+ 1 Administrators None 8204912  Oct  4 20:32 classes.dex
-rwx------+ 1 Administrators None  720048   Oct  4 20:32 classes2.dex
drwx------+ 1 Administrators None       0       Oct  4 20:54 com
-rwx------+ 1 Administrators None  146498  Oct  4 20:32 font_metrics.properties
drwx------+ 1 Administrators None       0      Oct  4 20:54 org
drwx------+ 1 Administrators None       0      Oct  4 20:54 res
-rwx------+ 1 Administrators None  264448  Oct  4 20:05 resources.arsc

So the dx already supports creating multiple dex files. What about consuming them?

As Anwar said in podcast, Android L will natively support multiple dex files. That's it.
What about devices with API lower than Android L? The support library code (supports API 4+) that Anwar talks about was already pushed about a year ago. However, it was not shipped in the latest support library (revision 20), but it is anticipated to be included in next version which will be released side by side with Android L.

What should we do until then?
Just grab support library's multidex files, create the android.support.multidex package in your application, and paste the files there. Otherwise you can use this github repository, which wrapped the multidex files as a gradle library that you can add as a dependency.
[Update - 10/17/2014] As anticipated, revision 21 of support library ships with multidex support. You can find the android-support-multidex.jar in /sdk/extras/android/support/multidex/library/libs folder.

Last step requires to call the multidex library code. If you're subclassing the Application class, then you'll have to either derive from MultiDexApplication (instead of Application)

public class MyApplication extends MultiDexApplication { .. }

or override attachBaseContext method and call MultiDex.install().

protected void attachBaseContext(Context base) {
super.attachBaseContext(base);
MultiDex.install(this);
}

Otherwise (if your application does not have custom Application implementation), declare MultiDexApplication as application implementation in your AndroidManifest.xml.

<application
    android:name="android.support.multidex.MultiDexApplication"
    .. >
..
</application>

The support library's multidex code patches the ClassLoader's path/pathList field to point on two (or more) dex files, instead of default classes.dex. This also means that when the application is starting, the support library's multidex code should be already present in the main dex file (instead of within one of the secondary dex files). This is the purpose of --main-dex-list parameter, which specifies a file that contains class names I want to be put in the main dex file:


Note that if you have a custom Application class, it must be present in the main dex as well (just add it to your main-dex-list file).
[Update - 10/24/2014] The main-dex-list file can be generated. See my post for more details.

Some additional observations (discussed in this github page as well):

• com.android.dex.DexException thrown during dexDebug task with "Library dex files are not supported in multi-dex mode" error. Dependencies pre-dexing is used to speed up the incremental builds. This option is enabled by default, and disabling it will cause no harm. So I disabled it (see build.gradle above).
• My Application-derived class made many initializations in onCreate callback method. The ClassLoader patch takes place in Application#attachBaseContext callback method, which is being called before Application#onCreate. This means that all the classes from secondary dex files should be known to ClassLoader at this point. However, for some reason, accessing a class, that is packaged in a secondary dex file, during Application#onCreate throws a ClassNotFoundException. The workaround for me was to move all the onCreate logic into a separate method in a separate class, and then call it from Application#onCreate. [Update - 10/24/2014] This is not relevant anymore, since you can generate the main-dex-list file.