Friday, October 24, 2014

Generating main-dex-list file

[Update - 10/31/2014] Gradle plugin v0.14.0 for Android adds support for Multi-Dex. Everything that is written below is now done automatically. All you need to do is to add this line in your build.gradle file:
android {
   defaultConfig {
      ...
      multiDexEnabled = true
}

In my previous post I described the 65536 methods issue and how to solve it using multi-dex technique. In overall, this solution is pretty straightforward - you tell dx to operate in multi-dex mode, and add the multi-dex support library to patch the ClassLoader on application start-up.

The only tricky part is the main-dex-list file that specifies all the classes that you want to be included in the main dex file. dx takes this file as a parameter and generates appropriate dex files.

Failing to keep required classes in the main dex file will produce the following runtime exception:
java.lang.RuntimeException: Unable to instantiate application info.osom.multidex2.Application: java.lang.ClassNotFoundException: Didn't find class "info.osom.multidex2.Application" on path: DexPathList[[zip file "/data/app/info.osom.multidex2-1.apk"],nativeLibraryDirectories=[/data/app-lib/info.osom.multidex2-1, /vendor/lib, /system/lib]]
            ..
     Caused by: java.lang.ClassNotFoundException: Didn't find class "info.osom.multidex2.Application" on path: DexPathList[[zip file "/data/app/info.osom.multidex2-1.apk"],nativeLibraryDirectories=[/data/app-lib/info.osom.multidex2-1, /vendor/lib, /system/lib]]
            at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:497)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:457)
Now, you may ask, which exact classes should be included in main dex file? And if I have many classes, do I need to write all those class names by hand?

I had no clear answer to those questions, so I took a further look on dx source code. Thanks God (and Google) it's open source :)

After searching a bit, I found this commit, whose descriptions says:
Introduction of a tool to find main dex classes.
This is for legacy application wanting to allow --multi-dex on dx and load the multiple files using the com.android.multidex.installer library. The mainDexClasses script will provide the content of the file to give to dx in --main-dex-list.
Seems like there's a script (called mainDexClasses) that generates the exact class names that should be included in main-dex-list file!

Important note: use it at your own risk. This script is not officially released, and I haven't found any references in the Internet. It worked for me well, but if you encountered any bugs - I would love to hear!

The script (comes in two versions - bash script and Windows's batch) is located in /dx/etc folder. Download two files from there - mainDexClasses (or mainDexClasses.bat for Windows) and mainDexClasses.rules. Place both files in your /sdk/build-tools/[build-tools-version] folder. The minimum supported build-tools version is 20 (I will explain later why).

Invoking the script without parameters echoes its usage:
Usage : mainDexClasses [--output <output file>] <application path>
Output file is obviously your main-dex-list. The second parameter (application path), requires to specify all the zip files and directories (separated by path separator) that dx receives as input. To find the paths for your project, run gradlew with --info parameter:
gradlew clean assembleDebug --info
and find the command for your dexDebug task. For example (relevant jars and directories coloured):
All input files are considered out-of-date for incremental task ':app:dexDebug'.
command: dx.bat --dex --num-threads=4 --multi-dex --main-dex-list=D:\multidex2\app/multidex.keep --output D:\multidex2\app\build\intermediates\dex\debug D:\multidex2\app\build\intermediates\classes\debug D:\multidex2\app\build\intermediates\dependency-cache\debug D:\multidex2\app\libs\android-support-multidex.jar
Note that the script expects class files within jar. By default, preDexDebug task is enabled, and it turns all the dependencies into dex files. Therefore, you have to disable library pre-dexing, by specifying dexOptions.preDexLibraries = false.

The mainDexClasses script does two things.

First, it invokes proguard to shrink all the classes according to rules defined in mainDexClasses.rules file (we downloaded it previously, remember?):
call "%proguard%" -injars %params% -dontwarn -forceprocessing  -outjars "%tmpJar%" -libraryjars "%shrinkedAndroidJar%" -dontoptimize -dontobfuscate -dontpreverify -include "%baserules%" 1>nul
The %params% (-injars parameter) is the second parameter that the script receives. The script expects to find shrinkedAndroid.jar in /sdk/build-tools/[build-tools-version]/lib folder. There's no file with such name there, so to make it work I copied the android.jar from /sdk/platforms/[api-version] and renamed it to shrinkedAndroid.jar.

Secondly, the script invokes dx.jar's com.android.multidex.ClassReferenceListBuilder class. This is a command-line utility that described as following:
This is a command line tool used by mainDexClasses script to find direct class references to other classes. First argument of the command line is an archive, each class file contained in this archive is used to identify a class whose references are to be searched, those class files are not opened by this tool only their names matter. Other arguments must be zip files or directories, they constitute in a classpath in with the classes named by the first argument will be searched. Each searched class must be found. On each of this classes are searched for their dependencies to other classes. Finally the tools prints on standard output a list of class files names suitable as content of the file argument --main-dex-list of dx.
The description is self-explanatory, so I have nothing to add here. This class was added to dx.jar in build-tools 20 (and yes, this is why it is a minimum supported version).

Last thing is to invoke the script:
mainDexClasses.bat --output mutlidex.keep "D:\multidex2\app\build\intermediates\classes\debug;D:\multidex2\app\build\intermediates\dependency-cache\debug;D:\multidex2\app\libs\android-support-multidex.jar"
.. and this is the output:


As expected!

Last notes:
  • As this script relies on Proguard, it naturally won't keep your classes that are being invoked via reflection API - you will have to add them manually
  • The generation of course can be simply extracted into a gradle task that runs before dexDebug task. I will post example project in the near future, so stay tuned!

15 comments:

  1. How do you execute the preDex step when your project requires --multi-dex due to the 64k limit, and that option requires libraries to *not* be preDexed?

    ReplyDelete
    Replies
    1. You're right. You shouldn't use preDexDebug. I've edited the relevant part. Thanks for pointing it out to me!

      Delete
  2. I ran into some issues where the script was trying to add virtually the entire project's classpath to the main-dex-classes list, even though they are not required for the verify step of loading the Application class. I ended up wrapping this entire process with a gradle task that automatically downloads the files listed above and also attempts to strip out known packages that we *do not want* in the main dex file. The automatic download makes it possible for developers (and even build servers) to take advantage of it without needing to go through this tutorial manually. And since the main-dex-classes list is generated every time we need to re-dex, there's no need to manually maintain a list of classes in some arbitrary file as part of the development process. Here's the gradle file[1], which we apply with `apply from: file('gradle/multidex.gradle')`

    [1] http://privatepaste.com/74f013aabc

    ReplyDelete
    Replies
    1. Looks terrific.. thanks for sharing!

      Delete
    2. Hi Joe/Alex,

      We've been experiencing the problem described here and this solution is working awesomely for us so far but for a small hiccup.

      We integrated the gradle script to automate all this and debug builds work flawlessly. However for release builds we are seeing a missing classes.jar from proguard.

      :app:generateMainDexListLivestoreReleaseQA
      Error: Can't read [/Users/nlai/Android/workspace/Keek/app/build/intermediates/classes-proguard/livestore/releaseQA/classes.jar] (No such file or directory)


      When I disable this script and re-run our build I see the classes.jar get generated. Then when I re-enable the script and run it again it works fine using the now generated classes.jar file.

      I'm thinking I need to modify these lines:

      mainDexTask.dependsOn variant.javaCompile
      dx.dependsOn mainDexTask

      so that the mainDexTask only runs after the proguard task on release builds but not really sure how to accomplish this, especially since debug builds don't have this step.

      Thanks,
      Nic

      Delete
    3. First of all, as I already wrote (see note at the top), Gradle plugin for Android (v.0.14) supports multi-dex. They basically implemented everything that I wrote above in set of gradle tasks. For more info, check here: http://developer.android.com/tools/building/multidex.html . If you still have issues, please post them here.

      Delete
    4. HI Joe,

      Thanks for sharing the code. Please let me know on how to execute this custom task before dexDebug task.

      Br,
      Suman

      Delete
  3. hi.
    i will try the gradle task of joe later on but i have a question to the ApplicationWrapper approach.

    I used the --minimal-main-dex option along with a keep file like yours. This results in 12 classes in my main dex file whcih is ok. I than use a library uses the following code to list all classes in the dexfile but just gets the entries of the main "clesses.dex" and not also of all other loaded dex files:
    private static List getPaths(final String[] sourcePaths) {
    List result = new ArrayList();

    for (String s : sourcePaths) {
    try {
    DexFile dexfile = new DexFile(s);
    Enumeration entries = dexfile.entries();

    while (entries.hasMoreElements()) {
    result.add(entries.nextElement());
    }
    } catch (IOException ioe) {
    Log.w(TAG, "cannot open file=" + s + ";Exception=" + ioe.getMessage());
    }
    }

    return result;
    }

    the for now single path gets determined with: application.getApplicationContext().getApplicationInfo().sourceDir;
    which results to somthing like /data/../myapplicationname.apk

    Is there a possibility to get all dex files listed?

    ReplyDelete
    Replies
    1. See how support library's code finds all the dex files: https://android.googlesource.com/platform/frameworks/multidex/+/master/library/src/android/support/multidex/MultiDexExtractor.java

      Delete
    2. thanks for the hint, i looked at it before. As its all private i will have to copy paste the code or use it via reflection, which is not ideal but looks like the only possible way as google didn't built in an official way to do this.

      Delete
  4. Hi!
    I try to use the new android multidex support for a project. I have some problem with this exception:
    Execution failed for task ':packageAllDebugClassesForMultiDex'.
    > java.util.zip.ZipException: duplicate entry: org/apache/maven/wagon/providers/file/FileWagon.class
    about the problem. I use 2 different jar package as dependency, and some class will be duplicated in classes.dex because both jars contains they. Do you have any idea?
    of course, not just 2 jar affected, so i try to find a robust solution.
    so far, i used maven with failOnDuplicatePackages=false flag

    ReplyDelete
    Replies
    1. From what I understood, you're using two fat-jars, and both of them depend on maven-wagon. I'm not aware of something similar to failOnDuplicatePackages in Gradle. Why not just delete the wagon classes from one of fat-jars?

      Delete
    2. Hi!
      In fact, i needed to different jar, maven-wagon and maven-wagon-provider-api. In this case, i can remove one of them, but i have several similar conflict with dependencies, so i would like to solve it without the jars removing. In time, one of the removed jars will be affected the app

      Delete
    3. Hi David, Did you find the solution for this?

      Delete
  5. When I run this
    ./mainDexClasses --output 1.txt ~/Desktop/coohua/CooHuaClient/app/
    I met the problem.How can I resolve it

    Note: there were 8436 duplicate class definitions.
    You should check if you need to specify additional program jars.
    Warning: can't write resource [.readme] (Duplicate zip entry [build/intermediates/exploded-aar/com.android.support/design/23.2.1/jars/classes.jar:.readme])
    Warning: can't write resource [.readme] (Duplicate zip entry [build/intermediates/exploded-aar/com.android.support/gridlayout-v7/23.2.1/jars/classes.jar:.readme])
    Warning: can't write resource [.readme] (Duplicate zip entry [build/intermediates/exploded-aar/com.android.support/multidex/1.0.1/jars/classes.jar:.readme])
    Warning: can't write resource [android/support/v7/widget/annotations.xml] (Duplicate zip entry [build/intermediates/exploded-aar/com.android.support/recyclerview-v7/23.2.1/annotations.zip:android/support/v7/widget/annotations.xml])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.coohua.framework/CoohuaFramework/2.6/jars/classes.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.coohua.framework/CoohuaFramework/2.6/jars/libs/tbs_sdk_v1.5.1.1040_25435_obfs_20160222_143032.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.coohua.framework/CoohuaFramework/2.6/jars/libs/TencentLocationSDK_v4.1.1_r175129.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.coohua.nativepro/NativeProject/1.3/jars/classes.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.facebook.fresco/drawee/0.9.0/jars/classes.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.facebook.fresco/fbcore/0.9.0/jars/classes.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.facebook.fresco/fresco/0.9.0/jars/classes.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.facebook.fresco/imagepipeline/0.9.0/jars/classes.jar:META-INF/MANIFEST.MF])
    Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [build/intermediates/exploded-aar/com.facebook.fresco/imagepipeline-base/0.9.0/jars/classes.jar:META-INF/MANIFEST.MF])
    Error: Can't write [/private/tmp/mainDexClasses-3127.tmp.jar] (Can't read [/Users/jiangbin/Desktop/coohua/CooHuaClient/app] (Duplicate zip entry [build/intermediates/multi-dex/online/debug/componentClasses.jar:android/support/multidex/MultiDex$V14.class]))

    ReplyDelete