Too many methods in main-dex?

Notes:
1. The content of this post is relevant only if your minSdk < 21. On Lollipop, no ClassLoader patching required.
2. This solution worked for me well, but it might not work for you in the same way. So please use it on your own risk. Nevertheless, I would be happy to know of your problems - so please post in comments!


Right after the release of multidex support we removed our custom solutions (for 64k methods problem) in favor of multidex support library. Gradle for Android plugin v0.14.0+ simplified the process even more - it implemented all the manual tasks (I wrote about some of them in my previous post) that we had to do previously.

Lately we encountered another issue: we have reached the 64k methods limit in main dex as well. It comes in form of the following error when we're building our project:

UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dex.DexException: Too many classes in --main-dex-list, main dex capacity exceeded
  at com.android.dx.command.dexer.Main.processAllFiles(Main.java:494)
  at com.android.dx.command.dexer.Main.runMultiDex(Main.java:332)
  at com.android.dx.command.dexer.Main.run(Main.java:243)
  at com.android.dx.command.dexer.Main.main(Main.java:214)
  at com.android.dx.command.Main.main(Main.java:106)

You can say: man, something is wrong with your design if you're having such a problem!
Well, you are quite right, and we're thinking hard on how to split various features of our application into separate apps. But for short-term, we had to find a solution for this specific issue.

Generating main-dex-list file

[Update - 10/31/2014] Gradle plugin v0.14.0 for Android adds support for Multi-Dex. Everything that is written below is now done automatically. All you need to do is to add this line in your build.gradle file:
android {
   defaultConfig {
      ...
      multiDexEnabled = true
}

In my previous post I described the 65536 methods issue and how to solve it using multi-dex technique. In overall, this solution is pretty straightforward - you tell dx to operate in multi-dex mode, and add the multi-dex support library to patch the ClassLoader on application start-up.

The only tricky part is the main-dex-list file that specifies all the classes that you want to be included in the main dex file. dx takes this file as a parameter and generates appropriate dex files.

Failing to keep required classes in the main dex file will produce the following runtime exception:

java.lang.RuntimeException: Unable to instantiate application info.osom.multidex2.Application: java.lang.ClassNotFoundException: Didn't find class "info.osom.multidex2.Application" on path: DexPathList[[zip file "/data/app/info.osom.multidex2-1.apk"],nativeLibraryDirectories=[/data/app-lib/info.osom.multidex2-1, /vendor/lib, /system/lib]]
            ..
     Caused by: java.lang.ClassNotFoundException: Didn't find class "info.osom.multidex2.Application" on path: DexPathList[[zip file "/data/app/info.osom.multidex2-1.apk"],nativeLibraryDirectories=[/data/app-lib/info.osom.multidex2-1, /vendor/lib, /system/lib]]
            at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:497)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:457)

Now, you may ask, which exact classes should be included in main dex file? And if I have many classes, do I need to write all those class names by hand?

Multi-dex to rescue from the infamous 65536 methods limit

You landed on this page because you've probably received the following stack trace when you tried to build your Android project:

UNEXPECTED TOP-LEVEL EXCEPTION:
java.lang.IllegalArgumentException: method ID not in [0, 0xffff]: 65536
at com.android.dx.merge.DexMerger$6.updateIndex(DexMerger.java:501)
at com.android.dx.merge.DexMerger$IdMerger.mergeSorted(DexMerger.java:282)
at com.android.dx.merge.DexMerger.mergeMethodIds(DexMerger.java:490)
at com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:167)
at com.android.dx.merge.DexMerger.merge(DexMerger.java:188)
at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:439)
at com.android.dx.command.dexer.Main.runMonoDex(Main.java:287)
at com.android.dx.command.dexer.Main.run(Main.java:230)
at com.android.dx.command.dexer.Main.main(Main.java:199)
at com.android.dx.command.Main.main(Main.java:103)

or this stack trace if you're using dx version 1.8 (shipped with SDK build tools 19.0 and above):

trouble writing output: Too many method references: [num of methods]; max is 65536.
You may try using --multi-dex option.

If this is the first time you see this message, then you'll be surprised to know that Dalvik bytecode has a fundamental limitation which allows to invoke a maximum of 2^16 (65536) methods (per dex file). One glance on a list of StackOverflow questions on this topic is enough to realize how painful this limitation for us.

Managing started service's lifecycle efficiently

So you decided to run your time-consuming operation in service component because you want to ensure that it will continue to run even if the user interface is no longer visible. You chose to extend Service, rather than IntentService because you want your service to handle multiple requests in parallel. In this post I will share a few tips for implementing your service slightly more efficiently.

Symmetric encryption issue in Android 4.3

About a year ago, a bit after 4.3 release, I had to fix an error in one of my applications. I patched it then, but wished to return and understand better the underlying reason. The other day I came back to this code and dug a little deeper. This post is about what I found.

In one of my applications I use symmetric-key algorithm for encryption and decryption of data. Nothing fancy here, just a standard Java APIs everyone uses for such things.

Migrating to SQLCipher for Android 3.x

In my last post I wrote why you have seriously consider upgrading to SQLCipher for Android 3.0.2. Version 3.x introduced new default configuration that is not out-of-the-box compatible with databases created with previous versions (1.x/2.x). Particularly, the default key derivation function's (KDF) iteration count increased from 4000 to 64000 (yes, x16), and therefore some kind of migration of the current database required.

You won't be able to open the existing database file if you'll try to upgrade the SQLCipher libraries without migrating it (the database file) first. The following error will be thrown in attempt to open existing database:

03-29 16:14:04.199  18107-18107/info.osom.sqlciphertest I/Database? sqlite returned: error code = 26, msg = file is encrypted or is not a database
03-29 16:14:04.199  18107-18107/info.osom.sqlciphertest E/Database? CREATE TABLE android_metadata failed
03-29 16:14:04.199  18107-18107/info.osom.sqlciphertest E/Database? Failed to setLocale() when constructing, closing the database
    net.sqlcipher.database.SQLiteException: file is encrypted or is not a database
            at net.sqlcipher.database.SQLiteDatabase.native_setLocale(Native Method)
            at net.sqlcipher.database.SQLiteDatabase.setLocale(SQLiteDatabase.java:2092)
            at net.sqlcipher.database.SQLiteDatabase.(SQLiteDatabase.java:1958)
            at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:875)
            at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:907)
            at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:132)
            at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:99)
            at info.osom.sqlciphertest.MainActivity$PlaceholderFragment$1.onClick(MainActivity.java:69)
            at android.view.View.performClick(View.java:4202)
            at android.view.View$PerformClick.run(View.java:17340)
            at android.os.Handler.handleCallback(Handler.java:725)
            at android.os.Handler.dispatchMessage(Handler.java:92)
            at android.os.Looper.loop(Looper.java:137)
            at android.app.ActivityThread.main(ActivityThread.java:5039)
            at java.lang.reflect.Method.invokeNative(Native Method)
            at java.lang.reflect.Method.invoke(Method.java:511)
            at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)
            at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:560)
            at dalvik.system.NativeStart.main(Native Method)

Why you should upgrade SQLCipher to latest (3.0.2) version

If you're using SQLCipher (for Android) version <= 3.0.1, then your application may potentially leak sensitive data into the log system. That's because in case of error in its underlying native library, the (SQLCipher) Android wrapper library was printing the original query values into the log system.
This issue was fixed on February 1st, 2014 and integrated in version 3.0.2.

I'm including here a short demo that demonstrates the problem.

I've created a new basic project using Android Studio. Then downloaded SQLCipher for Android version 2.2.1, unzipped and added all the content to the project. This is the structure:

Guarding enumeration classes from ProGuard

During my application tests I received exception with the following call stack:

02-09 15:18:06.478 26166 26403 E TAG: Caused by: java.lang.NullPointerException
02-09 15:18:06.478 26166 26403 E TAG:   at java.lang.Enum$1.create(Enum.java:43)
02-09 15:18:06.478 26166 26403 E TAG:   at java.lang.Enum$1.create(Enum.java:35)
02-09 15:18:06.478 26166 26403 E TAG:   at libcore.util.BasicLruCache.get(BasicLruCache.java:54)
02-09 15:18:06.478 26166 26403 E TAG:   at java.lang.Enum.getSharedConstants(Enum.java:209)
02-09 15:18:06.478 26166 26403 E TAG:   at java.util.EnumSet.noneOf(EnumSet.java:48)
02-09 15:18:06.478 26166 26403 E TAG:   at af.a(SourceFile:115)

The exception occurred only in release version, while in debug everything worked fine. So I immediately suspected that ProGuard, whose processing is part of a release build, is somehow involved here.

From ProGuard's documentation:

If your application, applet, servlet, library, etc., contains enumeration classes, you'll have to preserve some special methods. Enumerations were introduced in Java 5. The java compiler translates enumerations into classes with a special structure. Notably, the classes contain implementations of some static methods that the run-time environment accesses by introspection (Isn't that just grand? Introspection is the self-modifying code of a new generation). You have to specify these explicitly, to make sure they aren't removed or obfuscated: 

-keepclassmembers,allowoptimization enum * {
      public static **[] values();
      public static ** valueOf(java.lang.String);
}

OK, problem found: I just need to add this 'keepclassmembers' exception in ProGuard project configuration file.

But what are values() and valueof(java.lang.String) needed for? I still was curious how exactly I received NullPointerException in java.lang.Enum entrails. Let's better understand it.

Programmatically filtering your application's logs from Logcat output

Note: this post discusses filtering Logcat output on devices running Android prior to version 4.1. Starting from Android 4.1 applications are allowed to read only their own logs.

Logcat isn't much convenient when it comes to filtering your application logs. In fact, if you look at the logcat documentation, you will find that you can filter logcat output only by priority and/or tag fields:

A filter expression follows this format tag:priority ..., where tag indicates the tag of interest and priority indicates the minimum level of priority to report for that tag. Messages for that tag at or above the specified priority are written to the log. You can supply any number of tag:priority specifications in a single filter expression. The series of specifications is whitespace-delimited.

The android.util.Log class is your interface for printing your application logs to the Logcat system. Common print statement looks like this:

Log.e(TAG, "something bad happened");

Invoked method specifies the priority of the log record - in this example e stands for error, and TAG specifies well.. the tag.
 I commonly set the class name as a TAG using this statement:

private static final String TAG = ClassName.class.getCanonicalName();

Resolving the class name at run-time allows me to stay flexible and be adjusted to all the obfuscations made by Proguard processing.

But here's a problem: given a project with a hundred classes, how would you filter Logcat's output to print just your application's logs?

Solving errors in Android Studio's new project creation

The other day I upgraded my Android Studio bundle to latest stable version: android-studio-bundle-132.893413-windows.

Shortly afterwards, I noticed that each time I tried to create a new project, it ended with the following error message:

Cause: error in opening zip file. Consult IDE log for more details (Help | Show Log)

Hmmmm.. which error file exactly and what's wrong with it?