Thursday, April 23, 2015

Commons Codec on Android

If your application depends on Commons Codec library (version 1.4 and above), and you are not aware that Android framework already includes this library, then there's a chance that your application will crash with a runtime exception.


Why does the Android framework includes the Commons Codec library?

Android framework shipped with HttpClient (version 4.0 pre-BETA, an ancient 2008 release). HttpClient depends on Commons Codec. Therefore, AOSP had to include Commons Codec as well.


OK, so can I use framework's Commons Codec library in my application?

Unfortunately no. This API is hidden and not exposed in Android SDK for application developers. You can invoke the library's classes using reflection of course, but by doing so you take the risk of relying on undocumented APIs, which is not recommended at all.


What is the version of Commons Codec library that shipped with Android framework?

Version 1.3 (released in July, 2004), but without org.apache.commons.codec.digest.DigestUtils class.


What's wrong with including more advanced version (i.e. v1.9) as a library in my application?

Your application's Commons Codec classes will collide with those that are loaded as part of the framework. The dexopt process will identify conflicts and won't allow loading your classes.

Fortunately, for each such conflict dexopt will print a warning to logcat (when using Dalvik runtime; on ART those logs are omitted for some reason):
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/Decoder;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/DecoderException;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/Encoder;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/EncoderException;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/StringEncoderComparator;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/DoubleMetaphone$DoubleMetaphoneResult;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/SoundexUtils;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/RFC1522Codec;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/BinaryDecoder;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/BinaryEncoder;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/StringDecoder;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/StringEncoder;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/binary/BinaryCodec;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/binary/Hex;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/DoubleMetaphone;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/Metaphone;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/RefinedSoundex;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/Soundex;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/BCodec;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/QCodec;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/QuotedPrintableCodec;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/URLCodec;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/binary/Base64;' has an earlier definition; blocking out
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/Decoder;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/DecoderException;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/Encoder;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/EncoderException;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/StringEncoderComparator;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Hex;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/DoubleMetaphone$DoubleMetaphoneResult;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/SoundexUtils;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/RFC1522Codec;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/DecoderException;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/BinaryDecoder;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/BinaryEncoder;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/StringDecoder;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/StringEncoder;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Base64;'
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Base64;'
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Base64;'
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/DecoderException;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/binary/BinaryCodec;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/binary/Hex;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/DoubleMetaphone;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/Metaphone;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/RefinedSoundex;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/Soundex;': multiple definitions
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/BCodec;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/QCodec;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/QuotedPrintableCodec;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/URLCodec;': multiple definitions
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/binary/Base64;': multiple definitions

Even if you're compiling against newer Commons Codec version, your application will use framework's version 1.3 classes at runtime. This happens because version 1.3 classes exist in boot classpath, and therefore classloader will always load them in preference to the version you've packaged in your application.

That said, everything may work well if you don't use functionality that was introduced in version 1.4 or later. But what if you do use? For example, version 1.4 introduced new method -Base64#encodeBase64String. If I'll add dependency to Commons Codec 1.9 and call this method in my application, the compiler won't complain. But executing this code on the device will produce the following runtime exception:
18917-18917/info.osom.sandbox E/AndroidRuntime? FATAL EXCEPTION: main
    Process: info.osom.sandbox, PID: 18917
    java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeBase64String
            at info.osom.sandbox.MainActivity.onCreate(MainActivity.java:48)
            at android.app.Activity.performCreate(Activity.java:5231)
            ...
            at dalvik.system.NativeStart.main(Native Method)
I copied the above error log from a device running KitKat with Dalvik. Lollipop's (with ART) error is a bit more verbose:
898-898/info.osom.sandbox E/AndroidRuntime? FATAL EXCEPTION: main
    Process: info.osom.sandbox, PID: 898
    java.lang.NoSuchMethodError: No static method encodeBase64String([B)Ljava/lang/String; in class Lorg/apache/commons/codec/binary/Base64; or its super classes (declaration of 'org.apache.commons.codec.binary.Base64' appears in /system/framework/ext.jar)
            at info.osom.sandbox.MainActivity.onCreate(MainActivity.java:48)
            at android.app.Activity.performCreate(Activity.java:5990)
            ...
            at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)


So what shall I do?

You can avoid the aforementioned conflicts by changing the classes namespace, that is, the Commons Codec package. Either:
  • do it manually - grab the Commons Codec (version 1.9 for example) sources and then refactor the code by changing the package name (you can find an example for someone who took this approach here); 
  • or repackage the official release jar file using jarjar tool
jarjar basically allows to alter the bytecode according to specified rules. One of the available rules provides the ability to rename class names. Here's an example for repackaging version 1.9 of Commons Codec:

Get the latest jarjar jar from here, and run (I'm using jarjar-1.4) this command in terminal:
java -jar jarjar-1.4.jar process commons-codec.rules commons-codec-1.9.jar commons-codec-1.9-rep.jar
where commons-codec.rules is a plain text file with the following single line:
rule org.apache.** android.org.apache.@1
This rule (see more on rules here) tells the jarjar tool to add android prefix to all the class names within org.apache package, and put those classes in a new jar called commons-codec-1.9-rep.jar. For example, org.apache.commons.codec.binary.Base64 class will be renamed to android.org.apache.commons.codec.binary.Base64.

Add the repackaged jar to your application instead of original Commons Codec jar. This will allow both framework's Base64 class and your (newer version) Base64 class to coexist within application.

You can even create automated task and use it as part of your build process: check out the jarjar-gradle plugin for Gradle or even this task if you still use ant.

10 comments:

  1. Great Explanation..But i am stil getting the same erro

    05-27 15:03:51.087: E/AndroidRuntime(31659): java.lang.NoSuchMethodError: No virtual method decode(Ljava/lang/String;)[B in class Lorg/apache/commons/codec/binary/Base64; or its super classes (declaration of 'org.apache.commons.codec.binary.Base64' appears in /system/framework/ext.jar)

    ReplyDelete
    Replies
    1. @John You're trying to use org.apache.commons.codec.binary.Base64#decode method that receives String as parameter and returns byte array. This function was added in version 1.4, and therefore won't be available at runtime. To solve it, see the suggested solution above in the post.

      Delete
  2. Thanks man - you saved my day!
    I added a little GH-Repo with script and rule to generate the new JAR. Repo shows also how to generate the according POM for a private Mave-Repo:
    https://github.com/MikeMitterer/tools-repackage-commons-codec

    ReplyDelete
  3. Thank you so much man!

    I had the same issue with Apache Commons Lang jar (3.2 and above) - but only on all phones from the vendor Xiomi. Apparently, these phones have an old version of Apache Commons Lang as part of system runtime. So I used to get exceptions like:

    STACK_TRACE=java.lang.NoSuchMethodError: org.apache.commons.lang3.mutable.MutableBoolean.setTrue

    STACK_TRACE=java.lang.NoSuchMethodError: org.apache.commons.lang3.StringEscapeUtils.escapeXml10

    Following your advice, I used jarjar tool to rename the package namespace to something unique to my app.

    Great work.

    ReplyDelete
  4. Same error reported in android app using google.cloud.speech.v1p1beta1 but not with google.cloud.speech.v1. Not sure why the beta version doesn't use the same dependencies. Is there a way to force the beta version to use latest commons codec versus legacy version?
    Thanks,
    Gary

    ReplyDelete
  5. Great work! But what if I'm using a third part lib which use newer version Common-Codec and met the same issue? Do you have any suggestion to address this problem?

    ReplyDelete
    Replies
    1. In this case, I'd probably download both the third party library and its dependencies as jars, jarjar the libraries that reference commons-codec and add them to my project as jars (as opposed to maven dependency).
      There're also gradle plugins that automate that process - as an example see how I'm doing that in one of my projects - https://github.com/alipov/ews-android-api/blob/master/ews-android-api/build.gradle#L11

      Delete
    2. Example for such plugin: https://github.com/vRallev/jarjar-gradle

      Delete