Compiling libcurl for Android

Curl is a pretty nifty package and it can be useful on lots of platforms to provide a fairly sane way to contacting and downloading network stuff (webpages for example).  Now to only get this to work on Android for games or whatever...

This was a bit tricky.

I used the standalone toolchain with API level 9 (gingerbread, Android 2.3) and did not use ndk-build script.

Two things ultimately had to be done before libcurl would successfully compile for Android using the toolchain.  I opted for using a Makefile and good old fashioned make... but there was some "convincing" needed to be done to make this happen.

First, I defined two symbols during compile time.  This would be -DANDROID_NDK and -DHAVE_CONFIG_H.  You will want the second one as is, the first can be renamed to whatever you want so long as you know this is selecting Android specific code during compilation and this is the same symbol to check for below.

Now, you need to add a section in [curl]/include/curlbuild.h

I added the following code above the "penultimate" section for MSVC.  These defines MUST go in curlbuild.h unless you want to make curl angry with you.

#elif defined(ANDROID_NDK)
#  define CURL_SIZEOF_LONG           4
#  define CURL_TYPEOF_CURL_OFF_T     long long
#  define CURL_FORMAT_CURL_OFF_T     "lld"
#  define CURL_FORMAT_CURL_OFF_TU    "llu"
#  define CURL_FORMAT_OFF_T          "%lld"
#  define CURL_SIZEOF_CURL_OFF_T     8
#  define CURL_SUFFIX_CURL_OFF_T     LL
#  define CURL_SUFFIX_CURL_OFF_TU    ULL
#  define CURL_TYPEOF_CURL_SOCKLEN_T socklen_t
#  define CURL_SIZEOF_CURL_SOCKLEN_T 4
#  define CURL_PULL_SYS_TYPES_H      0
#  define CURL_PULL_SYS_SOCKET_H     0

This defines some macros needed to build curl for Android, but wait, there's more.

We also have to make curl know we want to use a curl_config.h file (which is normally generated with ./configure, but we like to do things our own way).

Because we defined HAVE_CONFIG_H (-DHAVE_CONFIG_H), curl is now instructed to look for this file when compiling.

It took some trial and error, and this probably isn't totally correct, but I got curl to compile and seemingly functional with this manually constructed file.  It originally was the windows config file I think warped to a more linux-like configuration.  At the end of the night (read: week), it resulted as follows:

---

#ifndef HEADER_CURL_CONFIG_H
#define HEADER_CURL_CONFIG_H

/* ================================================================ */
/*               Hand crafted config file for Android               */
/* ================================================================ */

/* ---------------------------------------------------------------- */
/*                          HEADER FILES                            */
/* ---------------------------------------------------------------- */

/* Define if you have the <arpa/inet.h> header file. */
#define HAVE_ARPA_INET_H 1 */

/* Define if you have the <assert.h> header file. */
#define HAVE_ASSERT_H 1

/* Define if you have the <crypto.h> header file. */
/* #define HAVE_CRYPTO_H 1 */

#define HAVE_PERROR 1
#define HAVE_PIPE 1
#define HAVE_POLL 1

#define HAVE_GETADDRINFO 1
#define HAVE_FREEADDRINFO 1

#define HAVE_PWD_H 1

/* Define if you have the <errno.h> header file. */
#define HAVE_ERRNO_H 1

/* Define if you have the <err.h> header file. */
#define HAVE_ERR_H 1 */

/* Define if you have the <fcntl.h> header file. */
#define HAVE_FCNTL_H 1

/* Define to 1 if you have a working fcntl O_NONBLOCK function. */
#define HAVE_FCNTL_O_NONBLOCK 1

/* Define if you have the <getopt.h> header file. */
#define HAVE_GETOPT_H 1

/* Define if you have the <io.h> header file. */
/* #define HAVE_IO_H 1 */

/* Define if you have the <limits.h> header file. */
#define HAVE_LIMITS_H 1

/* Define if you have the <locale.h> header file. */
#define HAVE_LOCALE_H 1

/* Define if you have the <netdb.h> header file. */
#define HAVE_NETDB_H 1

/* Define if you have the <netinet/in.h> header file. */
#define HAVE_NETINET_IN_H 1

/* Define if you have the <process.h> header file. */
/* #define HAVE_PROCESS_H 1 */

/* Define if you have the <signal.h> header file. */
#define HAVE_SIGNAL_H 1

/* Define if you have the <sgtty.h> header file. */
/* #define HAVE_SGTTY_H 1 */

/* Define if you have the <ssl.h> header file. */
/* #define HAVE_SSL_H 1 */

/* Define if you have the <stdlib.h> header file. */
#define HAVE_STDLIB_H 1

/* Define if you have the <sys/param.h> header file. */
#define HAVE_SYS_PARAM_H 1

/* Define if you have the <sys/select.h> header file. */
#define HAVE_SYS_SELECT_H 1

/* Define if you have the <sys/socket.h> header file. */
#define HAVE_SYS_SOCKET_H 1

/* Define if you have the <sys/sockio.h> header file. */
/* #define HAVE_SYS_SOCKIO_H 1 */

/* Define if you have the <sys/stat.h> header file. */
#define HAVE_SYS_STAT_H 1

/* Define if you have the <sys/time.h> header file. */
#define HAVE_SYS_TIME_H 1

/* Define if you have the <sys/types.h> header file. */
#define HAVE_SYS_TYPES_H 1

/* Define if you have the <sys/utime.h> header file. */
/* #define HAVE_SYS_UTIME_H 1 */

/* Define if you have the <termio.h> header file. */
/* #define HAVE_TERMIO_H 1 */

/* Define if you have the <termios.h> header file. */
/* #define HAVE_TERMIOS_H 1 */

/* Define to 1 if you have the <sys/uio.h> header file. */
#define HAVE_SYS_UIO_H 1

/* Define to 1 if you have the <sys/un.h> header file. */
#define HAVE_SYS_UN_H 1

/* Define to 1 if you have the <sys/utime.h> header file. */
#define HAVE_SYS_UTIME_H 1

/* Define to 1 if you have the <termios.h> header file. */
#define HAVE_TERMIOS_H 1

/* Define to 1 if you have the <termio.h> header file. */
/* #undef HAVE_TERMIO_H */

/* Define to 1 if you have the <time.h> header file. */
#define HAVE_TIME_H 1

/* Define if you have the <unistd.h> header file. */
#define HAVE_UNISTD_H 1

/* ---------------------------------------------------------------- */
/*                        OTHER HEADER INFO                         */
/* ---------------------------------------------------------------- */

/* Define if sig_atomic_t is an available typedef. */
#define HAVE_SIG_ATOMIC_T 1

/* Define if you have the ANSI C header files. */
#define STDC_HEADERS 1

/* Define if you can safely include both <sys/time.h> and <time.h>. */
#define TIME_WITH_SYS_TIME 1

/* ---------------------------------------------------------------- */
/*                             FUNCTIONS                            */
/* ---------------------------------------------------------------- */

/* Define if you have the closesocket function. */
/* #define HAVE_CLOSESOCKET 1 */

/* Define if you don't have vprintf but do have _doprnt. */
/* #define HAVE_DOPRNT 1 */

/* Define if you have the ftruncate function. */
#define HAVE_FTRUNCATE 1

/* Define if you have the gethostbyaddr function. */
#define HAVE_GETHOSTBYADDR 1

/* Define if you have the gethostname function. */
#define HAVE_GETHOSTNAME 1

/* Define if you have the getpass function. */
#define HAVE_GETPASS 1

/* Define if you have the getservbyname function. */
#define HAVE_GETSERVBYNAME 1

/* Define if you have the getprotobyname function. */
#define HAVE_GETPROTOBYNAME

/* Define if you have the gettimeofday function. */
#define HAVE_GETTIMEOFDAY 1

/* Define if you have the inet_addr function. */
#define HAVE_INET_ADDR 1

/* Define if you have the ioctlsocket function. */
/* #define HAVE_IOCTLSOCKET 1 */

/* Define if you have a working ioctlsocket FIONBIO function. */
/* #define HAVE_IOCTLSOCKET_FIONBIO 1 */

/* Define if you have the perror function. */
#define HAVE_PERROR 1

/* Define if you have the RAND_screen function when using SSL. */
#define HAVE_RAND_SCREEN 1

/* Define if you have the `RAND_status' function when using SSL. */
#define HAVE_RAND_STATUS 1

/* Define if you have the `CRYPTO_cleanup_all_ex_data' function.
   This is present in OpenSSL versions after 0.9.6b */
#define HAVE_CRYPTO_CLEANUP_ALL_EX_DATA 1

/* Define if you have the select function. */
#define HAVE_SELECT 1

/* Define if you have the setlocale function. */
#define HAVE_SETLOCALE 1

/* Define if you have the setmode function. */
#define HAVE_SETMODE 1

/* Define if you have the setvbuf function. */
#define HAVE_SETVBUF 1

/* Define if you have the socket function. */
#define HAVE_SOCKET 1

/* Define if you have the strcasecmp function. */
#define HAVE_STRCASECMP 1

/* Define if you have the strdup function. */
#define HAVE_STRDUP 1

/* Define if you have the strftime function. */
#define HAVE_STRFTIME 1

/* Define if you have the stricmp function. */
#define HAVE_STRICMP 1

/* Define if you have the strncasecmp function. */
#define HAVE_STRNCASECMP 1

/* Define if you have the strnicmp function. */
#define HAVE_STRNICMP 1

/* Define if you have the strstr function. */
#define HAVE_STRSTR 1

/* Define if you have the strtoll function. */
#define HAVE_STRTOLL 1

/* Define if you have the tcgetattr function. */
#define HAVE_TCGETATTR 1

/* Define if you have the tcsetattr function. */
#define HAVE_TCSETATTR 1

/* Define if you have the utime function. */
#define HAVE_UTIME 1

/* Define to the type qualifier of arg 1 for getnameinfo. */
#define GETNAMEINFO_QUAL_ARG1 const

/* Define to the type of arg 1 for getnameinfo. */
#define GETNAMEINFO_TYPE_ARG1 struct sockaddr *

/* Define to the type of arg 2 for getnameinfo. */
#define GETNAMEINFO_TYPE_ARG2 socklen_t

/* Define to the type of args 4 and 6 for getnameinfo. */
#define GETNAMEINFO_TYPE_ARG46 size_t

/* Define to the type of arg 7 for getnameinfo. */
#define GETNAMEINFO_TYPE_ARG7 int

/* Define if you have the recv function. */
#define HAVE_RECV 1

/* Define to the type of arg 1 for recv. */
#define RECV_TYPE_ARG1 int

/* Define to the type of arg 2 for recv. */
#define RECV_TYPE_ARG2 void *

/* Define to the type of arg 3 for recv. */
#define RECV_TYPE_ARG3 size_t

/* Define to the type of arg 4 for recv. */
#define RECV_TYPE_ARG4 int

/* Define to the function return type for recv. */
#define RECV_TYPE_RETV ssize_t

/* Define if you have the recvfrom function. */
#define HAVE_RECVFROM 1

/* Define to the type of arg 1 for recvfrom. */
#define RECVFROM_TYPE_ARG1 int

/* Define to the type pointed by arg 2 for recvfrom. */
#define RECVFROM_TYPE_ARG2 void

/* Define to the type of arg 3 for recvfrom. */
#define RECVFROM_TYPE_ARG3 size_t

/* Define to the type of arg 4 for recvfrom. */
#define RECVFROM_TYPE_ARG4 int

/* Define to the type pointed by arg 5 for recvfrom. */
#define RECVFROM_TYPE_ARG5 struct sockaddr

/* Define to the type pointed by arg 6 for recvfrom. */
#define RECVFROM_TYPE_ARG6 socklen_t

/* Define to the function return type for recvfrom. */
#define RECVFROM_TYPE_RETV ssize_t

/* Define if you have the send function. */
#define HAVE_SEND 1

/* Define to the type of arg 1 for send. */
#define SEND_TYPE_ARG1 int

/* Define to the type qualifier of arg 2 for send. */
#define SEND_QUAL_ARG2 const

/* Define to the type of arg 2 for send. */
#define SEND_TYPE_ARG2 void *

/* Define to the type of arg 3 for send. */
#define SEND_TYPE_ARG3 size_t

/* Define to the type of arg 4 for send. */
#define SEND_TYPE_ARG4 int

/* Define to the function return type for send. */
#define SEND_TYPE_RETV ssize_t

/* ---------------------------------------------------------------- */
/*                       TYPEDEF REPLACEMENTS                       */
/* ---------------------------------------------------------------- */

/* Define to the return type of signal handlers (int or void). */
#define RETSIGTYPE void

/* ---------------------------------------------------------------- */
/*                            TYPE SIZES                            */
/* ---------------------------------------------------------------- */

/* Define to the size of `int', as computed by sizeof. */
#define SIZEOF_INT 4

#define SIZEOF_LONG 4

/* Define to the size of `long double', as computed by sizeof. */
#define SIZEOF_LONG_DOUBLE 12

/* Define to the size of `long long', as computed by sizeof. */
#define SIZEOF_LONG_LONG 8

/* Define to the size of `short', as computed by sizeof. */
#define SIZEOF_SHORT 2

/* Define to the size of `size_t', as computed by sizeof. */
#define SIZEOF_SIZE_T 4

/* ---------------------------------------------------------------- */
/*                          STRUCT RELATED                          */
/* ---------------------------------------------------------------- */

/* Define if you have struct sockaddr_storage. */
#define HAVE_STRUCT_SOCKADDR_STORAGE 1

/* Define if you have struct timeval. */
#define HAVE_STRUCT_TIMEVAL 1

/* Define if struct sockaddr_in6 has the sin6_scope_id member. */
#define HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID 1

/* ---------------------------------------------------------------- */
/*                        COMPILER SPECIFIC                         */
/* ---------------------------------------------------------------- */

/* Define to nothing if compiler does not support 'const' qualifier. */
/* #define const */

/* Define to nothing if compiler does not support 'volatile' qualifier. */
/* #define volatile */

/* Windows should not have HAVE_GMTIME_R defined */
/* #undef HAVE_GMTIME_R */

/* Define if the compiler supports C99 variadic macro style. */
#define HAVE_VARIADIC_MACROS_C99 1

/* gcc variadic macros */
#define HAVE_VARIADIC_MACROS_GCC 1

/* Define if the compiler supports the 'long long' data type. */
#define HAVE_LONGLONG 1

#define SIZEOF_TIME_T 4

/* ---------------------------------------------------------------- */
/*                       DNS RESOLVER SPECIALTY                     */
/* ---------------------------------------------------------------- */

/*
 * Undefine both USE_ARES and USE_THREADS_WIN32 for synchronous DNS.
 */

/* Define to enable c-ares asynchronous DNS lookups. */
/* #define USE_ARES 1 */

/* Define to enable threaded asynchronous DNS lookups. */
/* #define USE_THREADS_WIN32 1 */

#if defined(USE_ARES) && defined(USE_THREADS_WIN32)
#  error "Only one DNS lookup specialty may be defined at most"
#endif

/* ---------------------------------------------------------------- */
/*                           LDAP SUPPORT                           */
/* ---------------------------------------------------------------- */

#define CURL_DISABLE_LDAP 1
#define CURL_DISABLE_LDAPS 1

/* ---------------------------------------------------------------- */
/*                       ADDITIONAL DEFINITIONS                     */
/* ---------------------------------------------------------------- */

/* Define cpu-machine-OS */
#undef OS
#define OS "arm-android-linux"

/* Name of package */
#define PACKAGE "curl"

/* If you want to build curl with the built-in manual */
#define USE_MANUAL 1

#define ENABLE_IPV6 1

#endif /* HEADER_CURL_CONFIG_H */

Java obfuscation, ndk, jni, and proguard.

We recently had the pleasure of posting an update to one of our apps - Bubble Zing - on to a couple Android markets.  Everything looked great on the development side and everything was pretty much tested and ready to go, so we exported our apk in Eclipse and sent it out!

Then the problems started trickling in (of course).

This was quite baffling that the same code running in Eclipse was completely breaking when running from an exported/signed APK file.  What was going on here?!  I googled around for various errors and almost all of them talked about missing libraries and such.  Well, none of those things were my issue.

I finally started getting some useful information from logcat about ClassNotFound and being unable to get new instances of *my activity* or various other failures about FindClass failing.

I was pretty stumped by the whole thing, so I ended up opening the APK like a zip file to check out what was actually IN the apk.  Everything looked as it should have been.  All my resources were there, the class path stuff was set as well (really confusing me why all this classnotfound stuff was happening now).  It wasn't until I ran apktool on my apk to get some actual additional information about my java files that I started to trace down the issue.  I noticed that all of my Java classes which were called from my native library were named "a.java" and "b.java", etc.  As opposed to all the normal java files which were not interacting my native shared library were fine.  So, it seems like we narrowed down what sort of thing was happening here.

Ok, somewhere in the export process, I was was losing my actual java names... and with things like fully-qualified names everywhere, I suspect this was a major problem.

So, I found that Proguard was obfuscating away my class names during the export process, meaning once my app was in apk form, calls from native code were completely unable to find half of what it needed to talk to for it to work!

For those interested, I added the following lines to proguard.cfg to inform proguard that I would like to keep most of my stuff unmangled please:


-keep public class [FULLY.QUALIFIED.PACKAGE.NAME].* {
public static *;
public static *;
private static *;
}

For us, only static methods were being called from native code.  We also set *all* of our classes in our package to be saved with the "-keep public class" line.


In retrospect, this totally makes sense, since during the apk packaging process, it realized that at no point were some of my chunks of java code being called by any other java code, so proguard figured it was ok to munge those up in to various cryptic alternate symbols.  I suppose it wasn't proguard's fault, but really my own misunderstanding of how the export process works and what proguard did for you automatically (or why).

I'm glad that was over, lesson learned?!  Always test an APK on my devices BEFORE sending it out to the market, even if we completely ran through all our tests before the export process.

Welcome to iOS

We've been sitting on our iOS developer license for a little while since we've been fairly occupied catering to the problematic nature of our Android apps.  As soon as we can get our existing products on Android to an acceptable level we'll begin porting Bubble Zing and possibly Simple Match over to iOS.

One of the major changes in our process and code was to put as much game logic in to portable C code for Bubble Zing so that, in theory, we could just wrap the game code in to iOS objective-c wrappers and retain the java wrappers for Android.  Of course, this major shift in philosophy warranted a huge time undertaking on our Android ports as we essentially had to recode everything... meaning we may as well put a lot of code in to libraries and that whole thing.

On the plus side, all this "nativeification" of game code has excellent performance indicators and should increase our product exposure to older devices that needed all the speedups (fairly badly might I add).

I suppose this process was inevitable anyways.  I mean after all, Bubble Zing 1.0.3 (the latest public version as of this writing), showed that just drawing a background graphic during gameplay - JUST DRAWING THE BACKGROUND - took 60% of the processor in several test devices.  That was horrendous and completely unavoidable due to the nature of java, dalvikvm, and opengl es.  The only way around it was to go native code and start screwing with lower level stuff... and this has already paid off in huge ways... but it was not without its own drawbacks and tradeoffs.  Specifically, things that use to work great now require extensive rework and compromises.  Some things now look "worse" than in 1.0.3, but perform a ton better, yet the overall gameplay experience has been increased a lot.

Anyways, once we get Bubble Zing 1.1.0 up and in to public hands, our iOS adventure shall begin!  We took a look at XCode and their iOS stuff already, and I must say it was quite welcoming from a developer's perspective and I'm going to assume from a business perspective as well once we get there!

Hopefully from here on out, all our products will see a simultaneous Android and iOS release at the same time!

So many Android devices to cater to...

In follow up to my earlier post about being envious of software houses of yesteryear, I have to admit that in a sense, they had it easier.

When they made their games and products, they had a single target system with known specs and limitations.  Code that relied on *timing* of events and things like this can be planned for meticulously.  Developers also knew 100% ahead of time what features a system did or did not have.  Yet, they also dealt with 8-bit / 16-bit processors, no FPU, no advanced graphics pipeline, and all sorts of "limitations", and they still cranked out great products after great products all without needing all these patches and what not.

Well, fast forward a bunch of years, and here we're in the age of the smartphone.  In some regards, we have it easy - a powerful CPU (relatively speaking), an amazing graphics processor, and sometimes a FPU.  Wait, sometimes?!  Yes, that's right, if you're a developer for the Android platform, you can't be guaranteed to have a floating point unit.  You also have no guarantee of having any given amount of RAM, disk space, or really much of anything.  Sure, you can *use* float data types, but on devices with no dedicated float processor, it is software emulated!!! Talk about a massive performance hit if you want to do updates and renders at any sane framerate!

Our earlier game, Bubble Zing, ran amazingly awesome on newer Android phones, but on older phones, it ran terribly.  With the 1.0.3 release of the game last month, we thought all major compatibility issues were resolved; well, that was only partially correct.  The major bugs seemingly were ironed out, and the core game "worked", but with no guarantee of how well it would work - sometimes outright unplayable on some devices!

So, we delve further in to the "why is this running like terrible on x,y,z,q,f devices, but great on others".  Well, this is where the things Android does is both great and bad at the same time.  We coded 1.0.3 in 100% Java as this was seemingly the correct way to make any app on Android devices.  In theory, it sounded good since Java apps are suppose to work 100% on devices with a compatible JVM (ie, all Android devices) - clearly something we wanted, and it is fairly easy to crank a Java program much faster than an equivalent C/C++ program, so it sounded like a great idea at the time.

Our test system was a Droid RAZR with Android 2.3.6.  Our initial testing was running well, FPS was over 40 and everything seemed great.  The emulator running an AVD 1.5 seemed sluggish but that could be written off as being an emulator and running not on the most powerful host computer.

Things got really confusing and exciting when we tested 1.0.1 and 1.0.2 on a Samsung Galaxy tablet -- that's when all hell broke loose.  Everything that should have been working in theory wasn't.  On top of that, the frame rate was a dismal 10 to 15!  What the flying expletive!  How can a program go from 40+ FPS to 10!  It wasn't an issue with the version of the Android OS, it must have been something device specific.

I ran a traceview to track what was eating all the processing time and it came down to rendering a full screen background PNG file.  On the phone, this wasn't such a big deal, but on the tablet, it was eating almost 60% of all CPU time!

We're sitting here with a complete Java game that worked ok on an emulator and great on our target test phone, yet are dumbfounded with why the same exact Java code on the tablet ran so terribly.

A lot of trial and error later we figured that to get things working faster and better, the only way was to ditch Java and start programming on a closer-to-the-hardware level as we could to squeeze out a lot more performance and manage memory more effectively.  Java made it very frustrating with garbage collecting at some fairly inconvenient times and this compounded other issues making the game run very badly at certain instances.  It was time to bring on the NDK.

Frustration is what is in store for Android developers.  It can be done, but good apps will need a very persistent set of people to make these products and make them well.

Oh, and I'm still in awe of game programmers from decades ago on consoles, but at the same time, I'm jealous of their "single architecture" deployment platform as that probably simplified a lot of these problems thousands fold.  I suppose this is another reason Apple is winning the smartphone wars with developers as it is "easier" to make apps on iOS with very heavily controlled hardware features by Apple.

Android NDK loading resources and PNGs

After following the nice code sample from http://androgeek.info/?p=275 about how to load a PNG file by accessing the Android APK file directly using only the NDK, everything seemed awesome.

It uses libzip and libpng to manually load the APK, then load a png file right from the APK itself without using any Java or the Android SDK.

Why would you want to do this?  Well, aside from liking complicated stuff, it is useful for us because now we can put all of our "portable" resource loading code in c/c++ files so it is more easily moved to iOS, windows, whatever, without being bound to using only the Android SDK, thus having to recode loading logic over and over again for each target deployment architecture.

Everything worked well, until something didn't go quite right.  Some of my textures were corrupted and worse, some actually crashed my app!

It took 4 days to track down what exactly was happening.

Common problems usually indicate I was doing something wrong with opengl and the textures themselves...

* All my textures were perfect powers of two (64x64, 128x128, etc)
* All my textures were 32-bit RGBA files
* I fixed a warning and error from the given sample code to correctly read the libzip'd apk file

So, it was time to dig deeper with what was going on.

First, I noticed that my glGenTextures calls were giving me very large opengl texture ids so I thought this was a huge problem as it was not continuous numbers, and in C, any time you have generating wild numbers, it usually means a bad pointer somewhere.  I spent about a day analyzing and referencing code and started to realize that the generated numbers were always the same (this usually rules out a memory corruption issue if the results are *always* the same).

I noticed that the generated opengl texture ids always followed this pattern on my Droid RAZR:
100271
315638026
534244737
1505553200
-1563003837
...
etc.

Yet, on my Android emulator, the texture id's were:
1
2
3
4
5
...
etc

I had found this thread on google groups that I thought was my problem since it sounded really similar, but it didn't actually help.

Knowing that glGenTextures expects an *unsigned* integer as the data type, I realized that the negative number was an overflowed signed integer and actually nothing to worry about.  In my debugging print code I needed to change "%d" to "%u" to see the actual unsigned value.

Since these numbers repeated on every subsequent run of the program, I decided to test the idea that maybe my code had nothing at all to do with the "corrupt" large texture ids.  Indeed, I made a program that literally only called glGenTextures about 100 times right at startup time and did nothing else.  The result?  The exact same large numbers showed up in my device, and was "normal" on the emulator, as expected.  This left me with the conclusion that whatever logic is in my Motorola device in terms of picking a unique texture ID that this was normal - next.

Yet, this didn't answer my question as to why I was seeing an app crash occur and seemingly only when loading a certain texture or two.  My first order of business was to rule out if my other code was at fault, or was it somehow related to ONLY those sets of textures (somehow). Strangely, dozens of tests indicated that it was only the problem of 3 specific textures being loaded as I added lots of additional textures, even loaded some textures multiple times, then finally ONLY loading one of those 3 problematic textures.  The app would only crash at loading any of the 3.

Frustrated, I ended up putting a ton of extra debugging code all over my apps trying to track down what in the world was wrong.  All indicators were pointing to the idea that a problem was with either libzip or libpng since libc.so itself was crashing in a call to a libpng function -- great, time to track down potential bugs in those libraries.

Now, let me put a disclaimer up here, it is almost never a good idea to assume a mature code library has a bug in it; especially one this severe, yet here I am with a consistent crash always calling a specific line of code that libpng was using leaving me to believe this could be the case.

First, I downloaded the most recent version of each library and replaced the old versions that were provided in the linked post above.  I was now using libzip 0.10.1 and libpng 1.5.9.  This had promise since each of those had a bit of fixes/enhancements along the way.  I was hopeful that this would magically fix my problems.

Well, it did - sort of.

Whatever wizardry was happening internally in those libraries helped indicate that libpng was crashing specifically on this line:

png_read_info(png_ptr, info_ptr);

Great, all this work only to help reinforce the idea that libpng was broken for my special case.  Yet, I still couldn't rule out that libzip somehow corrupted my image data while decompressing my APK so libpng might not be at fault trying to load corrupted data.  Worse, all error checking came up empty for all libzip and libpng calls!  I even checked my opengl calls for errors using glError -- NOTHING!

Well, the next step was to dive in to those library's source code and start tracking stuff down, the last resort I guess was to find the problem in one of these libraries and fix it myself and give the respective team a patch!

One of my testing tangents included making a stand alone C project to test very specific sets of code to completely rule out any Android Java JNI mysticism causing problems.  Everything was pointing to only those textures being a problem in the libpng loading code.  So I ran the libpng test program on those 3 test pngs and the only thing that sounded remotely threatening was:

Files aaaa.png and pngout.png are different
Was aaaa.png written with the same maximum IDAT chunk size (8192 bytes), filtering heuristic (libpng default), compression level (zlib default), and zlib version (1.2.5)?

It certainly sounded menacing and I was convinced that somehow my textures were corrupted.  I ended up regenerating those textures using the latest version of Gimp (2.6.11 at the time) -- plugged those in to my app in excitement; and they still crashed.

Strangely, the libpng test program generated pngout.png which was seemingly the same exact copy of my original texture, but a different filesize; something was different, and different was good at this point.  For giggles, I put that pngout.png file in to my Android's APK res/raw folder in place of my original texture and much to my surprise the texture loaded!!!  Ah-hah, the answer finally!

Unfortunately, things weren't over yet, the texture loaded completely corrupted.  Everything was misaligned, the colors were wrong and the texture was completely garbled with random colors all over it.  Basically, I've gone nowhere.

3 days in and I'm still struggling with what is wrong here.  I'm still baffled, why is it that my 10 other textures work fine, yet these 3 don't want to work at all?!  I decided to write my own texture loading function from scratch to ensure I understood exactly every step of logic that was taking place.  The same result happened.  I did all sorts of strange additional tests, hex dumps, you name it.  I was at the end of my rope on this one, I was getting angry.

Finally, I saw an interesting article somewhere that said I needed to rename my png files to mp3 files.  I thought to myself, that's the dumbest thing I ever heard - why would I ever want to do that?!  Well, if you're programming for Android, you might be crazy enough to hear this out.

Apparently, during the APK packing process *SOME* of your png textures are automagically compressed and others are not.  Renaming your png files tells the apk packing process to not mangle/compress that file, and my happy 3 textures were some of the lucky selected ones for this process; which completely broke the texture loading code.  In a move of desperation, I did rename my files to mp3 to verify this was the case, and they worked perfectly.  I was enraged and ecstatic at the same time.

On the bright side I got a better understanding of both libpng and libzip (we got a little familiar with each other, if you know what I mean).

Alright, so the magic bullet answer is that I need to rename all my PNG files to MP3 (test.png.mp3 if you will) or simply dump them all in to a game data file of sorts (probably the better idea).

So, I make a quick resource packing program (using zlib) and I find out that during the APK packing process again, my GZ files are also modified!!!

I quickly found this link explaining it a little bit.

I wonder what other goodies are in store for me to discover by accident!?

Hope this article is found by anyone else stuck in my situation and save you all a couple days of madness.