Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Tuesday, January 15, 2013

Back to Basics and Understanding Application Performance

The article I'm linking here was written back in 2001, it is "old" in technology terms, yet is still 100% completely accurate and relevant!

http://joelonsoftware.com/articles/fog0000000319.html

Java, C#, .NET, database languages like MySQL, and things like this all benefit from someone with a nice, solid understanding of how "lower level" technology works.  The linked article delves in to a bit of C code in which all these other languages implement on some level.

I will be forthcoming on this, I do not agree, even today, that universities should remove a C programming class from any Computer Science curriculum.  I believe it teaches vital "low level" understanding of fundamental concepts of how software works.  Sure, one level further down and we get to assembly and all that good stuff, and while I think that should be taught as well to some degree - maybe as an elective - C is probably the "best" starting point.

I also agree with Joel, the article's author, with the essence of this quote.

I am actually physically disgusted that so many computer science programs think that Java is a good introductory language, because it's "easy" and you don't get confused with all that boring string/malloc stuff but you can learn cool OOP stuff which will make your big programs ever so modular. This is a pedagogical disaster waiting to happen. Generations of graduates are descending on us and creating Shlemiel The Painter algorithms right and left and they don't even realize it, since they fundamentally have no idea that strings are, at a very deep level, difficult, even if you can't quite see that in your perl script. If you want to teach somebody something well, you have to start at the very lowest level.

Monday, January 7, 2013

C as King of the Programming Languages

Disclaimer: I've always been a fan of the C programming language.  I adore the power and flexibility it has always afforded me, the programmer.

I was a bit surprised at first to see that C ranked #1 in 2012 according to TIOBE.

http://www.i-programmer.info/news/98-languages/5298-the-top-languages-of-2012.html

Then it kind of dawned on me that C was one of the major languages involved with smartphone and tablet development, and that seems to be the hot topic of the day.  For iOS, we have Apple's flavor of C in the form of Objective-C (which takes a while to get use to for the uninitiated) which rose sharply as well giving credit to this theory (since Obj-C really has no other use aside from Apple specific solutions).

Java was hogging the top spot for a while, but it was refreshing to see C going in and out of the #1 spot since at least 1988 according to:

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

So, why is this so?  As a software guy, I've always been told that everyone should be using Java, C# or other "managed" languages.  Object-Oriented is *THE* way to go and that's that.  Dynamic typing, garbage collection, and other things are stuff we should want...

Yet, C somehow retains its value?  How and why?

In one of my more subjective rants, I'll say that it is because C let's the programmer be a programmer.  People don't have to learn assembly (though it never hurts of course) and still have platform agnostic source code all while cranking out amazingly optimized (or disastrous) binaries.

It is not for the feint of heart.  Many trivial things need solutions done over and over again.  C taught me lessons constantly that almost always showed that any "general solution" to a problem will be slower than a meticulously crafted one for whatever we're doing.  C gives me this power for better or for worse.

Of course, C can be quite cryptic, excessive, and takes a long time to go from design to solution (usually), and in general, it requires a more skilled and competent developer to actually make a program to solve some problem.  All these things usually mean more money and time invested from a business perspective, so I can understand why the push and drive for "easier" languages has been around for a while.

Don't get me wrong, I like Java and C# as well for what they are, but I would never give a blanket statement that one language is the end all be all answer to everything.

For a binary distribution where performance is needed, C definitely ranks in my top recommended answers.  At the same time, if you have a program that needs to work on multiple platforms, C is well adopted across many platforms and should allow porting with few issues (though when porting issues do come up, they can get complicated).

Languages like Java tried to solve this exact platform portability problem with the idea that you "compile once, run anywhere" (among other things).

Each language has its ups and downs given some situation.  For things like games, I can't recommend enough using C and/or C++.  These languages will *allow* a competent programmer to deploy a well crafted, albeit probably complex, solution.  Will there be problems?  Well, of course!

The *potential* for great software exists with C/C++, but the real issue here is whether a company thinks it is really worth shelling out a substantially larger amount of money for a good programmer just for a chance to make their software run xx% better/faster than some other "easier" language that probably could be deployed in a much faster timeframe.

As a programmer myself, this potential I mentioned is why C should remain king for a long time.  It allows me the ability to do what I want to do and what I can do in to one volatile package.  The learning curve can be steep, and it will force a developer sooner or later to learn more about how CPU's and computers actually work, but this will make them better at their craft at the end of the day.

I may blog more about specifics of this later.


Thursday, March 29, 2012

Android NDK loading resources and PNGs

After following the nice code sample from http://androgeek.info/?p=275 about how to load a PNG file by accessing the Android APK file directly using only the NDK, everything seemed awesome.

It uses libzip and libpng to manually load the APK, then load a png file right from the APK itself without using any Java or the Android SDK.

Why would you want to do this?  Well, aside from liking complicated stuff, it is useful for us because now we can put all of our "portable" resource loading code in c/c++ files so it is more easily moved to iOS, windows, whatever, without being bound to using only the Android SDK, thus having to recode loading logic over and over again for each target deployment architecture.

Everything worked well, until something didn't go quite right.  Some of my textures were corrupted and worse, some actually crashed my app!

It took 4 days to track down what exactly was happening.

Common problems usually indicate I was doing something wrong with opengl and the textures themselves...

* All my textures were perfect powers of two (64x64, 128x128, etc)
* All my textures were 32-bit RGBA files
* I fixed a warning and error from the given sample code to correctly read the libzip'd apk file

So, it was time to dig deeper with what was going on.

First, I noticed that my glGenTextures calls were giving me very large opengl texture ids so I thought this was a huge problem as it was not continuous numbers, and in C, any time you have generating wild numbers, it usually means a bad pointer somewhere.  I spent about a day analyzing and referencing code and started to realize that the generated numbers were always the same (this usually rules out a memory corruption issue if the results are *always* the same).

I noticed that the generated opengl texture ids always followed this pattern on my Droid RAZR:
100271
315638026
534244737
1505553200
-1563003837
...
etc.

Yet, on my Android emulator, the texture id's were:
1
2
3
4
5
...
etc

I had found this thread on google groups that I thought was my problem since it sounded really similar, but it didn't actually help.

Knowing that glGenTextures expects an *unsigned* integer as the data type, I realized that the negative number was an overflowed signed integer and actually nothing to worry about.  In my debugging print code I needed to change "%d" to "%u" to see the actual unsigned value.

Since these numbers repeated on every subsequent run of the program, I decided to test the idea that maybe my code had nothing at all to do with the "corrupt" large texture ids.  Indeed, I made a program that literally only called glGenTextures about 100 times right at startup time and did nothing else.  The result?  The exact same large numbers showed up in my device, and was "normal" on the emulator, as expected.  This left me with the conclusion that whatever logic is in my Motorola device in terms of picking a unique texture ID that this was normal - next.

Yet, this didn't answer my question as to why I was seeing an app crash occur and seemingly only when loading a certain texture or two.  My first order of business was to rule out if my other code was at fault, or was it somehow related to ONLY those sets of textures (somehow). Strangely, dozens of tests indicated that it was only the problem of 3 specific textures being loaded as I added lots of additional textures, even loaded some textures multiple times, then finally ONLY loading one of those 3 problematic textures.  The app would only crash at loading any of the 3.

Frustrated, I ended up putting a ton of extra debugging code all over my apps trying to track down what in the world was wrong.  All indicators were pointing to the idea that a problem was with either libzip or libpng since libc.so itself was crashing in a call to a libpng function -- great, time to track down potential bugs in those libraries.

Now, let me put a disclaimer up here, it is almost never a good idea to assume a mature code library has a bug in it; especially one this severe, yet here I am with a consistent crash always calling a specific line of code that libpng was using leaving me to believe this could be the case.

First, I downloaded the most recent version of each library and replaced the old versions that were provided in the linked post above.  I was now using libzip 0.10.1 and libpng 1.5.9.  This had promise since each of those had a bit of fixes/enhancements along the way.  I was hopeful that this would magically fix my problems.

Well, it did - sort of.

Whatever wizardry was happening internally in those libraries helped indicate that libpng was crashing specifically on this line:

png_read_info(png_ptr, info_ptr);

Great, all this work only to help reinforce the idea that libpng was broken for my special case.  Yet, I still couldn't rule out that libzip somehow corrupted my image data while decompressing my APK so libpng might not be at fault trying to load corrupted data.  Worse, all error checking came up empty for all libzip and libpng calls!  I even checked my opengl calls for errors using glError -- NOTHING!

Well, the next step was to dive in to those library's source code and start tracking stuff down, the last resort I guess was to find the problem in one of these libraries and fix it myself and give the respective team a patch!

One of my testing tangents included making a stand alone C project to test very specific sets of code to completely rule out any Android Java JNI mysticism causing problems.  Everything was pointing to only those textures being a problem in the libpng loading code.  So I ran the libpng test program on those 3 test pngs and the only thing that sounded remotely threatening was:

Files aaaa.png and pngout.png are different
Was aaaa.png written with the same maximum IDAT chunk size (8192 bytes), filtering heuristic (libpng default), compression level (zlib default), and zlib version (1.2.5)?


It certainly sounded menacing and I was convinced that somehow my textures were corrupted.  I ended up regenerating those textures using the latest version of Gimp (2.6.11 at the time) -- plugged those in to my app in excitement; and they still crashed.

Strangely, the libpng test program generated pngout.png which was seemingly the same exact copy of my original texture, but a different filesize; something was different, and different was good at this point.  For giggles, I put that pngout.png file in to my Android's APK res/raw folder in place of my original texture and much to my surprise the texture loaded!!!  Ah-hah, the answer finally!

Unfortunately, things weren't over yet, the texture loaded completely corrupted.  Everything was misaligned, the colors were wrong and the texture was completely garbled with random colors all over it.  Basically, I've gone nowhere.

3 days in and I'm still struggling with what is wrong here.  I'm still baffled, why is it that my 10 other textures work fine, yet these 3 don't want to work at all?!  I decided to write my own texture loading function from scratch to ensure I understood exactly every step of logic that was taking place.  The same result happened.  I did all sorts of strange additional tests, hex dumps, you name it.  I was at the end of my rope on this one, I was getting angry.

Finally, I saw an interesting article somewhere that said I needed to rename my png files to mp3 files.  I thought to myself, that's the dumbest thing I ever heard - why would I ever want to do that?!  Well, if you're programming for Android, you might be crazy enough to hear this out.

Apparently, during the APK packing process *SOME* of your png textures are automagically compressed and others are not.  Renaming your png files tells the apk packing process to not mangle/compress that file, and my happy 3 textures were some of the lucky selected ones for this process; which completely broke the texture loading code.  In a move of desperation, I did rename my files to mp3 to verify this was the case, and they worked perfectly.  I was enraged and ecstatic at the same time.

On the bright side I got a better understanding of both libpng and libzip (we got a little familiar with each other, if you know what I mean).

Alright, so the magic bullet answer is that I need to rename all my PNG files to MP3 (test.png.mp3 if you will) or simply dump them all in to a game data file of sorts (probably the better idea).

So, I make a quick resource packing program (using zlib) and I find out that during the APK packing process again, my GZ files are also modified!!!

I quickly found this link explaining it a little bit.

I wonder what other goodies are in store for me to discover by accident!?

Hope this article is found by anyone else stuck in my situation and save you all a couple days of madness.

Thursday, March 1, 2012

Design to Product - Software Testing (15 / 17)

When is a software product "finished"?  This all depends on who you're asking and in what aspect.  One of the crucial aspects of software houses is testing.  Unlike play testing where people have the potential to have fun with the product, software testing is mostly the opposite.

There are several types of software testing, and this article isn't meant to be a definitive guide on how to perform this job, but I will give a quick run down of what it means and what we do.

Data Validation
During any testing, are the proper inputs handled?  This should be one of the first questions that arise.  Of course, what is good input data, if we can't throw some bad input data in there to spice things up.  Data validation is the process of testing software against good, bad, and unknown data.  What happens when that 0 or 1 input was actually a -1 as input?  Does the program blow up?  We won't know unless we test it!

Unit Testing
This is basically the smallest form of testing.  Do small units of logic perform as expected.  This usually means I test a specific function or method and if the resultant output from that logic is what is expected, I could say that this unit test passed -- it performed as expected.  This testing is more or less automatically performed by the programmer as code is written, or at least it should be!

Performance Testing
Does the software do what it is suppose to do, and do it fast enough?  We may make the next best game ever, but if it runs at 8 frames a second... that's probably bad.

Compatibility Testing
This can sometimes go overlooked (especially on Android phones!), does the software work on multiple different configurations?  Are there weird effects?  Specifically for Android, will the product work on a phone, and an Android tablet?


Acceptance Testing
This is like Software Quality Assurance or Quality Control.  Does the software deliver, on a high level, what the design is?  Think of this like a marriage of software testing with play testing.  Does the product allow the player to actually play the game as designed?  Does the code actually allow for the player to launch new marbles and perform all the functions needed to play?  If these are both yes, then it passed both the play test, and the software test, this the product can be accepted as the product.

For more information about testing, validation, methods to perform them, etc... really, we encourage you to look at the wikipedia article here: Software Testing.  The trick is to find what works for you and use that.

Wednesday, February 22, 2012

Design to Product - Software Design (12 / 17)

Finally, let's talk about how the code should be set up.

We mentioned earlier that we took the game design, analyzed it, and figured that nouns become classes and verbs become methods.  This sounds good on paper, and there are even tools available to help this process.  We could sit around and chart UML all day, every day in regards to the design, but is it necessary?  That depends.

Because our product isn't an enterprise solution, our team is small, and one guy is basically doing everything, a massive UML chart isn't really necessary.  Of course, we could never discount the idea that someday in the future these things could become useful (like if we sold our products off to another company, as an example).

Regardless, the best way forward for us is to pick apart the major components of the design and start designing what logic should go where and what it should be called.

Honestly, because of our development methodology (or lack there of, depending on who you ask), we take more of an extreme programming measure in this regard.  We make classes as we need them.  We start with a basic skeleton framework, then start adding new classes as they are needed in the code itself as we program the meat and bones of the product.

We start with the largest classes in the skeleton.  Specifically, for Android programming, we start with the GameActivity class.  This class acts, more or less, as the "starting point" for the program and interfaces with the OS.  Because of how Android itself is designed, we must make a GameThread class that contains the top level logic for the flow of the game itself.  We also will need an extension of a View class so that we can interact with the screen and display stuff.

Right away we have:
GameActivity
GameThread
GameView

Without really even going further, based on the game design, we know we already have 2 major components to the product, a Marble, and a Bubble.  These playing pieces are what are the basic, interactable things in the game itself.  Because both things follow the same physics, we decided that it is best they share a common parent class; PlayingPiece.  This parent class defines movement vectors, collision information, the abstract methods of how the pieces are updated and that each child class should know how to draw itself to the screen.

At the same time, we can't really play the game without some sort of logical thing to interact with and where the playing pieces do their magic.  Enter the PlayingField, a logical container that contains everything the player can interact with and see.

The game design indicates the need for persistent data over game launches, so we made another class Gamedata that is our "middle man" that handles data saved and loaded between game sessions.

We also ended up needing a class called MarbleSlot, a place where players can touch and interact with the playing field in which marbles can be spawned from.

All that said we now have:
Bubble
Gamedata
GameThread
GameView
Marble
MarbleSlot
PlayingField
PlayingPiece

There are one or two utility classes that are used to assist in the passing of data and messages between classes, but those are fairly technical and don't have anything to do with the direct design of the content of the game.  Suffice to say, those can take a while to design and program as well.  Just because a game design is "simple" doesn't mean the work behind it will take a set number of hours based on the relative ease of that design.  Lots of "stuff" can go on behind the scenes that can take days or even weeks to implement that have no actual, obvious visual effect on the product itself.

For example, a programmer could spend a month just drawing some graphics on a display.  They are very proud of this accomplishment, but to non-programmers they usually have the attitude of "you spent a month on this, and only have this to show for it?!".  Get used to scenarios like this because it happens more often than not.

Anyways, now that we have our basic classes determined, the essentials are fleshed out.

Tune in next time for elaboration of the class designs and the overall flow of the logic for some of them.

Thursday, February 9, 2012

Design to Product on Android - Game Design (7 / 17)

Finally, we can start making a game!!! Let's crack open our favorite Integrated Development Environment (Eclipse in this case) and start forging all that hot code all over the place... err wait, what am I going to make?!  Oh who cares, I can make a game/product on the spot!

Whoah easy there code vigilante!  Let's take a moment to actually design what we're doing first.  Again, in software engineering, lots of people assume that programmers just make code.  Well, that's nice, but there probably should be at least some documentation in terms of what are the features needed in a project to make. Programmers use this as a roadmap (and even a task list) of what they need to make.  Once made, it serves doubly useful as a checklist of what to test to make sure the product was made correctly.  Remember, at this point, we're not even talking about software design, just the content of the product itself -- not even code yet!

Of course, testing the product is a fairly large topic in itself as well and will likely get its own article.  Suffice to say, an acceptance test looks at a design, looks at a software product and then validates and verifies that the product is what was desired in the first place -- thus it is accepted as the product.

We will come back to game design on and off throughout the series.

*** Updated section ***
Originally, we did not post the game design here because it wasn't finalized, plus we didn't want the "spoilers" of the actual game getting out before we really thought the whole game through.  Since the product was posted, the finalized game design can be found here:
BubbleZing game design

This game design document was first linked when we released the product at the end of this series, but is now retroactively linked here for convenience.
*** End new section ***

Next up, let's start analyzing what kind of art specifications and styles we want to incorporate in the game.  Once our design was completed, we started toying with ideas of what it should look like, the theme, how to go about generating/acquiring the art assets.  So, next up is the design of the art itself.