Thursday, June 19, 2014

The supporting libraries of a custom game engine

While creating my own game engine, it became pretty apparent fairly quickly that I couldn't possibly do everything that would be required of me to reprogram "from scratch".  We are talking about stuff like opening PNG files and extracting the RGB data out of them!  (As a side note, I DID actually read the RFC  and realized very quickly this isn't something I wanted to do).

PNG files weren't the only thing I wasn't interested in figuring out myself.  There are perfectly good libraries out there that have weathered the long haul and perform very well.  Let's go ahead and admit we can use some help from others in our ambitious goals of writing a game engine from the ground up.

Oh, and a caveat, it had to be free.  I'm sure there are some great commercial libraries out there, but that's not my fancy.  Of all the years gaming technology has existed, there just HAS to be free solutions out there somewhere just for this adventure!

The libraries that I have thus settled for (at least for now!) are:

  • libcurl - CURL is a useful library for dealing with network data transfer over various protocols.  This is useful for stuff like downloading things from HTTP, FTP, and many, many others.  For my intents and purposes, I found it useful that I could post, say, patch files on a webserver and have my game clients download those patches using features of this library.  Technically, my game engine has "enough" features to be able to talk HTTP with a server but I never got around to formalizing it to be as robust as libcurl.
  • lua - Lua is a fantastic and powerful scripting language and feature to integrate in to a game engine.  What better way to extend games than allowing other people, or even the public, to create, modify, and execute their own scripts!  We integrated liblua in to the game engine and can now embed scripts directly (the engine incorporated the lua parser so it can run lua scripts in the engine!)
  • OpenAL - Really, when it comes to sound, it becomes excessively problematic dealing with the tons of options out there; many of which are platform specific.  OpenAL was kind of the only sane way of dealing with audio output in a cross-platform setup.  I attempted to write my own audio mixer and all that, got pretty far, but then I realized I needed more than just mixing, I would need to somehow talk to the sound driver(s) on each target platform as well -- way over my head and just too much to learn and do well. 
  • libpng - This one is a granddaddy library that fuels a surprising number of applications out there, may as well adopt it!  What can I say, it let's us load up PNG files and extract the much-needed RGB data from them.  It took a little while to learn how to interact with the API, but eventually I got it - perhaps I'll make an article just for this someday.  At the end of the day, this is a fantastic library to integrate with a game engine.
  • zlib - Similar to libpng, zlib is almost everywhere on the planet.  It's in your smartphones, in your computer, and in almost every application and Operating System out there!  In fact, I'd be surprised if you haven't interacted with something zlib related, even today!  It deals with compression of data files and data streams.  As well, of course, with inflating data from compressed states -- something insanely useful.
  • btrxml - An XML parser I wrote with a constrained memory footprint in mind while retaining cross-platform capabilities.  I open sourced it under the zlib license so other people can use it for whatever they want (permissable with the license, of course).  Originally this was going to be part of the game engine itself after I looked in to other XML parsers, but at the end of the day, I opened it up and generalized it in to its own project!
That's it!

Using the libraries mentioned above, combined with the other engine code, everything I can do is made possible.  There may be some additional libraries to consider in the future.  I've dabbled with a couple 3d data libraries like lib3ds and things like that to help with handling 3d data (models, animation, etc), but I haven't quite figured out how I want to handle many things yet with that aspect.  Plus, for now, my engine really only caters to 2d games.  It does have super basic support for 3d things, can even render cubes, spheres, etc, but anything beyond that is no go yet.

Speaking of new libraries to consider, I think the next library to incorporate is likely ffmpeg or some of its dependency libraries to help with audio/video playback in a cross-platform manner.



Wednesday, June 18, 2014

The dawn of a new game engine

After my [de]motivating blog post earlier about making your own game engine, this will be THE post talking about how I got started!

A long, long time ago, I set out to create my own engine.  I've always been one to need to figure everything out to understand how things work, and game engines were no exception.  In my moments of weakness, I did consider pre-made things (hey, I even used RPG Maker wayyy back in the day).  If you're curious, I even went on and on about Unity and why I didn't use it here.

It's been a while since I started the adventure of this engine, so I'm looking back on the commit history of that first legendary day of when my engine was first put in to version control (protip: use version control software when making your own engine - I MEAN IT).

According to my own notes, version 0.0.0 of my engine had the following capacities:

  • Integrated libpng (for loading png files)
  • Integrated zlib (for compression/decompression of data)
  • Engine definitions (TRUE = 1, FALSE = 0, etc)
  • Device capability reading (number of cores in the CPU, display resolution...)
  • Engine init and deinit code, engine version numbers, etc
  • Fixed math placeholders (float replacements)
  • Real basic and rudimentary bitmap font loading
  • A very, very broken, misguided, and naive set of game configuration routines
  • A box structure, used to define a region for 2d interaction on the screen (think buttons)
  • An actual button structure, the intention was to be able to just put clickable buttons anywhere
  • OpenGL initialization stuff
  • Life cycle logic (useful mainly for mobile platforms)
  • Platform agnostic logging routines (to stdout, or to a file)
  • Very, very basic network socket operations
  • Very broken sound code that required extensive rework later
  • Very broken storage routines (come to find out, this is much more complicated in cross-platform)
  • Super basic time and date handling routines
  • A generic "timer" structure to keep track of elapsed time (in milliseconds)
  • Some engine utility functions (stuff like check if integer is even, get process id, check for file exist)
  • Basic 2d vector structure
In theory, I could drop this v0.0.0 of my engine with some game logic and a real basic game could be made.  How wrong I was.

According to my version control, the notes I left for myself were mostly unhelpful at first, but it is quite apparent that I grew frustrated with myself with how broken a lot of stuff was; especially with my early cross-platform issues.  At first, this engine was to be used for Android development specifically to replace the very clunky Java solutions out there.  What I was finding out was that there is a crazy amount of problems to be solved here at these fundamental levels for a couple platforms, not just Android.

It looks like the first thing I did was make some test functions to specifically help with testing new engine code.  This way I could make an engine testing program that can test engine functionality in a sandbox-like environment -- definitely want a controlled system when testing engine logic; many times I've driven myself to the brink of insanity debugging engine code when I came to find out that it was a bug in a game's logic that actually was bugged, not the engine!  Then again, some of these sanity checks lead to more robust and fault tolerant code in the engine, so it wasn't ALL for naught.

I just realized I should back up a bit; hey there's a bit of history here and I'm getting old!

Let's go back to the design goals of the engine.
  • It should have fantastic performance
  • It must be cross-platform to the major platforms (windows, android, iOS, OSX, linux)
  • It must cater to doing as little work across platforms as possible for games using it
  • It should help me later when making games by doing a lot of the less... interesting things
  • It should help me later when making games by doing a lot of the tedious things
To cater to the first two points, I settled on making the engine in C.  Yes, straight C programming.  Not even C++ (though one of my early iterations of an engine I did utilized C++, and I certainly miss some of those features).  C's performance has the potential to destroy the competition in the compiled languages sphere (well, short of FORTRAN, I suppose, though the race is close!).

Performance issues aside, C is also one of the few (maybe only?) languages that is supported by all the mentioned platforms.  The goal here is such that I need to be able to do as little rework as possible across all platforms.

I couldn't do Java, because iOS doesn't support it (a decision I agree with!).  Getting it to work on OSX can also be a bit finicky.  I certainly have a fair share of less-than pleasurable experiences with JVMs on Windows as well... and dalvik with Android... with its extensive device fragmentation issues is a whole other terribad experience I don't recommend to anyone.  Those things ruled Java out pretty quickly -- not to mention early versions of a Java engine I attempted were met with very bad performance metrics (especially on Android).  As a game engine, the fastest performance possible should be a high priority (I'll make an article about this later in much more elaborate detail).  Java was out.

C# was a candidate.  Unity uses it, it's not completely terrible.  The Mono framework supports it on non-windows solutions.  I dabbled with it briefly, but I found myself throwing features out that were suppose to be "pros" of C#.  Stuff like garbage collection, exceptions, reference counting, running unsafe code, etc... all these things were starting to add up against C#.  Though, unlike Java, C# at least seems to be deployable to the major platforms in some way... yet I found myself too often "going native" and making the hardcore components with C/C++ anyways.

So, to cater to this newfound information, I pretty much settled that the engine must be C/C++ flat out.  It's not all bad, I mean, every major Operating System in the world uses it.  Games on consoles use it.  How bad can it be?  Thankfully, these are my primary languages as well, so there's a comfort factor involved too.  Well, time will be a casualty with how complicated "going native" is for each supported platform... the tradeoff, though, is fantastic performance, and surprisingly little administrative overhead for "write code once, run anywhere"... which is a common advantage for Java/C# -- and here we are emulating that slogan with C/C++!  While this sounds good, there are platform intrinsics that we have to manually cater to, which is far from fun and anything but straight forward.

The other bullets mentioned I will be getting to as we go forward with these articles.

This was a lot of stuff.  I settled on straight C for maximum control, portability, and performance.  I dropped a couple of months on trivial things, setting things up for the engine (the first bullet list in this article reflects that effort).  There was already a bit of cross-platform support right from the get go as reading back certain information, like getting the current process id, were platform dependent.

Remember, each platform is fueled by different system capacities and standard libraries for their respecitve operating systems.  I will be taking full advantage of POSIX functions and things in the standard C libraries when possible, and I recommend doing the same for your own game engine for maximum performance, predictability, accuracy, and compliance with those standard behaviors.

Not sure what to cover next, I'd like to talk about big picture things and super technical things... so if anyone has a request for something, by all means leave a comment and I'll focus on that specifically.


So, you want to create a game engine?

Don't.

Ok, now that we have that out of the way, and I haven't convinced you not to, let's get this party started.

I'll be blogging a bit about my own experiences with making our own game engine.  It will be part technical, maybe part code, a bit of code architecture, motivating stories, demotivating stories, nuances, and all sorts of other things.  If anything, this will be used as a self-counseling and rambling type session, and if you want to follow along, feel free.

Usually you'll hear a sizable amount of pessimism from other people when you mention a desire to make a game engine. This is probably rightfully so.  As a solo developer myself, I will have to admit I didn't truly appreciate the sheer complex, massive task that this would encompass.  As of this writing, my own engine isn't "done", it isn't "feature complete", and has a substantial amount of things todo left.  However, my engine currently DOES fuel the backbone of a couple of cross-platform, published games - so it IS functional and even usable.

I suppose the first thing we need to focus on is just WHAT exactly IS a game engine.  I could probably list out a dozen or so references and copy pasta, but I generally like to rephrase things in my own words as I feel that once I believe I understand something, I should be able to reiterate it off the top of my head in somewhat plain English.

So, without further ado, a game engine is a large, complex collection of software that other software products (in this case games) utilize to deal with inconvenient and time consuming aspects of development.

When we save time, we save money.  An endeavor to make a game engine is a non-trivial obligation and decision to invest a substantial amount of time to save you time and money later.  Return on investment is the name of the game here.

Speaking of return on investment.  There are plenty of ready made engines out there.  You will likely see people advocating their use.  Things like Unity, Cocos2d, GameMaker, and various others exist.  If you are convinced you want to make an engine, you should STILL download some/all of them and look at them.  Look at the service they provide, look at the source code (where applicable), play with them a bit.  Get a feel for how they work and what they give you.  If you are to make your own engine, YOU will be making a comparable product, with similar interfaces, with similar needs and goals.  Think of this as competitive research.

If you look at the comparable engines out there, I hope you begin to look at how overwhelming the goal of making your own is.  "Look at all this stuff, I certainly won't need all this clutter!".  Trust me, I know what you're saying.  Chances are, you're right, you likely won't need a large portion of "all that stuff".  Sadly, there is a lot of it you will need, but you won't know that until you're well in to development (likely several months or even years).  So, look at these large projects and realize one thing.  There exists the potential that YOU will ultimately make something very similar in scope and complexity, you just don't know it yet!

"But all I really need is something to load up graphics and put them on the screen... my game code will handle the rest".  This is incredibly naive and very, very far from the truth.  I know, because I thought like this once.

The small laundry list of things you think you need will grow ever so slightly at first, but it grows at a consistent pace.  Oh, I need some abstract thing that handles generic box-like shaped regions (think clickable zones, usually useful for things like buttons)... yea, we'll need that... but what about font and text handling... don't really need TTF format support... or do I?  Should I want to load in custom bitmap fonts?  Oh, I need to be able to load bitmap formats (bmp, png, jpg?).  It will probably be good to be able to display those graphics too -- hmm, OpenGL, DirectX?  Is my engine going to be cross-platform?  If so, there are major implications for choices that I can make.

Surely, we COULD support DirectX with a cross-platform engine... but that technology would only be supported for the Windows versions of our games.  This sounds fairly straightforward, but it complicates things in your graphics code, build process, and even workflow when messing with your engine, and then ultimately building games using that engine - which of course is the actual goal when rolling your own engine.

I'm going to make more articles about specifics as I go, so don't worry.

For now, know this.  The game engine I'm currently working on, and actually using, is my 4th attempt at making an engine.  It is by far the most complex and furthest developed one I've ever made (none of the other ones actually fueled any games!).

I highly recommend you not grow too overly attached to any engine you make right away.  The first engine or two you make should really be seen as a learning experience because you will need a deep appreciation for the overarching goal and understanding of what it is you need to do and actually CAN do before you can make something actually usable.

Look in to version control software.  I'm using the very awesome mercurial to track changes in my engine code (and to keep my own general sanity).  According to my commits, my current engine was first submitted to version control on 17 August, 2012.  As of this writing, it will be hitting its 2nd birthday soon.  It is still not "complete".  We have a ways to go still; especially in 3d support and having support for playing back avi/mpg files in games.  There are plenty of other things left to do as well, some of which I probably don't even know yet, but we'll get there.

What I hope you take away from this article is several things.

One, making a game engine is not trivial.

Two, making a game engine is vastly time consuming, likely taking years of fairly hard work and lots of research.

Three, making a game engine is a lot more complicated than you realize.

Four, be ready to admit your ways are wrong, to incorporate new ideas, and be ready to gut things that aren't working in favor of newer, better ways of doing things.

Five, if you're weak of heart and don't have borderline OCD, this probably isn't for you.  If you're even somewhat on the fence about this whole thing, you will likely only find disappointment.

Oh, and lastly, you should consider using other game engines still... but if you're still not actually convinced and you still want to venture forth in to this crazy goal... then by all means, let us proceed!

Next article we'll talk about choice of technology and actually getting started making your own game engine through my ramblings about how I got started with my own.

It is not for the feint of heart, but here you go, the dawn of a new game engine!







Wednesday, September 18, 2013

When game projects die.

Welcome to another installment from one of those crazy indie game dev guys.  Today, I'd like to talk about when a project goes awry, and ends up being nixed.

About a year ago, around September of 2012, I began designing a tower defense game.  I figured it was going to be "easy enough" to design and develop.  Now remember, I'm basically working by myself and on top of the proposed game itself, I also make my own game engine, which is, of course, a never ending task.  Anyone with a decent amount of game industry experience (indie or otherwise) already knows that these are red flags, but hey, I'm stubborn and persistent.

The design seemingly went ok, the theme is to take control of plastic soldiers and defend human living spaces from pesky animals and things like that.  Once I sat down and wrote down everything about the game that I could think of; stuff like the player's pieces, the enemies, the general gameplay, etc, I decided I needed an artist to make the project come to life.  My lacklustre art talent was several years lacking to be "good enough" to do it myself.  So offloading the art would free me up to actively start programming the logic and game system, while still working on core game engine features in the background.  Now mind you, my game engine had already been in development for around 9 months at this time and had quite a bit of features "ready-made" (and published products using it), so I wasn't starting from scratch, but at the same time also put a technical limit as to what the game could be designed to handle.

Knowing that the budget is tight, the only real options I could consider was working with local college students that needed portfolio work, or outsource to Asia.  I generally like to try to keep things local and to support our own citizens, so I opted for the earlier option.  This turned out to be quite problematic, as with all things, apparently.

I may sound like a terrible person, but the people I met had some combination of no motivation to get anything done, couldn't follow directions, simply disappeared off the face of the planet, or just didn't have the required talent to make it happen.  The people I met and talked to just simply were not qualified.  Now, what did I expect, I was scalping college students.  Eventually I did meet up with 2 or 3 potentially qualified people.  After going over the plan, we drafted up a monetary agreement -- any "good" talent is worth money.  The amount of money offered, of course, couldn't be much, relatively speaking as compared to, say, established art houses, or even asian production companies, but hey, what do these guys expect, I'm an indie game dev with barely enough cash to survive with.  This conundrum is worthy of its own blog or even book so I'll leave it at this for now.

While there are other arrangements, like profit sharing, etc, that I have entertained in the past, I wasn't at that point with this project and tried to emphasize a cash payment, *on release of the product* to potential artists.  Some of the earlier failures with contracting ended up with some work being done and having the artist disappear, leaving me high and dry, so to protect myself, I insisted heavily on "no payment until release".  The game project wasn't something that could be done in a week or two, I understood that, so it definitely warranted some cash.

The amount of cash, of course, becomes the problem.  However, an agreement had been made between myself and an artist that seemed to have the right skillset.

The first couple of months went by with an acceptable amount of artwork production (definitely better than earlier contacts), and development proceeded in a fairly smooth manner -- but artwork production soon tapered off after about a month and a half.  I understand people have other commitments, but this was starting to get a little alarming.  I try to give people slack and not push *too* hard, but hey, things can only go so long, and plus, I'm not offering a huge cash payout, so I could see how and why my project would lose priority.

However, things got more complicated.  I had been in contact with Kongregate (a publisher) over the project.  This could have been the money infusion I'd been looking for, but this has its own ups and downs, dealing with publishers that is.  If we were to go this route, I'd have to increase game production, and art, several times over... something I don't think went over very well with the artist.

So, I ended up losing a bit of credibility here, and told Kongregate that we wouldn't be capable of producing something for them in any reasonable amount of time.  At the time, I was also pushing to release a demo of the product, or gameplay videos, but they told me not to as that could jeopardize our potential arrangement.

Time for alternative plans. An indie dev competition thing that was happening a couple months away came to my attention.  The dev builds of the game were going pretty well, but I'd need to get the game to an almost releaseable status to really be able to compete; the development was fine, but the art was slacking quite far behind.

I asked the artist for a status and estimate for how long it would be before I'd get "the rest of it".  He mentioned a time, but I almost laughed.  I had been tracking rough production time and my number was *vastly* different than his for artwork.  He mentioned something like a month, whereas my numbers suggested potentially 6 months.  Apparently, he didn't appreciate the discrepancy notice.

I think I got one additional piece of art from him since, and that was almost 6 months ago.  So, rewind the clock to that time, I pretty much knew his number was never going to happen.  Knowing that, I went ahead and did what any programmer would do, bury himself with more coding to "get it done".  I kept plowing away at the core game and the engine itself when a potential tech change occurred to me.

The entire game, at this point, is using 2d technology.  Yet, there was a lot of convoluted zooming in and out, with complex scaling, floating point stuff, strange math, things like this to produce a mock 3d-esque type player experience, using a core 2d tech system.  In the background, I was already toiling with 2.5d tech in the game engine and thought, hmm, this could be the perfect chance for me to incorporate this 2.5d stuff in to the game since I'm not going to be getting art anytime real soon anyways.

"It would be so much easier if I changed over everything to this new stuff".  This line alone could probably sink entire projects, and it certainly didn't do ours well either.   3 months later, most of the game had been transitioned over to the "new" 2.5d system, utilizing more of the game engine, which was also under heavy development, especially in this area.  There now exists a mostly broken game using new tech, old art, tattered nerves, little meaningful communication, and little funds.  I'd say time was short also, but there's never enough time, so that's a constant problem.

At this point, I had to sit down and ask myself where I was at with the project.  I had gotten so far to see over the peak of the mountain to realize I've gotten myself in to a bit of a pickle.  I had to reflect on myself where did things start going wrong.

Ultimately, failure rests on the project leadership, which lucky for me, was me.  The only way to grow was to reflect.

I personally believe that it all started back at the design of the project.  I described the game with words, not with concept art.  I didn't DRAW what the game would look like, I described it.  I used reference artwork, sure, but I didn't give explicit visual direction to what the artist was to create.  I didn't understand what the difference between what a game artist did versus what a game designer did.  I (wrongly) thought I could pass descriptions over to an artist and they would make the game come to life.  I was wrong.  When questions came up over intention and meaning of things, I'd answer with words, not drawings.  Visuals speak more clearly than words, especially in the media, and artists apparently are no different.

It wasn't really until I was working on other projects during this mess that I had realized that working with designers is what I was missing here right from the get go.  Two designers (unrelated to this project and each other) had sent me conceptual art for their own projects to develop and what they sent to me was visuals.  Artwork mock ups.  I could envision exactly what things would look like on a screen.  We had a hard, set, requirement for what displays and aspect ratios we would target.  I had basically no questions about what to do or how to go about doing it, and this, became quite apparent that this is where I was so sorely weak in my own project.

In fact, I don't even really blame the artist I was working with; sure, there might have been a bit of his own motivation issue at hand, but as a project lead, that's also my problem.  I think it probably didn't help that there wasn't a solid plan and design he could follow to find his own success, and again, this was my fault.  It would have been easy to just say "he sucks" and call it good, but it is more complicated than that, and there's no way I would grow from here if I did that anyways.

So, here we are about a year later from the very beginning with almost nothing to show for it. It is quite the sobering experience.  I'd like to get this project out the door one of these days, but I think I need to send it back to the drawing board and redesign it from essentially scratch, this time with a proper design.  It's hard to let something like this go; especially after "wasting" almost an entire year on it, but I guess this is one of those things that you have to love enough to let go when need be.

Friday, July 19, 2013

Game Programming and 2d Art, Part 3 (Texture Atlases)

Welcome to the final part of my series on game programming with 2d art assets!

The previous 2 entries are here:
Part 1
Part 2

If you don't know already, a texture atlas (also known as a spritesheet) helps game developers to display neat 2d graphics in to their games.  There is usually some sort of organization sanity happening with them as well, for example, all of the frames of animation for a player character would be in one atlas and its file name could be "herowalking.png" or something like that.

Of course, this isn't limited in any way, I've often used atlases that are just a collection of User Interface components usually in a file called "gui.png" -- it has all the buttons, graphs, meters, check marks, and all sorts of doodads, all conveniently packed in to one GUI atlas file.

At any rate, assembling a texture atlas can be extremely tedious.  You have to make sure nothing overlaps anything else, that all your texture objects can fit, you will even need to remember where you put each object so that a game engine can subsequently extract each object.  It is so much planning and tracking!

Thankfully, there is a program out there called Texture Packer.  It is a commercial product, but all in all, it's a nice, convenient tool to have in your tool belt.  I'll even go so far to say that if you make 2d games commercially, you probably shouldn't be without it!  It's reasonably priced as well for how much time and money it would save you.

Let's put it this way, I spent about 2 weeks making a command line program to extract texture sizes and coordinates and it was a pain to deal with.  On top of that, I was still relying on manually entering texture coordinates in to our games from that program's output.  If an artist changed any graphics, it would be a nightmare changing things all over the place.

This program keeps track of all that stuff, and allows you to output all that relevant information in to various game engine formats; or for us, even a generic XML data format!

Below is a screencap of what it looks like:





Honestly, it's pretty good.  There are a couple of features that would be nice to have; like manually being able to move texture objects around (or swap places with each other), but what the tool does is automatically find a place for your sprites in the empty atlas for you, so you don't have to sit around with a magnifying glass lining pixels up manually!

There are quite a few neat features as well about it, like automatically generating different scaled versions of your atlas (click AutoSD) for catering to different resolutions, being able to specify atlas geometry (like powers of two, max width/height, etc), and it can even import SVG files.

It takes a little while getting use to, but all in all, it's a pretty useful tool and saves us quite a bit of time dealing with artwork.

I have noticed that the windows and linux ports are a little behind the mac port, and again, it is missing a couple neat features, but all in all, it is useable and useful.

They offer a trial version of the program also, so you can check it out without having to buy it first.