In follow up to my earlier post about being envious of software houses of yesteryear, I have to admit that in a sense, they had it easier.
When they made their games and products, they had a single target system with known specs and limitations. Code that relied on *timing* of events and things like this can be planned for meticulously. Developers also knew 100% ahead of time what features a system did or did not have. Yet, they also dealt with 8-bit / 16-bit processors, no FPU, no advanced graphics pipeline, and all sorts of "limitations", and they still cranked out great products after great products all without needing all these patches and what not.
Well, fast forward a bunch of years, and here we're in the age of the smartphone. In some regards, we have it easy - a powerful CPU (relatively speaking), an amazing graphics processor, and sometimes a FPU. Wait, sometimes?! Yes, that's right, if you're a developer for the Android platform, you can't be guaranteed to have a floating point unit. You also have no guarantee of having any given amount of RAM, disk space, or really much of anything. Sure, you can *use* float data types, but on devices with no dedicated float processor, it is software emulated!!! Talk about a massive performance hit if you want to do updates and renders at any sane framerate!
Our earlier game, Bubble Zing, ran amazingly awesome on newer Android phones, but on older phones, it ran terribly. With the 1.0.3 release of the game last month, we thought all major compatibility issues were resolved; well, that was only partially correct. The major bugs seemingly were ironed out, and the core game "worked", but with no guarantee of how well it would work - sometimes outright unplayable on some devices!
So, we delve further in to the "why is this running like terrible on x,y,z,q,f devices, but great on others". Well, this is where the things Android does is both great and bad at the same time. We coded 1.0.3 in 100% Java as this was seemingly the correct way to make any app on Android devices. In theory, it sounded good since Java apps are suppose to work 100% on devices with a compatible JVM (ie, all Android devices) - clearly something we wanted, and it is fairly easy to crank a Java program much faster than an equivalent C/C++ program, so it sounded like a great idea at the time.
Our test system was a Droid RAZR with Android 2.3.6. Our initial testing was running well, FPS was over 40 and everything seemed great. The emulator running an AVD 1.5 seemed sluggish but that could be written off as being an emulator and running not on the most powerful host computer.
Things got really confusing and exciting when we tested 1.0.1 and 1.0.2 on a Samsung Galaxy tablet -- that's when all hell broke loose. Everything that should have been working in theory wasn't. On top of that, the frame rate was a dismal 10 to 15! What the flying expletive! How can a program go from 40+ FPS to 10! It wasn't an issue with the version of the Android OS, it must have been something device specific.
I ran a traceview to track what was eating all the processing time and it came down to rendering a full screen background PNG file. On the phone, this wasn't such a big deal, but on the tablet, it was eating almost 60% of all CPU time!
We're sitting here with a complete Java game that worked ok on an emulator and great on our target test phone, yet are dumbfounded with why the same exact Java code on the tablet ran so terribly.
A lot of trial and error later we figured that to get things working faster and better, the only way was to ditch Java and start programming on a closer-to-the-hardware level as we could to squeeze out a lot more performance and manage memory more effectively. Java made it very frustrating with garbage collecting at some fairly inconvenient times and this compounded other issues making the game run very badly at certain instances. It was time to bring on the NDK.
Frustration is what is in store for Android developers. It can be done, but good apps will need a very persistent set of people to make these products and make them well.
Oh, and I'm still in awe of game programmers from decades ago on consoles, but at the same time, I'm jealous of their "single architecture" deployment platform as that probably simplified a lot of these problems thousands fold. I suppose this is another reason Apple is winning the smartphone wars with developers as it is "easier" to make apps on iOS with very heavily controlled hardware features by Apple.
Showing posts with label software testing. Show all posts
Showing posts with label software testing. Show all posts
Monday, April 2, 2012
Friday, March 2, 2012
Awe at software development houses of yesteryear
After making another small game, I can't help but be in total stupor over how... easy software houses have it today. I mean, seriously, back in the day when I was playing things like Nintendo way more often than I should have been, I never even thought about how much time was invested in to making all those games I played.
Back then, it's not like game dev houses could just send out a patch to fix things over the Internet. When products were shipped, they were shipped for real! Imagine, this had to be the final product, no going back; we're about to manufacture millions of cartridges and customers will get their hands on all our hard work, period, no going back. Dev houses had to be pretty damn sure that what they were packaging up was 99.99% perfect.
Ok, a company probably could have done a product recall if the game was horrendously broken, but honestly, we just "shipped" Bubble Zing yesterday, and I already found 2 or 3 small bugs that should be fixed! Talk about maddening for the devs of yesteryear! Now granted, we made that game in 3 weeks with a really small team, but still... it is mind boggling to know that these guys did it, and did it well is stupendous... well, for the most part... games like Action 52 did exist, and that was... well... a broken down mess incurring the wrath of gamers lucky enough to play it.
If you're unsure about Action 52, check out James Rolfe's review of it on cinemassacre, it is pretty funny. James, being the "angry video game nerd", can be a tad vulgar (just a little), so if bad words offend you, you might want to stay away from his review.
On a side note, I guess this would help explain why no version numbers were prevalent in old games, no such thing was important because what you got was it! In rare cases, a second or third production run was accomplished and it was during this time a dev house could slip in a revision copy of the game, fixing a couple things, but still, we almost never saw any indication of what version number our copies were running.
So yea, one of our products just shipped and I'm already getting a laundry list of things to put in to a patch within the first day of release!
Back then, it's not like game dev houses could just send out a patch to fix things over the Internet. When products were shipped, they were shipped for real! Imagine, this had to be the final product, no going back; we're about to manufacture millions of cartridges and customers will get their hands on all our hard work, period, no going back. Dev houses had to be pretty damn sure that what they were packaging up was 99.99% perfect.
Ok, a company probably could have done a product recall if the game was horrendously broken, but honestly, we just "shipped" Bubble Zing yesterday, and I already found 2 or 3 small bugs that should be fixed! Talk about maddening for the devs of yesteryear! Now granted, we made that game in 3 weeks with a really small team, but still... it is mind boggling to know that these guys did it, and did it well is stupendous... well, for the most part... games like Action 52 did exist, and that was... well... a broken down mess incurring the wrath of gamers lucky enough to play it.
If you're unsure about Action 52, check out James Rolfe's review of it on cinemassacre, it is pretty funny. James, being the "angry video game nerd", can be a tad vulgar (just a little), so if bad words offend you, you might want to stay away from his review.
On a side note, I guess this would help explain why no version numbers were prevalent in old games, no such thing was important because what you got was it! In rare cases, a second or third production run was accomplished and it was during this time a dev house could slip in a revision copy of the game, fixing a couple things, but still, we almost never saw any indication of what version number our copies were running.
So yea, one of our products just shipped and I'm already getting a laundry list of things to put in to a patch within the first day of release!
Thursday, March 1, 2012
Design to Product - Software Testing (15 / 17)
When is a software product "finished"? This all depends on who you're asking and in what aspect. One of the crucial aspects of software houses is testing. Unlike play testing where people have the potential to have fun with the product, software testing is mostly the opposite.
There are several types of software testing, and this article isn't meant to be a definitive guide on how to perform this job, but I will give a quick run down of what it means and what we do.
Data Validation
During any testing, are the proper inputs handled? This should be one of the first questions that arise. Of course, what is good input data, if we can't throw some bad input data in there to spice things up. Data validation is the process of testing software against good, bad, and unknown data. What happens when that 0 or 1 input was actually a -1 as input? Does the program blow up? We won't know unless we test it!
Unit Testing
This is basically the smallest form of testing. Do small units of logic perform as expected. This usually means I test a specific function or method and if the resultant output from that logic is what is expected, I could say that this unit test passed -- it performed as expected. This testing is more or less automatically performed by the programmer as code is written, or at least it should be!
Performance Testing
Does the software do what it is suppose to do, and do it fast enough? We may make the next best game ever, but if it runs at 8 frames a second... that's probably bad.
Compatibility Testing
This can sometimes go overlooked (especially on Android phones!), does the software work on multiple different configurations? Are there weird effects? Specifically for Android, will the product work on a phone, and an Android tablet?
Acceptance Testing
This is like Software Quality Assurance or Quality Control. Does the software deliver, on a high level, what the design is? Think of this like a marriage of software testing with play testing. Does the product allow the player to actually play the game as designed? Does the code actually allow for the player to launch new marbles and perform all the functions needed to play? If these are both yes, then it passed both the play test, and the software test, this the product can be accepted as the product.
For more information about testing, validation, methods to perform them, etc... really, we encourage you to look at the wikipedia article here: Software Testing. The trick is to find what works for you and use that.
There are several types of software testing, and this article isn't meant to be a definitive guide on how to perform this job, but I will give a quick run down of what it means and what we do.
Data Validation
During any testing, are the proper inputs handled? This should be one of the first questions that arise. Of course, what is good input data, if we can't throw some bad input data in there to spice things up. Data validation is the process of testing software against good, bad, and unknown data. What happens when that 0 or 1 input was actually a -1 as input? Does the program blow up? We won't know unless we test it!
Unit Testing
This is basically the smallest form of testing. Do small units of logic perform as expected. This usually means I test a specific function or method and if the resultant output from that logic is what is expected, I could say that this unit test passed -- it performed as expected. This testing is more or less automatically performed by the programmer as code is written, or at least it should be!
Performance Testing
Does the software do what it is suppose to do, and do it fast enough? We may make the next best game ever, but if it runs at 8 frames a second... that's probably bad.
Compatibility Testing
This can sometimes go overlooked (especially on Android phones!), does the software work on multiple different configurations? Are there weird effects? Specifically for Android, will the product work on a phone, and an Android tablet?
Acceptance Testing
This is like Software Quality Assurance or Quality Control. Does the software deliver, on a high level, what the design is? Think of this like a marriage of software testing with play testing. Does the product allow the player to actually play the game as designed? Does the code actually allow for the player to launch new marbles and perform all the functions needed to play? If these are both yes, then it passed both the play test, and the software test, this the product can be accepted as the product.
For more information about testing, validation, methods to perform them, etc... really, we encourage you to look at the wikipedia article here: Software Testing. The trick is to find what works for you and use that.
Design to Product - Play Testing (14 / 17)
We constantly do general "play testing" - or rather, a general form of playing the game and noting any discrepancies that happen. This could be referred to as finding bugs, but really it isn't. It serves as a way to see if the game is any fun, if the design is "good" and what may have been overseen as pitfalls earlier in the design stages of production. Can things be improved? Does everything "feel" right? Are there overlapping aspects? Simply put, this form of testing finds design flaws not software flaws.
This method of testing happens all throughout the development of the product. This is by no means the only form of testing (nor should it be), but instead gives us the ability to second guess some of our designs as it is being developed and explore some options before publishing the product - more on software testing coming later.
One of the fairly humorous aspects of this is that back in our article about the graphics design where we showed the proposed layout for the game, we had big black circles at the bottom of the screen above the ads that indicated where marbles would occupy. These "slots" were to be used to launch marbles from.
During the development, we ended up shrinking those slots to about half that size so that the launcher graphic underneath could be seen more; since we made those graphics, may as well have people see them, right? Well over dozens of play tests, we found that it was difficult to correctly touch and drag marbles from those slots and be able to actually play the game. We ended up increasing their sizes back to the original layout sizes - taking up the entire height of the launcher.
The gameplay became much smoother and easier to drag marbles out, but the launcher graphic isn't seen easily, yet overall, the game is now smoother and the user should have an easier time actually playing the game; rather than seeing some eye candy. Aesthetics versus function is the issue here, and we chose gameplay over showing off artwork.
This detail was really only found after performing a lot of play tests and from different people. The general feedback was "we like this... but...". This should be taken seriously and addressed during development. Yes, the catch phrase of what users say can take a lot of extra time to implement. Fortunately for us, the slot sizes in this case was easy to change because only two variables controlled the width and height of the slot which "magically" handled both the visual display, and the collision detection of the user input. If these values were hardcoded, this could have been a fun hour or two adventure hunting down and "fixing" all the math that referenced the slot sizes.
When your friends say they want to be game testers, this is generally what they mean without actually knowing it. They want to be play testers, not software testers. Unfortunately for them, usually play testing is a team effort and isn't a dedicated job. For example, this is why software publishers utilize beta tests, they literally outsource play testing to users and gather feedback about the design of the game in a distributed manner -- if they find actual software bugs as well, that's a nice side benefit, but in reality the primary purpose is to assess the value of the design of the product. It is also a nice way to get the word out and serves as a nice marketing gimmick, but that's a whole other issue.
This method of testing happens all throughout the development of the product. This is by no means the only form of testing (nor should it be), but instead gives us the ability to second guess some of our designs as it is being developed and explore some options before publishing the product - more on software testing coming later.
One of the fairly humorous aspects of this is that back in our article about the graphics design where we showed the proposed layout for the game, we had big black circles at the bottom of the screen above the ads that indicated where marbles would occupy. These "slots" were to be used to launch marbles from.
During the development, we ended up shrinking those slots to about half that size so that the launcher graphic underneath could be seen more; since we made those graphics, may as well have people see them, right? Well over dozens of play tests, we found that it was difficult to correctly touch and drag marbles from those slots and be able to actually play the game. We ended up increasing their sizes back to the original layout sizes - taking up the entire height of the launcher.
The gameplay became much smoother and easier to drag marbles out, but the launcher graphic isn't seen easily, yet overall, the game is now smoother and the user should have an easier time actually playing the game; rather than seeing some eye candy. Aesthetics versus function is the issue here, and we chose gameplay over showing off artwork.
This detail was really only found after performing a lot of play tests and from different people. The general feedback was "we like this... but...". This should be taken seriously and addressed during development. Yes, the catch phrase of what users say can take a lot of extra time to implement. Fortunately for us, the slot sizes in this case was easy to change because only two variables controlled the width and height of the slot which "magically" handled both the visual display, and the collision detection of the user input. If these values were hardcoded, this could have been a fun hour or two adventure hunting down and "fixing" all the math that referenced the slot sizes.
When your friends say they want to be game testers, this is generally what they mean without actually knowing it. They want to be play testers, not software testers. Unfortunately for them, usually play testing is a team effort and isn't a dedicated job. For example, this is why software publishers utilize beta tests, they literally outsource play testing to users and gather feedback about the design of the game in a distributed manner -- if they find actual software bugs as well, that's a nice side benefit, but in reality the primary purpose is to assess the value of the design of the product. It is also a nice way to get the word out and serves as a nice marketing gimmick, but that's a whole other issue.
Subscribe to:
Posts (Atom)