Rollicking OpenGL ES Performance Adventure

I just got done with a crazy weeklong stint of development on my new game. Really, what I did was untangle a neverending mess that started with the downloading of a new Provisioning Profile for BoatGame. This is a good story for those who are curious what exactly a game developer does for a living.

First off, let me tell you a strange story that is true. Code rots. Just like logs in the woods. This log is RopeBurn, my ill fated but altogether lovely little game which is earning me nearly $4 a quarter, waiting for someone to rediscover on an emulator in 50 years. I'll warn you, its easier to get a new log than restore the old one.

Last year, the utterly amazing and horrifying 2011, was a "growth" year, meaning that I did a bunch of partying and socializing and swapping out some long time friends for new ones (usually involuntary) and minimal extra efforts on any extracurricular activities. On Halloween 2011, I arrive in Vietnam to fulfill a client contract for my employer and become good friends with a European 3D artist who has had a freakishly similar upbringing and career to my own. He motivated me to start a new project with him, and 2012 seems like a pretty good year to be productive in.

A few days after Christmas, I start the new project, using the tried and true methods of anyone familiar with IT, by copying and pasting a directory and searching and replacing some names, generally turning "RopeBurn" into "BoatGame". iOS development comes with this crazy locked down system of certificates that requires cryptic console commands and many order dependent steps. iOS development is easy until it does not work, and then it is difficult to figure out what is breaking, but what's new with software development tools. That's always been how it is. We're supposed to be making snappy lightweight programs, but we write them on these bloated monoliths that topple this way and that. I then fiddled with the XCode project to use the new certificates. I got stuck for awhile trying to understand why the debugger refused to believe that BoatGame was on the iPad, even though I saw the icon and could run it, but then I remembered a sneaky way out.

I could just install XCode 4. It would be like a big reset of whatever mess I left on this laptop. I could just follow the directions and viola! I could have everything working in the same few hours, except this would involve drinking wine and watching a progress bar and seeing who the roommates brought in. You know, I used to be of this mindset not to upgrade ever, and to maintain a static working environment, but I have learned over the years that upgrades are often easy fixes to mystifying problems. I learned that mystifying problems are often bugs in the dependencies. That's what I get for not writing my own compiler.

So I installed XCode 4 and suddenly everything is working, I can debug on the iPad, except that I keep having to reset the iPad every one or two debug runs. A debug run is when I write the tiniest bit of new code that would cause a visible change, and then I run the game to see how the new code changed what I see on the screen. It is important to be able to perform these little changes quickly, because I do not have written notes about the algorithm that I am debugging, just a bunch of code to stare at and random thoughts of trees and grids and whatnot in my head. Resetting the iPad takes two minutes. A quick run can be done in about 20 seconds. Two minutes feels like forever.

I try to restore the iPad, but there are none of my previously installed iOS images available in the XCode drop down. I take the plunge, updating the iPad to whatever the latest and greatest happens to be. I know that this version of iOS is probably designed to work better with XCode and I wouldn't have to reset the iPad constantly.

Well, I did change a few things by this point. Remember, everything that I just told you is just the stuff that I have to do to do the stuff that I need to do. I like to write game code and graphics code. I want to make a visible thing, and then make it do something. I don't like to google error messages and restore iPads to factory settings and find bugs in the Windows NVidia OpenGL drivers. I like to program. I had changed a number of graphics, and turned the tapering grass and ceiling style terrain to a curvy beach island. The new algorithm took up significantly more triangles than the old one. Now that I could switch between the debugger and the profiler easily, it would be simple to optimize my game, which was good since it was no longer hugging a solid 60 frames per second.

Well, here's the funny thing. Once the tool could tell me what was going on, it became very clear I had some serious unresolved issues. This new iOS had some new graphics driver inefficiencies, and my redundant state commands needed to be removed. The problem is essentially this. Say that I give you a paintbrush and blue paint. I ask you to paint a board on a wall blue. Then I ask you to wash the brush clean, and I'll think about what color I want you to paint next. I think we both can imagine a more efficient method of working together. I wrote an algorithm to manage multiple types of paints and layers, bringing the game from 34 fps to 39 fps.

Then, the panick hit. I brought the hundreds of redundant commands per frame down to less than 10, and the framerate didn't change very much. The last little bit wouldn't be worth the effort and the framerate was still far short of 60. I found a nifty new tool that came with XCode 4, called the OpenGL ES Performance Detective. It is very easy to use. I tell it which program to run on the iPad, and when I am ready, I click a button on the Mac and suddenly a progress bar starts going while the iPad freezes up and appears to crash. I see a message on the Mac warning me not to mess with the iPad and the iPad displays a funky little screen with this wierd gear lock key thingie, for developers only I suppose. Come to my house sometime and I'll show you, but I warn you that unless you are drunk, its going to be a disappointment.

Anyhow, the OpenGL ES Performance Detective finishes chomping on whatever data it managed to glean from the iPad and displays a text readout with suggestions on how to speed up the program. OpenGL was at fault, not the CPU. I knew that from the 2% usage, but thanks. There were other interesting hints, and something about Vertex Array Objects. Well, this is a funny thing in programmer land, but we have no problem making up confusing names, but we do it differently than doctors. Doctors like to make up long funny names from languages no one uses, producing words that don't appear to have any meaning, but supposedly we can trust them that the whole word does. Programmers like to do something like that, but we like to take small sets of words that have a specific meaning and change one word to come up with another meaning for the phrase, and to top it all off, toss in some abbreviations. VBOs and VAOs are not the same thing. One is a vertex buffer object, an earlier lesson in using pointless refactoring to limbo the entire game under some crappy graphics driver's limitations. The other is a vertex array object, which stores just a little bit more information about a triangle mesh and allows me to use one command to perform the usual extra steps in one command. This ended up being a dead end, although I did get the game up to 42 fps with a minimal change in just one part of the graphics engine.

I start googling and discovered that the panic was well founded. The iPad barely has enough fill rate to do 60 frames per second. This didn't quite seem right since Carmack supposedly got Rage to run on the iPad at 60fps, but that's what the article was saying and I was seeing. I was even able to find good confirmation of this fill rate limitation. It didn't seem right, because RopeBurn was less efficiently written than what I currently had BoatGame doing, and it always ran at a solid 60. I could turn off just the water, which was nothing more than a single giant quad with a GL_REPEAT water texture, and get a solid 60 fps.

As a last ditch effort, I added a depth buffer. If the grass is drawn above the water, first, then the depth buffer can be used to stop some of the water from getting drawn. A little more fps gain after I get past the white screen problem. That problem ended up being an undocumented OpenGL ES requirement, that the last render buffer bound is the color buffer, not the depth buffer, which shouldn't have mattered because I already bound the frame buffer.

I sprinkled more fine tuned OpenGL state control and used it to allow me to skip some slightly costly operations. Opaque objects can be drawn quicker than transparent objects. Objects that aren't used to block other objects don't need to write to the depth buffer. The depth and color buffers can be discarded at the end of each frame. There is this special little discard buffer command and sure enough, I got another fps.

Right now, my game runs at 50 fps. I can't conceive how to make it any more efficient. I am frustrated, because I now know that iOS 5 has some serious OpenGL performance bottlenecks, and that this is a widely reported problem on older iOS devices. I don't have a convenient downgrade path back to a performant iPad 1. I have an engine that is nominally optimized for newer iOSes. I could live with [displayLink setFrameInterval:2]; or get an iPad 2. Decisions, decisions. People are biting at my $350 offer for the iPad, so maybe I can make this sale and justify spending $100 for a solid 60 fps.