Your App’s Not As Fast As You Think

June 6, 2014

It was all going so well. Until suddenly it wasn’t.

I have an app that started as a experiment. A toy. It eventually evolved into a full fledged game. Because it started this way it was built using things I know well. Specifically CALayers and other assorted Core Animation goodies. Despite reading the occasional warning that I should be using OpenGL ES or one of the many 2D game engines, I pressed on. After all, periodic spot checking of performance revealed no issues. In fact the frame rate was great and I appeared to have plenty of headroom. Plenty.

Actually, I did at one point decide to create a small test app just to see how OpenGL might perform in comparison and was surprised to discover that it seemed to require more CPU. Seeing as I was fairly ignorant about OpenGL I chalked this up to Apple doing clever things to somehow optimize CA in certain situations. “Lucky me!”, I thought!

So I pressed on. At the start of development my primary phone was an iPhone 4S. Later I upgraded to a 5s, but continued to test on my old phone and all seemed well. Then I tried it on my son’s iPhone 4. That’s when the trouble started. The game, which was buttery smooth on my device, was unplayable on the 4. Granted the iPhone 4S offered a big boost in performance over the iPhone 4, but was it really that big of a difference?!

I fired up Instruments to get to the bottom of the problem and Instruments told me… everything is fine. CPU usage was maybe 30-40%. Graphics device utilization fine. Everything’s awesome! And yet clearly it wasn’t. What the?

On a hunch I decided to use the Activity Monitor in Instruments to look at CPU usage for all processes and discovered that backboardd was using ~70% of the CPU.

So what the hell is backboardd and why does it hate me?!

Google says backboardd is primarily responsible for event handling in iOS but it’s also seemingly responsible for doing much of the heavy lifting required by Core Animation. Remember back when I mentioned how my OpenGL experiment seemed to indicate it was slower than Core Animation? Well that’s because I was only looking at half of the picture.

So at this point I have a big problem. I suppose I could somehow limit support to iPhone 4S or newer and it would appear to run fine, but I’d be releasing an app that I now know to be fantastically inefficient. Or I can completely replace my rendering engine (such as it is). I decided to break down and learn Open GL ES. I could have also looked at Cocos2D or Unity but I’ve had “Learn OpenGL” on my todo list for far too long and I really want to understand what’s going on at the lowest levels. Mmm, shaders.

This was all not quite three weeks ago. At that time, on my iPhone 5s, the most complex level was using roughly 35% CPU while backboardd used 70%. GPU device utilization was around 10%. Since then I’ve learned enough OpenGL to be dangerous and replaced most of the Core Animation with my own rendering engine. I’m not quite finished and there is still tuning to do, but now I’m seeing about 40% CPU usage and 20% GPU utilization. Using more of the GPU is actually a good thing since I’m offloading work from the CPU that the GPU can perform more efficiently. Meanwhile backboardd has dropped to about 15%. Yeah there are still some lingering CALayers laying about that I’m not easily able to rid myself of yet.

So the takeaways from all this:

  • Really, seriously, spot check performance on all the devices you plan to support and don’t assume. My assumptions were based on an incomplete picture of performance data.
  • Keep an eye on CPU usage of all processes when running performance tests. With only one app visible at a time it’s easy to forget that your app doesn’t have the whole place to itself and needs to play well with others.
  • Pay closer attention to those that have gone before you. I read many times how Open GL or a purpose built 2D rendering engine was the way to go but shrugged it off. I should have tried harder to discover why my results seemed so much better than expected.
  • Open GL ES and Xcode’s Open GL analysis is pretty nice. I wish there were a Core Animation counterpart.
  • Don’t write a game next time.
1 Comment

Analytics and Logging Ideas

April 10, 2014

A couple weeks back I threatened to start making more regular posts. This is a threat I make from time to time and sometimes, like now, I actually follow through with it. In that same post I also alluded to some of the ways I’ve been using Parse and Dropbox while developing my game and that’s what I wanted to write about today.

First, Parse. I just want to touch on it briefly. Parse is primarily used to store / sync your app’s data in the cloud but offers an assortment of additional features to make things like push notifications and in app purchase easier to implement. It does these things well, but I’m using the data storage mechanism to create custom analytics. Parse encapsulates arbitrary key-value pairs in PFObjects which can be easily pushed up to their servers in near real time assuming you have an active network connection. Okay so perhaps this is an obvious use case, but I’m using PFObjects to note an assortment of information about each level that’s being played in the game I’m working on. It’s super easy to add a name, score, game-time, wall-time, result, etc., etc. to an object and store it for each attempt made on a level. I can then filter and analyze these results using their dashboard and see how long it takes a player to complete a level, or if they’ve giving up, got killed, etc., etc. Whatever information I feel is valuable is stored and easily accessible as I iterate over game and level design. Very easy to use and extensible. Plus it’s free (up to a point). Free is nice.

Okay next I really wanted to talk about how I used Dropbox to enable logging from multiple devices. So my problem was this: I needed to debug several devices simultaneously and it’s a bit of a pain to do this in Xcode especially when those devices may not be in the same room / city / state. I think TestFlight and maybe some others offer remote logging but these solutions weren’t exactly what I needed. Specifically I needed to start a multiplayer match with N-players and have an individual log created by each of the players in the match and deposited in a central location so I could reconcile any differences between them. I also needed this information as soon as the match completed.

It turns out that I was able to use Dropbox to do exactly this. I put together a demonstration project on github that you can grab and follow along if you like. Once you have that, head over to dropbox’s developer page and add a new app. Set up your app using the Dropbox API with Files & Datastores limited to its own private folder. I called mine QSLog. I don’t think you can reuse that name but if not it doesn’t really matter. Call it whatever. The important bits are the App key and App secret that you’ll find after you create the app. Copy those values and update the corresponding defines at the top of QSAppDelegate.m. Also replace db-appkey in QSLog’s URL Types with db-whateveryouractualappkey is. So far we’re pretty much following this basic Dropbox tutorial.

At this point you should be able to compile and run the app from Xcode. When you do, it should present you with a Dropbox connect screen. Log in and then check the console in Xcode for the User ID, accessToken, and accessTokenSecret. Take these values and again update the corresponding defines at the top of QSAppDelegate.m. Lastly change line 18 QSLog-Prefix.pch to #define HAVE_DROPBOX_TOKENS 1. Recompile and run the app again. Tap some of the Log Event buttons. When your ready tap the Upload / Reset button. That should upload a history of the button presses and reset the log. You should have a folder for your app in your Dropbox/Apps folder containing this log. And you should be able to run this app from any device and have the logs appear in the same folder.

Voila! Which is French for… Ta Da!

Okay a couple final thoughts. First, you really don’t want this in any shipping code. Actually I’m not even sure you want it in test code for an extended period of time. And second, the logging class in the sample project is really simple and you’ll probably want to replace it with something a little better. This is just something I whipped up for demonstration purposes.

Leave a comment

WWDCs Past (2014 edition)

March 31, 2014

It’s that time of year. Here’s an updated table of information about WWDC dates and announcements over the last 9 years.

Year Announce Date Announce Day of Week Conference Date Week In June (full) Days Notice Time to Sell Out
2005 Feb 15, 2005 Tuesday Jun 6, 2005 1st 111 days n/a
2006 Mar 8, 2006 Wednesday Aug 7, 2006 n/a 152 days n/a
2007 Feb 7, 2007 Wednesday Jun 11, 2007 2nd 124 days n/a
2008 Mar 13, 2008 Thursday Jun 9, 2008 2nd 88 days 60 days
2009 Mar 26, 2009 Thursday Jun 8, 2009 1st 74 days 30 days
2010 Apr 28, 2010 Wednesday Jun 7, 2010 1st 40 days 8 days
2011 Mar 28, 2011 Monday Jun 6, 2011 1st 70 days 12 hours
2012 Apr 25, 2012 Wednesday Jun 11, 2012 2nd 47 days 1h 43m
2013 Apr 24, 2013 Wednesday Jun 10, 2013 2nd 47 days 2 minutes
2014 Apr 3, 2014 Thursday Jun 2, 2014 1st 60 days lottery
Announced on
Sun Mon Tue Wed Thu Fri Sat
0 times 1 times 1 times 5 times 3 times 0 times 0 times
Leave a comment

What I’ve Been Up To

March 27, 2014

Where to begin…

You probably can’t tell from looking at my site, but I’ve been really busy over the last year or so. Most of that time was spent creating a game that I hope to release soon, but the last few months that work has taken a backseat to a diversion into crypto-currency (bitcoin, litecoin, etc). The shininess is at last fading from crypto and I’m starting to spend more and more time on my game again, so hopefully things are getting back on track and I’ll actually ship one day.

During the time working on my game I thought about keeping a development journal here, but as development progressed I never knew quite where to start or how to catch up with all that had gone on, so it just never happened. The same thing happened when I went off on the bitcoin tangent. I have so many thoughts about bitcoin and crypto in general but they’re all over the place and I just couldn’t take the time to untangle them and write them down here.

While it might have been interesting to follow along as I learned about bitcoin, it would have been even more worthwhile to have documented the creation of my game. Besides maybe helping others by sharing the resources and pitfalls I’ve encountered along the way, I’d have had a chance to get some feedback on decisions that I’ve made and maybe someone would have said, um it’s been four months since we’ve heard anything about your game, did you give up? Of course expanding my group of beta testers to more than three people might help too.

So, what I think I’ll do is create a condensed timeline of my goings on since development started and later maybe post more regular updates. I won’t get into too many details about the gameplay itself yet because I think I might have something bordering on an original idea and I’d like to wait until I’m closer to release before I dive too deeply into that. So here’s what I’ve been up to:

May 2, 2013

Initial commit in proof of concept project. Not yet playable.

May 6, 2013

The commit message on this date was “should commit at some point”. So helpful. It occurs to me while skimming through my commits that I’ve never had a repo where I said to myself, “this repo has way too many commits” or “these commit messages are far too descriptive”. I find that I’m far more lax about commits and commit messages in my personal projects than I am elsewhere. Definitely an area I need to improve upon. Side note: This is also the date of the first commit for the level editor repo.

May 9, 2013

I believe this is the first date that the game is truly playable. The game is built on top of box2d and at this point was using mostly CAShapeLayers for rendering since I didn’t (don’t) really know OpenGL. Nor did I know that SpriteKit was just around the corner.

May 30, 2013

v0.1: Most of the game elements are already present in this release and haven’t changed significantly since. The shape layers are gone at this point, but all elements are still CALayers. At this point, the most complex level only has 80 elements (layers). But, as I’m managing the the images that back them, I can be very efficient with memory usage. From the Core Animation Programming Guide:

Assigning an image to the contents property prevents the layer from allocating memory for a backing store. Instead, the layer uses the image you provide as its backing store. When several layers use the same image, this means that all of those layers are sharing the same memory rather than allocating a copy of the image for themselves.

June 10 – 14, 2013

WWDC and iOS 7. Still not thrilled with iOS 7, but I’ve made peace with it I guess. Kind of funny since my game was designed from the beginning with a sort of flat / minimalist look. As for the conference, I probably should have showed the game around more while I was there, but only a handful saw it. There never really seemed to be time for a full critique either. Just short demos. Regardless it was good to see everyone again. And, as WWDC announcement time approaches again this year, I’m beginning to waffle on my threat to quit WWDC.

July 3, 2013

v0.2: I got a lot of feedback about that initial release. Far and away the most popular complaint was that the game was too hard. Initially I wanted it to be really hard. Some of the most satisfying moments I’ve had gaming are finally beating a level / solving a puzzle that you’ve been banging your head against forever. Eventually I conceded that those times are probably past. With so many distractions available today players are probably more likely to just give up completely rather than keep working at it. The original 11 levels were scrapped. 17 new ones took their place. Here I also start trying to figure out how to get levels to progress smoothly in difficulty. It’s really hard to judge when you’ve been playing for nearly two months and I’m still struggling with it. The levels also start to grow in size here. In the original set the biggest level was about 900 x 900 points in size with most about a quarter of that. The new levels are generally much larger with as many as 500 CALayers. v0.2 also added a few more game elements, scoring, and a lot of optimization. Normally I wouldn’t be too concerned with optimization at this point, but I need to have an idea what sort of limitations I’m going to have when it comes to level design and hardware support.

August 4 – 11, 2013

v0.3 – v0.3.2: Another go round with levels although they are mostly evolutionary at this point. Also a new level picker, better camera control and zooming. Zooming was another feature that was added after listening to feedback from testers. As the level designer I know exactly how each is laid out so not being able to see the whole map didn’t bother me as much. However, I still wanted some element of discovery so decided to limit how far you can zoom out. Also decided that I would not add the ability to pan. You get a very brief glimpse at the entire level as it loads but I’m not yet convinced there is a need to see the entire level on demand.

September – October 2013

v0.4 – v0.4.3 (aka two months in hell): It all started innocently enough when I asked my wife “I wonder how hard it would be to add multiplayer support?” This is my first real experience creating a game (unless you count some of the nonsense I wrote in high school a million years ago) and along the way I made pretty much every mistake it was possible to make. If you’re thinking of creating a game, run, don’t walk, to this site and read all of the words in the Game Physics and Networking sections. Before I could even get to networking issues I had to separate my rendering and integration. I also had a variable timestep. Because rendering and integration were combined it was the only way to reduce frame rate if needed. So basically the whole game engine had to be re-written. Once that was complete I remained convinced that I could somehow write networking code that could basically see into the future. It turns out this is not possible. I finally came to the realization that multiplayer would not work unless I used lockstep synchronization. Sadly this means I have a ~200 mS minimum control delay when playing multiplayer, but it’s not terribly noticeable in practice. Lastly, floating point determinism. I won’t get into details in this post but basically the solution required me to add a separate set of transcendental functions (pow, exp, atan2, etc) and use them in place of those included in the standard C library. Perhaps there was another way, but it’s done now. There were many times I nearly bailed on multiplayer and to be honest it could still use some fine tuning, but what I have is a good start and a lot of fun.

November 19, 2013

v0.4.4: This release should have been v0.5 but it didn’t involve much change to the actual code itself. By v0.4.3 I was up to 24 levels but I still wasn’t happen with the progression. And there were also some inconsistencies I wanted to address. v0.4.4 actually had fewer levels but they were all completely reworked. I think I am, at last, pretty happy with the level design and progression. Maybe.

November – March

The Great Bitcoin Interruption: I’m not going to get into many details about crypto-currency, except to say that I find it pretty fascinating for a variety of reasons. I have no idea if it survives and actually becomes useful as a currency, but it’s a lot of fun to play with. Coincidentally it solves the Byzantine Generals Problem that I was running up against while trying to work out networking for my game, although not in a way that I can use.

February 10, 2014

v0.5: Somewhere in between the mining, writing my own wallet, contributing to dogecoin, etc, I managed to push out another update. This one adds some more refinements and levels bringing the total up to 29. Nine of these levels span an area greater than 1,500 points2 including one massive level with ~1300 elements spanning ~2300 x ~2600 points.

Ongoing

During that time I also sprinkled in five updates to Goalposts and three for Super Speller. So much to do. So little time.

Now

So where does that leave things? I figure at a minimum a have the following items left:

  • 2 more single player levels and 1 multiplayer level bringing the total to 32.
  • Tutorial section.
  • Soundtrack / Sound effects.
  • Scoring changes and rework for Game Center leaderboards.
  • Lots of refinement. Especially multiplayer.
  • Pricing decisions. Probably IAP. I would much rather charge up front but I don’t think I’d get any traction this way. This could be the topic of a post in itself.

Besides those items I’d really like to loop back and record some more thoughts here. Including using Parse to gather analytics during testing, a very handy technique using Dropbox to record logs from multiple devices into a single Dropbox folder (invaluable while I was working out multiplayer), thoughts on designing levels (it’s hard), floating point determinism (grr), and how I’m loading levels and whether or not to release the level designer so people can create their own. 90% of the work is done but we all know about that last 10%.

Leave a comment

Fragmentation

June 21, 2013
93% of customers are using iOS 6.
93% of customers are using iOS 6.

So Apple’s published this chart recently that I’m sure most of you have seen. It claims that 93% of customers are running iOS 6 and 99% are running iOS 5 or later. If accurate it’s pretty amazing. And why wouldn’t it be accurate? Well it’s possible that this only includes those devices that are capable of running iOS 6.

In any case it got me thinking, especially as I believe this is meant to encourage developers to adopt iOS 7 sooner rather than later, just how many sales have there been for each iPhone model? (note: I only looked at iPhone sales and not all iOS devices which may also be the reason Apple’s chart tends to skew higher) Turns out this isn’t really easy to find, but you can kind of / sort get an idea by looking through Apple’s quarterly reports. Here’s a breakdown of sales, that I’m pretty sure is accurate, grouped together by iPhone models as of Q2-2013:

Model/s Period Sales (thousands) % of Total
iPhone Q3-07 – Q3-08 6,124 2%
iPhone 3G Q4-08 – Q3-09 20,254 6%
iPhone 3G / 3GS Q4-09 – Q3-10 33,254 9%
iPhone 3GS / 4 Q4-10 – Q4-11 86,395 24%
iPhone 3GS / 4 / 4S Q1-12 – Q4-12 125,046 35%
iPhone 4 / 4S / 5 Q1-13 – Q2-13 85,219 24%
Total Q3-07 – Q2-13 356,292 100%

That’s a lot of phones. But notice that of the 350 million or so iPhones sold, at the very least, 26 million (8%) are incapable of running anything later than iOS 4.2.1. And that doesn’t include those iPhone 3Gs (not 3GSs… I don’t know how to make that plural). sold between Q4-2009 and Q3-2010. A fairly conservative guess puts that number closer to 40 million or 11%. That doesn’t really line up with Apple’s chart. Of course a lot of these phones are probably no longer functional or in service but 90% of them? Another possibility is that the people using these phones just don’t visit the App Store that often.

So, for various reasons, we’re not seeing many original iPhone and iPhone 3G users represented in Apple’s chart, but will we see the same adoption rate with iOS 7? I personally see a lot of iPhone 3GS users still out there. It’s not even been a year since it was discontinued after all.

It difficult to know exactly how many iPhone 3GSs were sold, but I think a reasonable estimate is around 73 million, or 21% of the total iPhones. If you add in the old crusty iPhone and iPhone 3G models it’s almost one third of all of the iPhones ever sold. And even if, let’s say, only half of all of these phones are still in use, that would work out to about 11% of active iPhone users that won’t be able to make the jump to iOS 7 even if they wanted to. Now add to that the people that won’t update because… who knows (these are the 7% in the current chart) and those that refuse to update because they don’t like the new version (yay polarization!) and it could be a while before iOS 7’s adoption rate gets anywhere near where iOS 6 is today.

Leave a comment