An Unexpected Botnet

“With great power comes great responsibility.”

iOS 7 introduced a fantastic feature called Background Fetch (Excellent overview by David Caunt here). This allows applications to opt-in to being woken up in the background periodically to fetch new data. This is perfect for applications that display periodical information that is constantly changing. You can, in theory, always have something fresh waiting for your user when they launch your app. No more spinning refresh wheels on launch.

In my own experience this feature has caused a subtle but definite improvement in my own experience using my iPhone. Everything feels slightly faster. I spend less time waiting for data. Yay.

There is, however, an intrinsic danger in applying this ability without fully thinking through the implications. When enabled within your applications you are essentially building a massively distributed botnet. Each copy of your application will be periodically awoken and sent on a mission to seek and assimilate internet content with only the OS safeguards holding it back. As your app grows in popularity this can lead to some rather significant increases in activity.

Examples

My first example of this was when I added Background Fetch to Check the Weather. A weather app’s primary function is displaying up-to-the-minute, constantly changing data so in my initial iOS 7 update I experimented with adding highly frequent background updates. The result was far more dramatic than I’d expected. Here are my weather API requests (which cost 0.01¢ per request) per day once the update went live. I saw an immediate jump in traffic, roughly 16x normal. Suffice to say I immediately had to scale back on my requested update frequency.

I ran into another example of this earlier this week while working on the server-side podcast aggregation that is part of Pod Wrangler. Pod Wrangler makes use of server based aggregation of podcast RSS feeds to know when new podcast episodes are available. The tricky part of server-side aggregation is that you need to determine a reasonable frequency to poll for feed updates (typically around 15 minutes for Pod Wrangler).

I was curious how often other clients were checking and so looked at the request logs for Developing Perspective.

NB: This isn’t intended as an attack on these applications, they are solidly built by solid people. They are simply a well encapsulated example of the broader point.

The results were rather surprising. Here are the feed request frequencies for various Background Fetch enabled podcast clients¹. Keep in mind that Developing Perspective is a weekly show with a modest audience. I’d expect the numbers for more popular shows to be even more dramatic.

For an RSS feed that changes only once per week just these apps produce 126k web requests each week (out of 160k across all aggregators ). The feed itself is 450KB (49KB gzipped). Where it not for HTTP caching/compression (discussed later) this would be generating 56GB of almost entirely unnecessary downloads each week.

Recommendations

So what should we as developers do to rein in these botnets we are creating? Here are some recommendations I’ve found from my own experiences that help a lot.

1. Be thoughtful about the value you give when configuring your apps fetch interval. Don’t just pass in UIApplicationBackgroundFetchIntervalMinimum and hope for the best. Think about how often the data your user is looking for is likely to change and how much increasing the interval will improve the user’s experience. The OS will throttle back your frequency based on actual use but you need to give it useful hints about what you are trying to fetch².

2. Be honest when your application didn’t get data from a fetch. Whenever you are awoken for a Background Fetch you are required to provide back to the OS whether you actually found new data. It can then use this as part of its heuristics about when to wake you up. I know I’ve been very tempted to lazily always return UIBackgroundFetchResultNewData every time, rather than actually building in the complexity of tracking when data was returned. This will artificially raise your frequency beyond what is actually needed or useful.

3. Make sure you are taking advantage of HTTP caching and compression. Judicious use of If-None-Match, If-Modified-Since and gzip can dramatically reduce the costs of frequently accessing the same resource. Some of this is done transparently for you by the OS but it is a good idea to verify that it is actually happening by using a tool like Charles Proxy to inspect what you are actually sending over the ‘wire’. Mattt Thompson has a good overview of NSURLCache over on NSHipster which should also be required reading.

4. When appropriate, use silent push notifications to avoid fetch entirely. Another feature of iOS 7 is the ability to proactively wake up your application with a silent push notification when your servers know that new data is available. This approach is intrinsically more efficient since the client apps are only awoken when there is a guarantee of useful work to be done. While not appropriate for every application if your servers already keep track of new content consider using them to also offload this resource collection.

We are building applications that run on handheld, battery powered computers that connect to the internet over wireless networks. Every watt that we can save will improve our customers’ experience.

---