DND 21: Byte Compression for Watch Connectivity

Design Notes Diary

One of the features in the next update of Pedometer++ is that you can build a list of GPS routes on your iPhone (either from previous workouts or imported GPX files). You can then view these routes as overlays on a map during your walks.

Additionally, you can then transfer these routes to your watch to be shown on the Watch’s workout map display.

The challenge in building this feature is working out how to reliably get the routes to sync from the iPhone to the Watch. Superficially the answer is a simple one…just use Watch Connectivity. Which is the correct answer, but one that carries with it a tremendous amount of luggage.

Watch Connectivity is one of those APIs that feels like working with an extremely sharp knife. Super capable, powerful and useful…but if you don’t use it 100% correctly can be super dangerous. I’ve used Watch Connectivity on several projects now and I feel like I know where most of the gotchas and pitfalls are. But I still find it one of the more challenging setups to get right.

In my case the question was which of the several connectivity pathways to use. For watch connectivity transfers you can send them over as either:

  • File Transfers
  • User Activity Dictionaries
  • Application Context Dictionaries
  • “Realtime” Messages

Each of these transfer methods carry with them different guarantees, expectations and limitations.

File transfers let you move large files to the watch, but with the drawback of completely unpredictable transfer times. Whereas, messaging is much more reliable and actual ‘best effort’ transfers but come with content size limitations.

For my use I decided to go with the messaging approach. I can have the Apple Watch request the routes from the iPhone when you are starting a workout which will wake up your iPhone app to get a response. The request are typically sent and received very quickly and perhaps more importantly will fail obviously if there are troubles.

The issue I ran into is that the payload of a message is limited in size (seems to be around 65KB but this isn’t specifically documented). I’m moving over arbitrarily large route files so this presents a challenge.

For my testing and development I used the route file that I generated on the last day of my Watch Ultra Review, where I walked 26.2 miles in a single session. The GPX file for this walk is 5.5MB and contains 39,323 individual coordinates. So I have some work to do to get this to fit into a message. But I figure that if I can get this file to work then more typically sized files should be fine.

The first thing I did was to run the route file through the Ramer–Douglas–Peucker line simplifying algorithm. This essentially looks for segments of the route where it is following a straight line and throws out all but the endpoints of these linear segments. You loose a bit of accuracy but not much.

Here is an image of the result in action. You can see that the simplified (blue line) cuts a few little corners and doesn’t follow the original perfectly but is incredibly close. For my purposes this seemed good enough.

In my test case this reduced the point count from 39,323 down to 1,591. Which is a massive win.

Next was how to actually package up the data for transfer. I could just put them in an array and pass that to Watch Connectivity but I suspect very large arrays like that wouldn’t be a good idea, and I also don’t like that I can’t easily predict what the final payload size is.

So instead what I decided to do was to use a binary plist encoder which generally does a good job of creating a compact representation. I’m sure there is a more compact encoder I could write myself, but pragmatically this seemed to generate solid, “good enough” results with minimal effort…my favorite kind of solution.

Rather than encoding the coordinates as tuples of data or Swift Structs, I instead opted to encode them as two arrays: one for latitudes and the other for longitudes. I can then re-merge them back together on the other side. Other formats seem to bloat up the size and were needlessly complex.

Another approach I took was to send over the data as integers rather than doubles. I multiply the double values by 1,000,000 and store the rounded value. This is more byte efficient and the expense of a very minor loss in accuracy. In my testing it resulted in an average point location error of around 1.3inches…which is probably small enough to never even shift the path by a pixel in display.

Combining all these approaches together I can now represent the original 5.5MB GPX file in a format that is just 34,902 bytes a 157x “compression” factor. That gives me roughly 2X headroom for the messaging limits and means that for shorter, more typical routes I’ll be very, very under the limit.

In practice, this has been super fast and efficient to transfer and works perfectly. For a feature that I originally went into with some trepidation (any time I work with Watch Connectivity I’m nervous), the result was solid and reliable.

Today’s diary entry was way more down in the weeds than typical…so I have a little reward for those who got this far. The first place I’m going to post a link to the TestFlight build for the update is right here. If you are interested enough in this project to read all that, you deserve first dibs on trying it out.

I’ll be opening up the beta more generally next week.

David Smith