This is a draft list of what we're thinking about measuring in Etsy's native apps.
Currently we're looking at how to measure these things with Espresso and Kif (or if each metric is even possible to measure in an automated way). We'd like to build internal dashboards and alerts around regressions in these metrics using automated tests. In the future, we'll want to measure most of these things with RUM too.
- App launch time - how long does it take between tapping the icon and being able to interact with the app?
- Time to complete critical flows - using automated testing, how long does it take a user to finish the checkout flow, etc.?
- Battery usage, including radio usage and GPS usage
- Peak memory allocation
- Frame rate - we need to figure out where we're dropping frames (and introducing scrolling jank). We should be able to dig into render, execute, and draw times.
- Memory leaks - using automated testing, can we find flows or actions that trigger a memory leak?
- An app version of Speed Index - visible completion of the above-the-fold screen over time.
- Time it takes for remote images to appear on the screen
- Time between tapping a link and being able to do something on the next screen
- Average time looking at spinners
- API performance
- Webview Performance
Lots of good content and ideas here. It's important to note that there are different audiences looking for app performance data. Each audience has a slightly different pivot from the necessary data - and each likely could have levels as guypo suggests.
One audience who is interested in how the app performs from action to action. How long does it take to load the app? How long does it take for the animation to complete? How much cpu, memory are used for an action? etc.
The second audience are those connected to network activities. User does X, causes Y network request(s), results in Z display change. In this scenario we are looking at it very much in the same light as web page timings. In fact most of the existing w3c spec would naturally apply here. You want to capture all the nitty gritty things like dns time, ttfb, and finish animations. For this audience some interesting things to track would include:
There is a third audience here as wel:. Those that are concerned about the impact of usability and usage of the app. How much time does the user spend in the app. What is the abandonment rate? In the world of web, there are many incidental artifacts that allow us to track (from logs) the users progress through a transaction (finding an item, putting it in the cart, checking out, etc). With native apps, these loggable artifacts aren't always available and therefore require more explicit capture and beaconed back.
/colin