Native app performance metrics

app-perf.md

Native app performance metrics

This is a draft list of what we're thinking about measuring in Etsy's native apps.

Currently we're looking at how to measure these things with Espresso and Kif (or if each metric is even possible to measure in an automated way). We'd like to build internal dashboards and alerts around regressions in these metrics using automated tests. In the future, we'll want to measure most of these things with RUM too.

Overall app metrics

• App launch time - how long does it take between tapping the icon and being able to interact with the app?
• Time to complete critical flows - using automated testing, how long does it take a user to finish the checkout flow, etc.?
• Battery usage, including radio usage and GPS usage
• Peak memory allocation

Metrics per screen

• Frame rate - we need to figure out where we're dropping frames (and introducing scrolling jank). We should be able to dig into render, execute, and draw times.
• Memory leaks - using automated testing, can we find flows or actions that trigger a memory leak?
• An app version of Speed Index - visible completion of the above-the-fold screen over time.
• Time it takes for remote images to appear on the screen
• Time between tapping a link and being able to do something on the next screen
• Average time looking at spinners

Additional considerations

• API performance
• Webview Performance

Hi,

Great discussion. I realize this is more than a year old now, but has anyone come across any more tools or methods to measure and maybe benchmark rendering in native mobile.

I am going to give the visual metrics video measurement from https://github.com/WPO-Foundation/visualmetrics a try.

Has anyone had any luck with this?

Thanks,

Nice thread. Does anyone have any pointers/product suggestions on how to measure mobile app performance?

Thanks!

Just wanted to share this resource: User-centric performance metrics

I found this to be a very powerful, perception-centered, and platform-independent way of thinking about app performance. I'd love everyone's help in thinking through how these should map to lower-level performance metrics, however. Here's how far I've gotten:

The experience	Notes	Metric
Is it happening?	Did the navigation start successfully? Has the server responded?	First Paint (FP) / First Contentful Paint (FCP) / Measuring any jank related to loading spinners or scrolling while UI is "busy"
Is it useful?	Has enough content rendered that users can engage with it?	First Meaningful Paint (FMP) / Hero Element Timing / Speed index and also the link above from @andydavies
Is it usable?	Can users interact with the page, or is it still busy loading?	Time to Interactive (TTI) / Not sure of native app metric...
Is it delightful?	Are the interactions smooth and natural, free of lag and jank?	Long Tasks (technically the absence of long tasks) / Frame rate / Jank per session (number of clusters of dropped frames per user session?)