Activity Stream Metrics

as_android_data_events.md

Session

Whenever the AS panel is displayed we start a session with the name "activitystream.1". All events triggered while we show the panel will have this session name attached. Additionally we send a probe whenever the panel is shown/hidden.

"New tab" panel shown

• action = show.1
• method = panel
• extras = as_newtab

"New tab" panel hidden

• action = cancel.1
• method = panel
• extras = as_newtab

Loading URLS

Loading Top Sites URL

• action = loadurl.1
• method = listitem
• extras = as_top_sites

Loading Highlights URL

• action = loadurl.1
• method = listitem
• extras = as_highlights

Showing Context Menu

Top Sites Context Menu

• action = show.1
• method = contextmenu
• extras = as_top_sites

Highlights Context Menu

• action = show.1
• method = contextmenu
• extras = as_highlights

Context Menu Actions

Share

• action = share.1
• method = list
• extras = as_contextmenu

Copy

?

Add to home screen

• action = action.1
• method = contextmenu
• extras = as_add_to_launcher

Open new tab

• action = loadurl.1
• method = contextmenu
• extras = as_new_tab

Open new private tab

• action = loadurl.1
• method = contextmenu
• extras = as_private_tab

Dismiss

?

Delete (history)

?

This seems like a decent place for a discussion :-) I've started thinking about what kind of insights we want to gather from this telemetry, and documented some thoughts in https://bugzilla.mozilla.org/show_bug.cgi?id=1319245 -> and then I saw then there's already some AS telemetry that's landed!

However, looking at the events, I'm wondering if this actually useful enough. We'll get some basic usage data out of this - "are people clicking on top sites? on highlights? which menu items are they interacting with?" - but it shouldn't be too much effort to expand our telemetry further.

One thing to note is that we have two types of items, and thus two types of menus - even though it's just one context menu, people might be interacting with them differently. It might make sense to track that as a separate method.

Another change that I want to explore is tracking additional data for the events we're recording. For example, for highlight interactions, it would be good to know that, for example, people aren't interacting with "recently bookmarked" items we're showing them. Or that pinned top sites aren't actually being used as much as other top sites. You can imagine many other similar questions; also, we have bugs on file to track if sync is enabled/disabled, and "sync state" of individual items, whatever that means.

In short, we need to be able to track richer data than just a label in the extras field. A possibly simple workaround for this is to use a simple JSON data structure.

For example, consider this event:

• user with sync enabled clicks on a third highlight's (whose origin is 'bookmarked' and it's not pinned) menu icon, and presses dismiss.

We can track it like so:

{
 "event": "action",
 "method": "as_highlight_contextmenu",
 "extras": {
  "item": "dismiss",
  "position": 3,
  "highlight_origin": "bookmarked",
  "pinned": false,
  "sync_enabled": true
 }
}

I think we should able to query against data like this pretty easily via https://prestodb.io/docs/current/functions/json.html, although I'm clarifying that in #telemetry.

Confirmed with folks at #telemetry that storing and querying against json strings should work just fine. I'll land a patch to see how well this works!

@grigoryk Yeah, from my point of view there are "two kinds of data" we are looking for:

• (1) Telemetry parity with the existing panels and other entry points: For comparing the new "New tab" panel with the existing ones and to see if it changes user behavior.
• (2) Adding new telemetry probes to help us improve and iterate on the new panel.

For now all telemetry that landed was focused on (1) and copied the structure of our existing probes in other places. I agree that this is not sufficient for (2) and we want to track more detailed actions. My goal was to finish (1) and then talk to barbara and marina again so that they can start creating the reports in re:dash and let us know what additional data they are interested in; so that we have at least data parity with the desktop add-on.

Your format is looking good though. That's btw. the format and data the add-on collects:
https://github.com/mozilla/activity-stream/blob/master/data_events.md

I wonder if we can use the JSON functions of presotdb in re:dash or if we need to create our own spark jobs in order to use them. However it should be fairly easy to test that: Our telemetry for preferences already sends JSON formatted "extras".