Jayson Whelpley

I’ve been assigned to a project at work that is going to require a significant amount of quantitative and qualitative user data to complete. Our main product is an app that was released in 2012, it’s on it’s 3 major version. Some significant bits about it:

• Localized in 18 languages.
• Content in 1600+ languages.
• 120,000 total videos.
• Android and iOS app.
• Separate contractor development teams.
• Managed by same internal person.
• Android: 65% of users (~67,000 in 2017)
• iOS: 35% of users (~36,000 in 2017)

Analytics Platforms

• GA
• Adobe
• Crashalytics
• ME2
• Flurry
• Currently testing AppSee and UXCam

As I’ve dug through data, I’ve begun to realize that our event reporting is inconsistent between OSes. Event labels have not been given precise and clear definitions. This is going to be a significant hurdle, so I’ve been seeking to sort that out first.

Signs of Trouble

The first sign of problems was discovered during the final feature build before the research project was to begin. One part of the feature integration was adding an icon/button to an already-crowded interface. The new feature was one that was to be highlighted as it was a part of an integration with an app that we wanted to promote the use of. It was clear that we were going to need to consider placing one or two previously-existing icons within a context menu to make room for the new one. We did not have the time resources to run a round of user testing to help determine how which ones were going to be harder to find, so I was asked to make recommendations based on analytics alone.

From the collected event data in GA, I discovered the following anomalies:

• Of all of the events possible to perform with the icons on the taskbar in question, the “Sharing – Generic” action was reported as 90% of all actions.
• Of the “Sharing – Generic” actions, 90% of those were Android (Android user share was ~65%).
• For all other actions on this taskbar, 90% of the events were iOS.
• “Sharing – Generic” was 98.5% of all Android actions for this taskbar.
• “Sharing – Generic” was 30% of all iOS actions for this taskbar.

The UI for the two OSes are different but do not seem different enough to warrant such vastly different event reporting numbers.
I met with the development teams and discovered that they were using “Share – Generic” in completely different ways. For Android every single time a user touched this button it logged “Share – Generic” and only that event. For iOS they were logging it only if it was successfully shared using an app outside of the ones that had their own specific sharing event assigned, e.g. “Sharing – Facebook”, and some recent versions of the app logged the string of the app name, e.g. “com.apple.reminders.RemindersEditorExtension”.

Digging into the data

Prior to this point, I didn’t have access to some of the analytics platforms. I had previously been assigned only as a UI designer and was still onboarding into the UX role. I was granted access to Adobe Analytics and Flurry. The learning curve on AA was too steep for the quick turnaround that was necessary for the project, and I discovered that Adobe wasn’t consistently logging events to Flurry at all.
I decided to stick with GA to pull out the data that I could to begin a full event-logging audit.

I generated two reports and pulled them into a spreadsheet:

• All events logged in 2017.
• All events logged by any app version released 1 January – 31 May  2018 (This was to ensure completely different data sets.)

Using Google’s scripting features I scripts determine & extract:

• “All Event Labels 2017” – Unique event labels and total events logged for each across all OSes and Versions used in 2017.
• “All Event Labels 2018 Releases” – Unique event labels and total logged in each app version released in the first 5 months of 2018.
• “Android Only”/“iOS Only”/“Both” – Which events were common or unique to either OS, that is which events were only logged in only Android, only iOS, or to both.
• “None in 2018” – Which events were logged last year, but not logged at all in the first 5 months of 2018

I line-by-lined through “None in 2018” comparing it with “All Event Labels 2017” to determine which ones were likely to be deprecated intentionally, and which ones seemed to be accidentally dropped. Some of these were confirmed with conversations and live-testing in GA’s “Live View.” (Neither developer team had a comprehensive list of currently integrated events.)

Pursuing Solutions

I compiled a spreadsheet with the following categories:

• iOS Only Reported
• Android Only Reported
• Both Reported
• Reported in 2017, but not 2018
• Confirmed Deprecated
• Not reporting, but should be

I have asked the developers to look through the lists and confirm which ones should belong where along with notes of data parameters that are currently being collected with each.

After determining which ones are currently in use, I will work with the developers to define clearly what should trigger each event, which events need to be added, and how to sync up the two platforms to make sure they’re reporting the same, or at least functionally equivalent data.