Testing & Release strategy for Native Android & iOS Apps

Predicting / knowing overall Quality of Native (Android / iOS) Apps before release is challenging for various reasons. Post release is when you ‘may’ realize the problem, but by then it is already too late. Learn how I managed testing, risk & releases for such apps, reducing impact to our users!

We know the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

However, how many know about - * The release process of Native Apps on Android & iOS, and its challenges and implications on the user-base * How is releasing a Native App different from Web / backend deployments?

In this session, we will focus on the Testing Strategy and the Release process an Android & iOS Native app, in the media / entertainment domain (case study example), used by millions of users across 10+ countries.

The app itself is not very unique - but its criteria for releases (apart from being a Native App) is very stringent. And that makes it special.

Criteria for App Releases:

• Apps have limited unit testing. Some e2e automated tests exist for Android
• Business runs on Analytics - to know the feature usage, consumer engagement, revenue, etc. Any miss in event reporting means “hell breaks loose for engineering team”
• No app releases can happen from 25th-5th of the calendar dates - as it coincides with pay-day - meaning users can (re-)subscribe to the App that month. So minimize disruption for the user.
• Development & Testing in house is based on a variety of 25-30 devices. Actual usage of Apps by consumers is on a 10K+ unique Device + OS + Network combinations. Testing can never be close to complete for this variety of usage

Testing Strategy & Release Approach

Given the above constraints and challenges, we will discuss what approaches and strategies can work, or not in such an ecosystem. I will also share points related to in this discussion:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

Note: Some of the above are anti-patterns. I would love to hear your thoughts on what could have been done differently in the context and at the time to avoid the same!

This is a new talk and speaks about the challenges in releasing a native app to Google’s Play Store and Apple’s App Store. I will be discussing approaches I had to take, which may not necessarily be the right decisions from a holistic perspective, but would be engaging and interacting with the audience to understand their thoughts on the same. This approach will also make the talk more interactive