Experience Building Mobile Test Automation Framework for Playback Mobile SDK (Android & IOS) Part 1
on 02-08-201711:41 AM - last edited on 02-08-201712:18 PM by thomas
Written by Bindu Sondur 1/10/17
The first project I worked on at Ooyala was to build a robust, sustainable and scalable mobile test automation framework for playback SDK (Android and iOS) as part of the playback automation team. Because we were starting with a clean slate, we initially came up with a few goals of what we expect the framework to do.
Goals of the Test Framework
Broadly, below are some of the goals we expected the framework to achieve initially:
1) Ability to automate a sizable chunk of existing test cases across multiple ecosystems ( Android and iOS).
Impact - a) Reduces the time and manual effort for testing during normal release week as well as during an off-cycle deploy.
Impact - b) Gets more coverage across different device and OS combinations. Device ecosystem is very fragmented esp, Android. There are a ton of devices of different OS, versions, physical shape and size.
2) Ability to run automation on both emulators and actual devices.
3) Ability to easily maintain the automated test scripts.
Impact - Over time , when number of test scripts grows, needed a sustainable process to manage test scripts and to continue to keep it relevant.
4) Ability to easily debug failures and classify them as an actual software bug.
5) Ability to scale up test execution.
Selection of Test Automation tool
Evaluation of available open source tools :
There are plenty of open source UI automation tools available in the market. Some of the tools considered were:
As we have both Android, iOS SDK and sample apps to test, we wanted a tool which will work on both Android and iOS ecosystems ( goal 1 ). We also wanted a tool which runs on both emulators and real devices (goal 2). Performed an initial POC using Appium tool, it executed reasonably well given our goals for the project. Hence, we settled with Appium (http://appium.io/).
Making Ooyala SDK Test Automation Friendly
Needed something more than UI Automation, verifying ooyala player sdk events :
Our test framework had to have the ability to verify the events generated by the Ooyala SDK which was not possible with the out of the box Appium tool.
For example, one use case is to click on the play button and verify if the Ooyala SDK triggered the corresponding playStarted event .
In order to achieve this, we had to make modifications to both Android, iOS sample apps and our test framework .
Changes to Android Sample Apps
We added automation hooks to the Android Sample Apps repo using which we were able to write the events generated by Ooyala SDK in a separate file on the device if the file existed . The test framework was modified to first create the file on the device before the test runs, then read the events written on that file by sample apps. If the file did not exist, no events were written by the SDK. (Customer who uses sample app is not exposed to this functionality.)
Changes to iOS Sample Apps
With iOS being a more closed ecosystem, we could not use file as a common interface to write and read SDK events as in Android Hence, we created QA mode switch for iOS sample apps. In the QA mode, the events generated by the Ooyala SDK is written in a textbox on the device screen itself below the video player. The test framework in turn reads the events from the text box for validation purposes.
Now that we have listed the broad goals of the test framework, selected which test automation tool to make use of and also made Ooyala SDK to be test automation friendly ( by adding the events automation hooks ), the next part of the journey is to actually build the test framework and test scripts. I would be love to share further about the same in the subsequent blogs.