ABSTRACT:
We advocate a Robotic Testing approach to address the profound shift we are currently witnessing, from desktop to mobile computation. This is a trend which is projected to gather pace, accelerated by a concomitant shift from desktop to mobile ownership. Automated software testing is needed more than ever in this emerging mobile world, yet we may need to rethink some of the principles of software testing, even fundamentals, such as what it means to be truly ‘black box’ when testing. Mobile devices enable rich user interaction inputs such as gestures via touch screens and various signals via sensors (GPS, accelerometer, barometer, NFC, etc.).They serve a wide range of users in heterogeneous and dynamic contexts such as geographical locations and networking infrastructures. Complex interactions with various sensors under a wide range of testing contexts are required, to adequately explore and uncover bugs.
A recent survey work on mobile app development indicated that current practical mobile app testing relies heavily on manual testing, with its inherent inefficiencies and biases. There are frameworks, such as Appium, Robotium, UI Automator that can partly support automatic test execution, but they rely upon on human test script design, thereby creating a bottleneck.
Fortunately, there have been many recent advances in automated Android testing research [2, 3, 4, 5]. However, all of these techniques use intrusive (fully, or partly white box) approaches to execute the generated test cases. They also assume that testing tools will enjoy developer-level permissions; an assumption that does not always hold.
Many such techniques need to modify the app code or even the mobile operating system, while even the most ‘black box’ of approaches rely on communicating
with the app under test through a test harness. This is not ‘truly’ black box, because it relies on the machine-to-machine interface between test harness and app under test.
A truly black box approach would make no assumptions, relying only upon the device-level cyber-physical interface between human and the app. Testing at this level of abstraction is not only truly black box, but it makes fewer assumptions and more closely emulates the experience of the real user. It may, therefore, yield more realistic test cases. Furthermore, such truly black box approach to testing is inherently device independent; a considerable benefit in the world with more than 2,000 different devices under test.
