There has been a lot of material presented over the last four parts. In this final part, we will finally go into some detail about our automated testing system for Android. We will also finally get an opportunity to bring everything together by looking a little more how our shell scripts orchestrate the test run and connect components from the previous parts.

Android was a lot easier to setup automated on-device tests than iOS because the entire toolchain is command line driven. But ironically, actually running the tests has been more unreliable for us, mostly due to some bug related to adb. For some reason we can’t explain, adb will hang on us and it will not allow us to communicate with our device. This has happened with all the Android devices we’ve tried and changing our host computer didn’t help either. Killing and restarting adb does nothing to fix the problem and the only solution is to disconnect and reconnect the cable. We discovered that killing the adb server after we complete our tests seems to reduce the chance of this problem occurring, but this still happens for us, and it happens more frequently than our iOS/Xcode process failing for random reasons.

Despite these problems, conceptually, Android automation is a lot more straightforward.

 

Install and Execute

As described in Part 1, we build our app from source code. We have a local build process for Corona that also builds the Lua projects as part of this build process. The stuff from Part 3 is also incorporated into this part. This is achieved mostly through shell scripts which invoke the standard Android toolchain like ndk-build and ant. While the approach is pretty generic, the scripts are pretty specific to our process and not very useful to anybody else so we’ll move on to the next step.

Once we have an .apk ready to install, we need to:

1) Install the apk

2) Launch the TAP server listener (From Part 3)

3) Launch the app

4) Uninstall the apk (clean up)

Below is an excerpt from one of our (bash) shell scripts responsible for running tests. It is a function that does the above steps after the app is built.


# bash shell script excerpt:
function InstallAndExecuteTest()
{
	# Make sure there isn't a stale apk on the device
	eval "${adb_cmd} ${device_flags} uninstall com.ansca.test.Corona"

	echo "Installing"
	eval "${adb_cmd} ${device_flags} install -r '${i}/${test_name}.apk'"
	INSTALL_STATUS=$?
	if [ $INSTALL_STATUS -ne 0 ]; then
		# Need to write error report
		echo "Bail out! Install failed." >> "../test-output/${test_name}.tap"
		continue
	fi

	echo "Launching App"
	# Launch the app. The process returns immediately.
	# adb -d shell am start -a android.intent.action.MAIN -n com.ansca.test.HelloWorld/com.ansca.test.HelloWorld.MyCoronaActivity
	# I can't seem to get an exit status code other than 0 from adb so no point in checking.
	# Adding a sleep to try to avoid race condition that might be sometimes affecting us.
	sleep 2 && eval "${adb_cmd} ${device_flags} shell am start -a android.intent.action.MAIN -n com.ansca.test.Corona/com.ansca.corona.CoronaActivity" &

	# There may be a race condition here. The server needs to be present before the app tries to connect.
	# But we rely on the server blocking to let us know when the # process is finished.
	# So far, we have not been bitten.
	echo "Launching Server"
	# lua TapSocketListener.lua "*" ${PORT} "../test-output/${test_name}.tap" ${ECHO_TAPSOCKET_LISTENER}
	eval "${TAPSOCKETCMD}"
	LISTENER_STATUS=$?

	# remove the application from the device. (This should also kill it if it is still running.)
	eval "${adb_cmd} ${device_flags} uninstall com.ansca.test.Corona"
}

${adb_cmd} is just the command line tool adb. See the ‘adb Recipes’ towards the end of this article.

Our ${TAPSOCKETCMD} is defined as:

TAPSOCKETCMD='$timeout_exec "${TIMEOUT}" $lua_exec TapSocketListener.lua "*" ${PORT} "../test-output/${test_name}.tap" ${ECHO_TAPSOCKET_LISTENER}'

$timeout_exec is a command line tool called gtimeout which is supposed to abort with an error code if the process doesn’t complete within the allotted time (${TIMEOUT}).

So the above line reduces to something like:

gtimeout 300s lua TapSocketListener.lua "*" 12345 ../test-output/audio.tap true

(The TapSocketListener parameters were described in Part 3.)

You might notice the sleep command in the script. That is there to avoid a race condition. We technically need to start the TAP server listener before we start the app. But we want our shell script to block on the server and only continue processing once the server is finished. So we need to invoke the command to launch the app as a non-blocking background process (hence the ‘&’ symbol at the end of the command) before we invoke the server. To make sure the application doesn’t start before the server is ready, we inject a sleep call before it to give the server enough time to start up.

We also look at the return status codes and write additional information to the TAP file output when there is a failure. This helps guarantee that the test output contains explicit information that the test failed plus additional information about what happened which is sometimes useful for debugging. We save the return status code in the global variable LISTENER_STATUS which we examine when the function returns.

The Master Script

For completeness, the following is an excerpt of the shell script that calls the above function which is our main script that orchestrates the test run. A lot of the code is very specific to our environment so it may have limited usefulness to you. But there are several things worth noting:

• We check the LISTENER_STATUS variable in this case and append information to the TAP output so we can clearly mark type type of failure (as described in Part 1).

• We repeat the tests up to a certain number of times if they fail (look for $loop_count and $do_loop) to try to weed out random tool chain/device communication failures (as described in Part 1)

• We show how we capture a port number via FindFreeSocketPort.lua and use that port number for the TapSocketListener.lua and create/write to the TestMoreOutputServerInfo.lua file (as described in Part 3).

• We copy all our lua-TestMore files into our app’s project directory right before we build it so we only need to have one version of the code to maintain (as described in Part 3). We do a remove because we used to use symlinks which are still in our repository, but we are trying to move away from symlinks because of Windows not supporting them.

• The big for-loop is just our way of looping through all the test directories we want. We actually have additional tests that focus on specific types of devices or skins which are omitted here for brevity.

• Jenkins has yet more scripts that invoke this script so builds and tests can be triggered automatically on events such as timers or code check-ins.

• It is worth noting that we have very similar functions/scripts for iOS. The way we deal with the TapSocketListener.lua is identical. The major changes in our iOS tests are that we call our various Scripting Bridge scripts or xcodebuild/iphonesim utilities previously discussed.



#####
#Run all the non skin specific tests
#####
for i in $( find . -type d -maxdepth 1 ! -name *-* ! -empty ! -name .* ); do
  test_name=`echo $i | cut -c 3-`
#	echo $i
	echo "Starting next test: $test_name"

	# Figure out the IP address
	# Select a port number
	# Write a file called TestMoreOutputServerInfo.lua
	# module("TestMoreOutputServerInfo", package.seeall)
	# Host = "192.168.192.125"
	# Port = 12345
	# TimeOut = 300 * 1000  -- in milliseconds
	IPADDRESS=`ifconfig en0 | grep -E 'inet.[0-9]' | awk '{ print $2}'`
	PORT=`$lua_exec FindFreeSocketPort.lua ${IPADDRESS}`
	echo "module('TestMoreOutputServerInfo', package.seeall)" > TestMoreOutputServerInfo.lua
	echo "Host = '${IPADDRESS}'" >> TestMoreOutputServerInfo.lua
	echo "Port = '${PORT}'" >> TestMoreOutputServerInfo.lua
	echo "TimeOut = '${TIMEOUT_FOR_APP}'" >> TestMoreOutputServerInfo.lua

	# Copying Test files
	rm -f "$i/Builder.lua"
	rm -f "$i/More.lua"
	rm -f "$i/SocketOutput.lua"
	rm -f "$i/NoOutput.lua"
	rm -f "$i/Tester.lua"
	rm -f "$i/CoronaTest.lua"
	rm -f "$i/TestMoreOutputServerInfo.lua"

	cp -f Builder.lua "$i/Builder.lua"
	cp -f More.lua "$i/More.lua"
	cp -f SocketOutput.lua "$i/SocketOutput.lua"
	cp -f NoOutput.lua "$i/NoOutput.lua"
	cp -f Tester.lua "$i/Tester.lua"
	cp -f CoronaTest.lua "$i/CoronaTest.lua"
	cp -f TestMoreOutputServerInfo.lua "$i/TestMoreOutputServerInfo.lua"

	echo "Building"
	# Paramaters are: path_with_ratatouille.xcodeproj
	# path_where_assets_dir_needs_to_be directory_of_test xcode_target_to_build_or_run build_configuration os_or_simulator should_build should_run

	# /Volumes/CoronaBigDMG/CoronaBigDMG/platform/android/build_app_for_automated_testing.sh /Applications/CoronaSDK.2011.484/SampleCode/GettingStarted/HelloWorld
	eval "${BUILD_APP}" $i "${NDK_ROOT}" "${SDK_ROOT}" automation release incremental
	BUILD_STATUS=$?
	if [ $BUILD_STATUS -eq 0 ]; then
		echo "Build succeeded"
	else
		echo "Build failed"
		# Need to write error report
		echo "Bail out! Build failed." >> "../test-output/${test_name}.tap"
		continue
	fi

	do_loop=1
	loop_count=1
	while [ $do_loop -ne 0 ]; do
		# we are re-running a test, delete the past run
		rm -f "../test-output/${test_name}.tap"

		InstallAndExecuteTest
		if [ $LISTENER_STATUS -eq 0 ]; then
			echo "Run succeeded"
			do_loop=0
		elif [ $LISTENER_STATUS -eq 124 ]; then
			echo "gtimeout killed the test"
			# Need to write error report
			echo -e "nBail out! gtimeout killed the test" >> "../test-output/${test_name}.tap"
		elif [ $LISTENER_STATUS -eq 1 ]; then
			echo "TapSocketListener aborted due to receiving App-auto-timeout notification"
			echo -e "nBail out! TapSocketListener aborted due to receiving App-auto-timeout notification" >> "../test-output/${test_name}.tap"
		elif [ $LISTENER_STATUS -eq 2 ]; then
			echo "TapSocketListener aborted due to unexpected disconnect"
			echo -e "nBail out! TapSocketListener aborted due to unexpected disconnect" >> "../test-output/${test_name}.tap"
		else
			# Need to write error report
			echo "TapSocketListener aborted due and returned status $?"
			echo -e "nBail out! TapSocketListener aborted due and returned status $?" >> "../test-output/${test_name}.tap"
		fi

		loop_count=$loop_count+1
		# After a few failed attempts, give up and move on
		if [ $loop_count -gt 2 ]; then
			do_loop=0
		fi
	done

	# print a newline to segment next test
	echo -e "n"
#	exit 0

done

# experiment to see if killing the adb server will help avoid the dead-server hang problem
eval "${adb_cmd} kill-server"

#####
#exit without passing an error code
#####
exit 0

This is the video of the automated test run on a Nexus S from Part 1. If you watch the terminal output closely, you will see the script lines described in this article echoed. You will also see the TAP output echoed.

 

Android adb Recipes:


adb -d

Means talk to the connected device. If you have multiple devices connected, you must use -s <serial number> instead of -d.

adb -d uninstall com.ansca.test.Corona

Uninstall a package (You need to provide the package name)

adb shell pm list packages | grep com.ansca | cut -d : -f 2 | tr -d 'r' | xargs -t -o -n 1 adb -d uninstall

Delete all apps matching a pattern (for quickly freeing up space):

The first part (pm list packages) returns all the packages on the connected device.

The second part (grep) filters for packages that contain the name “com.ansca”. Change this filter depending on your needs.

The third part (cut) removes the “package:” that begins each string.

The forth part (tr) removes an invisible carriage return that confuses adb uninstall if not removed.

The fifth part (xargs) passes each package to adb uninstall which removes the package

adb -d install -r foo.apk

Install a package. The -r will overwrite any package previously installed with the same name.

adb -d shell am start -a android.intent.action.MAIN -n com.ansca.test.Corona/com.ansca.corona.CoronaActivity"

Launch an application: You must provide the package name and activity name.

Closing Thoughts

As you can see over the last 5 parts, our automated testing is quite involved. But we believe it has a lot of potential moving forward. We hope sharing this material might spark more interest in the community and begin more collaboration on how to improve automated testing on mobile devices. If you have your own automated test systems or have ideas for this one, please drop us a line. We would love to hear about them.

For us, some of our future direction considerations with this system are:

• Get a working solution for Xcode 4 for installing/running on iOS devices

• Look at Apple’s UI Automation for doing UI tests on iOS

• Look at developing a more generic event system for Corona so we can do UI testing across all our platforms

• Look at using Bonjour/Zeroconf to replace the manual IP address and port number settings

• Make it possible/better to support multiple attached devices to the same computer

• Rendering image/screenshot comparison/validation tests

  1. Hi again! Just finished reading this series. Grand material, thanks for sharing. I was looking for some examples of Lua with TAP for a presentation and found your blog. Keep posting interesting content like this, please :-)

  2. Thanks for writing! We’re glad you found it useful. Please spread the word. The more people that use this stuff, the better it is for all of us :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>