We allow Corona developers to also build for the Xcode iOS Simulator. Sometimes the simulator is preferable to our Mac or Windows simulator because the Xcode Simulator behaves more like a real device. Since we officially support the Xcode Simulator, we run our automated tests on the Xcode iOS Simulator to help verify our stuff actually works.

While we could theoretically reuse the same process of scripting Xcode that we described in Part 2, we opted for a slightly different approach. As described in Part 2, Xcode 4 broke everything so we didn’t want to put this in the same critical path. Furthermore, Xcode 4 has some very nice speed improvements and reduces our build times to almost half.

So instead, we simply use the command line tool, xcodebuild to build our binaries for us. To invoke the simulator, there are already people and projects that have documented and written code on how to invoke the Xcode simulator through private frameworks. We grabbed iphonesim which provides a command line tool for launching the simulator and use it in our automation solution.

Our actual tests are essentially the same on the simulator as the device. We use LuaSocket as described in Part 3 to send our test results to our server for processing/reporting.


xcodebuild

Using xcodebuild is mostly straightforward, but coming from pre-Xcode 3, we still haven’t completely figured out how to map our project in terms of Workspaces and Schemes for what we need to do. At least for now, the legacy xcodebuild switches still work so we can specify Targets and Configurations directly.

One other complication for us is that Xcode now does out-of-source builds in which all the build products are placed somewhere else on the system. I personally like out-of-source builds and have had a lot of experience with them using tools like CMake. But the difficulty with Xcode is that there is no good way to query Xcode about where build components are placed from external calling processes. To workaround this problem, we also set the DSTROOT variable and use the install command with xcodebuild. This instructs Xcode to place the final built products to the directory of our choosing, with a few additional gotchas.

One gotcha is that the products will be placed in a subdirectory determined by the target’s install destination in the project, e.g. /usr/local/bin. Another gotcha is that things get weirder when anchors like @rpath are used (e.g. LuaCocoa does this); products seem to go to parallel directories that start with “@rpath”. And cross-project dependencies and Workspaces seem to have funny relationships where the product first gets built in its own separate build directory and then may get relocated. But in a rebuild or clean sometimes can really mess up Xcode’s dependency tracking and break the build.

So in our shell scripts, our xcodebuild command looks like:

xcodebuild -project “${CORONA_IPHONE_PROJECT}” -target “${WHICH_XCODE_TARGET}” -configuration Release -sdk iphonesimulator${SDK_VERSION_NUMBER} DSTROOT=${CORONA_IPHONE_INSTALL_TMP_DIR} install

In terms of “MySampleProject” from Part 2, the command might look like something like:

xcodebuild -project “MySampleProject.xcodeproj” -target “OpenGLES2″ -configuration Release -sdk iphonesimulator4.3 DSTROOT=/tmp/MyKnownBuildDir install

Controlling the iOS Simulator: Return to Scripting Bridge/LuaCocoa, System Events

While the private frameworks and the tool iphonesim are a great starting point for launching our apps in the iOS Simulator, we needed to do a few things that went beyond those capabilities. One thing we needed to do is make sure to delete the application we just ran and all the user preferences it might save. We didn’t want future runs to pick up stale data left by older runs. Second, since the simulator supports iPhone, iPhone 4, and iPad skins, we felt compelled to run the tests through each skin. Unfortunately, as far as we could tell, the private framework doesn’t allow us to specify the device family for iPhone 4. Thus we were unable to fully control the skin we launch with with through the command line tool.

So it was back to Scripting Bridge and LuaCocoa.

Unfortunately, the Xcode iOS Simulator has virtually no scripting dictionary so scripting it is painful. Instead of targeting the application directly, we end up targeting Apple’s System Events services via Scripting Bridge which allow us to manipulate generic UI elements (via generating key presses and mouse clicks) belonging to the app. Our strategy is to find the menu item that controls changing the skin, and pick the desired option.

This was my first experience with System Events and don’t feel particularly versed enough to talk in detail about it. Our Scripting Bridge script is available for reference. The script is called ScriptingBridge_iOSSimulatorQuit.lua and is with the XcodeScriptingBridge download from Part 2. (I just added the file, so you may need to pull the update.) But I will try to highlight some of the key points.

The first thing to note is that we first use iphonesim to launch and run our app and ScriptingBridge_iOSSimulatorQuit.lua is only run after the iOS Simulator completes our test app. The purpose of the script is to erase all traces of the app we just ran and then setup the simulator skin for the next time we run an application. The iOS Simulator will remember the last skin option and launch as that one.

We can know that the test app is complete because of the way our socket server from Part 3 works. Our socket server quits when the test ends. In our controlling shell scripts, we allow the server call to block until completion so we can simply invoke ScriptingBridge_iOSSimulatorQuit.lua right after.

So in ScriptingBridge_iOSSimulatorQuit.lua itself:

To access System Events:


local system_events = SBApplication:applicationWithBundleIdentifier_("com.apple.systemevents")

To get focus on the running Xcode iOS Simulator, we search through a list of processes provided by System Events and bring it to the foreground:


local processes = system_events:processes()
local ios_sim_process = nil
for i=1, #processes do
	if tostring(processes[i]:name()) == "iPhone Simulator" then
		ios_sim_process = processes[i]
	end
end

-- Bring the app to the foreground:
ios_sim_process:setFrontmost_(true)

So now we want to clean up the application we just ran. The simulator has a menu option named “Reset Content and Settings…”.

To select this menu item, we are going to need search through the list of menus and menu items to find the object we need so we can invoke it.


local menu_items = ios_sim_process:menuBars()[1]:menus()[2]:menuItems()
local reset_option = nil
local quit_option = nil
--print(menu_items)
for i=1, #menu_items do
--	NSLog("%@", menu_items[i]:title())
	-- Note the ellipses is not 3 dots but Alt-Semicolon
	if  tostring(menu_items[i]:title()) == "Reset Content and Settings…" then
		reset_option = menu_items[i]
	elseif tostring(menu_items[i]:title()) == "Quit iOS Simulator" then
		quit_option = menu_items[i]
	end
end
if not reset_option then
	print("Warning: Could not find 'Reset Content and Settings…', setting to last known index position")
	reset_option = menu_items[3]
end
if not quit_option then
	print("Warning: Could not find 'Quit iOS Simulator', setting to last known index position")
	quit_option = menu_items[7]
end

The line ios_sim_process:menuBars()[1]:menus()[2]:menuItems() returns an array of menu items. menuBars() returns an array of menuBars and [1] means to take the first one. (Please note that Lua by convention uses array indices that start at 1 instead of 0.) I don’t know when there are cases of not having exactly 1 menubar. The menus() returns an array of menus. The [2] means the get the second item in the array which is the menu that is is one over from the left. (The first one is the Apple (logo) menu.)

The code following that line is mostly paranoia checks. We traverse the menu items until we find the “Reset Content and Settings…” option by string name compares. In principle, we know the index position, so we don’t need to do this. But if Apple changes the menu order by adding new items or removing them in future releases, the code will break. But arguably, this code will break if Apple renames the menu items. Localization is also a potential problem. And if Apple moves the item to another menu (e.g. File), this breaks too. But my personal estimation is that Apple is less likely to rename the item or move it to another menu.

You will also notice I am looking for the “Quit iOS Simulator” option and saving that. We will invoke that at the very end of our script. I save it here since it is in the same menu as “Reset”.

To invoke the menu item, we do:


local ret_val = reset_option:clickAt_(reset_option:position())

But as an additional gotcha, when you invoke this method, you get a confirmation dialog asking if you are sure you want to reset everything.

We are forced to put a sleep in our code to allow enough time for this dialog to appear. Then we must generate a key event to hit the correct button. But this key event will only work if you enable on keyboard navigation for buttons in Accessibility.

Because we don’t get any code notifications about this prompt box appearing, we must force our script to wait for a long enough period of time for this box to appear. Once the box appears, we need to press the correct button. It turns out that once the correct Accessibility options are enabled, the button we need to press will be triggered by the spacebar. So we just need to invoke a space key press.


-- Give the system enough time for the "Are you sure?" dialog to appear
os.execute("sleep 1")

-- This requires that keyboard navigation for buttons is on in Accessibility.
-- I believe this sends a spacebar press which is sufficient to cause the button to press.
system_events:keystroke_using_(" ", 0)

Next, we want to change the skin. Again, we need to access this by navigating the menus.

Here is the code to select the menu items:


menu_items = ios_sim_process:menuBars()[1]:menus()[5]:menuItems()
ios_device = menu_items[1]:menus()[1]:menuItems()[next_skin_index]
ios_device:clickAt_(ios_device:position())

The index=5 for menus()[5] represents the fifth menu item over to the right which is the category “Hardware”

menu_items[1]:menus()[1] gets us to the Device submenu under Hardware and menuItems()[next_skin_index] lets us pick the specific device option. The menus()[1] seems a little strange to me and I’m not sure I can explain this. I did this through lots of trial and error and experimentation and all I know is that this works.

next_skin_index should be 1 for iPad, 2 for iPhone, and 3 for iPhone 4. Our controlling shell script which is orchestrating the entire test run keeps track of the skin we want to run next and passes this desired value to this script.


quit_option:clickAt_(quit_option:position())

One final thing worth noting is that because we are reliant on System Events, it is very sensitive to things like the current frontmost process. That basically means you should not interact with the GUI while tests are running. Since we have a headless machine to do builds and tests, this is usually not a big issue for us, but does reiterate the need for Apple to provide us better tools.

This is the video of the iOS Simulator from Part 1. Notice the test is run first, then how System Events are controlled to manipulate the application focus, trigger the menu options, and dismiss the “Are you sure?” dialog. The video shows all 3 skins being run for the same test program.

Next Time

In the next part, we will look at automation on Android.

  1. Testing had often been underrated, but not any more due to rapid introduction of new capabilities and features. Thanks for thorough 4-part series on Corona SDK testing that also underscores the importance of continual process improvement and optimization. Looking forward to your automation on Android next.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>